Add support for SVM atomics in Nouveau

The behaviour of try_to_unmap_one() is difficult to follow because it performs different operations based on a fairly large set of flags used in different combinations.

TTU_MUNLOCK is one such flag. However it is exclusively used by try_to_munlock() which specifies no other flags. Therefore rather than overload try_to_unmap_one() with unrelated behaviour split this out into it's own function and remove the flag.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Ralph Campbell <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v9:
* Improved comments

v8:
* Renamed try_to_munlock to page_mlock to better reflect what the function actually does.
* Removed the TODO from the documentation that this patch addresses.

v7:
* Added Christoph's Reviewed-by

v4:
* Removed redundant check for VM_LOCKED

---
 Documentation/vm/unevictable-lru.rst | 33 ++++++---------
 include/linux/rmap.h                 |  3 +-
 mm/mlock.c                           | 10 ++---
 mm/rmap.c                            | 61 ++++++++++++++++++++--------
 4 files changed, 63 insertions(+), 44 deletions(-)

Documentation/vm/unevictable-lru.rst +13 -20

index 0e1490524f53..eae3af17f2d9 100644
@@ -389,14 +389,14 @@ mlocked, munlock_vma_page() updates that zone statistics for the number of
 mlocked pages.  Note, however, that at this point we haven't checked whether
 the page is mapped by other VM_LOCKED VMAs.
 
-We can't call try_to_munlock(), the function that walks the reverse map to
+We can't call page_mlock(), the function that walks the reverse map to
 check for other VM_LOCKED VMAs, without first isolating the page from the LRU.
-try_to_munlock() is a variant of try_to_unmap() and thus requires that the page
+page_mlock() is a variant of try_to_unmap() and thus requires that the page
 not be on an LRU list [more on these below].  However, the call to
-isolate_lru_page() could fail, in which case we couldn't try_to_munlock().  So,
+isolate_lru_page() could fail, in which case we can't call page_mlock().  So,
 we go ahead and clear PG_mlocked up front, as this might be the only chance we
-have.  If we can successfully isolate the page, we go ahead and
-try_to_munlock(), which will restore the PG_mlocked flag and update the zone
+have.  If we can successfully isolate the page, we go ahead and call
+page_mlock(), which will restore the PG_mlocked flag and update the zone
 page statistics if it finds another VMA holding the page mlocked.  If we fail
 to isolate the page, we'll have left a potentially mlocked page on the LRU.
 This is fine, because we'll catch it later if and if vmscan tries to reclaim

@@ -545,31 +545,24 @@ munlock or munmap system calls, mm teardown (munlock_vma_pages_all), reclaim,
 holepunching, and truncation of file pages and their anonymous COWed pages.
 
 
-try_to_munlock() Reverse Map Scan
+page_mlock() Reverse Map Scan
 ---------------------------------
 
-.. warning::
-   [!] TODO/FIXME: a better name might be page_mlocked() - analogous to the
-   page_referenced() reverse map walker.
-
 When munlock_vma_page() [see section :ref:`munlock()/munlockall() System Call
 Handling <munlock_munlockall_handling>` above] tries to munlock a
 page, it needs to determine whether or not the page is mapped by any
 VM_LOCKED VMA without actually attempting to unmap all PTEs from the
 page.  For this purpose, the unevictable/mlock infrastructure
-introduced a variant of try_to_unmap() called try_to_munlock().
+introduced a variant of try_to_unmap() called page_mlock().
 
-try_to_munlock() calls the same functions as try_to_unmap() for anonymous and
-mapped file and KSM pages with a flag argument specifying unlock versus unmap
-processing.  Again, these functions walk the respective reverse maps looking
-for VM_LOCKED VMAs.  When such a VMA is found, as in the try_to_unmap() case,
-the functions mlock the page via mlock_vma_page() and return SWAP_MLOCK.  This
-undoes the pre-clearing of the page's PG_mlocked done by munlock_vma_page.
+page_mlock() walks the respective reverse maps looking for VM_LOCKED VMAs. When
+such a VMA is found the page is mlocked via mlock_vma_page(). This undoes the
+pre-clearing of the page's PG_mlocked done by munlock_vma_page.
 
-Note that try_to_munlock()'s reverse map walk must visit every VMA in a page's
+Note that page_mlock()'s reverse map walk must visit every VMA in a page's
 reverse map to determine that a page is NOT mapped into any VM_LOCKED VMA.
 However, the scan can terminate when it encounters a VM_LOCKED VMA.
-Although try_to_munlock() might be called a great many times when munlocking a
+Although page_mlock() might be called a great many times when munlocking a
 large region or tearing down a large address space that has been mlocked via
 mlockall(), overall this is a fairly rare event.
 

@@ -602,7 +595,7 @@ inactive lists to the appropriate node's unevictable list.
 shrink_inactive_list() should only see SHM_LOCK'd pages that became SHM_LOCK'd
 after shrink_active_list() had moved them to the inactive list, or pages mapped
 into VM_LOCKED VMAs that munlock_vma_page() couldn't isolate from the LRU to
-recheck via try_to_munlock().  shrink_inactive_list() won't notice the latter,
+recheck via page_mlock().  shrink_inactive_list() won't notice the latter,
 but will pass on to shrink_page_list().
 
 shrink_page_list() again culls obviously unevictable pages that it could

include/linux/rmap.h +1 -2

index def5c62c93b3..38a746787c2f 100644
@@ -87,7 +87,6 @@ struct anon_vma_chain {
 
 enum ttu_flags {
 	TTU_MIGRATION		= 0x1,	/* migration mode */
-	TTU_MUNLOCK		= 0x2,	/* munlock mode */
 
 	TTU_SPLIT_HUGE_PMD	= 0x4,	/* split huge PMD if any */
 	TTU_IGNORE_MLOCK	= 0x8,	/* ignore mlock */

@@ -239,7 +238,7 @@ int page_mkclean(struct page *);
  * called in munlock()/munmap() path to check for other vmas holding
  * the page mlocked.
  */
-void try_to_munlock(struct page *);
+void page_mlock(struct page *page);
 
 void remove_migration_ptes(struct page *old, struct page *new, bool locked);
 

mm/mlock.c +5 -5

index df590fda5688..a518d4c48e65 100644
@@ -108,7 +108,7 @@ void mlock_vma_page(struct page *page)
 /*
  * Finish munlock after successful page isolation
  *
- * Page must be locked. This is a wrapper for try_to_munlock()
+ * Page must be locked. This is a wrapper for page_mlock()
  * and putback_lru_page() with munlock accounting.
  */
 static void __munlock_isolated_page(struct page *page)

@@ -118,7 +118,7 @@ static void __munlock_isolated_page(struct page *page)
 	 * and we don't need to check all the other vmas.
 	 */
 	if (page_mapcount(page) > 1)
-		try_to_munlock(page);
+		page_mlock(page);
 
 	/* Did try_to_unlock() succeed or punt? */
 	if (!PageMlocked(page))

@@ -158,7 +158,7 @@ static void __munlock_isolation_failed(struct page *page)
  * munlock()ed or munmap()ed, we want to check whether other vmas hold the
  * page locked so that we can leave it on the unevictable lru list and not
  * bother vmscan with it.  However, to walk the page's rmap list in
- * try_to_munlock() we must isolate the page from the LRU.  If some other
+ * page_mlock() we must isolate the page from the LRU.  If some other
  * task has removed the page from the LRU, we won't be able to do that.
  * So we clear the PageMlocked as we might not get another chance.  If we
  * can't isolate the page, we leave it for putback_lru_page() and vmscan

@@ -168,7 +168,7 @@ unsigned int munlock_vma_page(struct page *page)
 {
 	int nr_pages;
 
-	/* For try_to_munlock() and to serialize with page migration */
+	/* For page_mlock() and to serialize with page migration */
 	BUG_ON(!PageLocked(page));
 	VM_BUG_ON_PAGE(PageTail(page), page);
 

@@ -205,7 +205,7 @@ static int __mlock_posix_error_return(long retval)
  *
  * The fast path is available only for evictable pages with single mapping.
  * Then we can bypass the per-cpu pvec and get better performance.
- * when mapcount > 1 we need try_to_munlock() which can fail.
+ * when mapcount > 1 we need page_mlock() which can fail.
  * when !page_evictable(), we need the full redo logic of putback_lru_page to
  * avoid leaving evictable page in unevictable list.
  *

mm/rmap.c +0 -6

index bc08c4d4b58a..e88966903e1e 100644
@@ -1405,10 +1405,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 	struct mmu_notifier_range range;
 	enum ttu_flags flags = (enum ttu_flags)(long)arg;
 
-	/* munlock has nothing to gain from examining un-locked vmas */
-	if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED))
-		return true;
-
 	if (IS_ENABLED(CONFIG_MIGRATION) && (flags & TTU_MIGRATION) &&
 	    is_zone_device_page(page) && !is_device_private_page(page))
 		return true;

@@ -1469,8 +1465,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
 				page_vma_mapped_walk_done(&pvmw);
 				break;
 			}
-			if (flags & TTU_MUNLOCK)
-				continue;
 		}
 
 		/* Unexpected PMD-mapped THP? */

mm/rmap.c +40 -11

+static bool page_mlock_one(struct page *page, struct vm_area_struct *vma,
+				 unsigned long address, void *unused)
+{
+	struct page_vma_mapped_walk pvmw = {
+		.page = page,
+		.vma = vma,
+		.address = address,
+	};
+
+	/* An un-locked vma doesn't have any pages to lock, continue the scan */
+	if (!(vma->vm_flags & VM_LOCKED))
+		return true;
+
+	while (page_vma_mapped_walk(&pvmw)) {
+		/* PTE-mapped THP are never mlocked */
+		if (!PageTransCompound(page))
+			mlock_vma_page(page);
+		page_vma_mapped_walk_done(&pvmw);
+
+		/*
+		 * no need to continue scanning other vma's if the page has
+		 * been locked.
+		 */
+		return false;
+	}
+
+	return true;
+}
+
 /**
- * try_to_munlock - try to munlock a page
- * @page: the page to be munlocked
+ * page_mlock - try to mlock a page
+ * @page: the page to be mlocked
  *
- * Called from munlock code.  Checks all of the VMAs mapping the page
- * to make sure nobody else has this page mlocked. The page will be
- * returned with PG_mlocked cleared if no other vmas have it mlocked.
+ * Called from munlock code. Checks all of the VMAs mapping the page and mlocks
+ * the page if any are found. The page will be returned with PG_mlocked cleared
+ * if it is not mapped by any locked vmas.
+ *
+ * mmap_lock should be held for read or write.
  */
-
-void try_to_munlock(struct page *page)
+void page_mlock(struct page *page)
 {
 	struct rmap_walk_control rwc = {
-		.rmap_one = try_to_unmap_one,
-		.arg = (void *)TTU_MUNLOCK,
+		.rmap_one = page_mlock_one,
 		.done = page_not_mapped,
 		.anon_lock = page_lock_anon_vma_read,
 

@@ -1849,7 +1876,7 @@ static struct anon_vma *rmap_walk_anon_lock(struct page *page,
  * Find all the mappings of a page using the mapping pointer and the vma chains
  * contained in the anon_vma struct it points to.
  *
- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma
+ * When called from page_mlock(), the mmap_lock of the mm containing the vma
  * where the page was found will be held for write.  So, we won't recheck
  * vm_flags for that VMA.  That should be OK, because that vma shouldn't be
  * LOCKED.

@@ -1901,7 +1928,7 @@ static void rmap_walk_anon(struct page *page, struct rmap_walk_control *rwc,
  * Find all the mappings of a page using the mapping pointer and the vma chains
  * contained in the address_space struct it points to.
  *
- * When called from try_to_munlock(), the mmap_lock of the mm containing the vma
+ * When called from page_mlock(), the mmap_lock of the mm containing the vma
  * where the page was found will be held for write.  So, we won't recheck
  * vm_flags for that VMA.  That should be OK, because that vma shouldn't be
  * LOCKED.

Signature · 1 line

2.20.1

Quoted 5 lines

+/*
+ * Walks the vma's mapping a page and mlocks the page if any locked vma's are
+ * found. Once one is found the page is locked and the scan can be terminated.
+ */

Can you please add that this requires the mmap_sem() lock to the comments?

Quoted 28 lines

mm/rmap.c +28 -0

+static bool page_mlock_one(struct page *page, struct vm_area_struct *vma,
+                              unsigned long address, void *unused)
+{
+     struct page_vma_mapped_walk pvmw = {
+             .page = page,
+             .vma = vma,
+             .address = address,
+     };
+
+     /* An un-locked vma doesn't have any pages to lock, continue the scan */
+     if (!(vma->vm_flags & VM_LOCKED))
+             return true;
+
+     while (page_vma_mapped_walk(&pvmw)) {
+             /* PTE-mapped THP are never mlocked */
+             if (!PageTransCompound(page))
+                     mlock_vma_page(page);
+             page_vma_mapped_walk_done(&pvmw);
+
+             /*
+              * no need to continue scanning other vma's if the page has
+              * been locked.
+              */
+             return false;
+     }
+
+     return true;
+}

Quoted 1 line · Liam Howlett

[...]

Quoted 6 lines

Quoted 5 lines

+/*
+ * Walks the vma's mapping a page and mlocks the page if any locked vma's are
+ * found. Once one is found the page is locked and the scan can be terminated.
+ */

Quoted 3 lines

Can you please add that this requires the mmap_sem() lock to the comments?

Why does this require mmap_sem() lock? Also mmap_sem() lock of which mm_struct?

Quoted 21 lines

Quoted 21 lines

mm/rmap.c +20 -0

+static bool page_mlock_one(struct page *page, struct vm_area_struct *vma,
+                              unsigned long address, void *unused)
+{
+     struct page_vma_mapped_walk pvmw = {
+             .page = page,
+             .vma = vma,
+             .address = address,
+     };
+
+     /* An un-locked vma doesn't have any pages to lock, continue the scan */
+     if (!(vma->vm_flags & VM_LOCKED))
+             return true;
+
+     while (page_vma_mapped_walk(&pvmw)) {
+             /* PTE-mapped THP are never mlocked */
+             if (!PageTransCompound(page))
+                     mlock_vma_page(page);
+             page_vma_mapped_walk_done(&pvmw);
+
+             /*

Quoted 8 lines

mm/rmap.c bool try_to_unmap(struct page *page, enum ttu_flags flags) · line 1812

Quoted 8 lines

mm/rmap.c +8 -0

+              * no need to continue scanning other vma's if the page has
+              * been locked.
+              */
+             return false;
+     }
+
+     return true;
+}

Quoted 13 lines · Shakeel Butt

Quoted 1 line · Liam Howlett

[...]

Quoted 9 lines

Quoted 5 lines

+/*
+ * Walks the vma's mapping a page and mlocks the page if any locked vma's are
+ * found. Once one is found the page is locked and the scan can be terminated.
+ */

Can you please add that this requires the mmap_sem() lock to the comments?

Quoted 1 line

Why does this require mmap_sem() lock? Also mmap_sem() lock of which mm_struct?

Doesn't the mlock_vma_page() require the mmap_sem() for reading? The mm_struct in vma->vm_mm;

From what I can see, at least the following paths have mmap_lock held for writing:

munlock_vma_pages_range() from __do_munmap()
munlokc_vma_pages_range() from remap_file_pages()

Quoted 15 lines · Liam Howlett

Quoted 13 lines · Shakeel Butt

Quoted 1 line · Liam Howlett

[...]

Quoted 9 lines

Quoted 5 lines

+/*
+ * Walks the vma's mapping a page and mlocks the page if any locked vma's are
+ * found. Once one is found the page is locked and the scan can be terminated.
+ */

Can you please add that this requires the mmap_sem() lock to the comments?

Quoted 6 lines

Quoted 1 line

Why does this require mmap_sem() lock? Also mmap_sem() lock of which mm_struct?

Doesn't the mlock_vma_page() require the mmap_sem() for reading? The mm_struct in vma->vm_mm;

We are traversing all the vmas where this page is mapped of possibly different mm_structs. I don't think we want to take mmap_sem() of all those mm_structs. The commit b87537d9e2fe ("mm: rmap use pte lock not mmap_sem to set PageMlocked") removed exactly that.

Quoted 4 lines

From what I can see, at least the following paths have mmap_lock held for writing:

Quoted 3 lines

munlock_vma_pages_range() from __do_munmap()
munlokc_vma_pages_range() from remap_file_pages()

The following path does not hold mmap_sem:

exit_mmap() -> munlock_vma_pages_all() -> munlock_vma_pages_range().

I would really suggest all to carefully read the commit message of b87537d9e2fe ("mm: rmap use pte lock not mmap_sem to set PageMlocked").

Particularly the following paragraph:
...
Vlastimil Babka points out another race which this patch protects against.
try_to_unmap_one() might reach its mlock_vma_page() TestSetPageMlocked a
moment after munlock_vma_pages_all() did its Phase 1 TestClearPageMlocked:
leaving PageMlocked and unevictable when it should be evictable. mmap_sem is ineffective because exit_mmap() does not hold it; page lock ineffective because __munlock_pagevec() only takes it afterwards, in Phase 2; pte lock is effective because __munlock_pagevec_fill() takes it to get the page, after VM_LOCKED was cleared from vm_flags, so visible to try_to_unmap_one.
...

Alistair, please bring back the VM_LOCKED check with pte lock held and the comment "Holding pte lock, we do *not* need mmap_lock here".

One positive outcome of this cleanup patch is the removal of unnecessary invalidation (unmapping for kvm case) of secondary mmus.

Quoted 12 lines

Particularly the following paragraph:
...
Vlastimil Babka points out another race which this patch protects against.
try_to_unmap_one() might reach its mlock_vma_page() TestSetPageMlocked a
moment after munlock_vma_pages_all() did its Phase 1 TestClearPageMlocked:
leaving PageMlocked and unevictable when it should be evictable. mmap_sem is ineffective because exit_mmap() does not hold it; page lock ineffective because __munlock_pagevec() only takes it afterwards, in Phase 2; pte lock is effective because __munlock_pagevec_fill() takes it to get the page, after VM_LOCKED was cleared from vm_flags, so visible to try_to_unmap_one.
...

So this is saying the race with exit_mmap() isn't benign after all?

External email: Use caution opening links or attachments

Quoted 3 lines · Liam Howlett

Quoted 1 line · Shakeel Butt

On Tue, May 25, 2021 at 11:40 AM Liam Howlett <[email protected]>

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

[...]

Quoted 3 lines

Quoted 3 lines

+/*
+ * Walks the vma's mapping a page and mlocks the page if any locked

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

+ * found. Once one is found the page is locked and the scan can be

Quoted 7 lines

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

+ */

Can you please add that this requires the mmap_sem() lock to the comments?

Why does this require mmap_sem() lock? Also mmap_sem() lock of which

Quoted 5 lines

Doesn't the mlock_vma_page() require the mmap_sem() for reading? The mm_struct in vma->vm_mm;

We are traversing all the vmas where this page is mapped of possibly different mm_structs. I don't think we want to take mmap_sem() of all those mm_structs. The commit b87537d9e2fe ("mm: rmap use pte lock not mmap_sem to set PageMlocked") removed exactly that.

Quoted 7 lines

From what I can see, at least the following paths have mmap_lock held for writing:

munlock_vma_pages_range() from __do_munmap()
munlokc_vma_pages_range() from remap_file_pages()

The following path does not hold mmap_sem:

exit_mmap() -> munlock_vma_pages_all() -> munlock_vma_pages_range().

I would really suggest all to carefully read the commit message of b87537d9e2fe ("mm: rmap use pte lock not mmap_sem to set PageMlocked").

Particularly the following paragraph:
...
Vlastimil Babka points out another race which this patch protects

try_to_unmap_one() might reach its mlock_vma_page() TestSetPageMlocked

moment after munlock_vma_pages_all() did its Phase 1

leaving PageMlocked and unevictable when it should be evictable.

is ineffective because exit_mmap() does not hold it; page lock

because __munlock_pagevec() only takes it afterwards, in Phase 2; pte

is effective because __munlock_pagevec_fill() takes it to get the page, after VM_LOCKED was cleared from vm_flags, so visible to

...

Alistair, please bring back the VM_LOCKED check with pte lock held and the comment "Holding pte lock, we do *not* need mmap_lock here".

One positive outcome of this cleanup patch is the removal of unnecessary invalidation (unmapping for kvm case) of secondary mmus.

include/linux/mmu_notifier.h +3 -2

@@ -521,14 +521,14 @@ static inline void mmu_notifier_range_init(struct mmu_notifier_range *range,
 	range->flags = flags;
 }
 
-static inline void mmu_notifier_range_init_migrate(
-			struct mmu_notifier_range *range, unsigned int flags,
+static inline void mmu_notifier_range_init_owner(
+			struct mmu_notifier_range *range,
+			enum mmu_notifier_event event, unsigned int flags,

Currently if copy_nonpresent_pte() returns a non-zero value it is assumed to be a swap entry which requires further processing outside the loop in copy_pte_range() after dropping locks. This prevents other values being returned to signal conditions such as failure which a subsequent change requires.

Instead make copy_nonpresent_pte() return an error code if further processing is required and read the value for the swap entry in the main loop under the ptl.

Signed-off-by Alistair Popple <[email protected]>

---

v9:

New for v9 to allow device exclusive handling to occur in copy_nonpresent_pte().

---
 mm/memory.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

mm/memory.c +1 -5

index 2fb455c365c2..e061cfa18c11 100644
@@ -718,7 +718,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 
 	if (likely(!non_swap_entry(entry))) {
 		if (swap_duplicate(entry) < 0)
-			return entry.val;
+			return -EAGAIN;
 
 		/* make sure dst_mm is on swapoff's mmlist. */
 		if (unlikely(list_empty(&dst_mm->mmlist))) {

@@ -974,11 +974,13 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			continue;
 		}
 		if (unlikely(!pte_present(*src_pte))) {
-			entry.val = copy_nonpresent_pte(dst_mm, src_mm,
-							dst_pte, src_pte,
-							src_vma, addr, rss);
-			if (entry.val)

Quoted 54 lines · Alistair Popple

Currently if copy_nonpresent_pte() returns a non-zero value it is assumed to be a swap entry which requires further processing outside the loop in copy_pte_range() after dropping locks. This prevents other values being returned to signal conditions such as failure which a subsequent change requires.

Instead make copy_nonpresent_pte() return an error code if further processing is required and read the value for the swap entry in the main loop under the ptl.

Signed-off-by Alistair Popple <[email protected]>

---

v9:

New for v9 to allow device exclusive handling to occur in copy_nonpresent_pte().
---

 mm/memory.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

mm/memory.c +0 -0

index 2fb455c365c2..e061cfa18c11 100644
@@ -718,7 +718,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct

mm_struct *src_mm,>
if (likely(!non_swap_entry(entry))) {

if (swap_duplicate(entry) < 0)

mm/memory.c +1 -1

-                     return entry.val;
+                     return -EAGAIN;
 
              /* make sure dst_mm is on swapoff's mmlist. */
              if (unlikely(list_empty(&dst_mm->mmlist))) {

@@ -974,11 +974,13 @@ copy_pte_range(struct vm_area_struct *dst_vma,

struct vm_area_struct *src_vma,>

                      continue;
              
              }
              if (unlikely(!pte_present(*src_pte))) {

mm/memory.c +4 -4

-                     entry.val = copy_nonpresent_pte(dst_mm, src_mm,
-                                                     dst_pte, src_pte,
-                                                     src_vma, addr, rss);
-                     if (entry.val)
+                     ret = copy_nonpresent_pte(dst_mm, src_mm,
+                                             dst_pte, src_pte,
+                                             src_vma, addr, rss);
+                     if (ret == -EAGAIN) {

mm/memory.c copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, · line 986

Quoted 10 lines

mm/memory.c +2 -0

+                             entry = pte_to_swp_entry(*src_pte);
 
                              break;
 
+                     }
 
                      progress += 8;
                      continue;
 
              }

Note that -EAGAIN was previously used by copy_present_page() for early cow use. Here later although we check entry.val first:

        if (entry.val) {
                if (add_swap_count_continuation(entry, GFP_KERNEL) < 0) {
                        ret = -ENOMEM;
                        goto out;
                }
                entry.val = 0;
        } else if (ret) {
                WARN_ON_ONCE(ret != -EAGAIN);
                prealloc = page_copy_prealloc(src_mm, src_vma, addr);
                if (!prealloc)
                        return -ENOMEM;
                /* We've captured and resolved the error. Reset, try again.
*/ ret = 0;
        }

We didn't reset "ret" in entry.val case (maybe we should?). Then in the next round of "goto again" if "ret" is unluckily untouched, it could reach the 2nd if check, and I think it could cause an unexpected page_copy_prealloc().

--
Peter Xu

Quoted 84 lines · AEST Peter Xu

Quoted 54 lines · Alistair Popple

Currently if copy_nonpresent_pte() returns a non-zero value it is assumed to be a swap entry which requires further processing outside the loop in copy_pte_range() after dropping locks. This prevents other values being returned to signal conditions such as failure which a subsequent change requires.

Instead make copy_nonpresent_pte() return an error code if further processing is required and read the value for the swap entry in the main loop under the ptl.

Signed-off-by Alistair Popple <[email protected]>

---

v9:

New for v9 to allow device exclusive handling to occur in copy_nonpresent_pte().
---

 mm/memory.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

mm/memory.c +0 -0

index 2fb455c365c2..e061cfa18c11 100644
@@ -718,7 +718,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct

mm_struct *src_mm,>
if (likely(!non_swap_entry(entry))) {

if (swap_duplicate(entry) < 0)

mm/memory.c +1 -1

-                     return entry.val;
+                     return -EAGAIN;
 
              /* make sure dst_mm is on swapoff's mmlist. */
              if (unlikely(list_empty(&dst_mm->mmlist))) {

@@ -974,11 +974,13 @@ copy_pte_range(struct vm_area_struct *dst_vma,

struct vm_area_struct *src_vma,>

                      continue;
              
              }
              if (unlikely(!pte_present(*src_pte))) {

mm/memory.c +4 -4

-                     entry.val = copy_nonpresent_pte(dst_mm, src_mm,
-                                                     dst_pte, src_pte,
-                                                     src_vma, addr, rss);
-                     if (entry.val)
+                     ret = copy_nonpresent_pte(dst_mm, src_mm,
+                                             dst_pte, src_pte,
+                                             src_vma, addr, rss);
+                     if (ret == -EAGAIN) {

mm/memory.c copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, · line 986

Quoted 10 lines

mm/memory.c +2 -0

+                             entry = pte_to_swp_entry(*src_pte);
 
                              break;
 
+                     }
 
                      progress += 8;
                      continue;
 
              }

Note that -EAGAIN was previously used by copy_present_page() for early cow use. Here later although we check entry.val first:

        if (entry.val) {
                if (add_swap_count_continuation(entry, GFP_KERNEL) < 0) {
                        ret = -ENOMEM;
                        goto out;
                }
                entry.val = 0;
        } else if (ret) {
                WARN_ON_ONCE(ret != -EAGAIN);
                prealloc = page_copy_prealloc(src_mm, src_vma, addr);
                if (!prealloc)
                        return -ENOMEM;
                /* We've captured and resolved the error. Reset, try again.
*/ ret = 0;
        }

Quoted 4 lines

We didn't reset "ret" in entry.val case (maybe we should?). Then in the next round of "goto again" if "ret" is unluckily untouched, it could reach the 2nd if check, and I think it could cause an unexpected page_copy_prealloc().

Thanks, I had considered that but saw "ret" was always set either by copy_nonpresent_pte() or copy_present_pte(). However missed the "unlucky" case at the start of the loop:

	if (progress >= 32) {
		progress = 0;
		if (need_resched() ||
				spin_needbreak(src_ptl) || pin_needbreak(dst_ptl))
			break;

Looking at this again though checking different variables to figure out what to do outside the locks and reusing error codes seems error prone. I reused - EAGAIN for copy_nonpresent_pte() simply because that seemed the most sensible error code, but I don't think that aids readability and it might be better to use a unique error code for each case needing extra handling.

So it might be better if I update this patch to:
1) Use unique error codes for each case requiring special handling outside the lock.
2) Only check "ret" to determine what to do outside locks (ie. not entry.val)
3) Document these.
4) Always reset ret after handling.

Thoughts?

Some devices require exclusive write access to shared virtual memory (SVM) ranges to perform atomic operations on that memory. This requires CPU page tables to be updated to deny access whilst atomic operations are occurring.

In order to do this introduce a new swap entry
type (SWP_DEVICE_EXCLUSIVE). When a SVM range needs to be marked for exclusive access by a device all page table mappings for the particular range are replaced with device exclusive swap entries. This causes any CPU access to the page to result in a fault.

Faults are resovled by replacing the faulting entry with the original mapping. This results in MMU notifiers being called which a driver uses to update access permissions such as revoking atomic access. After notifiers have been called the device will no longer have exclusive access to the region.

Walking of the page tables to find the target pages is handled by get_user_pages() rather than a direct page table walk. A direct page table walk similar to what migrate_vma_collect()/unmap() does could also have been utilised. However this resulted in more code similar in functionality to what get_user_pages() provides as page faulting is required to make the PTEs present and to break COW.

...

 Documentation/vm/hmm.rst     |  17 ++++
 include/linux/mmu_notifier.h |   6 ++
 include/linux/rmap.h         |   4 +
 include/linux/swap.h         |   7 +-
 include/linux/swapops.h      |  44 ++++++++-
 mm/hmm.c                     |   5 +
 mm/memory.c                  | 128 +++++++++++++++++++++++-
 mm/mprotect.c                |   8 ++
 mm/page_vma_mapped.c         |   9 +-
 mm/rmap.c                    | 186 +++++++++++++++++++++++++++++++++++
 10 files changed, 405 insertions(+), 9 deletions(-)

Quoted 14 lines

...

  Documentation/vm/hmm.rst     |  17 ++++
  include/linux/mmu_notifier.h |   6 ++
  include/linux/rmap.h         |   4 +
  include/linux/swap.h         |   7 +-
  include/linux/swapops.h      |  44 ++++++++-
  mm/hmm.c                     |   5 +
  mm/memory.c                  | 128 +++++++++++++++++++++++-
  mm/mprotect.c                |   8 ++
  mm/page_vma_mapped.c         |   9 +-
  mm/rmap.c                    | 186 +++++++++++++++++++++++++++++++++++
  10 files changed, 405 insertions(+), 9 deletions(-)

This is quite a lot of code added to core MM for a single driver.

Is there any expectation that other drivers will use this code?

Is there a way of reducing the impact (code size, at least) for systems which don't need this code?

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

Quoted 14 lines · Andrew Morton

Quoted 13 lines

...

  Documentation/vm/hmm.rst     |  17 ++++
  include/linux/mmu_notifier.h |   6 ++
  include/linux/rmap.h         |   4 +
  include/linux/swap.h         |   7 +-
  include/linux/swapops.h      |  44 ++++++++-
  mm/hmm.c                     |   5 +
  mm/memory.c                  | 128 +++++++++++++++++++++++-
  mm/mprotect.c                |   8 ++
  mm/page_vma_mapped.c         |   9 +-
  mm/rmap.c                    | 186 +++++++++++++++++++++++++++++++++++
  10 files changed, 405 insertions(+), 9 deletions(-)

Quoted 3 lines

This is quite a lot of code added to core MM for a single driver.

Is there any expectation that other drivers will use this code?

Yes! This should work for GPUs (and potentially, other devices) that support OpenCL SVM atomic accesses on the device. I haven't looked into how amdgpu works in any detail, but that's certainly at the top of the list of likely additional callers.

Quoted 2 lines

Is there a way of reducing the impact (code size, at least) for systems which don't need this code?

I'll leave this question to others for the moment, in order to answer the "do we need it at all" points.

Quoted 3 lines

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

So this is interesting. Right now, OpenCL support in Nouveau is rather new and so probably not a huge impact yet. However, we've built up enough experience with CUDA and OpenCL to learn that atomic operations, as part of the user space programming model, are a super big deal. Atomic operations are so useful and important that I'd expect many OpenCL SVM users to be uninterested in programming models that lack atomic operations for GPU compute programs.

Again, this doesn't rule out future, non-GPU accelerator devices that may come along.

Atomic ops are just a really important piece of high-end multi-threaded programming, it turns out. So this is the beginning of support for an important building block for general purpose programming on devices that have GPU-like memory models.

thanks,

Quoted 26 lines · On Mon, 24 May 2021 23:27:22 +1000 Alistair Popple <[email protected]> wrote:

Some devices require exclusive write access to shared virtual memory (SVM) ranges to perform atomic operations on that memory. This requires CPU page tables to be updated to deny access whilst atomic operations are occurring.

In order to do this introduce a new swap entry
type (SWP_DEVICE_EXCLUSIVE). When a SVM range needs to be marked for exclusive access by a device all page table mappings for the particular range are replaced with device exclusive swap entries. This causes any CPU access to the page to result in a fault.

Faults are resovled by replacing the faulting entry with the original mapping. This results in MMU notifiers being called which a driver uses to update access permissions such as revoking atomic access. After notifiers have been called the device will no longer have exclusive access to the region.

Walking of the page tables to find the target pages is handled by get_user_pages() rather than a direct page table walk. A direct page table walk similar to what migrate_vma_collect()/unmap() does could also have been utilised. However this resulted in more code similar in functionality to what get_user_pages() provides as page faulting is required to make the PTEs present and to break COW.

...

Quoted 12 lines

 Documentation/vm/hmm.rst     |  17 ++++
 include/linux/mmu_notifier.h |   6 ++
 include/linux/rmap.h         |   4 +
 include/linux/swap.h         |   7 +-
 include/linux/swapops.h      |  44 ++++++++-
 mm/hmm.c                     |   5 +
 mm/memory.c                  | 128 +++++++++++++++++++++++-
 mm/mprotect.c                |   8 ++
 mm/page_vma_mapped.c         |   9 +-
 mm/rmap.c                    | 186 +++++++++++++++++++++++++++++++++++
 10 files changed, 405 insertions(+), 9 deletions(-)

This is quite a lot of code added to core MM for a single driver.

Is there any expectation that other drivers will use this code?

Is there a way of reducing the impact (code size, at least) for systems which don't need this code?

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

Quoted 3 lines

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

That is a very good question! I've not reviewed the code, but a sample program with the described use case would make things easy to parse. I suspect that is not easy to build at the moment?

I wonder how we co-ordinate all the work the mm is doing, page migration, reclaim with device exclusive access? Do we have any numbers for the worst case page fault latency when something is marked away for exclusive access?

I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would

only impact the address space of programs using the GPU. Should the exclusively marked range live in the unreclaimable list and recycled back to active/in-active to account for the fact that

1. It is not reclaimable and reclaim will only hurt via page faults?
2. It ages the page correctly or at-least allows for that possibility when the page is used by the GPU.

On 5/25/21 4:51 AM, Balbir Singh wrote:
...

Quoted 7 lines

Quoted 3 lines

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

That is a very good question! I've not reviewed the code, but a sample program with the described use case would make things easy to parse. I suspect that is not easy to build at the moment?

The cover letter says this:

This has been tested with upstream Mesa 21.1.0 and a simple OpenCL program which checks that GPU atomic accesses to system memory are atomic. Without this series the test fails as there is no way of write-protecting the page mapping which results in the device clobbering CPU writes. For reference the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/

Further testing has been performed by adding support for testing exclusive access to the hmm-tests kselftests.

...so that seems to cover the "sample program" request, at least.

Quoted 4 lines

I wonder how we co-ordinate all the work the mm is doing, page migration, reclaim with device exclusive access? Do we have any numbers for the worst case page fault latency when something is marked away for exclusive access?

CPU page fault latency is approximately "terrible", if a page is resident on the GPU. We have to spin up a DMA engine on the GPU and have it copy the page over the PCIe bus, after all.

Quoted 2 lines

I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would

Yes, for now.

Quoted 8 lines

only impact the address space of programs using the GPU. Should the exclusively marked range live in the unreclaimable list and recycled back to active/in-active to account for the fact that

1. It is not reclaimable and reclaim will only hurt via page faults?
2. It ages the page correctly or at-least allows for that possibility when the>
page is used by the GPU.

I'm not sure that that is *necessarily* something we can conclude. It depends upon access patterns of each program. For example, a "reduction" parallel program sends over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back to the CPU. In that case, freeing the physical page on the CPU is actually the best decision for the OS to make (if the OS is sufficiently prescient).

thanks,

On 5/25/21 4:51 AM, Balbir Singh wrote:
...

Quoted 8 lines

Quoted 3 lines

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

That is a very good question! I've not reviewed the code, but a sample program with the described use case would make things easy to parse. I suspect that is not easy to build at the moment?

The cover letter says this:

This has been tested with upstream Mesa 21.1.0 and a simple OpenCL program which checks that GPU atomic accesses to system memory are atomic. Without this series the test fails as there is no way of write-protecting the page mapping which results in the device clobbering CPU writes. For reference the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/

Further testing has been performed by adding support for testing exclusive access to the hmm-tests kselftests.

...so that seems to cover the "sample program" request, at least.

Quoted 4 lines

I wonder how we co-ordinate all the work the mm is doing, page migration, reclaim with device exclusive access? Do we have any numbers for the worst case page fault latency when something is marked away for exclusive access?

CPU page fault latency is approximately "terrible", if a page is resident on the GPU. We have to spin up a DMA engine on the GPU and have it copy the page over the PCIe bus, after all.

Quoted 1 line

I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would

Yes, for now.

Quoted 7 lines

only impact the address space of programs using the GPU. Should the exclusively marked range live in the unreclaimable list and recycled back to active/in-active to account for the fact that

1. It is not reclaimable and reclaim will only hurt via page faults?
2. It ages the page correctly or at-least allows for that possibility when the page is used by the GPU.

I'm not sure that that is *necessarily* something we can conclude. It depends upon access patterns of each program. For example, a "reduction" parallel program sends over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back to the CPU. In that case, freeing the physical page on the CPU is actually the best decision for the OS to make (if the OS is sufficiently prescient).

Quoted 19 lines · John Hubbard

On 5/25/21 4:51 AM, Balbir Singh wrote:
...

Quoted 8 lines

Quoted 3 lines

How beneficial is this code to nouveau users? I see that it permits a part of OpenCL to be implemented, but how useful/important is this in the real world?

That is a very good question! I've not reviewed the code, but a sample program with the described use case would make things easy to parse. I suspect that is not easy to build at the moment?

The cover letter says this:

This has been tested with upstream Mesa 21.1.0 and a simple OpenCL program which checks that GPU atomic accesses to system memory are atomic. Without this series the test fails as there is no way of write-protecting the page mapping which results in the device clobbering CPU writes. For reference the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/

Quoted 4 lines

Further testing has been performed by adding support for testing exclusive access to the hmm-tests kselftests.

...so that seems to cover the "sample program" request, at least.

Thanks, I'll take a look

Quoted 21 lines

Quoted 4 lines

I wonder how we co-ordinate all the work the mm is doing, page migration, reclaim with device exclusive access? Do we have any numbers for the worst case page fault latency when something is marked away for exclusive access?

CPU page fault latency is approximately "terrible", if a page is resident on the GPU. We have to spin up a DMA engine on the GPU and have it copy the page over the PCIe bus, after all.

Quoted 1 line

I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would

Yes, for now.

Quoted 7 lines

only impact the address space of programs using the GPU. Should the exclusively marked range live in the unreclaimable list and recycled back to active/in-active to account for the fact that

1. It is not reclaimable and reclaim will only hurt via page faults?
2. It ages the page correctly or at-least allows for that possibility when the page is used by the GPU.

Quoted 6 lines

I'm not sure that that is *necessarily* something we can conclude. It depends upon access patterns of each program. For example, a "reduction" parallel program sends over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back to the CPU. In that case, freeing the physical page on the CPU is actually the best decision for the OS to make (if the OS is sufficiently prescient).

With a shared device or a device exclusive range, it would be good to get the device usage pattern and update the mm with that knowledge, so that the LRU can be better maintained. With your comment you seem to suggest that a page used by the GPU might be a good candidate for reclaim based on the CPU's understanding of the age of the page should not account for use by the device
(are GPU workloads - access once and discard?)

External email: Use caution opening links or attachments

Quoted 64 lines · Balbir Singh

Quoted 24 lines · John Hubbard

On 5/25/21 4:51 AM, Balbir Singh wrote:
...

Quoted 9 lines

Quoted 5 lines

How beneficial is this code to nouveau users? I see that it permits a
part of OpenCL to be implemented, but how useful/important is this in
the real world?

That is a very good question! I've not reviewed the code, but a sample program with the described use case would make things easy to parse. I suspect that is not easy to build at the moment?

The cover letter says this:

This has been tested with upstream Mesa 21.1.0 and a simple OpenCL program
which checks that GPU atomic accesses to system memory are atomic. Without
this series the test fails as there is no way of write-protecting the page
mapping which results in the device clobbering CPU writes. For reference the test is available at https://ozlabs.org/~apopple/opencl_svm_atomics/

Quoted 5 lines

Further testing has been performed by adding support for testing exclusive
access to the hmm-tests kselftests.

...so that seems to cover the "sample program" request, at least.

Thanks, I'll take a look

Quoted 25 lines

Quoted 6 lines

I wonder how we co-ordinate all the work the mm is doing, page migration,
reclaim with device exclusive access? Do we have any numbers for the worst
case page fault latency when something is marked away for exclusive access?

CPU page fault latency is approximately "terrible", if a page is resident on the GPU. We have to spin up a DMA engine on the GPU and have it copy the page over the PCIe bus, after all.

Quoted 2 lines

I presume for now this is anonymous memory only? SWP_DEVICE_EXCLUSIVE would

Yes, for now.

Quoted 8 lines

only impact the address space of programs using the GPU. Should the exclusively marked range live in the unreclaimable list and recycled back to active/in-active to account for the fact that

1. It is not reclaimable and reclaim will only hurt via page faults?
2. It ages the page correctly or at-least allows for that possibility when the> > >
page is used by the GPU.

Quoted 6 lines

I'm not sure that that is *necessarily* something we can conclude. It depends upon access patterns of each program. For example, a "reduction" parallel program sends over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back to the CPU. In that case, freeing the physical page on the CPU is actually the best decision for the OS to make (if the OS is sufficiently prescient).>

Quoted 7 lines

With a shared device or a device exclusive range, it would be good to get the device usage pattern and update the mm with that knowledge, so that the LRU can be better maintained. With your comment you seem to suggest that a page used by the GPU might be a good candidate for reclaim based on the CPU's understanding of the age of the page should not account for use by the device
(are GPU workloads - access once and discard?)

Hmm, besides the aging info, this reminded me: do we need to isolate the page from lru too when marking device exclusive access?

Afaict the current patch didn't do that so I think it's reclaimable. If we still have the rmap then we'll get a mmu notify CLEAR when unmapping that special pte, so device driver should be able to drop the ownership. However we dropped the rmap when marking exclusive. Now I don't know whether and how it'll work if page reclaim runs with the page being exclusively owned if without isolating the page..

--
Peter Xu

Reclaim won't run on the page due to the extra references from the special swap entries.

External email: Use caution opening links or attachments

Quoted 2 lines · Alistair Popple

Reclaim won't run on the page due to the extra references from the special swap entries.

That sounds reasonable, but I didn't find the point that stops it, probably due to my limited knowledge on the reclaim code. Could you elaborate?

--
Peter Xu

Quoted 8 lines · AEST Peter Xu

External email: Use caution opening links or attachments

Quoted 2 lines · Alistair Popple

Reclaim won't run on the page due to the extra references from the special swap entries.

That sounds reasonable, but I didn't find the point that stops it, probably due to my limited knowledge on the reclaim code. Could you elaborate?

Sure, it isn't immediately obvious but it ends up being detected at the start of is_page_cache_freeable() in the pageout code:

static pageout_t pageout(struct page *page, struct address_space *mapping) {

[...]

	if (!is_page_cache_freeable(page))
		return PAGE_KEEP;

Quoted 8 lines

Quoted 7 lines

only impact the address space of programs using the GPU. Should the exclusively marked range live in the unreclaimable list and recycled back to active/in-active to account for the fact that

1. It is not reclaimable and reclaim will only hurt via page faults?
2. It ages the page correctly or at-least allows for that possibility when the page is used by the GPU.

Quoted 6 lines

I'm not sure that that is *necessarily* something we can conclude. It depends upon access patterns of each program. For example, a "reduction" parallel program sends over lots of data to the GPU, and only a tiny bit of (reduced!) data comes back to the CPU. In that case, freeing the physical page on the CPU is actually the best decision for the OS to make (if the OS is sufficiently prescient).

With a shared device or a device exclusive range, it would be good to get the device usage pattern and update the mm with that knowledge, so that the LRU can be better

maintained. With your comment you seem to suggest that a page used by the GPU might be a good candidate for reclaim based on the CPU's understanding of the age of the page should not account for use by the device
(are GPU workloads - access once and discard?)

Some devices require exclusive write access to shared virtual memory (SVM) ranges to perform atomic operations on that memory. This requires CPU page tables to be updated to deny access whilst atomic operations are occurring.

In order to do this introduce a new swap entry
type (SWP_DEVICE_EXCLUSIVE). When a SVM range needs to be marked for exclusive access by a device all page table mappings for the particular range are replaced with device exclusive swap entries. This causes any CPU access to the page to result in a fault.

Faults are resovled by replacing the faulting entry with the original mapping. This results in MMU notifiers being called which a driver uses to update access permissions such as revoking atomic access. After notifiers have been called the device will no longer have exclusive access to the region.

Walking of the page tables to find the target pages is handled by get_user_pages() rather than a direct page table walk. A direct page table walk similar to what migrate_vma_collect()/unmap() does could also have been utilised. However this resulted in more code similar in functionality to what get_user_pages() provides as page faulting is required to make the PTEs present and to break COW.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v9:
* Split rename of migrate_pgmap_owner into a separate patch.
* Added comments explaining SWP_DEVICE_EXCLUSIVE_* entries.
* Renamed try_to_protect{_one} to page_make_device_exclusive{_one} based somewhat on a suggestion from Peter Xu. I was never particularly happy with try_to_protect() as a name so think this is better.
* Removed unneccesary code and reworded some comments based on feedback from Peter Xu.
* Removed the VMA walk when restoring PTEs for device-exclusive entries.
* Simplified implementation of copy_pte_range() to fail if the page cannot be locked. This might lead to occasional fork() failures but at this stage we don't think that will be an issue.

v8:
* Remove device exclusive entries on fork rather than copy them.

v7:
* Added Christoph's Reviewed-by.
* Minor cosmetic cleanups suggested by Christoph.
* Replace mmu_notifier_range_init_migrate/exclusive with mmu_notifier_range_init_owner as suggested by Christoph.
* Replaced lock_page() with lock_page_retry() when handling faults.
* Restrict to anonymous pages for now.

v6:
* Fixed a bisectablity issue due to incorrectly applying the rename of migrate_pgmap_owner to the wrong patches for Nouveau and hmm_test.

v5:
* Renamed range->migrate_pgmap_owner to range->owner.
* Added MMU_NOTIFY_EXCLUSIVE to allow passing of a driver cookie which allows notifiers called as a result of make_device_exclusive_range() to be ignored.
* Added a check to try_to_protect_one() to detect if the pages originally returned from get_user_pages() have been unmapped or not.
* Removed check_device_exclusive_range() as it is no longer required with the other changes.
* Documentation update.

v4:
* Add function to check that mappings are still valid and exclusive.
* s/long/unsigned long/ in make_device_exclusive_entry().

---
 Documentation/vm/hmm.rst     |  17 ++++
 include/linux/mmu_notifier.h |   6 ++
 include/linux/rmap.h         |   4 +
 include/linux/swap.h         |   7 +-
 include/linux/swapops.h      |  44 ++++++++-
 mm/hmm.c                     |   5 +
 mm/memory.c                  | 128 +++++++++++++++++++++++-
 mm/mprotect.c                |   8 ++
 mm/page_vma_mapped.c         |   9 +-
 mm/rmap.c                    | 186 +++++++++++++++++++++++++++++++++++
 10 files changed, 405 insertions(+), 9 deletions(-)

Documentation/vm/hmm.rst +17 -0

index 3df79307a797..a14c2938e7af 100644
@@ -405,6 +405,23 @@ between device driver specific code and shared common code:
 
    The lock can now be released.
 
+Exclusive access memory
+=======================
+
+Some devices have features such as atomic PTE bits that can be used to implement
+atomic access to system memory. To support atomic operations to a shared virtual
+memory page such a device needs access to that page which is exclusive of any
+userspace access from the CPU. The ``make_device_exclusive_range()`` function
+can be used to make a memory range inaccessible from userspace.
+
+This replaces all mappings for pages in the given range with special swap
+entries. Any attempt to access the swap entry results in a fault which is
+resovled by replacing the entry with the original mapping. A driver gets
+notified that the mapping has been changed by MMU notifiers, after which point
+it will no longer have exclusive access to the page. Exclusive access is
+guranteed to last until the driver drops the page lock and page reference, at
+which point any CPU faults on the page may proceed as described.
+
 Memory cgroup (memcg) and rss accounting
 ========================================
 

include/linux/mmu_notifier.h +0 -0

index 8e428eb813b8..d049e0f6f756 100644

include/linux/mmu_notifier.h +1 -0

  */
 enum mmu_notifier_event {
 	MMU_NOTIFY_UNMAP = 0,

@@ -51,6 +56,7 @@ enum mmu_notifier_event {
 	MMU_NOTIFY_SOFT_DIRTY,
 	MMU_NOTIFY_RELEASE,
 	MMU_NOTIFY_MIGRATE,
+	MMU_NOTIFY_EXCLUSIVE,
 };
 
 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)

include/linux/rmap.h +4 -0

index 0e25d829f742..3a1ce4ef9276 100644
@@ -193,6 +193,10 @@ int page_referenced(struct page *, int is_locked,
 bool try_to_migrate(struct page *page, enum ttu_flags flags);
 bool try_to_unmap(struct page *, enum ttu_flags flags);
 
+int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
+				unsigned long end, struct page **pages,
+				void *arg);
+
 /* Avoid racy checks */
 #define PVMW_SYNC		(1 << 0)
 /* Look for migarion entries rather than present PTEs */

include/linux/swap.h +0 -0

index a6d4505ecf73..306df39d7c67 100644

include/linux/swap.h +6 -1

+ *
+ * When a page is mapped by the device for exclusive access we set the CPU page
+ * table entries to special SWP_DEVICE_EXCLUSIVE_* entries.
  */
 #ifdef CONFIG_DEVICE_PRIVATE
-#define SWP_DEVICE_NUM 2
+#define SWP_DEVICE_NUM 4
 #define SWP_DEVICE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)
 #define SWP_DEVICE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)
+#define SWP_DEVICE_EXCLUSIVE_WRITE (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2)
+#define SWP_DEVICE_EXCLUSIVE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3)
 #else
 #define SWP_DEVICE_NUM 0
 #endif

include/linux/swapops.h +43 -1

index 4dfd807ae52a..4129bd2ff9d6 100644
@@ -120,6 +120,27 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry)
 {
 	return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);
 }
+
+static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(SWP_DEVICE_EXCLUSIVE_READ, offset);
+}
+
+static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(SWP_DEVICE_EXCLUSIVE_WRITE, offset);
+}
+
+static inline bool is_device_exclusive_entry(swp_entry_t entry)
+{
+	return swp_type(entry) == SWP_DEVICE_EXCLUSIVE_READ ||
+		swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE;
+}
+
+static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)
+{
+	return unlikely(swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE);
+}
 #else /* CONFIG_DEVICE_PRIVATE */
 static inline swp_entry_t make_readable_device_private_entry(pgoff_t offset)
 {

@@ -140,6 +161,26 @@ static inline bool is_writable_device_private_entry(swp_entry_t entry)
 {
 	return false;
 }
+
+static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(0, 0);
+}
+
+static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t offset)
+{
+	return swp_entry(0, 0);
+}
+
+static inline bool is_device_exclusive_entry(swp_entry_t entry)
+{
+	return false;
+}
+
+static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)
+{
+	return false;
+}
 #endif /* CONFIG_DEVICE_PRIVATE */
 
 #ifdef CONFIG_MIGRATION

@@ -219,7 +260,8 @@ static inline struct page *pfn_swap_entry_to_page(swp_entry_t entry)
  */
 static inline bool is_pfn_swap_entry(swp_entry_t entry)
 {
-	return is_migration_entry(entry) || is_device_private_entry(entry);
+	return is_migration_entry(entry) || is_device_private_entry(entry) ||
+	       is_device_exclusive_entry(entry);
 }
 
 struct page_vma_mapped_walk;

mm/hmm.c +5 -0

index 11df3ca30b82..fad6be2bf072 100644
@@ -26,6 +26,8 @@
 #include <linux/mmu_notifier.h>
 #include <linux/memory_hotplug.h>
 
+#include "internal.h"
+
 struct hmm_vma_walk {
 	struct hmm_range	*range;
 	unsigned long		last;

@@ -271,6 +273,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
 		if (!non_swap_entry(entry))
 			goto fault;
 
+		if (is_device_exclusive_entry(entry))
+			goto fault;
+
 		if (is_migration_entry(entry)) {
 			pte_unmap(ptep);
 			hmm_vma_walk->last = addr;

mm/memory.c +37 -0

index e061cfa18c11..c1d2d732f189 100644
@@ -700,6 +700,68 @@ struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
 }
 #endif
 
+static void restore_exclusive_pte(struct vm_area_struct *vma,
+				  struct page *page, unsigned long address,
+				  pte_t *ptep)
+{
+	pte_t pte;
+	swp_entry_t entry;
+
+	pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot)));
+	if (pte_swp_soft_dirty(*ptep))
+		pte = pte_mksoft_dirty(pte);
+
+	entry = pte_to_swp_entry(*ptep);
+	if (pte_swp_uffd_wp(*ptep))
+		pte = pte_mkuffd_wp(pte);
+	else if (is_writable_device_exclusive_entry(entry))
+		pte = maybe_mkwrite(pte_mkdirty(pte), vma);
+
+	set_pte_at(vma->vm_mm, address, ptep, pte);
+
+	/*
+	 * No need to take a page reference as one was already
+	 * created when the swap entry was made.
+	 */
+	if (PageAnon(page))
+		page_add_anon_rmap(page, vma, address, false);
+	else
+		/*
+		 * Currently device exclusive access only supports anonymous
+		 * memory so the entry shouldn't point to a filebacked page.
+		 */
+		WARN_ON_ONCE(!PageAnon(page));
+
+	if (vma->vm_flags & VM_LOCKED)
+		mlock_vma_page(page);
+
+	/*
+	 * No need to invalidate - it was non-present before. However

mm/memory.c +28 -0

+{
+	swp_entry_t entry = pte_to_swp_entry(*src_pte);
+	struct page *page = pfn_swap_entry_to_page(entry);
+
+	if (trylock_page(page)) {
+		restore_exclusive_pte(vma, page, addr, src_pte);
+		unlock_page(page);
+		return 0;
+	}
+
+	return -EBUSY;
+}
+
 /*
  * copy one vm_area from one task to the other. Assumes the page tables
  * already present in the new task to be cleared in the whole range

@@ -781,6 +843,17 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 				pte = pte_swp_mkuffd_wp(pte);
 			set_pte_at(src_mm, addr, src_pte, pte);
 		}
+	} else if (is_device_exclusive_entry(entry)) {
+		/*
+		 * Make device exclusive entries present by restoring the
+		 * original entry then copying as for a present pte. Device
+		 * exclusive entries currently only support private writable
+		 * (ie. COW) mappings.
+		 */
+		VM_BUG_ON(!is_cow_mapping(vma->vm_flags));
+		if (try_restore_exclusive_pte(src_mm, src_pte, vma, addr))
+			return -EBUSY;
+		return -ENOENT;
 	}
 	set_pte_at(dst_mm, addr, dst_pte, pte);
 	return 0;

@@ -980,9 +1053,18 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			if (ret == -EAGAIN) {
 				entry = pte_to_swp_entry(*src_pte);
 				break;
+			} else if (ret == -EBUSY) {
+				break;
+			} else if (!ret) {
+				progress += 8;

mm/memory.c +13 -2

 		}
 		/* copy_present_pte() will clear `*prealloc' if consumed */
 		ret = copy_present_pte(dst_vma, src_vma, dst_pte, src_pte,

@@ -1019,6 +1101,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
 			goto out;
 		}
 		entry.val = 0;
+	} else if (ret == -EBUSY) {
+		return -EBUSY;
 	} else if (ret) {
 		WARN_ON_ONCE(ret != -EAGAIN);
 		prealloc = page_copy_prealloc(src_mm, src_vma, addr);

@@ -1283,7 +1367,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 		}
 
 		entry = pte_to_swp_entry(ptent);
-		if (is_device_private_entry(entry)) {
+		if (is_device_private_entry(entry) ||
+		    is_device_exclusive_entry(entry)) {
 			struct page *page = pfn_swap_entry_to_page(entry);
 
 			if (unlikely(details && details->check_mapping)) {

@@ -1299,7 +1384,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 
 			pte_clear_not_present_full(mm, addr, pte, tlb->fullmm);
 			rss[mm_counter(page)]--;
-			page_remove_rmap(page, false);
+
+			if (is_device_private_entry(entry))
+				page_remove_rmap(page, false);
+
 			put_page(page);
 			continue;
 		}

@@ -3303,6 +3391,35 @@ void unmap_mapping_range(struct address_space *mapping,
 }
 EXPORT_SYMBOL(unmap_mapping_range);
 
+/*
+ * Restore a potential device exclusive pte to a working pte entry
+ */
+static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf)
+{

mm/memory.c +5 -0

+}
+
 /*
  * We enter with non-exclusive mmap_lock (to exclude vma changes,
  * but allow concurrent faults), and pte mapped but not yet locked.

@@ -3330,6 +3447,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 		if (is_migration_entry(entry)) {
 			migration_entry_wait(vma->vm_mm, vmf->pmd,
 					     vmf->address);
+		} else if (is_device_exclusive_entry(entry)) {
+			vmf->page = pfn_swap_entry_to_page(entry);
+			ret = remove_device_exclusive_entry(vmf);
 		} else if (is_device_private_entry(entry)) {
 			vmf->page = pfn_swap_entry_to_page(entry);
 			ret = vmf->page->pgmap->ops->migrate_to_ram(vmf);

mm/mprotect.c +8 -0

index ee5961888e70..883e2cc85cad 100644
@@ -165,6 +165,14 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
 				newpte = swp_entry_to_pte(entry);
 				if (pte_swp_uffd_wp(oldpte))
 					newpte = pte_swp_mkuffd_wp(newpte);
+			} else if (is_writable_device_exclusive_entry(entry)) {
+				entry = make_readable_device_exclusive_entry(
+							swp_offset(entry));
+				newpte = swp_entry_to_pte(entry);
+				if (pte_swp_soft_dirty(oldpte))
+					newpte = pte_swp_mksoft_dirty(newpte);
+				if (pte_swp_uffd_wp(oldpte))
+					newpte = pte_swp_mkuffd_wp(newpte);
 			} else {
 				newpte = oldpte;
 			}

mm/page_vma_mapped.c +6 -3

index a6a7febb4d93..f535bcb4950c 100644
@@ -41,7 +41,8 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw)
 
 				/* Handle un-addressable ZONE_DEVICE memory */
 				entry = pte_to_swp_entry(*pvmw->pte);
-				if (!is_device_private_entry(entry))
+				if (!is_device_private_entry(entry) &&
+				    !is_device_exclusive_entry(entry))
 					return false;
 			} else if (!pte_present(*pvmw->pte))
 				return false;

@@ -93,7 +94,8 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw)
 			return false;
 		entry = pte_to_swp_entry(*pvmw->pte);
 
-		if (!is_migration_entry(entry))
+		if (!is_migration_entry(entry) &&
+		    !is_device_exclusive_entry(entry))
 			return false;
 
 		pfn = swp_offset(entry);

@@ -102,7 +104,8 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw)
 
 		/* Handle un-addressable ZONE_DEVICE memory */
 		entry = pte_to_swp_entry(*pvmw->pte);
-		if (!is_device_private_entry(entry))
+		if (!is_device_private_entry(entry) &&
+		    !is_device_exclusive_entry(entry))
 			return false;
 
 		pfn = swp_offset(entry);

mm/rmap.c +15 -0

index 8ed1853060cf..fe062f63ef4d 100644
@@ -2008,6 +2008,192 @@ void page_mlock(struct page *page)
 	rmap_walk(page, &rwc);
 }
 
+struct make_exclusive_args {
+	struct mm_struct *mm;
+	unsigned long address;
+	void *owner;
+	bool valid;
+};
+
+static bool page_make_device_exclusive_one(struct page *page,
+		struct vm_area_struct *vma, unsigned long address, void *priv)
+{
+	struct mm_struct *mm = vma->vm_mm;
+	struct page_vma_mapped_walk pvmw = {
+		.page = page,
+		.vma = vma,
+		.address = address,

mm/rmap.c +5 -0

+				      vma->vm_mm, address, min(vma->vm_end,
+				      address + page_size(page)), args->owner);
+	mmu_notifier_invalidate_range_start(&range);
+
+	while (page_vma_mapped_walk(&pvmw)) {

mm/rmap.c +6 -0

+		address = pvmw.address;
+
+		/* Nuke the page table entry. */
+		flush_cache_page(vma, address, pte_pfn(*pvmw.pte));
+		pteval = ptep_clear_flush(vma, address, pvmw.pte);
+

mm/rmap.c +21 -0

+
+		/*
+		 * Check that our target page is still mapped at the expected
+		 * address.
+		 */
+		if (args->mm == mm && args->address == address &&
+		    pte_write(pteval))
+			args->valid = true;
+
+		/*
+		 * Store the pfn of the page in a special migration
+		 * pte. do_swap_page() will wait until the migration
+		 * pte is removed and then restart fault handling.
+		 */
+		if (pte_write(pteval))
+			entry = make_writable_device_exclusive_entry(
+							page_to_pfn(subpage));
+		else
+			entry = make_readable_device_exclusive_entry(
+							page_to_pfn(subpage));
+		swp_pte = swp_entry_to_pte(entry);

mm/rmap.c +89 -0

+ * @page: the page to replace page table entries for
+ * @mm: the mm_struct where the page is expected to be mapped
+ * @address: address where the page is expected to be mapped
+ * @owner: passed to MMU_NOTIFY_EXCLUSIVE range notifier callbacks
+ *
+ * Tries to remove all the page table entries which are mapping this page and
+ * replace them with special device exclusive swap entries to grant a device
+ * exclusive access to the page. Caller must hold the page lock.
+ *
+ * Returns false if the page is still mapped, or if it could not be unmapped
+ * from the expected address. Otherwise returns true (success).
+ */
+static bool page_make_device_exclusive(struct page *page, struct mm_struct *mm,
+				unsigned long address, void *owner)
+{
+	struct make_exclusive_args args = {
+		.mm = mm,
+		.address = address,
+		.owner = owner,
+		.valid = false,
+	};
+	struct rmap_walk_control rwc = {
+		.rmap_one = page_make_device_exclusive_one,
+		.done = page_not_mapped,
+		.anon_lock = page_lock_anon_vma_read,
+		.arg = &args,
+	};
+
+	/*
+	 * Restrict to anonymous pages for now to avoid potential writeback
+	 * issues.
+	 */
+	if (!PageAnon(page))
+		return false;
+
+	rmap_walk(page, &rwc);
+
+	return args.valid && !page_mapcount(page);
+}
+
+/**
+ * make_device_exclusive_range() - Mark a range for exclusive use by a device
+ * @mm: mm_struct of assoicated target process
+ * @start: start of the region to mark for exclusive device access
+ * @end: end address of region
+ * @pages: returns the pages which were successfully marked for exclusive access
+ * @owner: passed to MMU_NOTIFY_EXCLUSIVE range notifier to allow filtering
+ *
+ * Returns: number of pages found in the range by GUP. A page is marked for
+ * exclusive access only if the page pointer is non-NULL.
+ *
+ * This function finds ptes mapping page(s) to the given address range, locks
+ * them and replaces mappings with special swap entries preventing userspace CPU
+ * access. On fault these entries are replaced with the original mapping after
+ * calling MMU notifiers.
+ *
+ * A driver using this to program access from a device must use a mmu notifier
+ * critical section to hold a device specific lock during programming. Once
+ * programming is complete it should drop the page lock and reference after
+ * which point CPU access to the page will revoke the exclusive access.
+ */
+int make_device_exclusive_range(struct mm_struct *mm, unsigned long start,
+				unsigned long end, struct page **pages,
+				void *owner)
+{
+	unsigned long npages = (end - start) >> PAGE_SHIFT;
+	unsigned long i;
+
+	npages = get_user_pages_remote(mm, start, npages,
+				       FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD,
+				       pages, NULL, NULL);
+	for (i = 0; i < npages; i++, start += PAGE_SIZE) {
+		if (!trylock_page(pages[i])) {
+			put_page(pages[i]);
+			pages[i] = NULL;
+			continue;
+		}
+
+		if (!page_make_device_exclusive(pages[i], mm, start, owner)) {
+			unlock_page(pages[i]);
+			put_page(pages[i]);
+			pages[i] = NULL;
+		}
+	}
+
+	return npages;
+}
+EXPORT_SYMBOL_GPL(make_device_exclusive_range);
+
 void __put_anon_vma(struct anon_vma *anon_vma)
 {
 	struct anon_vma *root = anon_vma->root;

Signature · 1 line

2.20.1

Quoted 99 lines · Alistair Popple

Some devices require exclusive write access to shared virtual memory (SVM) ranges to perform atomic operations on that memory. This requires CPU page tables to be updated to deny access whilst atomic operations are occurring.

In order to do this introduce a new swap entry
type (SWP_DEVICE_EXCLUSIVE). When a SVM range needs to be marked for exclusive access by a device all page table mappings for the particular range are replaced with device exclusive swap entries. This causes any CPU access to the page to result in a fault.

Faults are resovled by replacing the faulting entry with the original mapping. This results in MMU notifiers being called which a driver uses to update access permissions such as revoking atomic access. After notifiers have been called the device will no longer have exclusive access to the region.

Walking of the page tables to find the target pages is handled by get_user_pages() rather than a direct page table walk. A direct page table walk similar to what migrate_vma_collect()/unmap() does could also have been utilised. However this resulted in more code similar in functionality to what get_user_pages() provides as page faulting is required to make the PTEs present and to break COW.

Signed-off-by Alistair Popple <[email protected]>

Reviewed-by Christoph Hellwig <[email protected]>

---

v9:
* Split rename of migrate_pgmap_owner into a separate patch.
* Added comments explaining SWP_DEVICE_EXCLUSIVE_* entries.
* Renamed try_to_protect{_one} to page_make_device_exclusive{_one} based

somewhat on a suggestion from Peter Xu. I was never particularly happy with try_to_protect() as a name so think this is better.

* Removed unneccesary code and reworded some comments based on feedback

from Peter Xu.

* Removed the VMA walk when restoring PTEs for device-exclusive entries.
* Simplified implementation of copy_pte_range() to fail if the page

cannot be locked. This might lead to occasional fork() failures but at this stage we don't think that will be an issue.

v8:
* Remove device exclusive entries on fork rather than copy them.

v7:
* Added Christoph's Reviewed-by.
* Minor cosmetic cleanups suggested by Christoph.
* Replace mmu_notifier_range_init_migrate/exclusive with

mmu_notifier_range_init_owner as suggested by Christoph.

* Replaced lock_page() with lock_page_retry() when handling faults.
* Restrict to anonymous pages for now.

v6:
* Fixed a bisectablity issue due to incorrectly applying the rename of

migrate_pgmap_owner to the wrong patches for Nouveau and hmm_test.

v5:
* Renamed range->migrate_pgmap_owner to range->owner.
* Added MMU_NOTIFY_EXCLUSIVE to allow passing of a driver cookie which

allows notifiers called as a result of make_device_exclusive_range() to be ignored.

* Added a check to try_to_protect_one() to detect if the pages originally

returned from get_user_pages() have been unmapped or not.

* Removed check_device_exclusive_range() as it is no longer required with

the other changes.

* Documentation update.

v4:
* Add function to check that mappings are still valid and exclusive.
* s/long/unsigned long/ in make_device_exclusive_entry().
---

 Documentation/vm/hmm.rst     |  17 ++++
 include/linux/mmu_notifier.h |   6 ++
 include/linux/rmap.h         |   4 +
 include/linux/swap.h         |   7 +-
 include/linux/swapops.h      |  44 ++++++++-
 mm/hmm.c                     |   5 +
 mm/memory.c                  | 128 +++++++++++++++++++++++-
 mm/mprotect.c                |   8 ++
 mm/page_vma_mapped.c         |   9 +-
 mm/rmap.c                    | 186 +++++++++++++++++++++++++++++++++++
 10 files changed, 405 insertions(+), 9 deletions(-)

Documentation/vm/hmm.rst between device driver specific code and shared common

Quoted 6 lines

Documentation/vm/hmm.rst +0 -0

index 3df79307a797..a14c2938e7af 100644
@@ -405,6 +405,23 @@ between device driver specific code and shared common 

Quoted 35 lines

Documentation/vm/hmm.rst +4 -0

    The lock can now be released.
 
+Exclusive access memory
+=======================
+
+Some devices have features such as atomic PTE bits that can be used to

implement +atomic access to system memory. To support atomic operations to a shared virtual +memory page such a device needs access to that page which is exclusive of any +userspace access from the CPU. The ``make_device_exclusive_range()`` function +can be used to make a memory range inaccessible from userspace.

Documentation/vm/hmm.rst +5 -0

+
+This replaces all mappings for pages in the given range with special swap
+entries. Any attempt to access the swap entry results in a fault which is
+resovled by replacing the entry with the original mapping. A driver gets
+notified that the mapping has been changed by MMU notifiers, after which

point +it will no longer have exclusive access to the page. Exclusive access is +guranteed to last until the driver drops the page lock and page reference, at +which point any CPU faults on the page may proceed as described.

Documentation/vm/hmm.rst +1 -0

+
 
 Memory cgroup (memcg) and rss accounting
 ========================================

include/linux/mmu_notifier.h +0 -0

index 8e428eb813b8..d049e0f6f756 100644
@@ -42,6 +42,11 @@ struct mmu_interval_notifier;
 
  * @MMU_NOTIFY_MIGRATE: used during migrate_vma_collect() invalidate to
  signal * a device driver to possibly ignore the invalidation if the
  * owner field matches the driver's device private pgmap owner.
 

Quoted 6 lines

include/linux/mmu_notifier.h +2 -0

+ *
+ * @MMU_NOTIFY_EXCLUSIVE: to signal a device driver that the device will

no + * longer have exclusive access to the page. May ignore the invalidation that's + * part of make_device_exclusive_range() if the owner field
+ * matches the value passed to make_device_exclusive_range().

Perhaps s/matches/does not match/?

Quoted 43 lines

*/

enum mmu_notifier_event {

MMU_NOTIFY_UNMAP = 0,

include/linux/mmu_notifier.h +1 -0

@@ -51,6 +56,7 @@ enum mmu_notifier_event {
 
      MMU_NOTIFY_SOFT_DIRTY,
      MMU_NOTIFY_RELEASE,
      MMU_NOTIFY_MIGRATE,
 
+     MMU_NOTIFY_EXCLUSIVE,
 

};

#define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0)

include/linux/rmap.h +4 -0

index 0e25d829f742..3a1ce4ef9276 100644
@@ -193,6 +193,10 @@ int page_referenced(struct page *, int is_locked,
 
 bool try_to_migrate(struct page *page, enum ttu_flags flags);
 bool try_to_unmap(struct page *, enum ttu_flags flags);
 
+int make_device_exclusive_range(struct mm_struct *mm, unsigned long
 start,
+                             unsigned long end, struct page **pages,
+                             void *arg);
+
 

/* Avoid racy checks */
#define PVMW_SYNC (1 << 0)
/* Look for migarion entries rather than present PTEs */

include/linux/swap.h +0 -0

index a6d4505ecf73..306df39d7c67 100644
@@ -63,11 +63,16 @@ static inline int current_is_kswapd(void)
 

include/linux/swap.h static inline int current_is_kswapd(void) · line 66

Quoted 3 lines

include/linux/swap.h +0 -0

  *
  * When a page is migrated from CPU to device, we set the CPU page table
  entry * to a special SWP_DEVICE_* entry.

s/SWP_DEVICE_*/SWP_DEVICE_{READ|WRITE}/? Since SWP_DEVICE_* covers all four too.

Quoted 192 lines

include/linux/swap.h +2 -0

+ *
+ * When a page is mapped by the device for exclusive access we set the

CPU page + * table entries to special SWP_DEVICE_EXCLUSIVE_* entries.

*/

#ifdef CONFIG_DEVICE_PRIVATE

include/linux/swap.h +2 -1

-#define SWP_DEVICE_NUM 2
+#define SWP_DEVICE_NUM 4
 
 #define SWP_DEVICE_WRITE
 (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM)
 #define SWP_DEVICE_READ
 (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+1)> 
+#define SWP_DEVICE_EXCLUSIVE_WRITE

(MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+2) +#define SWP_DEVICE_EXCLUSIVE_READ (MAX_SWAPFILES+SWP_HWPOISON_NUM+SWP_MIGRATION_NUM+3)>
#else
#define SWP_DEVICE_NUM 0
#endif

include/linux/swapops.h +0 -0

index 4dfd807ae52a..4129bd2ff9d6 100644
@@ -120,6 +120,27 @@ static inline bool

is_writable_device_private_entry(swp_entry_t entry)>
{

      return unlikely(swp_type(entry) == SWP_DEVICE_WRITE);

}

include/linux/swapops.h +2 -0

+
+static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t

offset) +{

include/linux/swapops.h +4 -0

+     return swp_entry(SWP_DEVICE_EXCLUSIVE_READ, offset);
+}
+
+static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t

offset) +{

include/linux/swapops.h +13 -0

+     return swp_entry(SWP_DEVICE_EXCLUSIVE_WRITE, offset);
+}
+
+static inline bool is_device_exclusive_entry(swp_entry_t entry)
+{
+     return swp_type(entry) == SWP_DEVICE_EXCLUSIVE_READ ||
+             swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE;
+}
+
+static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)
+{
+     return unlikely(swp_type(entry) == SWP_DEVICE_EXCLUSIVE_WRITE);
+}
 
 #else /* CONFIG_DEVICE_PRIVATE */
 static inline swp_entry_t make_readable_device_private_entry(pgoff_t
 offset) {

@@ -140,6 +161,26 @@ static inline bool

is_writable_device_private_entry(swp_entry_t entry)>
{

      return false;

}

include/linux/swapops.h +2 -0

+
+static inline swp_entry_t make_readable_device_exclusive_entry(pgoff_t

offset) +{

include/linux/swapops.h +4 -0

+     return swp_entry(0, 0);
+}
+
+static inline swp_entry_t make_writable_device_exclusive_entry(pgoff_t

offset) +{

include/linux/swapops.h +13 -1

+     return swp_entry(0, 0);
+}
+
+static inline bool is_device_exclusive_entry(swp_entry_t entry)
+{
+     return false;
+}
+
+static inline bool is_writable_device_exclusive_entry(swp_entry_t entry)
+{
+     return false;
+}
 
 #endif /* CONFIG_DEVICE_PRIVATE */
 
 #ifdef CONFIG_MIGRATION

@@ -219,7 +260,8 @@ static inline struct page
 *pfn_swap_entry_to_page(swp_entry_t entry)> 
  */
 
 static inline bool is_pfn_swap_entry(swp_entry_t entry)
 {
 
-     return is_migration_entry(entry) || is_device_private_entry(entry);
+     return is_migration_entry(entry) || is_device_private_entry(entry)

||

include/linux/swapops.h +1 -0

+            is_device_exclusive_entry(entry);
 
 }
 
 struct page_vma_mapped_walk;

mm/hmm.c +2 -0

index 11df3ca30b82..fad6be2bf072 100644
@@ -26,6 +26,8 @@
 
 #include <linux/mmu_notifier.h>
 #include <linux/memory_hotplug.h>
 
+#include "internal.h"
+
 
 struct hmm_vma_walk {

 
      struct hmm_range        *range;
      unsigned long           last;

mm/hmm.c +0 -0

@@ -271,6 +273,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk,

unsigned long addr,>

              if (!non_swap_entry(entry))
              
                      goto fault;

mm/hmm.c +3 -0

+             if (is_device_exclusive_entry(entry))
+                     goto fault;
+
 
              if (is_migration_entry(entry)) {
 
                      pte_unmap(ptep);
                      hmm_vma_walk->last = addr;

mm/memory.c +41 -0

index e061cfa18c11..c1d2d732f189 100644
@@ -700,6 +700,68 @@ struct page *vm_normal_page_pmd(struct vm_area_struct
 *vma, unsigned long addr,> 
 }
 #endif
 
+static void restore_exclusive_pte(struct vm_area_struct *vma,
+                               struct page *page, unsigned long address,
+                               pte_t *ptep)
+{
+     pte_t pte;
+     swp_entry_t entry;
+
+     pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot)));
+     if (pte_swp_soft_dirty(*ptep))
+             pte = pte_mksoft_dirty(pte);
+
+     entry = pte_to_swp_entry(*ptep);
+     if (pte_swp_uffd_wp(*ptep))
+             pte = pte_mkuffd_wp(pte);
+     else if (is_writable_device_exclusive_entry(entry))
+             pte = maybe_mkwrite(pte_mkdirty(pte), vma);
+
+     set_pte_at(vma->vm_mm, address, ptep, pte);
+
+     /*
+      * No need to take a page reference as one was already
+      * created when the swap entry was made.
+      */
+     if (PageAnon(page))
+             page_add_anon_rmap(page, vma, address, false);
+     else
+             /*
+              * Currently device exclusive access only supports anonymous
+              * memory so the entry shouldn't point to a filebacked page.
+              */
+             WARN_ON_ONCE(!PageAnon(page));
+
+     if (vma->vm_flags & VM_LOCKED)
+             mlock_vma_page(page);
+
+     /*
+      * No need to invalidate - it was non-present before. However
+      * secondary CPUs may have mappings that need invalidating.
+      */
+     update_mmu_cache(vma, address, ptep);
+}