Skip to contentSkip to conversation

mm: remove extra ZONE_DEVICE struct page refcount

RFC PATCH Superseded Ralph Campbell 4 participants 10 messages Sep 25, 2020 – Sep 29, 2020

Subsystems (17)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover mm: remove extra ZONE_DEVICE struct page refcount

Matthew Wilcox, Ira Weiny, and others have complained that ZONE_DEVICE struct page reference counting is ugly because they are "free" when the reference count is one instead of zero. This leads to explicit checks for ZONE_DEVICE pages in places like put_page(), GUP, THP splitting, and page migration which have to adjust the expected reference count when determining if the page is isolated or idle. This is my attempt to make ZONE_DEVICE pages be free when the reference count is zero and removing the special cases.

I'm only sending this out as a RFC since I'm not that familiar with the DAX, PMEM, XEN, and other uses of ZONE_DEVICE struct pages allocated with devm_memremap_pages() or memremap_pages() but my best reading of the code looks like it might be OK. I could use help testing these configurations.
I have been able to successfully run xfstests on ext4 with the memmap kernel boot option to simulate pmem.

One of the big changes in v2 is that devm_memremap_pages() and memremap_pages() now return the struct pages' reference count set to zero instead of one. Normally, get_page() will VM_BUG_ON_PAGE() if page->_refcount is zero. I didn't see any such warnings running the xfstests with dax/pmem but I'm not clear how the zero to one reference count is handled.

Other changes in v2:
Rebased to Linux-5.9.0-rc6 to include pmem fixes.
I added patch 1 to introduce a page refcount helper for ext4 and xfs as suggested by Christoph Hellwig.
I also applied Christoph Hellwig's other suggested changes for removing the devmap_managed_key, etc.

Ralph Campbell (2):
ext4/xfs: add page refcount helper
mm: remove extra ZONE_DEVICE struct page refcount

 arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
 fs/dax.c                               |  8 +--
 fs/ext4/inode.c                        |  2 +-
 fs/xfs/xfs_file.c                      |  2 +-
 include/linux/dax.h                    |  5 ++
 include/linux/memremap.h               |  7 ++-
 include/linux/mm.h                     | 44 --------------
 lib/test_hmm.c                         |  2 +-
 mm/gup.c                               | 44 --------------
 mm/internal.h                          |  8 +++
 mm/memremap.c                          | 82 ++++++--------------------
 mm/migrate.c                           |  5 --
 mm/page_alloc.c                        |  3 +
 mm/swap.c                              | 46 +++------------
 15 files changed, 54 insertions(+), 208 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/2 ext4/xfs: add page refcount helper

There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

---
 fs/dax.c            | 8 ++++----
 fs/ext4/inode.c     | 2 +-
 fs/xfs/xfs_file.c   | 2 +-
 include/linux/dax.h | 5 +++++
 4 files changed, 11 insertions(+), 6 deletions(-)
fs/dax.c +4 -4
index 994ab66a9907..8eddbcc0e149 100644
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+		WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));
 		WARN_ON_ONCE(page->mapping && page->mapping != mapping);
 		page->mapping = NULL;
 		page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		if (page_ref_count(page) > 1)
+		if (!dax_layout_is_idle_page(page))
 			return page;
 	}
 	return NULL;
@@ -560,11 +560,11 @@ static void *grab_mapping_entry(struct xa_state *xas,
 
 /**
  * dax_layout_busy_page - find first pinned page in @mapping
- * @mapping: address space to scan for a page with ref count > 1
+ * @mapping: address space to scan for a page with ref count > 0
  *
  * DAX requires ZONE_DEVICE mapped pages. These pages are never
  * 'onlined' to the page allocator so they are considered idle when
- * page->count == 1. A filesystem uses this interface to determine if
+ * page->count == 0. A filesystem uses this interface to determine if
  * any page in the mapping is busy, i.e. for DMA, or other
  * get_user_pages() usages.
  *
fs/ext4/inode.c +1 -1
index bf596467c234..d9f8ad55523a 100644
@@ -3927,7 +3927,7 @@ int ext4_break_layouts(struct inode *inode)
 			return 0;
 
 		error = ___wait_var_event(&page->_refcount,
-				atomic_read(&page->_refcount) == 1,
+				dax_layout_is_idle_page(page),
 				TASK_INTERRUPTIBLE, 0, 0,
 				ext4_wait_dax_page(ei));
 	} while (error == 0);
fs/xfs/xfs_file.c +1 -1
index a29f78a663ca..29ab96541bc1 100644
@@ -750,7 +750,7 @@ xfs_break_dax_layouts(
 
 	*retry = true;
 	return ___wait_var_event(&page->_refcount,
-			atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
+			dax_layout_is_idle_page(page), TASK_INTERRUPTIBLE,
 			0, 0, xfs_wait_dax_page(inode));
 }
 
include/linux/dax.h +5 -0
index 43b39ab9de1a..3f78ed78d1d6 100644
@@ -238,4 +238,9 @@ static inline bool dax_mapping(struct address_space *mapping)
 	return mapping->host && IS_DAX(mapping->host);
 }
 
+static inline bool dax_layout_is_idle_page(struct page *page)
+{
+	return page_ref_count(page) <= 1;
+}
+
 #endif
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 1/2] ext4/xfs: add page refcount helper

Quoted 80 lines · Ralph Campbell

There are several places where ZONE_DEVICE struct pages assume a reference count = 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

---
 fs/dax.c            | 8 ++++----
 fs/ext4/inode.c     | 2 +-
 fs/xfs/xfs_file.c   | 2 +-
 include/linux/dax.h | 5 +++++
 4 files changed, 11 insertions(+), 6 deletions(-)
fs/dax.c +4 -4
index 994ab66a9907..8eddbcc0e149 100644
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
        for_each_mapped_pfn(entry, pfn) {
                struct page *page = pfn_to_page(pfn);
 
-               WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+               WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));
                WARN_ON_ONCE(page->mapping && page->mapping != mapping);
                page->mapping = NULL;
                page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
        for_each_mapped_pfn(entry, pfn) {
                struct page *page = pfn_to_page(pfn);
 
-               if (page_ref_count(page) > 1)
+               if (!dax_layout_is_idle_page(page))
                        return page;
        }
        return NULL;
@@ -560,11 +560,11 @@ static void *grab_mapping_entry(struct xa_state *xas,
 
 /**
  * dax_layout_busy_page - find first pinned page in @mapping
- * @mapping: address space to scan for a page with ref count > 1
+ * @mapping: address space to scan for a page with ref count > 0
  *
  * DAX requires ZONE_DEVICE mapped pages. These pages are never
  * 'onlined' to the page allocator so they are considered idle when
- * page->count = 1. A filesystem uses this interface to determine if
+ * page->count = 0. A filesystem uses this interface to determine if
  * any page in the mapping is busy, i.e. for DMA, or other
  * get_user_pages() usages.
  *
fs/ext4/inode.c +1 -1
index bf596467c234..d9f8ad55523a 100644
@@ -3927,7 +3927,7 @@ int ext4_break_layouts(struct inode *inode)
                        return 0;
 
                error = ___wait_var_event(&page->_refcount,
-                               atomic_read(&page->_refcount) = 1,
+                               dax_layout_is_idle_page(page),
                                TASK_INTERRUPTIBLE, 0, 0,
                                ext4_wait_dax_page(ei));
        } while (error = 0);
fs/xfs/xfs_file.c +1 -1
index a29f78a663ca..29ab96541bc1 100644
@@ -750,7 +750,7 @@ xfs_break_dax_layouts(
 
        *retry = true;
        return ___wait_var_event(&page->_refcount,
-                       atomic_read(&page->_refcount) = 1, TASK_INTERRUPTIBLE,
+                       dax_layout_is_idle_page(page), TASK_INTERRUPTIBLE,
                        0, 0, xfs_wait_dax_page(inode));
 }
 
include/linux/dax.h +0 -0
index 43b39ab9de1a..3f78ed78d1d6 100644

include/linux/dax.h static inline bool dax_mapping(struct address_space *mapping) · line 243

Quoted 6 lines
include/linux/dax.h +3 -0
@@ -238,4 +238,9 @@ static inline bool dax_mapping(struct address_space *mapping)
        return mapping->host && IS_DAX(mapping->host);
 }
 
+static inline bool dax_layout_is_idle_page(struct page *page)
+{
+       return page_ref_count(page) <= 1;

Why convert the check from "= 1" to "<= 1" and then back to the = operator in the next patch? A refcount < 1 in this path before your other change is a bug.

#3 lore raw

Re: [PATCH 1/2] ext4/xfs: add page refcount helper

Quoted 84 lines · Dan Williams
Quoted 83 lines · Ralph Campbell

There are several places where ZONE_DEVICE struct pages assume a reference count = 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

---
  fs/dax.c            | 8 ++++----
  fs/ext4/inode.c     | 2 +-
  fs/xfs/xfs_file.c   | 2 +-
  include/linux/dax.h | 5 +++++
  4 files changed, 11 insertions(+), 6 deletions(-)
fs/dax.c +4 -4
index 994ab66a9907..8eddbcc0e149 100644
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
         for_each_mapped_pfn(entry, pfn) {
                 struct page *page = pfn_to_page(pfn);
 
-               WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+               WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));
                 WARN_ON_ONCE(page->mapping && page->mapping != mapping);
                 page->mapping = NULL;
                 page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
         for_each_mapped_pfn(entry, pfn) {
                 struct page *page = pfn_to_page(pfn);
 
-               if (page_ref_count(page) > 1)
+               if (!dax_layout_is_idle_page(page))
                         return page;
         }
         return NULL;
@@ -560,11 +560,11 @@ static void *grab_mapping_entry(struct xa_state *xas,
 
  /**
   * dax_layout_busy_page - find first pinned page in @mapping
- * @mapping: address space to scan for a page with ref count > 1
+ * @mapping: address space to scan for a page with ref count > 0
   *
   * DAX requires ZONE_DEVICE mapped pages. These pages are never
   * 'onlined' to the page allocator so they are considered idle when
- * page->count = 1. A filesystem uses this interface to determine if
+ * page->count = 0. A filesystem uses this interface to determine if
   * any page in the mapping is busy, i.e. for DMA, or other
   * get_user_pages() usages.
   *
fs/ext4/inode.c +1 -1
index bf596467c234..d9f8ad55523a 100644
@@ -3927,7 +3927,7 @@ int ext4_break_layouts(struct inode *inode)
                         return 0;
 
                 error = ___wait_var_event(&page->_refcount,
-                               atomic_read(&page->_refcount) = 1,
+                               dax_layout_is_idle_page(page),
                                 TASK_INTERRUPTIBLE, 0, 0,
                                 ext4_wait_dax_page(ei));
         } while (error = 0);
fs/xfs/xfs_file.c +1 -1
index a29f78a663ca..29ab96541bc1 100644
@@ -750,7 +750,7 @@ xfs_break_dax_layouts(
 
         *retry = true;
         return ___wait_var_event(&page->_refcount,
-                       atomic_read(&page->_refcount) = 1, TASK_INTERRUPTIBLE,
+                       dax_layout_is_idle_page(page), TASK_INTERRUPTIBLE,
                         0, 0, xfs_wait_dax_page(inode));
  }
 
include/linux/dax.h +0 -0
index 43b39ab9de1a..3f78ed78d1d6 100644
@@ -238,4 +238,9 @@ static inline bool dax_mapping(struct address_space *mapping)
         return mapping->host && IS_DAX(mapping->host);
  }
 
Quoted 8 lines

include/linux/dax.h static inline bool dax_mapping(struct address_space *mapping) · line 243

Quoted 3 lines
include/linux/dax.h +3 -0
+static inline bool dax_layout_is_idle_page(struct page *page)
+{
+       return page_ref_count(page) <= 1;

Why convert the check from "= 1" to "<= 1" and then back to the = operator in the next patch? A refcount < 1 in this path before your other change is a bug.

Mostly I was thinking > 1 was busy so <= 1 is idle. And yes, <=0 is never supposed to happen. Checking for = 1 is probably better though.

#4 lore raw

Re: [PATCH 1/2] ext4/xfs: add page refcount helper

In reply to Ralph Campbell
Quoted 8 lines · Ralph Campbell
fs/ext4/inode.c +2 -1
 		error = ___wait_var_event(&page->_refcount,
-				atomic_read(&page->_refcount) = 1,
+				dax_layout_is_idle_page(page),
 				TASK_INTERRUPTIBLE, 0, 0,
 				ext4_wait_dax_page(ei));
 
+++ b/fs/xfs/xfs_file.c

fs/ext4/inode.c xfs_break_dax_layouts( · line 755

Quoted 7 lines
fs/ext4/inode.c +1 -1
@@ -750,7 +750,7 @@ xfs_break_dax_layouts(
 
 	*retry = true;
 	return ___wait_var_event(&page->_refcount,
-			atomic_read(&page->_refcount) = 1, TASK_INTERRUPTIBLE,
+			dax_layout_is_idle_page(page), TASK_INTERRUPTIBLE,
 			0, 0, xfs_wait_dax_page(inode));
 }

I still think a litte helper macro would be nice here:

#define dax_wait_page(_inode, _page, _wait_cb) \

  	___wait_var_event(&(_page)->_refcount,				\
		atomic_read(&(_page)->_refcount) = 1,			\
		TASK_INTERRUPTIBLE, dax_layout_is_idle_page(_page),	\
		TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode));
#5 lore raw

Re: [PATCH 1/2] ext4/xfs: add page refcount helper

Quoted 19 lines · Christoph Hellwig
Quoted 8 lines · Ralph Campbell

error = ___wait_var_event(&page->_refcount,

fs/ext4/inode.c +2 -1
-				atomic_read(&page->_refcount) = 1,
+				dax_layout_is_idle_page(page),
  				TASK_INTERRUPTIBLE, 0, 0,
  				ext4_wait_dax_page(ei));
 
+++ b/fs/xfs/xfs_file.c

fs/ext4/inode.c xfs_break_dax_layouts( · line 755

Quoted 7 lines
fs/ext4/inode.c +1 -1
@@ -750,7 +750,7 @@ xfs_break_dax_layouts(
 
  	*retry = true;
  	return ___wait_var_event(&page->_refcount,
-			atomic_read(&page->_refcount) = 1, TASK_INTERRUPTIBLE,
+			dax_layout_is_idle_page(page), TASK_INTERRUPTIBLE,
  			0, 0, xfs_wait_dax_page(inode));
  }

I still think a litte helper macro would be nice here:

Quoted 5 lines
#define dax_wait_page(_inode, _page, _wait_cb)				\
   	___wait_var_event(&(_page)->_refcount,				\
		atomic_read(&(_page)->_refcount) = 1,			\
		TASK_INTERRUPTIBLE, dax_layout_is_idle_page(_page),	\
		TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode));

OK, I'll add it in v3.

#6 lore raw

PATCH v2 2/2 mm: remove extra ZONE_DEVICE struct page refcount

In reply to Ralph Campbell

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.

---
 arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
 include/linux/dax.h                    |  2 +-
 include/linux/memremap.h               |  7 ++-
 include/linux/mm.h                     | 44 --------------
 lib/test_hmm.c                         |  2 +-
 mm/gup.c                               | 44 --------------
 mm/internal.h                          |  8 +++
 mm/memremap.c                          | 82 ++++++--------------------
 mm/migrate.c                           |  5 --
 mm/page_alloc.c                        |  3 +
 mm/swap.c                              | 46 +++------------
 12 files changed, 44 insertions(+), 203 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
index 7705d5557239..e6ec98325fab 100644
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
 
 	dpage = pfn_to_page(uvmem_pfn);
 	dpage->zone_device_data = pvt;
-	get_page(dpage);
+	init_page_count(dpage);
 	lock_page(dpage);
 	return dpage;
 out_clear:
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -1
index 4e8112fde3e6..ca2e3c3edc36 100644
@@ -323,7 +323,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)
 			return NULL;
 	}
 
-	get_page(page);
+	init_page_count(page);
 	lock_page(page);
 	return page;
 }
include/linux/dax.h +1 -1
index 3f78ed78d1d6..8d29f38645aa 100644
@@ -240,7 +240,7 @@ static inline bool dax_mapping(struct address_space *mapping)
 
 static inline bool dax_layout_is_idle_page(struct page *page)
 {
-	return page_ref_count(page) <= 1;
+	return page_ref_count(page) == 0;
 }
 
 #endif
include/linux/memremap.h +4 -3
index e5862746751b..f9224f88e4cd 100644
@@ -65,9 +65,10 @@ enum memory_type {
 
 struct dev_pagemap_ops {
 	/*
-	 * Called once the page refcount reaches 1.  (ZONE_DEVICE pages never
-	 * reach 0 refcount unless there is a refcount bug. This allows the
-	 * device driver to implement its own memory management.)
+	 * Called once the page refcount reaches 0. The reference count
+	 * should be reset to one with init_page_count(page) before reusing
+	 * the page. This allows the device driver to implement its own
+	 * memory management.
 	 */
 	void (*page_free)(struct page *page);
 
include/linux/mm.h +0 -44

56 lines not on this page · raw message

lib/test_hmm.c +1 -1
index e7dc3de355b7..1033b19c9c52 100644
@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice)
 	}
 
 	dpage->zone_device_data = rpage;
-	get_page(dpage);
+	init_page_count(dpage);
 	lock_page(dpage);
 	return dpage;
 
mm/gup.c +0 -44

56 lines not on this page · raw message

mm/internal.h +8 -0
index 10c677655912..c9de39ad5f52 100644
@@ -620,4 +620,12 @@ struct migration_target_control {
 	gfp_t gfp_mask;
 };
 
+#ifdef CONFIG_DEV_PAGEMAP_OPS
+void free_zone_device_page(struct page *page);
+#else
+static inline void free_zone_device_page(struct page *page)
+{
+}
+#endif
+
 #endif	/* __MM_INTERNAL_H */
mm/memremap.c +18 -64

153 lines not on this page · raw message

mm/migrate.c +0 -5
index aecb1433cf3c..c51749c7cb9d 100644
@@ -380,11 +380,6 @@ static int expected_page_refs(struct address_space *mapping, struct page *page)
 {
 	int expected_count = 1;
 
-	/*
-	 * Device public or private pages have an extra refcount as they are
-	 * ZONE_DEVICE pages.
-	 */
-	expected_count += is_device_private_page(page);
 	if (mapping)
 		expected_count += thp_nr_pages(page) + page_has_private(page);
 
mm/page_alloc.c +3 -0
index fab5e97dc9ca..397b0765de97 100644
@@ -6070,6 +6070,9 @@ void __ref memmap_init_zone_device(struct zone *zone,
 
 		__init_single_page(page, pfn, zone_idx, nid);
 
+		/* ZONE_DEVICE pages start with a zero reference count. */
+		set_page_count(page, 0);
+
 		/*
 		 * Mark page reserved as it will need to wait for onlining
 		 * phase for it to be fully associated with a zone.
mm/swap.c +7 -39

75 lines not on this page · raw message

Signature · 1 line
2.20.1
#7 lore raw
Quoted 146 lines · Ralph Campbell

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.

---
 arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
 include/linux/dax.h                    |  2 +-
 include/linux/memremap.h               |  7 ++-
 include/linux/mm.h                     | 44 --------------
 lib/test_hmm.c                         |  2 +-
 mm/gup.c                               | 44 --------------
 mm/internal.h                          |  8 +++
 mm/memremap.c                          | 82 ++++++--------------------
 mm/migrate.c                           |  5 --
 mm/page_alloc.c                        |  3 +
 mm/swap.c                              | 46 +++------------
 12 files changed, 44 insertions(+), 203 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
index 7705d5557239..e6ec98325fab 100644
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
 
 	dpage = pfn_to_page(uvmem_pfn);
 	dpage->zone_device_data = pvt;
-	get_page(dpage);
+	init_page_count(dpage);
 	lock_page(dpage);
 	return dpage;
 out_clear:
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -1
index 4e8112fde3e6..ca2e3c3edc36 100644
@@ -323,7 +323,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)
 			return NULL;
 	}
 
-	get_page(page);
+	init_page_count(page);
 	lock_page(page);
 	return page;
 }
include/linux/dax.h +1 -1
index 3f78ed78d1d6..8d29f38645aa 100644
@@ -240,7 +240,7 @@ static inline bool dax_mapping(struct address_space *mapping)
 
 static inline bool dax_layout_is_idle_page(struct page *page)
 {
-	return page_ref_count(page) <= 1;
+	return page_ref_count(page) = 0;
 }
 
 #endif
include/linux/memremap.h +4 -3
index e5862746751b..f9224f88e4cd 100644
@@ -65,9 +65,10 @@ enum memory_type {
 
 struct dev_pagemap_ops {
 	/*
-	 * Called once the page refcount reaches 1.  (ZONE_DEVICE pages never
-	 * reach 0 refcount unless there is a refcount bug. This allows the
-	 * device driver to implement its own memory management.)
+	 * Called once the page refcount reaches 0. The reference count
+	 * should be reset to one with init_page_count(page) before reusing
+	 * the page. This allows the device driver to implement its own
+	 * memory management.
 	 */
 	void (*page_free)(struct page *page);
 
include/linux/mm.h +0 -44
index b2f370f0b420..2159c2477aa3 100644
@@ -1092,39 +1092,6 @@ static inline bool is_zone_device_page(const struct page *page)
 }
 #endif
 
-#ifdef CONFIG_DEV_PAGEMAP_OPS
-void free_devmap_managed_page(struct page *page);
-DECLARE_STATIC_KEY_FALSE(devmap_managed_key);
-
-static inline bool page_is_devmap_managed(struct page *page)
-{
-	if (!static_branch_unlikely(&devmap_managed_key))
-		return false;
-	if (!is_zone_device_page(page))
-		return false;
-	switch (page->pgmap->type) {
-	case MEMORY_DEVICE_PRIVATE:
-	case MEMORY_DEVICE_FS_DAX:
-		return true;
-	default:
-		break;
-	}
-	return false;
-}
-
-void put_devmap_managed_page(struct page *page);
-
-#else /* CONFIG_DEV_PAGEMAP_OPS */
-static inline bool page_is_devmap_managed(struct page *page)
-{
-	return false;
-}
-
-static inline void put_devmap_managed_page(struct page *page)
-{
-}
-#endif /* CONFIG_DEV_PAGEMAP_OPS */
-
 static inline bool is_device_private_page(const struct page *page)
 {
 	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
@@ -1171,17 +1138,6 @@ static inline void put_page(struct page *page)
 {
 	page = compound_head(page);
 
-	/*
-	 * For devmap managed pages we need to catch refcount transition from
-	 * 2 to 1, when refcount reach one it means the page is free and we
-	 * need to inform the device driver through callback. See
-	 * include/linux/memremap.h and HMM for details.
-	 */
-	if (page_is_devmap_managed(page)) {
-		put_devmap_managed_page(page);
-		return;
-	}
-
 	if (put_page_testzero(page))
 		__put_page(page);
 }
lib/test_hmm.c +0 -0
index e7dc3de355b7..1033b19c9c52 100644

lib/test_hmm.c static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) · line 567

Quoted 8 lines
lib/test_hmm.c +1 -1
@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice)
 	}
 
 	dpage->zone_device_data = rpage;
-	get_page(dpage);
+	init_page_count(dpage);
 	lock_page(dpage);
 	return dpage;
 

Doesn't test_hmm also need to reinitialize the refcount before freeing the page in hmm_dmirror_exit?

lib/test_hmm.c void *memremap_pages(struct dev_pagemap *pgmap, int nid) · line 173

Quoted 9 lines
lib/test_hmm.c +0 -1
 	int error, is_ram;
-	bool need_devmap_managed = true;
 
 	switch (pgmap->type) {
 	case MEMORY_DEVICE_PRIVATE:
@@ -217,11 +171,9 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)
 		}
 		break;
 	case MEMORY_DEVICE_GENERIC:

The MEMORY_DEVICE_PRIVATE cases loses the sanity check that the page_free method is set.

Otherwise this looks good.

#8 lore raw
Quoted 149 lines · Christoph Hellwig
Quoted 148 lines · Ralph Campbell

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.

---
  arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
  drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
  include/linux/dax.h                    |  2 +-
  include/linux/memremap.h               |  7 ++-
  include/linux/mm.h                     | 44 --------------
  lib/test_hmm.c                         |  2 +-
  mm/gup.c                               | 44 --------------
  mm/internal.h                          |  8 +++
  mm/memremap.c                          | 82 ++++++--------------------
  mm/migrate.c                           |  5 --
  mm/page_alloc.c                        |  3 +
  mm/swap.c                              | 46 +++------------
  12 files changed, 44 insertions(+), 203 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
index 7705d5557239..e6ec98325fab 100644
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
 
  	dpage = pfn_to_page(uvmem_pfn);
  	dpage->zone_device_data = pvt;
-	get_page(dpage);
+	init_page_count(dpage);
  	lock_page(dpage);
  	return dpage;
  out_clear:
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -1
index 4e8112fde3e6..ca2e3c3edc36 100644
@@ -323,7 +323,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)
  			return NULL;
  	}
 
-	get_page(page);
+	init_page_count(page);
  	lock_page(page);
  	return page;
  }
include/linux/dax.h +1 -1
index 3f78ed78d1d6..8d29f38645aa 100644
@@ -240,7 +240,7 @@ static inline bool dax_mapping(struct address_space *mapping)
 
  static inline bool dax_layout_is_idle_page(struct page *page)
  {
-	return page_ref_count(page) <= 1;
+	return page_ref_count(page) = 0;
  }
 
  #endif
include/linux/memremap.h +4 -3
index e5862746751b..f9224f88e4cd 100644
@@ -65,9 +65,10 @@ enum memory_type {
 
  struct dev_pagemap_ops {
  	/*
-	 * Called once the page refcount reaches 1.  (ZONE_DEVICE pages never
-	 * reach 0 refcount unless there is a refcount bug. This allows the
-	 * device driver to implement its own memory management.)
+	 * Called once the page refcount reaches 0. The reference count
+	 * should be reset to one with init_page_count(page) before reusing
+	 * the page. This allows the device driver to implement its own
+	 * memory management.
  	 */
  	void (*page_free)(struct page *page);
 
include/linux/mm.h +0 -44
index b2f370f0b420..2159c2477aa3 100644
@@ -1092,39 +1092,6 @@ static inline bool is_zone_device_page(const struct page *page)
  }
  #endif
 
-#ifdef CONFIG_DEV_PAGEMAP_OPS
-void free_devmap_managed_page(struct page *page);
-DECLARE_STATIC_KEY_FALSE(devmap_managed_key);
-
-static inline bool page_is_devmap_managed(struct page *page)
-{
-	if (!static_branch_unlikely(&devmap_managed_key))
-		return false;
-	if (!is_zone_device_page(page))
-		return false;
-	switch (page->pgmap->type) {
-	case MEMORY_DEVICE_PRIVATE:
-	case MEMORY_DEVICE_FS_DAX:
-		return true;
-	default:
-		break;
-	}
-	return false;
-}
-
-void put_devmap_managed_page(struct page *page);
-
-#else /* CONFIG_DEV_PAGEMAP_OPS */
-static inline bool page_is_devmap_managed(struct page *page)
-{
-	return false;
-}
-
-static inline void put_devmap_managed_page(struct page *page)
-{
-}
-#endif /* CONFIG_DEV_PAGEMAP_OPS */
-
  static inline bool is_device_private_page(const struct page *page)
  {
  	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
@@ -1171,17 +1138,6 @@ static inline void put_page(struct page *page)
  {
  	page = compound_head(page);
 
-	/*
-	 * For devmap managed pages we need to catch refcount transition from
-	 * 2 to 1, when refcount reach one it means the page is free and we
-	 * need to inform the device driver through callback. See
-	 * include/linux/memremap.h and HMM for details.
-	 */
-	if (page_is_devmap_managed(page)) {
-		put_devmap_managed_page(page);
-		return;
-	}
-
  	if (put_page_testzero(page))
  		__put_page(page);
  }
lib/test_hmm.c +0 -0
index e7dc3de355b7..1033b19c9c52 100644
@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice)
  	}
 
Quoted 9 lines

lib/test_hmm.c static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) · line 567

Quoted 6 lines
lib/test_hmm.c +1 -1
  	dpage->zone_device_data = rpage;
-	get_page(dpage);
+	init_page_count(dpage);
  	lock_page(dpage);
  	return dpage;
 

Doesn't test_hmm also need to reinitialize the refcount before freeing the page in hmm_dmirror_exit?

The dmirror_zero_page is dead code, it isn't used. There is a patch queued in linux-mm which removes it. Besides, it was allocated with alloc_page() so it isn't a device private struct page.

Quoted 12 lines

lib/test_hmm.c void *memremap_pages(struct dev_pagemap *pgmap, int nid) · line 173

Quoted 9 lines

int error, is_ram;

lib/test_hmm.c +0 -1
-	bool need_devmap_managed = true;
 
  	switch (pgmap->type) {
  	case MEMORY_DEVICE_PRIVATE:
@@ -217,11 +171,9 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)
  		}
  		break;
  	case MEMORY_DEVICE_GENERIC:

The MEMORY_DEVICE_PRIVATE cases loses the sanity check that the page_free method is set.

I'll add that back into memremap_pages() with other sanity checks in v3.

Quoted 1 line

Otherwise this looks good.

Thanks!

Tested-by on v2 2/2
#9 lore raw
In reply to Ralph Campbell
Quoted 27 lines · Ralph Campbell

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.

---
 arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
 include/linux/dax.h                    |  2 +-
 include/linux/memremap.h               |  7 ++-
 include/linux/mm.h                     | 44 --------------
 lib/test_hmm.c                         |  2 +-
 mm/gup.c                               | 44 --------------
 mm/internal.h                          |  8 +++
 mm/memremap.c                          | 82 ++++++--------------------
 mm/migrate.c                           |  5 --
 mm/page_alloc.c                        |  3 +
 mm/swap.c                              | 46 +++------------
 12 files changed, 44 insertions(+), 203 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +0 -0
index 7705d5557239..e6ec98325fab 100644

arch/powerpc/kvm/book3s_hv_uvmem.c static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm) · line 714

Quoted 5 lines
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
 
 	dpage = pfn_to_page(uvmem_pfn);
 	dpage->zone_device_data = pvt;
-	get_page(dpage);
+	init_page_count(dpage);

The powerpc change looks good. Passes a quick sanity test of booting/rebooting a secure guest on Power.

Regards,
Bharata.