Skip to contentSkip to conversation

start sorting out the ZONE_DEVICE refcount mess

PATCH In review Christoph Hellwig 10 participants 47 messages Feb 7, 2022 – Feb 10, 2022

Subsystems (24)
lore
Share & tools
lore

Christoph Hellwig

author maintainer
#0 lore raw

Hi all,

this series removes the offset by one refcount for ZONE_DEVICE pages that are freed back to the driver owning them, which is just device private ones for now, but also the planned device coherent pages and the ehanced p2p ones pending.

It does not address the fsdax pages yet, which will be attacked in a follow on series.

Diffstat:

 arch/arm64/mm/mmu.c                      |    1 
 arch/powerpc/kvm/book3s_hv_uvmem.c       |    1 
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c |    2 
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h    |    1 
 drivers/gpu/drm/drm_cache.c              |    2 
 drivers/gpu/drm/nouveau/nouveau_dmem.c   |    3 -
 drivers/gpu/drm/nouveau/nouveau_svm.c    |    1 
 drivers/infiniband/core/rw.c             |    1 
 drivers/nvdimm/pmem.h                    |    1 
 drivers/nvme/host/pci.c                  |    1 
 drivers/nvme/target/io-cmd-bdev.c        |    1 
 fs/Kconfig                               |    2 
 fs/fuse/virtio_fs.c                      |    1 
 include/linux/hmm.h                      |    9 ----
 include/linux/memremap.h                 |   22 +++++++++-
 include/linux/mm.h                       |   59 ++++-------------------------
 lib/test_hmm.c                           |    4 +
 mm/Kconfig                               |    4 -
 mm/internal.h                            |    2 
 mm/memcontrol.c                          |   11 +----
 mm/memremap.c                            |   63 ++++++++++++++++---------------
 mm/migrate.c                             |    6 --
 mm/swap.c                                |   49 ++----------------------
 23 files changed, 90 insertions(+), 157 deletions(-)
author maintainer
#1 lore raw

PATCH 1/8 mm: remove a pointless CONFIG_ZONE_DEVICE check in memremap_pages

memremap.c is only built when CONFIG_ZONE_DEVICE is set, so remove the superflous extra check.

---
 mm/memremap.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
mm/memremap.c +1 -2
index 6aa5f0c2d11fda..5f04a0709e436e 100644
@@ -328,8 +328,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)
 		}
 		break;
 	case MEMORY_DEVICE_FS_DAX:
-		if (!IS_ENABLED(CONFIG_ZONE_DEVICE) ||
-		    IS_ENABLED(CONFIG_FS_DAX_LIMITED)) {
+		if (IS_ENABLED(CONFIG_FS_DAX_LIMITED)) {
 			WARN(1, "File system DAX not supported\n");
 			return ERR_PTR(-EINVAL);
 		}
Signature · 1 line
2.30.2
Reviewed-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] mm: remove a pointless CONFIG_ZONE_DEVICE check in memremap_pages

Quoted 3 lines · Christoph Hellwig

memremap.c is only built when CONFIG_ZONE_DEVICE is set, so remove the superflous extra check.

Looks good to me.

Reviewed-by on v1 1/8
#4 lore raw

Re: [PATCH 1/8] mm: remove a pointless CONFIG_ZONE_DEVICE check in memremap_pages

In reply to Christoph Hellwig
Quoted 14 lines · Christoph Hellwig

memremap.c is only built when CONFIG_ZONE_DEVICE is set, so remove the superflous extra check.

---
  mm/memremap.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
mm/memremap.c +0 -0
index 6aa5f0c2d11fda..5f04a0709e436e 100644

mm/memremap.c void *memremap_pages(struct dev_pagemap *pgmap, int nid) · line 334

Quoted 9 lines
mm/memremap.c +1 -2
@@ -328,8 +328,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)
  		}
  		break;
  	case MEMORY_DEVICE_FS_DAX:
-		if (!IS_ENABLED(CONFIG_ZONE_DEVICE) ||
-		    IS_ENABLED(CONFIG_FS_DAX_LIMITED)) {
+		if (IS_ENABLED(CONFIG_FS_DAX_LIMITED)) {
  			WARN(1, "File system DAX not supported\n");
  			return ERR_PTR(-EINVAL);
  		}

Indeed it does have it in the makefile:-

root@dev mm (for-next) # grep memremap.o Makefile
obj-$(CONFIG_ZONE_DEVICE) += memremap.o

Looks good.

author maintainer
#6 lore raw

PATCH 2/8 mm: remove the __KERNEL__ guard from <linux/mm.h>

In reply to Christoph Hellwig

__KERNEL__ ifdefs don't make sense outside of include/uapi/.

---
 include/linux/mm.h | 4 ----
 1 file changed, 4 deletions(-)
include/linux/mm.h +0 -4
index 213cc569b19223..7b46174989b086 100644
@@ -3,9 +3,6 @@
 #define _LINUX_MM_H
 
 #include <linux/errno.h>
-
-#ifdef __KERNEL__
-
 #include <linux/mmdebug.h>
 #include <linux/gfp.h>
 #include <linux/bug.h>
@@ -3381,5 +3378,4 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
 }
 #endif
 
-#endif /* __KERNEL__ */
 #endif /* _LINUX_MM_H */
Signature · 1 line
2.30.2
Reviewed-by on v1 2/8
#7 lore raw

Re: [PATCH 2/8] mm: remove the __KERNEL__ guard from <linux/mm.h>

Quoted 2 lines · Christoph Hellwig

__KERNEL__ ifdefs don't make sense outside of include/uapi/.

Yes.

Reviewed-by on v1 2/8
#9 lore raw

Re: [PATCH 2/8] mm: remove the __KERNEL__ guard from <linux/mm.h>

In reply to Christoph Hellwig
Quoted 23 lines · Christoph Hellwig

__KERNEL__ ifdefs don't make sense outside of include/uapi/.

---
  include/linux/mm.h | 4 ----
  1 file changed, 4 deletions(-)
include/linux/mm.h +0 -3
index 213cc569b19223..7b46174989b086 100644
@@ -3,9 +3,6 @@
  #define _LINUX_MM_H
 
  #include <linux/errno.h>
-
-#ifdef __KERNEL__
-
  #include <linux/mmdebug.h>
  #include <linux/gfp.h>
  #include <linux/bug.h>

include/linux/mm.h madvise_set_anon_name(struct mm_struct *mm, unsigned long start, · line 3381

Quoted 5 lines
include/linux/mm.h +0 -1
@@ -3381,5 +3378,4 @@ madvise_set_anon_name(struct mm_struct *mm, unsigned long start,
  }
  #endif
 
-#endif /* __KERNEL__ */
  #endif /* _LINUX_MM_H */

Looks good.

author maintainer
#11 lore raw

PATCH 3/8 mm: remove pointless includes from <linux/hmm.h>

In reply to Christoph Hellwig

hmm.h pulls in the world for no good reason at all. Remove the includes and push a few ones into the users instead.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 1 +
 drivers/gpu/drm/nouveau/nouveau_dmem.c   | 1 +
 include/linux/hmm.h                      | 9 ++-------
 lib/test_hmm.c                           | 2 ++
 4 files changed, 6 insertions(+), 7 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +1 -0
index ed5385137f4831..cb835f95a76e66 100644
@@ -24,6 +24,7 @@
 #include <linux/hmm.h>
 #include <linux/dma-direction.h>
 #include <linux/dma-mapping.h>
+#include <linux/migrate.h>
 #include "amdgpu_sync.h"
 #include "amdgpu_object.h"
 #include "amdgpu_vm.h"
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
index 3828aafd3ac46f..e886a3b9e08c7d 100644
@@ -39,6 +39,7 @@
 
 #include <linux/sched/mm.h>
 #include <linux/hmm.h>
+#include <linux/migrate.h>
 
 /*
  * FIXME: this is ugly right now we are using TTM to allocate vram and we pin
include/linux/hmm.h +2 -7
index 2fd2e91d5107c0..d5a6f101f843e6 100644
@@ -9,14 +9,9 @@
 #ifndef LINUX_HMM_H
 #define LINUX_HMM_H
 
-#include <linux/kconfig.h>
-#include <linux/pgtable.h>
+#include <linux/mm.h>
 
-#include <linux/device.h>
-#include <linux/migrate.h>
-#include <linux/memremap.h>
-#include <linux/completion.h>
-#include <linux/mmu_notifier.h>
+struct mmu_interval_notifier;
 
 /*
  * On output:
lib/test_hmm.c +2 -0
index 767538089a62e4..396beee6b061d4 100644
@@ -26,6 +26,8 @@
 #include <linux/sched/mm.h>
 #include <linux/platform_device.h>
 #include <linux/rmap.h>
+#include <linux/mmu_notifier.h>
+#include <linux/migrate.h>
 
 #include "test_hmm_uapi.h"
 
Signature · 1 line
2.30.2
Reviewed-by on v1 3/8
#12 lore raw

Re: [PATCH 3/8] mm: remove pointless includes from <linux/hmm.h>

Quoted 10 lines · Christoph Hellwig

hmm.h pulls in the world for no good reason at all. Remove the includes and push a few ones into the users instead.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 1 +
 drivers/gpu/drm/nouveau/nouveau_dmem.c   | 1 +
 include/linux/hmm.h                      | 9 ++-------
 lib/test_hmm.c                           | 2 ++
 4 files changed, 6 insertions(+), 7 deletions(-)

Jason

author maintainer
#14 lore raw

PATCH 4/8 mm: move free_devmap_managed_page to memremap.c

In reply to Christoph Hellwig

free_devmap_managed_page has nothing to do with the code in swap.c, move it to live with the rest of the code for devmap handling.

---
 include/linux/mm.h |  1 -
 mm/memremap.c      | 21 +++++++++++++++++++++
 mm/swap.c          | 23 -----------------------
 3 files changed, 21 insertions(+), 24 deletions(-)
include/linux/mm.h +0 -1
index 7b46174989b086..91dd0bc786a9ec 100644
@@ -1092,7 +1092,6 @@ static inline bool is_zone_movable_page(const struct page *page)
 }
 
 #ifdef CONFIG_DEV_PAGEMAP_OPS
-void free_devmap_managed_page(struct page *page);
 DECLARE_STATIC_KEY_FALSE(devmap_managed_key);
 
 static inline bool page_is_devmap_managed(struct page *page)
mm/memremap.c +21 -0
index 5f04a0709e436e..55d23e9f5c04ec 100644
@@ -501,4 +501,25 @@ void free_devmap_managed_page(struct page *page)
 	page->mapping = NULL;
 	page->pgmap->ops->page_free(page);
 }
+
+void put_devmap_managed_page(struct page *page)
+{
+	int count;
+
+	if (WARN_ON_ONCE(!page_is_devmap_managed(page)))
+		return;
+
+	count = page_ref_dec_return(page);
+
+	/*
+	 * devmap page refcounts are 1-based, rather than 0-based: if
+	 * refcount is 1, then the page is free and the refcount is
+	 * stable because nobody holds a reference on the page.
+	 */
+	if (count == 1)
+		free_devmap_managed_page(page);
+	else if (!count)
+		__put_page(page);
+}
+EXPORT_SYMBOL(put_devmap_managed_page);
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
mm/swap.c +0 -23
index bcf3ac288b56d5..08058f74cae23e 100644
@@ -1153,26 +1153,3 @@ void __init swap_setup(void)
 	 * _really_ don't want to cluster much more
 	 */
 }
-
-#ifdef CONFIG_DEV_PAGEMAP_OPS
-void put_devmap_managed_page(struct page *page)
-{
-	int count;
-
-	if (WARN_ON_ONCE(!page_is_devmap_managed(page)))
-		return;
-
-	count = page_ref_dec_return(page);
-
-	/*
-	 * devmap page refcounts are 1-based, rather than 0-based: if
-	 * refcount is 1, then the page is free and the refcount is
-	 * stable because nobody holds a reference on the page.
-	 */
-	if (count == 1)
-		free_devmap_managed_page(page);
-	else if (!count)
-		__put_page(page);
-}
-EXPORT_SYMBOL(put_devmap_managed_page);
-#endif
Signature · 1 line
2.30.2
Reviewed-by on v1 4/8
#15 lore raw

Re: [PATCH 4/8] mm: move free_devmap_managed_page to memremap.c

Quoted 4 lines · Christoph Hellwig

free_devmap_managed_page has nothing to do with the code in swap.c, move it to live with the rest of the code for devmap handling.

Looks good.

Reviewed-by on v1 4/8
#16 lore raw

Re: [PATCH 4/8] mm: move free_devmap_managed_page to memremap.c

In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig

free_devmap_managed_page has nothing to do with the code in swap.c, move it to live with the rest of the code for devmap handling.

---
 include/linux/mm.h |  1 -
 mm/memremap.c      | 21 +++++++++++++++++++++
 mm/swap.c          | 23 -----------------------
 3 files changed, 21 insertions(+), 24 deletions(-)

Jason

Reviewed-by on v1 4/8
#17 lore raw

Re: [PATCH 4/8] mm: move free_devmap_managed_page to memremap.c

In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig

free_devmap_managed_page has nothing to do with the code in swap.c, move it to live with the rest of the code for devmap handling.

True, the only devmap code is present in the swap.c is couple of calls in my tree.

Looks good.

author maintainer
#19 lore raw

PATCH 5/8 mm: simplify freeing of devmap managed pages

In reply to Christoph Hellwig

Make put_devmap_managed_page return if it took charge of the page or not and remove the separate page_is_devmap_managed helper.

---
 include/linux/mm.h | 34 ++++++++++------------------------
 mm/memremap.c      | 20 +++++++++-----------
 mm/swap.c          | 10 +---------
 3 files changed, 20 insertions(+), 44 deletions(-)
include/linux/mm.h +10 -24
index 91dd0bc786a9ec..26baadcef4556b 100644
@@ -1094,33 +1094,24 @@ static inline bool is_zone_movable_page(const struct page *page)
 #ifdef CONFIG_DEV_PAGEMAP_OPS
 DECLARE_STATIC_KEY_FALSE(devmap_managed_key);
 
-static inline bool page_is_devmap_managed(struct page *page)
+bool __put_devmap_managed_page(struct page *page);
+static inline bool put_devmap_managed_page(struct page *page)
 {
 	if (!static_branch_unlikely(&devmap_managed_key))
 		return false;
 	if (!is_zone_device_page(page))
 		return false;
-	switch (page->pgmap->type) {
-	case MEMORY_DEVICE_PRIVATE:
-	case MEMORY_DEVICE_FS_DAX:
-		return true;
-	default:
-		break;
-	}
-	return false;
+	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
+	    page->pgmap->type != MEMORY_DEVICE_FS_DAX)
+		return false;
+	return __put_devmap_managed_page(page);
 }
 
-void put_devmap_managed_page(struct page *page);
-
 #else /* CONFIG_DEV_PAGEMAP_OPS */
-static inline bool page_is_devmap_managed(struct page *page)
+static inline bool put_devmap_managed_page(struct page *page)
 {
 	return false;
 }
-
-static inline void put_devmap_managed_page(struct page *page)
-{
-}
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
 
 static inline bool is_device_private_page(const struct page *page)
@@ -1220,16 +1211,11 @@ static inline void put_page(struct page *page)
 	struct folio *folio = page_folio(page);
 
 	/*
-	 * For devmap managed pages we need to catch refcount transition from
-	 * 2 to 1, when refcount reach one it means the page is free and we
-	 * need to inform the device driver through callback. See
-	 * include/linux/memremap.h and HMM for details.
+	 * For some devmap managed pages we need to catch refcount transition
+	 * from 2 to 1:
 	 */
-	if (page_is_devmap_managed(&folio->page)) {
-		put_devmap_managed_page(&folio->page);
+	if (put_devmap_managed_page(&folio->page))
 		return;
-	}
-
 	folio_put(folio);
 }
 
mm/memremap.c +9 -11
index 55d23e9f5c04ec..f41233a67edb12 100644
@@ -502,24 +502,22 @@ void free_devmap_managed_page(struct page *page)
 	page->pgmap->ops->page_free(page);
 }
 
-void put_devmap_managed_page(struct page *page)
+bool __put_devmap_managed_page(struct page *page)
 {
-	int count;
-
-	if (WARN_ON_ONCE(!page_is_devmap_managed(page)))
-		return;
-
-	count = page_ref_dec_return(page);
-
 	/*
 	 * devmap page refcounts are 1-based, rather than 0-based: if
 	 * refcount is 1, then the page is free and the refcount is
 	 * stable because nobody holds a reference on the page.
 	 */
-	if (count == 1)
+	switch (page_ref_dec_return(page)) {
+	case 1:
 		free_devmap_managed_page(page);
-	else if (!count)
+		break;
+	case 0:
 		__put_page(page);
+		break;
+	}
+	return true;
 }
-EXPORT_SYMBOL(put_devmap_managed_page);
+EXPORT_SYMBOL(__put_devmap_managed_page);
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
mm/swap.c +1 -9
index 08058f74cae23e..25b55c56614311 100644
@@ -930,16 +930,8 @@ void release_pages(struct page **pages, int nr)
 				unlock_page_lruvec_irqrestore(lruvec, flags);
 				lruvec = NULL;
 			}
-			/*
-			 * ZONE_DEVICE pages that return 'false' from
-			 * page_is_devmap_managed() do not require special
-			 * processing, and instead, expect a call to
-			 * put_page_testzero().
-			 */
-			if (page_is_devmap_managed(page)) {
-				put_devmap_managed_page(page);
+			if (put_devmap_managed_page(page))
 				continue;
-			}
 			if (put_page_testzero(page))
 				put_dev_pagemap(page->pgmap);
 			continue;
Signature · 1 line
2.30.2
Reviewed-by on v1 5/8
#20 lore raw

Re: [PATCH 5/8] mm: simplify freeing of devmap managed pages

Quoted 9 lines · Christoph Hellwig

Make put_devmap_managed_page return if it took charge of the page or not and remove the separate page_is_devmap_managed helper.

---
 include/linux/mm.h | 34 ++++++++++------------------------
 mm/memremap.c      | 20 +++++++++-----------
 mm/swap.c          | 10 +---------
 3 files changed, 20 insertions(+), 44 deletions(-)

Jason

Reviewed-by on v1 5/8
#22 lore raw

Re: [PATCH 5/8] mm: simplify freeing of devmap managed pages

In reply to Christoph Hellwig
Quoted 5 lines
include/linux/mm.h +2 -1
-static inline bool page_is_devmap_managed(struct page *page)
+bool __put_devmap_managed_page(struct page *page);
+static inline bool put_devmap_managed_page(struct page *page)
  {
  	if (!static_branch_unlikely(&devmap_managed_key))

include/linux/mm.h static inline bool is_zone_movable_page(const struct page *page) · line 1109

Quoted 10 lines
include/linux/mm.h +0 -7
  		return false;
  	if (!is_zone_device_page(page))
  		return false;
-	switch (page->pgmap->type) {
-	case MEMORY_DEVICE_PRIVATE:
-	case MEMORY_DEVICE_FS_DAX:
-		return true;
-	default:
-		break;
-	}

nit:- how some variant of following to makes all cases evident without having to look into memremap.h for other enum values ?

         switch (page->pgmap->type) {
         case MEMORY_DEVICE_PRIVATE:
         case MEMORY_DEVICE_FS_DAX:
                 return __put_devmap_managed_page(page);
         case MEMORY_DEVICE_GENERIC:
         case MEMORY_DEVICE_PCI_P2PDMA:
                 return false;
         default:
                 WARN_ON_ONCE(1);
                 return false;
         }

include/linux/mm.h static inline bool is_zone_movable_page(const struct page *page) · line 1107

Quoted 5 lines
include/linux/mm.h +4 -1
-	return false;
+	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
+	    page->pgmap->type != MEMORY_DEVICE_FS_DAX)
+		return false;
+	return __put_devmap_managed_page(page);

nit:- we are only returning true value from __put_devmap_managed_page() in this patch. Perhaps make it __put_dev_map_managed_page() return void and return true from above ?

or maybe someone can send a cleanup once this is merged.

Quoted 2 lines
  }

Irrespective of above comment(s), looks good.

author maintainer
#23 lore raw

PATCH 6/8 mm: don't include <linux/memremap.h> in <linux/mm.h>

In reply to Christoph Hellwig

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

---
 arch/arm64/mm/mmu.c                    |  1 +
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h  |  1 +
 drivers/gpu/drm/drm_cache.c            |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  1 +
 drivers/gpu/drm/nouveau/nouveau_svm.c  |  1 +
 drivers/infiniband/core/rw.c           |  1 +
 drivers/nvdimm/pmem.h                  |  1 +
 drivers/nvme/host/pci.c                |  1 +
 drivers/nvme/target/io-cmd-bdev.c      |  1 +
 fs/fuse/virtio_fs.c                    |  1 +
 include/linux/memremap.h               | 18 ++++++++++++++++++
 include/linux/mm.h                     | 20 --------------------
 lib/test_hmm.c                         |  1 +
 mm/memremap.c                          |  6 +++++-
 14 files changed, 34 insertions(+), 22 deletions(-)
arch/arm64/mm/mmu.c +1 -0
index acfae9b41cc8c9..580abae6c0b93f 100644
@@ -17,6 +17,7 @@
 #include <linux/mman.h>
 #include <linux/nodemask.h>
 #include <linux/memblock.h>
+#include <linux/memremap.h>
 #include <linux/memory.h>
 #include <linux/fs.h>
 #include <linux/io.h>
drivers/gpu/drm/amd/amdkfd/kfd_priv.h +1 -0
index ea68f3b3a4e9cb..6d643b4b791d87 100644
@@ -25,6 +25,7 @@
 
 #include <linux/hashtable.h>
 #include <linux/mmu_notifier.h>
+#include <linux/memremap.h>
 #include <linux/mutex.h>
 #include <linux/types.h>
 #include <linux/atomic.h>
drivers/gpu/drm/drm_cache.c +1 -1
index f19d9acbe95936..50b8a088f763a6 100644
@@ -27,11 +27,11 @@
 /*
  * Authors: Thomas Hellström <thomas-at-tungstengraphics-dot-com>
  */
-
 #include <linux/dma-buf-map.h>
 #include <linux/export.h>
 #include <linux/highmem.h>
 #include <linux/cc_platform.h>
+#include <linux/ioport.h>
 #include <xen/xen.h>
 
 #include <drm/drm_cache.h>
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
index e886a3b9e08c7d..a5cdfbe32b5e54 100644
@@ -39,6 +39,7 @@
 
 #include <linux/sched/mm.h>
 #include <linux/hmm.h>
+#include <linux/memremap.h>
 #include <linux/migrate.h>
 
 /*
drivers/gpu/drm/nouveau/nouveau_svm.c +1 -0
index 266809e511e2c1..090b9b47708cca 100644
@@ -35,6 +35,7 @@
 #include <linux/sched/mm.h>
 #include <linux/sort.h>
 #include <linux/hmm.h>
+#include <linux/memremap.h>
 #include <linux/rmap.h>
 
 struct nouveau_svm {
drivers/infiniband/core/rw.c +1 -0
index 5a3bd41b331c93..4d98f931a13ddd 100644
@@ -2,6 +2,7 @@
 /*
  * Copyright (c) 2016 HGST, a Western Digital Company.
  */
+#include <linux/memremap.h>
 #include <linux/moduleparam.h>
 #include <linux/slab.h>
 #include <linux/pci-p2pdma.h>
drivers/nvdimm/pmem.h +1 -0
index 59cfe13ea8a85c..1f51a23614299b 100644
@@ -3,6 +3,7 @@
 #define __NVDIMM_PMEM_H__
 #include <linux/page-flags.h>
 #include <linux/badblocks.h>
+#include <linux/memremap.h>
 #include <linux/types.h>
 #include <linux/pfn_t.h>
 #include <linux/fs.h>
drivers/nvme/host/pci.c +1 -0
index 6a99ed68091589..ab15bc72710dbe 100644
@@ -15,6 +15,7 @@
 #include <linux/init.h>
 #include <linux/interrupt.h>
 #include <linux/io.h>
+#include <linux/memremap.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
drivers/nvme/target/io-cmd-bdev.c +1 -0
index 70ca9dfc1771a9..a141446db1bea3 100644
@@ -6,6 +6,7 @@
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 #include <linux/blkdev.h>
 #include <linux/blk-integrity.h>
+#include <linux/memremap.h>
 #include <linux/module.h>
 #include "nvmet.h"
 
fs/fuse/virtio_fs.c +1 -0
index 9d737904d07c0b..86b7dbb6a0d43e 100644
@@ -8,6 +8,7 @@
 #include <linux/dax.h>
 #include <linux/pci.h>
 #include <linux/pfn_t.h>
+#include <linux/memremap.h>
 #include <linux/module.h>
 #include <linux/virtio.h>
 #include <linux/virtio_fs.h>
include/linux/memremap.h +18 -0
index 1fafcc38acbad6..514ab46f597e5c 100644
@@ -1,6 +1,8 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 #ifndef _LINUX_MEMREMAP_H_
 #define _LINUX_MEMREMAP_H_
+
+#include <linux/mm.h>
 #include <linux/range.h>
 #include <linux/ioport.h>
 #include <linux/percpu-refcount.h>
@@ -129,6 +131,22 @@ static inline unsigned long pgmap_vmemmap_nr(struct dev_pagemap *pgmap)
 	return 1 << pgmap->vmemmap_shift;
 }
 
+static inline bool is_device_private_page(const struct page *page)
+{
+	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+		IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
+		is_zone_device_page(page) &&
+		page->pgmap->type == MEMORY_DEVICE_PRIVATE;
+}
+
+static inline bool is_pci_p2pdma_page(const struct page *page)
+{
+	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+		IS_ENABLED(CONFIG_PCI_P2PDMA) &&
+		is_zone_device_page(page) &&
+		page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
+}
+
 #ifdef CONFIG_ZONE_DEVICE
 void *memremap_pages(struct dev_pagemap *pgmap, int nid);
 void memunmap_pages(struct dev_pagemap *pgmap);
include/linux/mm.h +0 -20
index 26baadcef4556b..80fccfe31c3444 100644
@@ -23,7 +23,6 @@
 #include <linux/err.h>
 #include <linux/page-flags.h>
 #include <linux/page_ref.h>
-#include <linux/memremap.h>
 #include <linux/overflow.h>
 #include <linux/sizes.h>
 #include <linux/sched.h>
@@ -1101,9 +1100,6 @@ static inline bool put_devmap_managed_page(struct page *page)
 		return false;
 	if (!is_zone_device_page(page))
 		return false;
-	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
-	    page->pgmap->type != MEMORY_DEVICE_FS_DAX)
-		return false;
 	return __put_devmap_managed_page(page);
 }
 
@@ -1114,22 +1110,6 @@ static inline bool put_devmap_managed_page(struct page *page)
 }
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
 
-static inline bool is_device_private_page(const struct page *page)
-{
-	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-		IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
-		is_zone_device_page(page) &&
-		page->pgmap->type == MEMORY_DEVICE_PRIVATE;
-}
-
-static inline bool is_pci_p2pdma_page(const struct page *page)
-{
-	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-		IS_ENABLED(CONFIG_PCI_P2PDMA) &&
-		is_zone_device_page(page) &&
-		page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
-}
-
 /* 127: arbitrary random number, small enough to assemble well */
 #define folio_ref_zero_or_close_to_overflow(folio) \
 	((unsigned int) folio_ref_count(folio) + 127u <= 127u)
lib/test_hmm.c +1 -0
index 396beee6b061d4..e5fc14ba71f33e 100644
@@ -12,6 +12,7 @@
 #include <linux/kernel.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
+#include <linux/memremap.h>
 #include <linux/mutex.h>
 #include <linux/rwsem.h>
 #include <linux/sched.h>
mm/memremap.c +5 -1
index f41233a67edb12..a0ece2344c2cab 100644
@@ -4,7 +4,7 @@
 #include <linux/io.h>
 #include <linux/kasan.h>
 #include <linux/memory_hotplug.h>
-#include <linux/mm.h>
+#include <linux/memremap.h>
 #include <linux/pfn_t.h>
 #include <linux/swap.h>
 #include <linux/mmzone.h>
@@ -504,6 +504,10 @@ void free_devmap_managed_page(struct page *page)
 
 bool __put_devmap_managed_page(struct page *page)
 {
+	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
+	    page->pgmap->type != MEMORY_DEVICE_FS_DAX)
+		return false;
+
 	/*
 	 * devmap page refcounts are 1-based, rather than 0-based: if
 	 * refcount is 1, then the page is free and the refcount is
Signature · 1 line
2.30.2
#24 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

Quoted 5 lines · Christoph Hellwig

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

I've noticed mm/memcontrol.c uses is_device_private_page() and also needs a memremap.h include added to compile with my configuration.

Logan

Reviewed-by on v1 6/8
#25 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Quoted 16 lines
---
 arch/arm64/mm/mmu.c                    |  1 +
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h  |  1 +
 drivers/gpu/drm/drm_cache.c            |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  1 +
 drivers/gpu/drm/nouveau/nouveau_svm.c  |  1 +
 drivers/infiniband/core/rw.c           |  1 +
 drivers/nvdimm/pmem.h                  |  1 +
 drivers/nvme/host/pci.c                |  1 +
 drivers/nvme/target/io-cmd-bdev.c      |  1 +
 fs/fuse/virtio_fs.c                    |  1 +
 include/linux/memremap.h               | 18 ++++++++++++++++++
 include/linux/mm.h                     | 20 --------------------
 lib/test_hmm.c                         |  1 +
 mm/memremap.c                          |  6 +++++-
 14 files changed, 34 insertions(+), 22 deletions(-)

Jason

maintainer Acked-by on v1 6/8
#26 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

In reply to Christoph Hellwig

Am 2022-02-07 um 01:32 schrieb Christoph Hellwig:

Quoted 5 lines

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

The amdkfd part looks good to me.

It looks like this patch is not based on Alex Sierra's coherent memory series. He added two new helpers is_device_coherent_page and is_dev_private_or_coherent_page that would need to be moved along with is_device_private_page and is_pci_p2pdma_page.

Quoted 259 lines
---
  arch/arm64/mm/mmu.c                    |  1 +
  drivers/gpu/drm/amd/amdkfd/kfd_priv.h  |  1 +
  drivers/gpu/drm/drm_cache.c            |  2 +-
  drivers/gpu/drm/nouveau/nouveau_dmem.c |  1 +
  drivers/gpu/drm/nouveau/nouveau_svm.c  |  1 +
  drivers/infiniband/core/rw.c           |  1 +
  drivers/nvdimm/pmem.h                  |  1 +
  drivers/nvme/host/pci.c                |  1 +
  drivers/nvme/target/io-cmd-bdev.c      |  1 +
  fs/fuse/virtio_fs.c                    |  1 +
  include/linux/memremap.h               | 18 ++++++++++++++++++
  include/linux/mm.h                     | 20 --------------------
  lib/test_hmm.c                         |  1 +
  mm/memremap.c                          |  6 +++++-
  14 files changed, 34 insertions(+), 22 deletions(-)
arch/arm64/mm/mmu.c +1 -0
index acfae9b41cc8c9..580abae6c0b93f 100644
@@ -17,6 +17,7 @@
  #include <linux/mman.h>
  #include <linux/nodemask.h>
  #include <linux/memblock.h>
+#include <linux/memremap.h>
  #include <linux/memory.h>
  #include <linux/fs.h>
  #include <linux/io.h>
drivers/gpu/drm/amd/amdkfd/kfd_priv.h +1 -0
index ea68f3b3a4e9cb..6d643b4b791d87 100644
@@ -25,6 +25,7 @@
 
  #include <linux/hashtable.h>
  #include <linux/mmu_notifier.h>
+#include <linux/memremap.h>
  #include <linux/mutex.h>
  #include <linux/types.h>
  #include <linux/atomic.h>
drivers/gpu/drm/drm_cache.c +1 -1
index f19d9acbe95936..50b8a088f763a6 100644
@@ -27,11 +27,11 @@
  /*
   * Authors: Thomas Hellström <thomas-at-tungstengraphics-dot-com>
   */
-
  #include <linux/dma-buf-map.h>
  #include <linux/export.h>
  #include <linux/highmem.h>
  #include <linux/cc_platform.h>
+#include <linux/ioport.h>
  #include <xen/xen.h>
 
  #include <drm/drm_cache.h>
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
index e886a3b9e08c7d..a5cdfbe32b5e54 100644
@@ -39,6 +39,7 @@
 
  #include <linux/sched/mm.h>
  #include <linux/hmm.h>
+#include <linux/memremap.h>
  #include <linux/migrate.h>
 
  /*
drivers/gpu/drm/nouveau/nouveau_svm.c +1 -0
index 266809e511e2c1..090b9b47708cca 100644
@@ -35,6 +35,7 @@
  #include <linux/sched/mm.h>
  #include <linux/sort.h>
  #include <linux/hmm.h>
+#include <linux/memremap.h>
  #include <linux/rmap.h>
 
  struct nouveau_svm {
drivers/infiniband/core/rw.c +1 -0
index 5a3bd41b331c93..4d98f931a13ddd 100644
@@ -2,6 +2,7 @@
  /*
   * Copyright (c) 2016 HGST, a Western Digital Company.
   */
+#include <linux/memremap.h>
  #include <linux/moduleparam.h>
  #include <linux/slab.h>
  #include <linux/pci-p2pdma.h>
drivers/nvdimm/pmem.h +1 -0
index 59cfe13ea8a85c..1f51a23614299b 100644
@@ -3,6 +3,7 @@
  #define __NVDIMM_PMEM_H__
  #include <linux/page-flags.h>
  #include <linux/badblocks.h>
+#include <linux/memremap.h>
  #include <linux/types.h>
  #include <linux/pfn_t.h>
  #include <linux/fs.h>
drivers/nvme/host/pci.c +1 -0
index 6a99ed68091589..ab15bc72710dbe 100644
@@ -15,6 +15,7 @@
  #include <linux/init.h>
  #include <linux/interrupt.h>
  #include <linux/io.h>
+#include <linux/memremap.h>
  #include <linux/mm.h>
  #include <linux/module.h>
  #include <linux/mutex.h>
drivers/nvme/target/io-cmd-bdev.c +1 -0
index 70ca9dfc1771a9..a141446db1bea3 100644
@@ -6,6 +6,7 @@
  #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
  #include <linux/blkdev.h>
  #include <linux/blk-integrity.h>
+#include <linux/memremap.h>
  #include <linux/module.h>
  #include "nvmet.h"
 
fs/fuse/virtio_fs.c +1 -0
index 9d737904d07c0b..86b7dbb6a0d43e 100644
@@ -8,6 +8,7 @@
  #include <linux/dax.h>
  #include <linux/pci.h>
  #include <linux/pfn_t.h>
+#include <linux/memremap.h>
  #include <linux/module.h>
  #include <linux/virtio.h>
  #include <linux/virtio_fs.h>
include/linux/memremap.h +18 -0
index 1fafcc38acbad6..514ab46f597e5c 100644
@@ -1,6 +1,8 @@
  /* SPDX-License-Identifier: GPL-2.0 */
  #ifndef _LINUX_MEMREMAP_H_
  #define _LINUX_MEMREMAP_H_
+
+#include <linux/mm.h>
  #include <linux/range.h>
  #include <linux/ioport.h>
  #include <linux/percpu-refcount.h>
@@ -129,6 +131,22 @@ static inline unsigned long pgmap_vmemmap_nr(struct dev_pagemap *pgmap)
  	return 1 << pgmap->vmemmap_shift;
  }
 
+static inline bool is_device_private_page(const struct page *page)
+{
+	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+		IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
+		is_zone_device_page(page) &&
+		page->pgmap->type == MEMORY_DEVICE_PRIVATE;
+}
+
+static inline bool is_pci_p2pdma_page(const struct page *page)
+{
+	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+		IS_ENABLED(CONFIG_PCI_P2PDMA) &&
+		is_zone_device_page(page) &&
+		page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
+}
+
  #ifdef CONFIG_ZONE_DEVICE
  void *memremap_pages(struct dev_pagemap *pgmap, int nid);
  void memunmap_pages(struct dev_pagemap *pgmap);
include/linux/mm.h +0 -20
index 26baadcef4556b..80fccfe31c3444 100644
@@ -23,7 +23,6 @@
  #include <linux/err.h>
  #include <linux/page-flags.h>
  #include <linux/page_ref.h>
-#include <linux/memremap.h>
  #include <linux/overflow.h>
  #include <linux/sizes.h>
  #include <linux/sched.h>
@@ -1101,9 +1100,6 @@ static inline bool put_devmap_managed_page(struct page *page)
  		return false;
  	if (!is_zone_device_page(page))
  		return false;
-	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
-	    page->pgmap->type != MEMORY_DEVICE_FS_DAX)
-		return false;
  	return __put_devmap_managed_page(page);
  }
 
@@ -1114,22 +1110,6 @@ static inline bool put_devmap_managed_page(struct page *page)
  }
  #endif /* CONFIG_DEV_PAGEMAP_OPS */
 
-static inline bool is_device_private_page(const struct page *page)
-{
-	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-		IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
-		is_zone_device_page(page) &&
-		page->pgmap->type == MEMORY_DEVICE_PRIVATE;
-}
-
-static inline bool is_pci_p2pdma_page(const struct page *page)
-{
-	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-		IS_ENABLED(CONFIG_PCI_P2PDMA) &&
-		is_zone_device_page(page) &&
-		page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
-}
-
  /* 127: arbitrary random number, small enough to assemble well */
  #define folio_ref_zero_or_close_to_overflow(folio) \
  	((unsigned int) folio_ref_count(folio) + 127u <= 127u)
lib/test_hmm.c +1 -0
index 396beee6b061d4..e5fc14ba71f33e 100644
@@ -12,6 +12,7 @@
  #include <linux/kernel.h>
  #include <linux/cdev.h>
  #include <linux/device.h>
+#include <linux/memremap.h>
  #include <linux/mutex.h>
  #include <linux/rwsem.h>
  #include <linux/sched.h>
mm/memremap.c +5 -1
index f41233a67edb12..a0ece2344c2cab 100644
@@ -4,7 +4,7 @@
  #include <linux/io.h>
  #include <linux/kasan.h>
  #include <linux/memory_hotplug.h>
-#include <linux/mm.h>
+#include <linux/memremap.h>
  #include <linux/pfn_t.h>
  #include <linux/swap.h>
  #include <linux/mmzone.h>
@@ -504,6 +504,10 @@ void free_devmap_managed_page(struct page *page)
 
  bool __put_devmap_managed_page(struct page *page)
  {
+	if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
+	    page->pgmap->type != MEMORY_DEVICE_FS_DAX)
+		return false;
+
  	/*
  	 * devmap page refcounts are 1-based, rather than 0-based: if
  	 * refcount is 1, then the page is free and the refcount is
author maintainer
#27 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

Quoted 6 lines · Felix Kuehling

Am 2022-02-07 um 01:32 schrieb Christoph Hellwig:

Quoted 4 lines

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Quoted 8 lines
Quoted 1 line

The amdkfd part looks good to me.

It looks like this patch is not based on Alex Sierra's coherent memory series. He added two new helpers is_device_coherent_page and is_dev_private_or_coherent_page that would need to be moved along with is_device_private_page and is_pci_p2pdma_page.

Yes. I Naked that series because it spreads te mess with the refcount further in this latest version. My intent is that it gets rebased on top of this to avoid that spread. Same for the p2p series form Logan.

author maintainer
#28 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

In reply to Felix Kuehling
Quoted 6 lines · Felix Kuehling

Am 2022-02-07 um 01:32 schrieb Christoph Hellwig:

Quoted 4 lines

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Quoted 8 lines
Quoted 1 line

The amdkfd part looks good to me.

It looks like this patch is not based on Alex Sierra's coherent memory series. He added two new helpers is_device_coherent_page and is_dev_private_or_coherent_page that would need to be moved along with is_device_private_page and is_pci_p2pdma_page.

FYI, here is a branch that contains a rebase of the coherent memory related patches on top of this series:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/pgmap-refcount

I don't have a good way to test this, but I'll at least let the build bot finish before sending it out (probably tomorrow).

#29 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

Quoted 16 lines · AEDT Christoph Hellwig
Quoted 6 lines · Felix Kuehling

Am 2022-02-07 um 01:32 schrieb Christoph Hellwig:

Quoted 4 lines

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Quoted 8 lines
Quoted 1 line

The amdkfd part looks good to me.

It looks like this patch is not based on Alex Sierra's coherent memory series. He added two new helpers is_device_coherent_page and is_dev_private_or_coherent_page that would need to be moved along with is_device_private_page and is_pci_p2pdma_page.

Quoted 7 lines

FYI, here is a branch that contains a rebase of the coherent memory related patches on top of this series:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/pgmap-refcount

I don't have a good way to test this, but I'll at least let the build bot finish before sending it out (probably tomorrow).

Thanks, I ran up hmm-test which revealed a few minor problems with the rebase. Fixes below.

---

mm/gup.c +0 -1
index cbb49abb7992..8e85c9fb8df4 100644
@@ -2007,7 +2007,6 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
 	if (!ret && list_empty(&movable_page_list) && !isolation_error_count)
 		return nr_pages;
 
-	ret = 0;
 unpin_pages:
 	for (i = 0; i < nr_pages; i++)
 		if (!pages[i])
mm/migrate.c +3 -4
index f909f5a92757..1ae3e99baa50 100644
@@ -2686,12 +2686,11 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
 			swp_entry = make_readable_device_private_entry(
 						page_to_pfn(page));
 		entry = swp_entry_to_pte(swp_entry);
-	} else {
-		if (is_zone_device_page(page) &&
-		    is_device_coherent_page(page)) {
+	} else if (is_zone_device_page(page) &&
+		    !is_device_coherent_page(page)) {
 			pr_warn_once("Unsupported ZONE_DEVICE page type.\n");
 			goto abort;
-		}
+	} else {
 		entry = mk_pte(page, vma->vm_page_prot);
 		if (vma->vm_flags & VM_WRITE)
 			entry = pte_mkwrite(pte_mkdirty(entry));
author maintainer
#30 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

mm/gup.c static long check_and_migrate_movable_pages(unsigned long nr_pages, · line 2010

Quoted 10 lines · Alistair Popple
mm/gup.c +0 -1
index cbb49abb7992..8e85c9fb8df4 100644
@@ -2007,7 +2007,6 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,
 	if (!ret && list_empty(&movable_page_list) && !isolation_error_count)
 		return nr_pages;
 
-	ret = 0;
 unpin_pages:

This isn't quite correct as ret is initially set to -EFAULT now. I'll fix it by removing the early ret initialization and always using the goto. I've also added another refactoring patch for this messy function.

I've folded the inversion of the is_device_coherent_page check in migrate.c in as well, thanks!

maintainer
#31 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

In reply to Christoph Hellwig

Am 2022-02-09 um 12:48 schrieb Christoph Hellwig:

Quoted 13 lines · Felix Kuehling
Quoted 5 lines

Am 2022-02-07 um 01:32 schrieb Christoph Hellwig:

Quoted 4 lines

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Quoted 7 lines
Quoted 1 line

The amdkfd part looks good to me.

It looks like this patch is not based on Alex Sierra's coherent memory series. He added two new helpers is_device_coherent_page and is_dev_private_or_coherent_page that would need to be moved along with is_device_private_page and is_pci_p2pdma_page.

Quoted 7 lines

FYI, here is a branch that contains a rebase of the coherent memory related patches on top of this series:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/pgmap-refcount

I don't have a good way to test this, but I'll at least let the build bot finish before sending it out (probably tomorrow).

Thank you for taking care of this rebase! Alex tested it on one of our coherent memory systems and it passed our tests.

I see you also included these rebased patches in your latest 27-patch series. I'll try to review the changes in more detail over the weekend.

Regards,
  Felix

Reviewed-by on v1 6/8
#32 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Looks good to me assuming the compile bots agree.

Quoted 264 lines
---
 arch/arm64/mm/mmu.c                    |  1 +
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h  |  1 +
 drivers/gpu/drm/drm_cache.c            |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  1 +
 drivers/gpu/drm/nouveau/nouveau_svm.c  |  1 +
 drivers/infiniband/core/rw.c           |  1 +
 drivers/nvdimm/pmem.h                  |  1 +
 drivers/nvme/host/pci.c                |  1 +
 drivers/nvme/target/io-cmd-bdev.c      |  1 +
 fs/fuse/virtio_fs.c                    |  1 +
 include/linux/memremap.h               | 18 ++++++++++++++++++
 include/linux/mm.h                     | 20 --------------------
 lib/test_hmm.c                         |  1 +
 mm/memremap.c                          |  6 +++++-
 14 files changed, 34 insertions(+), 22 deletions(-)
arch/arm64/mm/mmu.c +1 -0
index acfae9b41cc8c9..580abae6c0b93f 100644
@@ -17,6 +17,7 @@
 #include <linux/mman.h>
 #include <linux/nodemask.h>
 #include <linux/memblock.h>
+#include <linux/memremap.h>
 #include <linux/memory.h>
 #include <linux/fs.h>
 #include <linux/io.h>
drivers/gpu/drm/amd/amdkfd/kfd_priv.h +1 -0
index ea68f3b3a4e9cb..6d643b4b791d87 100644
@@ -25,6 +25,7 @@
 
 #include <linux/hashtable.h>
 #include <linux/mmu_notifier.h>
+#include <linux/memremap.h>
 #include <linux/mutex.h>
 #include <linux/types.h>
 #include <linux/atomic.h>
drivers/gpu/drm/drm_cache.c +1 -1
index f19d9acbe95936..50b8a088f763a6 100644
@@ -27,11 +27,11 @@
 /*
  * Authors: Thomas Hellström <thomas-at-tungstengraphics-dot-com>
  */
-
 #include <linux/dma-buf-map.h>
 #include <linux/export.h>
 #include <linux/highmem.h>
 #include <linux/cc_platform.h>
+#include <linux/ioport.h>
 #include <xen/xen.h>
 
 #include <drm/drm_cache.h>
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
index e886a3b9e08c7d..a5cdfbe32b5e54 100644
@@ -39,6 +39,7 @@
 
 #include <linux/sched/mm.h>
 #include <linux/hmm.h>
+#include <linux/memremap.h>
 #include <linux/migrate.h>
 
 /*
drivers/gpu/drm/nouveau/nouveau_svm.c +1 -0
index 266809e511e2c1..090b9b47708cca 100644
@@ -35,6 +35,7 @@
 #include <linux/sched/mm.h>
 #include <linux/sort.h>
 #include <linux/hmm.h>
+#include <linux/memremap.h>
 #include <linux/rmap.h>
 
 struct nouveau_svm {
drivers/infiniband/core/rw.c +1 -0
index 5a3bd41b331c93..4d98f931a13ddd 100644
@@ -2,6 +2,7 @@
 /*
  * Copyright (c) 2016 HGST, a Western Digital Company.
  */
+#include <linux/memremap.h>
 #include <linux/moduleparam.h>
 #include <linux/slab.h>
 #include <linux/pci-p2pdma.h>
drivers/nvdimm/pmem.h +1 -0
index 59cfe13ea8a85c..1f51a23614299b 100644
@@ -3,6 +3,7 @@
 #define __NVDIMM_PMEM_H__
 #include <linux/page-flags.h>
 #include <linux/badblocks.h>
+#include <linux/memremap.h>
 #include <linux/types.h>
 #include <linux/pfn_t.h>
 #include <linux/fs.h>
drivers/nvme/host/pci.c +1 -0
index 6a99ed68091589..ab15bc72710dbe 100644
@@ -15,6 +15,7 @@
 #include <linux/init.h>
 #include <linux/interrupt.h>
 #include <linux/io.h>
+#include <linux/memremap.h>
 #include <linux/mm.h>
 #include <linux/module.h>
 #include <linux/mutex.h>
drivers/nvme/target/io-cmd-bdev.c +1 -0
index 70ca9dfc1771a9..a141446db1bea3 100644
@@ -6,6 +6,7 @@
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 #include <linux/blkdev.h>
 #include <linux/blk-integrity.h>
+#include <linux/memremap.h>
 #include <linux/module.h>
 #include "nvmet.h"
 
fs/fuse/virtio_fs.c +1 -0
index 9d737904d07c0b..86b7dbb6a0d43e 100644
@@ -8,6 +8,7 @@
 #include <linux/dax.h>
 #include <linux/pci.h>
 #include <linux/pfn_t.h>
+#include <linux/memremap.h>
 #include <linux/module.h>
 #include <linux/virtio.h>
 #include <linux/virtio_fs.h>
include/linux/memremap.h +18 -0
index 1fafcc38acbad6..514ab46f597e5c 100644
@@ -1,6 +1,8 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 #ifndef _LINUX_MEMREMAP_H_
 #define _LINUX_MEMREMAP_H_
+
+#include <linux/mm.h>
 #include <linux/range.h>
 #include <linux/ioport.h>
 #include <linux/percpu-refcount.h>
@@ -129,6 +131,22 @@ static inline unsigned long pgmap_vmemmap_nr(struct dev_pagemap *pgmap)
        return 1 << pgmap->vmemmap_shift;
 }
 
+static inline bool is_device_private_page(const struct page *page)
+{
+       return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+               IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
+               is_zone_device_page(page) &&
+               page->pgmap->type == MEMORY_DEVICE_PRIVATE;
+}
+
+static inline bool is_pci_p2pdma_page(const struct page *page)
+{
+       return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+               IS_ENABLED(CONFIG_PCI_P2PDMA) &&
+               is_zone_device_page(page) &&
+               page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
+}
+
 #ifdef CONFIG_ZONE_DEVICE
 void *memremap_pages(struct dev_pagemap *pgmap, int nid);
 void memunmap_pages(struct dev_pagemap *pgmap);
include/linux/mm.h +0 -20
index 26baadcef4556b..80fccfe31c3444 100644
@@ -23,7 +23,6 @@
 #include <linux/err.h>
 #include <linux/page-flags.h>
 #include <linux/page_ref.h>
-#include <linux/memremap.h>
 #include <linux/overflow.h>
 #include <linux/sizes.h>
 #include <linux/sched.h>
@@ -1101,9 +1100,6 @@ static inline bool put_devmap_managed_page(struct page *page)
                return false;
        if (!is_zone_device_page(page))
                return false;
-       if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
-           page->pgmap->type != MEMORY_DEVICE_FS_DAX)
-               return false;
        return __put_devmap_managed_page(page);
 }
 
@@ -1114,22 +1110,6 @@ static inline bool put_devmap_managed_page(struct page *page)
 }
 #endif /* CONFIG_DEV_PAGEMAP_OPS */
 
-static inline bool is_device_private_page(const struct page *page)
-{
-       return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-               IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
-               is_zone_device_page(page) &&
-               page->pgmap->type == MEMORY_DEVICE_PRIVATE;
-}
-
-static inline bool is_pci_p2pdma_page(const struct page *page)
-{
-       return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-               IS_ENABLED(CONFIG_PCI_P2PDMA) &&
-               is_zone_device_page(page) &&
-               page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
-}
-
 /* 127: arbitrary random number, small enough to assemble well */
 #define folio_ref_zero_or_close_to_overflow(folio) \
        ((unsigned int) folio_ref_count(folio) + 127u <= 127u)
lib/test_hmm.c +1 -0
index 396beee6b061d4..e5fc14ba71f33e 100644
@@ -12,6 +12,7 @@
 #include <linux/kernel.h>
 #include <linux/cdev.h>
 #include <linux/device.h>
+#include <linux/memremap.h>
 #include <linux/mutex.h>
 #include <linux/rwsem.h>
 #include <linux/sched.h>
mm/memremap.c +5 -2
index f41233a67edb12..a0ece2344c2cab 100644
@@ -4,7 +4,7 @@
 #include <linux/io.h>
 #include <linux/kasan.h>
 #include <linux/memory_hotplug.h>
-#include <linux/mm.h>
+#include <linux/memremap.h>
 #include <linux/pfn_t.h>
 #include <linux/swap.h>
 #include <linux/mmzone.h>
@@ -504,6 +504,10 @@ void free_devmap_managed_page(struct page *page)
 
 bool __put_devmap_managed_page(struct page *page)
 {
+       if (page->pgmap->type != MEMORY_DEVICE_PRIVATE &&
+           page->pgmap->type != MEMORY_DEVICE_FS_DAX)
+               return false;
+
        /*
         * devmap page refcounts are 1-based, rather than 0-based: if
         * refcount is 1, then the page is free and the refcount is
--

2.30.2

#33 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

Quoted 10 lines · Dan Williams
Quoted 4 lines · Christoph Hellwig

Move the check for the actual pgmap types that need the free at refcount one behavior into the out of line helper, and thus avoid the need to pull memremap.h into mm.h.

Looks good to me assuming the compile bots agree.

Yeah, same as Logan:

mm/memcontrol.c: In function ‘get_mctgt_type’:
mm/memcontrol.c:5724:29: error: implicit declaration of function ‘is_device_private_page’; did you mean ‘is_device_private_entry’? [-Werror=implicit-function-declaration]

 5724 |                         if (is_device_private_page(page))
      |                             ^~~~~~~~~~~~~~~~~~~~~~
      |                             is_device_private_entry

...needs:

mm/memcontrol.c +1 -0
index d1e97a54ae53..0ac7515c85f9 100644
@@ -62,6 +62,7 @@
 #include <linux/tracehook.h>
 #include <linux/psi.h>
 #include <linux/seq_buf.h>
+#include <linux/memremap.h>
 #include "internal.h"
 #include <net/sock.h>
 #include <net/ip.h>
author maintainer
#34 lore raw

Re: [PATCH 6/8] mm: don't include <linux/memremap.h> in <linux/mm.h>

Quoted 11 lines · Dan Williams

Yeah, same as Logan:

mm/memcontrol.c: In function ‘get_mctgt_type’:
mm/memcontrol.c:5724:29: error: implicit declaration of function ‘is_device_private_page’; did you mean ‘is_device_private_entry’? [-Werror=implicit-function-declaration]

 5724 |                         if (is_device_private_page(page))
      |                             ^~~~~~~~~~~~~~~~~~~~~~
      |                             is_device_private_entry

...needs:

Yeah, the buildbot also complained. I've fixed this up locally now.

author maintainer
#35 lore raw

PATCH 7/8 mm: remove the extra ZONE_DEVICE struct page refcount

In reply to Christoph Hellwig

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE pages.

Note that this excludes the special idle page wakeup for fsdax pages, which still happens at refcount 1. This is a separate issue and will be sorted out later. Given that only fsdax pages require the notifiacation when the refcount hits 1 now, the PAGEMAP_OPS Kconfig symbol can go away and be replaced with a FS_DAX check for this hook in the put_page fastpath.

Based on an earlier patch from Ralph Campbell <[email protected]>.

---
 arch/powerpc/kvm/book3s_hv_uvmem.c       |  1 -
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c |  1 -
 drivers/gpu/drm/nouveau/nouveau_dmem.c   |  1 -
 fs/Kconfig                               |  1 -
 include/linux/memremap.h                 | 12 +++--
 include/linux/mm.h                       |  6 +--
 lib/test_hmm.c                           |  1 -
 mm/Kconfig                               |  4 --
 mm/internal.h                            |  2 +
 mm/memcontrol.c                          | 11 ++---
 mm/memremap.c                            | 57 ++++++++----------------
 mm/migrate.c                             |  6 ---
 mm/swap.c                                | 16 ++-----
 13 files changed, 36 insertions(+), 83 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +0 -1
index e414ca44839fd1..8b6438fa18fc2b 100644
@@ -712,7 +712,6 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
 
 	dpage = pfn_to_page(uvmem_pfn);
 	dpage->zone_device_data = pvt;
-	get_page(dpage);
 	lock_page(dpage);
 	return dpage;
 out_clear:
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +0 -1
index cb835f95a76e66..e27ca375876230 100644
@@ -225,7 +225,6 @@ svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn)
 	page = pfn_to_page(pfn);
 	svm_range_bo_ref(prange->svm_bo);
 	page->zone_device_data = prange->svm_bo;
-	get_page(page);
 	lock_page(page);
 }
 
drivers/gpu/drm/nouveau/nouveau_dmem.c +0 -1
index a5cdfbe32b5e54..7ba66ad68a8a1e 100644
@@ -326,7 +326,6 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)
 			return NULL;
 	}
 
-	get_page(page);
 	lock_page(page);
 	return page;
 }
fs/Kconfig +0 -1
index 7a2b11c0b8036d..05efea674bffa0 100644
@@ -48,7 +48,6 @@ config FS_DAX
 	bool "File system based Direct Access (DAX) support"
 	depends on MMU
 	depends on !(ARM || MIPS || SPARC)
-	select DEV_PAGEMAP_OPS if (ZONE_DEVICE && !FS_DAX_LIMITED)
 	select FS_IOMAP
 	select DAX
 	help
include/linux/memremap.h +5 -7
index 514ab46f597e5c..d6a114dd5ea8b7 100644
@@ -68,9 +68,9 @@ enum memory_type {
 
 struct dev_pagemap_ops {
 	/*
-	 * Called once the page refcount reaches 1.  (ZONE_DEVICE pages never
-	 * reach 0 refcount unless there is a refcount bug. This allows the
-	 * device driver to implement its own memory management.)
+	 * Called once the page refcount reaches 0.  The reference count will be
+	 * reset to one by the core code after the method is called to prepare
+	 * for handing out the page again.
 	 */
 	void (*page_free)(struct page *page);
 
@@ -133,16 +133,14 @@ static inline unsigned long pgmap_vmemmap_nr(struct dev_pagemap *pgmap)
 
 static inline bool is_device_private_page(const struct page *page)
 {
-	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-		IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
+	return IS_ENABLED(CONFIG_DEVICE_PRIVATE) &&
 		is_zone_device_page(page) &&
 		page->pgmap->type == MEMORY_DEVICE_PRIVATE;
 }
 
 static inline bool is_pci_p2pdma_page(const struct page *page)
 {
-	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
-		IS_ENABLED(CONFIG_PCI_P2PDMA) &&
+	return IS_ENABLED(CONFIG_PCI_P2PDMA) &&
 		is_zone_device_page(page) &&
 		page->pgmap->type == MEMORY_DEVICE_PCI_P2PDMA;
 }
include/linux/mm.h +3 -3
index 80fccfe31c3444..ff9f149ca2017e 100644
@@ -1090,7 +1090,7 @@ static inline bool is_zone_movable_page(const struct page *page)
 	return page_zonenum(page) == ZONE_MOVABLE;
 }
 
-#ifdef CONFIG_DEV_PAGEMAP_OPS
+#if defined(CONFIG_ZONE_DEVICE) && defined(CONFIG_FS_DAX)
 DECLARE_STATIC_KEY_FALSE(devmap_managed_key);
 
 bool __put_devmap_managed_page(struct page *page);
@@ -1103,12 +1103,12 @@ static inline bool put_devmap_managed_page(struct page *page)
 	return __put_devmap_managed_page(page);
 }
 
-#else /* CONFIG_DEV_PAGEMAP_OPS */
+#else /* CONFIG_ZONE_DEVICE && CONFIG_FS_DAX */
 static inline bool put_devmap_managed_page(struct page *page)
 {
 	return false;
 }
-#endif /* CONFIG_DEV_PAGEMAP_OPS */
+#endif /* CONFIG_ZONE_DEVICE && CONFIG_FS_DAX */
 
 /* 127: arbitrary random number, small enough to assemble well */
 #define folio_ref_zero_or_close_to_overflow(folio) \
lib/test_hmm.c +0 -1
index e5fc14ba71f33e..cfe63204783918 100644
@@ -566,7 +566,6 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice)
 	}
 
 	dpage->zone_device_data = rpage;
-	get_page(dpage);
 	lock_page(dpage);
 	return dpage;
 
mm/Kconfig +0 -4
index 3326ee3903f330..a1901ae6d06293 100644
@@ -776,9 +776,6 @@ config ZONE_DEVICE
 
 	  If FS_DAX is enabled, then say Y.
 
-config DEV_PAGEMAP_OPS
-	bool
-
 #
 # Helpers to mirror range of the CPU page tables of a process into device page
 # tables.
@@ -790,7 +787,6 @@ config HMM_MIRROR
 config DEVICE_PRIVATE
 	bool "Unaddressable device memory (GPU memory, ...)"
 	depends on ZONE_DEVICE
-	select DEV_PAGEMAP_OPS
 
 	help
 	  Allows creation of struct pages to represent unaddressable device
mm/internal.h +2 -0
index d80300392a194f..a67222d17e5987 100644
@@ -718,4 +718,6 @@ void vunmap_range_noflush(unsigned long start, unsigned long end);
 int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
 		      unsigned long addr, int page_nid, int *flags);
 
+void free_zone_device_page(struct page *page);
+
 #endif	/* __MM_INTERNAL_H */
mm/memcontrol.c +3 -8
index 09d342c7cbd0d9..d1e97a54ae535e 100644
@@ -5503,17 +5503,12 @@ static struct page *mc_handle_swap_pte(struct vm_area_struct *vma,
 		return NULL;
 
 	/*
-	 * Handle MEMORY_DEVICE_PRIVATE which are ZONE_DEVICE page belonging to
-	 * a device and because they are not accessible by CPU they are store
-	 * as special swap entry in the CPU page table.
+	 * Handle device private pages that are not accessible by the CPU, but
+	 * stored as special swap entries in the page table.
 	 */
 	if (is_device_private_entry(ent)) {
 		page = pfn_swap_entry_to_page(ent);
-		/*
-		 * MEMORY_DEVICE_PRIVATE means ZONE_DEVICE page and which have
-		 * a refcount of 1 when free (unlike normal page)
-		 */
-		if (!page_ref_add_unless(page, 1, 1))
+		if (!get_page_unless_zero(page))
 			return NULL;
 		return page;
 	}
mm/memremap.c +19 -38

131 lines not on this page · raw message

mm/migrate.c +0 -6
index c7da064b4781b8..8e0370a73f8a43 100644
@@ -341,14 +341,8 @@ static int expected_page_refs(struct address_space *mapping, struct page *page)
 {
 	int expected_count = 1;
 
-	/*
-	 * Device private pages have an extra refcount as they are
-	 * ZONE_DEVICE pages.
-	 */
-	expected_count += is_device_private_page(page);
 	if (mapping)
 		expected_count += compound_nr(page) + page_has_private(page);
-
 	return expected_count;
 }
 
mm/swap.c +4 -12
index 25b55c56614311..c84d6817043257 100644
@@ -114,17 +114,9 @@ static void __put_compound_page(struct page *page)
 
 void __put_page(struct page *page)
 {
-	if (is_zone_device_page(page)) {
-		put_dev_pagemap(page->pgmap);
-
-		/*
-		 * The page belongs to the device that created pgmap. Do
-		 * not return it to page allocator.
-		 */
-		return;
-	}
-
-	if (unlikely(PageCompound(page)))
+	if (unlikely(is_zone_device_page(page)))
+		free_zone_device_page(page);
+	else if (unlikely(PageCompound(page)))
 		__put_compound_page(page);
 	else
 		__put_single_page(page);
@@ -933,7 +925,7 @@ void release_pages(struct page **pages, int nr)
 			if (put_devmap_managed_page(page))
 				continue;
 			if (put_page_testzero(page))
-				put_dev_pagemap(page->pgmap);
+				free_zone_device_page(page);
 			continue;
 		}
 
Signature · 1 line
2.30.2
Reviewed-by on v1 7/8
#36 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

Quoted 16 lines · Christoph Hellwig

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE pages.

Note that this excludes the special idle page wakeup for fsdax pages, which still happens at refcount 1. This is a separate issue and will be sorted out later. Given that only fsdax pages require the notifiacation when the refcount hits 1 now, the PAGEMAP_OPS Kconfig symbol can go away and be replaced with a FS_DAX check for this hook in the put_page fastpath.

Based on an earlier patch from Ralph Campbell <[email protected]>.

Quoted 15 lines
---
 arch/powerpc/kvm/book3s_hv_uvmem.c       |  1 -
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c |  1 -
 drivers/gpu/drm/nouveau/nouveau_dmem.c   |  1 -
 fs/Kconfig                               |  1 -
 include/linux/memremap.h                 | 12 +++--
 include/linux/mm.h                       |  6 +--
 lib/test_hmm.c                           |  1 -
 mm/Kconfig                               |  4 --
 mm/internal.h                            |  2 +
 mm/memcontrol.c                          | 11 ++---
 mm/memremap.c                            | 57 ++++++++----------------
 mm/migrate.c                             |  6 ---
 mm/swap.c                                | 16 ++-----
 13 files changed, 36 insertions(+), 83 deletions(-)

It looks like a good next step to me

include/linux/memremap.h enum memory_type { · line 73

Quoted 8 lines

struct dev_pagemap_ops {

include/linux/memremap.h +3 -3
 	/*
-	 * Called once the page refcount reaches 1.  (ZONE_DEVICE pages never
-	 * reach 0 refcount unless there is a refcount bug. This allows the
-	 * device driver to implement its own memory management.)
+	 * Called once the page refcount reaches 0.  The reference count will be
+	 * reset to one by the core code after the method is called to prepare
+	 * for handing out the page again.

I did prefer Ralph's version of this that kept the refcount at 0 while the page was on the free-list. I hope we can get there again after later series :)

Jason

Reviewed-by on v1 7/8
#37 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

In reply to Christoph Hellwig
Quoted 13 lines · Christoph Hellwig

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE pages.

Note that this excludes the special idle page wakeup for fsdax pages, which still happens at refcount 1. This is a separate issue and will be sorted out later. Given that only fsdax pages require the notifiacation when the refcount hits 1 now, the PAGEMAP_OPS Kconfig symbol can go away and be replaced with a FS_DAX check for this hook in the put_page fastpath.

Quoted 3 lines

Based on an earlier patch from Ralph Campbell <[email protected]>.

Thanks for working on this, definite step forward.

Reviewed-by on v1 7/8
#38 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

In reply to Christoph Hellwig

On Sun, Feb 6, 2022 at 10:33 PM Christoph Hellwig <[email protected]> wrote:
[..]

mm/memremap.c void free_devmap_managed_page(struct page *page) · line 489

Quoted 9 lines
mm/memremap.c +5 -0
@@ -500,28 +482,27 @@ void free_devmap_managed_page(struct page *page)
         */
        page->mapping = NULL;
        page->pgmap->ops->page_free(page);
+
+       /*
+        * Reset the page count to 1 to prepare for handing out the page again.
+        */
+       set_page_count(page, 1);

Interesting. I had expected that to really fix the refcount problem that fs/dax.c would need to start taking real page references as pages were added to a mapping, just like page cache.

This looks ok to me, and passes my tests. So given I'm still working my way back to fixing the references properly I'm ok for this hack to replace the more broken hack that is there presently.

author maintainer
#39 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

Quoted 3 lines · Dan Williams

Interesting. I had expected that to really fix the refcount problem that fs/dax.c would need to start taking real page references as pages were added to a mapping, just like page cache.

I think we should do that eventually. But I think this series that just attacks the device private type and extends to the device coherent and p2p enhacements is a good first step to stop the proliferation of the one off refcount and to allow to deal with the fsdax pages in another more focuessed series.

#40 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

Quoted 10 lines · Christoph Hellwig
Quoted 3 lines · Dan Williams

Interesting. I had expected that to really fix the refcount problem that fs/dax.c would need to start taking real page references as pages were added to a mapping, just like page cache.

I think we should do that eventually. But I think this series that just attacks the device private type and extends to the device coherent and p2p enhacements is a good first step to stop the proliferation of the one off refcount and to allow to deal with the fsdax pages in another more focuessed series.

It is nice, but the other series are still impacted by the fsdax mess
- they still stuff pages into ptes without proper refcounts and have to carry nonsense to dance around this problem.

I certainly would be unhappy if the amd driver, for instance, gained the fsdax problem as well and started pushing 4k pages into PMDs.

Jason

author maintainer
#41 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

Quoted 6 lines · Jason Gunthorpe

It is nice, but the other series are still impacted by the fsdax mess
- they still stuff pages into ptes without proper refcounts and have to carry nonsense to dance around this problem.

I certainly would be unhappy if the amd driver, for instance, gained the fsdax problem as well and started pushing 4k pages into PMDs.

As said before: I think this all needs to be fixed. But I'd rather fix it gradually and I think this series is a nice step forward. After that we can look at the pte mappings.

#42 lore raw

Re: [PATCH 7/8] mm: remove the extra ZONE_DEVICE struct page refcount

Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Jason Gunthorpe

It is nice, but the other series are still impacted by the fsdax mess
- they still stuff pages into ptes without proper refcounts and have to carry nonsense to dance around this problem.

I certainly would be unhappy if the amd driver, for instance, gained the fsdax problem as well and started pushing 4k pages into PMDs.

As said before: I think this all needs to be fixed. But I'd rather fix it gradually and I think this series is a nice step forward. After that we can look at the pte mappings.

Right, I agree with this

Jason

author maintainer
#43 lore raw

PATCH 8/8 fsdax: depend on ZONE_DEVICE || FS_DAX_LIMITED

In reply to Christoph Hellwig

Add a depends on ZONE_DEVICE support or the s390-specific limited DAX support, as one of the two is required at runtime for fsdax code to actually work.

---
 fs/Kconfig | 1 +
 1 file changed, 1 insertion(+)
fs/Kconfig +1 -0
index 05efea674bffa0..6e8818a5e53c45 100644
@@ -48,6 +48,7 @@ config FS_DAX
 	bool "File system based Direct Access (DAX) support"
 	depends on MMU
 	depends on !(ARM || MIPS || SPARC)
+	depends on ZONE_DEVICE || FS_DAX_LIMITED
 	select FS_IOMAP
 	select DAX
 	help
Signature · 1 line
2.30.2
Reviewed-by on v1 8/8
#44 lore raw

Re: [PATCH 8/8] fsdax: depend on ZONE_DEVICE || FS_DAX_LIMITED

Quoted 8 lines · Christoph Hellwig

Add a depends on ZONE_DEVICE support or the s390-specific limited DAX support, as one of the two is required at runtime for fsdax code to actually work.

---
 fs/Kconfig | 1 +
 1 file changed, 1 insertion(+)

Makes sense, but leaves me wonder why a kconfig randomizer didn't hit this.. Or maybe it means some of the function stubs on !ZONE_DEVICE are unnecessary now..

Jason

Reviewed-by
#45 lore raw
In reply to Christoph Hellwig
Quoted 27 lines · Christoph Hellwig

Hi all,

this series removes the offset by one refcount for ZONE_DEVICE pages that are freed back to the driver owning them, which is just device private ones for now, but also the planned device coherent pages and the ehanced p2p ones pending.

It does not address the fsdax pages yet, which will be attacked in a follow on series.

Diffstat:

 arch/arm64/mm/mmu.c                      |    1 
 arch/powerpc/kvm/book3s_hv_uvmem.c       |    1 
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c |    2 
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h    |    1 
 drivers/gpu/drm/drm_cache.c              |    2 
 drivers/gpu/drm/nouveau/nouveau_dmem.c   |    3 -
 drivers/gpu/drm/nouveau/nouveau_svm.c    |    1 
 drivers/infiniband/core/rw.c             |    1 
 drivers/nvdimm/pmem.h                    |    1 
 drivers/nvme/host/pci.c                  |    1 
 drivers/nvme/target/io-cmd-bdev.c        |    1 
 fs/Kconfig                               |    2 
 fs/fuse/virtio_fs.c                      |    1 
 include/linux/hmm.h                      |    9 ----
 include/linux/memremap.h                 |   22 +++++++++-
 include/linux/mm.h                       |   59 ++++-------------------------
Quoted 8 lines
 lib/test_hmm.c                           |    4 +
 mm/Kconfig                               |    4 -
 mm/internal.h                            |    2 
 mm/memcontrol.c                          |   11 +----
 mm/memremap.c                            |   63 ++++++++++++++++---------------
 mm/migrate.c                             |    6 --
 mm/swap.c                                |   49 ++----------------------
 23 files changed, 90 insertions(+), 157 deletions(-)

Looks good to me. I was wondering about the location of some of this code, so it's nice to see it cleaned up. Except for the one minor issue I noted on patch 6, it all looks good to me. I've reviewed all the patches and tested the series under my p2pdma series.

Logan

#46 lore raw
In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig

Hi all,

this series removes the offset by one refcount for ZONE_DEVICE pages that are freed back to the driver owning them, which is just device private ones for now, but also the planned device coherent pages and the ehanced p2p ones pending.

Many thanks for doing this, I remember the hard time when I read the relevant code. :)

Quoted 31 lines

It does not address the fsdax pages yet, which will be attacked in a follow on series.

Diffstat:

 arch/arm64/mm/mmu.c                      |    1 
 arch/powerpc/kvm/book3s_hv_uvmem.c       |    1 
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c |    2 
 drivers/gpu/drm/amd/amdkfd/kfd_priv.h    |    1 
 drivers/gpu/drm/drm_cache.c              |    2 
 drivers/gpu/drm/nouveau/nouveau_dmem.c   |    3 -
 drivers/gpu/drm/nouveau/nouveau_svm.c    |    1 
 drivers/infiniband/core/rw.c             |    1 
 drivers/nvdimm/pmem.h                    |    1 
 drivers/nvme/host/pci.c                  |    1 
 drivers/nvme/target/io-cmd-bdev.c        |    1 
 fs/Kconfig                               |    2 
 fs/fuse/virtio_fs.c                      |    1 
 include/linux/hmm.h                      |    9 ----
 include/linux/memremap.h                 |   22 +++++++++-
 include/linux/mm.h                       |   59 ++++-------------------------
 lib/test_hmm.c                           |    4 +
 mm/Kconfig                               |    4 -
 mm/internal.h                            |    2 
 mm/memcontrol.c                          |   11 +----
 mm/memremap.c                            |   63 ++++++++++++++++---------------
 mm/migrate.c                             |    6 --
 mm/swap.c                                |   49 ++----------------------
 23 files changed, 90 insertions(+), 157 deletions(-)

.