Skip to contentSkip to conversation

Support DEVICE_GENERIC memory in migrate_vma_*

PATCH Superseded Alex Sierra 7 participants 26 messages Jun 17, 2021 – Jul 30, 2021

Subsystems (18)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover Support DEVICE_GENERIC memory in migrate_vma_*

v1:
AMD is building a system architecture for the Frontier supercomputer with a coherent interconnect between CPUs and GPUs. This hardware architecture allows the CPUs to coherently access GPU device memory. We have hardware in our labs and we are working with our partner HPE on the BIOS, firmware and software for delivery to the DOE.

The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver looks it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC using devm_memremap_pages.

Now we're trying to migrate data to and from that memory using the migrate_vma_* helpers so we can support page-based migration in our unified memory allocations, while also supporting CPU access to those pages.

This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave correctly in the migrate_vma_* helpers. We are looking for feedback about this approach. If we're close, what's needed to make our patches acceptable upstream? If we're not close, any suggestions how else to achieve what we are trying to do (i.e. page migration and coherent CPU access to VRAM)?

This work is based on HMM and our SVM memory manager that was recently upstreamed to Dave Airlie's drm-next branch
https://lore.kernel.org/dri-devel/[email protected]/T/#r996356015e295780eb50453e7dbd5d0d68b47cbc
On top of that we did some rework of our VRAM management for migrations to remove some incorrect assumptions, allow partially successful migrations and GPU memory mappings that mix pages in VRAM and system memory.
https://patchwork.kernel.org/project/dri-devel/list/?series=489811

v2:
This patch series version has merged "[RFC PATCH v3 0/2]
mm: remove extra ZONE_DEVICE struct page refcount" patch series made by Ralph Campbell. It also applies at the top of these series, our changes to support device generic type in migration_vma helpers. This has been tested in systems with device memory that has coherent access by CPU.

Also addresses the following feedback made in v1:
- Isolate in one patch kernel/resource.c modification, based on Christoph's feedback.
- Add helpers check for generic and private type to avoid duplicated long lines.

v3:
- Include cover letter from v1
- Rename dax_layout_is_idle_page func to dax_page_unused in patch ext4/xfs: add page refcount helper

Patches 1-2 Rebased Ralph Campbell's ZONE_DEVICE page refcounting patches Patches 4-5 are for context to show how we are looking up the SPM memory and registering it with devmap.
Patches 3,6-8 are the changes we are trying to upstream or rework to make them acceptable upstream.

Alex Sierra (6):
kernel: resource: lookup_resource as exported symbol drm/amdkfd: add SPM support for SVM
drm/amdkfd: generic type as sys mem on migration to ram include/linux/mm.h: helpers to check zone device generic type
mm: add generic type support to migrate_vma helpers
mm: call pgmap->ops->page_free for DEVICE_GENERIC pages

Ralph Campbell (2):
ext4/xfs: add page refcount helper
mm: remove extra ZONE_DEVICE struct page refcount

 arch/powerpc/kvm/book3s_hv_uvmem.c       |  2 +-
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 ++++--
 drivers/gpu/drm/nouveau/nouveau_dmem.c   |  2 +-
 fs/dax.c                                 |  8 +--
 fs/ext4/inode.c                          |  5 +-
 fs/xfs/xfs_file.c                        |  4 +-
 include/linux/dax.h                      | 10 ++++
 include/linux/memremap.h                 |  7 +--
 include/linux/mm.h                       | 52 +++---------------
 kernel/resource.c                        |  2 +-
 lib/test_hmm.c                           |  2 +-
 mm/internal.h                            |  8 +++
 mm/memremap.c                            | 69 +++++++-----------------
 mm/migrate.c                             | 13 ++---
 mm/page_alloc.c                          |  3 ++
 mm/swap.c                                | 45 ++--------------
 16 files changed, 83 insertions(+), 164 deletions(-)
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

PATCH v3 1/8 ext4/xfs: add page refcount helper

From: Ralph Campbell <[email protected]>

There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

v2:
[AS]: rename dax_layout_is_idle_page func to dax_page_unused

---
 fs/dax.c            |  4 ++--
 fs/ext4/inode.c     |  5 +----
 fs/xfs/xfs_file.c   |  4 +---
 include/linux/dax.h | 10 ++++++++++
 4 files changed, 14 insertions(+), 9 deletions(-)
fs/dax.c +2 -2
index 26d5dcd2d69e..321f4ddc6643 100644
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+		WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));
 		WARN_ON_ONCE(page->mapping && page->mapping != mapping);
 		page->mapping = NULL;
 		page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		if (page_ref_count(page) > 1)
+		if (!dax_layout_is_idle_page(page))
 			return page;
 	}
 	return NULL;
fs/ext4/inode.c +1 -4
index c173c8405856..9ee00186412f 100644
@@ -3972,10 +3972,7 @@ int ext4_break_layouts(struct inode *inode)
 		if (!page)
 			return 0;
 
-		error = ___wait_var_event(&page->_refcount,
-				atomic_read(&page->_refcount) == 1,
-				TASK_INTERRUPTIBLE, 0, 0,
-				ext4_wait_dax_page(ei));
+		error = dax_wait_page(ei, page, ext4_wait_dax_page);
 	} while (error == 0);
 
 	return error;
fs/xfs/xfs_file.c +1 -3
index 5b0f93f73837..39565fe5f817 100644
@@ -782,9 +782,7 @@ xfs_break_dax_layouts(
 		return 0;
 
 	*retry = true;
-	return ___wait_var_event(&page->_refcount,
-			atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
-			0, 0, xfs_wait_dax_page(inode));
+	return dax_wait_page(inode, page, xfs_wait_dax_page);
 }
 
 int
include/linux/dax.h +10 -0
index b52f084aa643..8b5da1d60dbc 100644
@@ -243,6 +243,16 @@ static inline bool dax_mapping(struct address_space *mapping)
 	return mapping->host && IS_DAX(mapping->host);
 }
 
+static inline bool dax_page_unused(struct page *page)
+{
+	return page_ref_count(page) == 1;
+}
+
+#define dax_wait_page(_inode, _page, _wait_cb)				\
+	___wait_var_event(&(_page)->_refcount,				\
+		dax_page_unused(_page),				\
+		TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode))
+
 #ifdef CONFIG_DEV_DAX_HMEM_DEVICES
 void hmem_register_device(int target_nid, struct resource *r);
 #else
Signature · 1 line
2.17.1
Acked-by on v3 1/8
#2 lore raw

Re: [PATCH v3 1/8] ext4/xfs: add page refcount helper

Quoted 61 lines · Alex Sierra

From: Ralph Campbell <[email protected]>

There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

v2:
[AS]: rename dax_layout_is_idle_page func to dax_page_unused

---
 fs/dax.c            |  4 ++--
 fs/ext4/inode.c     |  5 +----
 fs/xfs/xfs_file.c   |  4 +---
 include/linux/dax.h | 10 ++++++++++
 4 files changed, 14 insertions(+), 9 deletions(-)
fs/dax.c +2 -2
index 26d5dcd2d69e..321f4ddc6643 100644
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+		WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));
 		WARN_ON_ONCE(page->mapping && page->mapping != mapping);
 		page->mapping = NULL;
 		page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		if (page_ref_count(page) > 1)
+		if (!dax_layout_is_idle_page(page))
 			return page;
 	}
 	return NULL;
fs/ext4/inode.c +1 -4
index c173c8405856..9ee00186412f 100644
@@ -3972,10 +3972,7 @@ int ext4_break_layouts(struct inode *inode)
 		if (!page)
 			return 0;
 
-		error = ___wait_var_event(&page->_refcount,
-				atomic_read(&page->_refcount) == 1,
-				TASK_INTERRUPTIBLE, 0, 0,
-				ext4_wait_dax_page(ei));
+		error = dax_wait_page(ei, page, ext4_wait_dax_page);
 	} while (error == 0);
 
 	return error;
fs/xfs/xfs_file.c +0 -0
index 5b0f93f73837..39565fe5f817 100644

fs/xfs/xfs_file.c xfs_break_dax_layouts( · line 785

Quoted 7 lines
fs/xfs/xfs_file.c +1 -3
@@ -782,9 +782,7 @@ xfs_break_dax_layouts(
 		return 0;
 
 	*retry = true;
-	return ___wait_var_event(&page->_refcount,
-			atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
-			0, 0, xfs_wait_dax_page(inode));
+	return dax_wait_page(inode, page, xfs_wait_dax_page);

Mechanically, this looks like a straightforward replacement, so:

--D

Quoted 27 lines
fs/xfs/xfs_file.c +0 -0
 }
 
 int
include/linux/dax.h +10 -0
index b52f084aa643..8b5da1d60dbc 100644
@@ -243,6 +243,16 @@ static inline bool dax_mapping(struct address_space *mapping)
 	return mapping->host && IS_DAX(mapping->host);
 }
 
+static inline bool dax_page_unused(struct page *page)
+{
+	return page_ref_count(page) == 1;
+}
+
+#define dax_wait_page(_inode, _page, _wait_cb)				\
+	___wait_var_event(&(_page)->_refcount,				\
+		dax_page_unused(_page),				\
+		TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode))
+
 #ifdef CONFIG_DEV_DAX_HMEM_DEVICES
 void hmem_register_device(int target_nid, struct resource *r);
 #else
Signature · 1 line
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

Re: [PATCH v3 1/8] ext4/xfs: add page refcount helper

In reply to Alex Sierra
Quoted 8 lines · Alex Sierra

From: Ralph Campbell <[email protected]>

There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

v2:
[AS]: rename dax_layout_is_idle_page func to dax_page_unused

Did you even compile test this?

Quoted 14 lines
---
 fs/dax.c            |  4 ++--
 fs/ext4/inode.c     |  5 +----
 fs/xfs/xfs_file.c   |  4 +---
 include/linux/dax.h | 10 ++++++++++
 4 files changed, 14 insertions(+), 9 deletions(-)
fs/dax.c +0 -0
index 26d5dcd2d69e..321f4ddc6643 100644

fs/dax.c static void dax_disassociate_entry(void *entry, struct address_space *mapping, · line 361

Quoted 5 lines
fs/dax.c +1 -1
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+		WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));

Because you still use dax_layout_is_idle_page() here, not dax_page_unused()...

fs/dax.c static struct page *dax_busy_page(void *entry) · line 375

Quoted 9 lines
fs/dax.c +1 -1
 		WARN_ON_ONCE(page->mapping && page->mapping != mapping);
 		page->mapping = NULL;
 		page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		if (page_ref_count(page) > 1)
+		if (!dax_layout_is_idle_page(page))

Here too.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw

Re: [PATCH v3 1/8] ext4/xfs: add page refcount helper

Quoted 10 lines · Dave Chinner
Quoted 8 lines · Alex Sierra

From: Ralph Campbell <[email protected]>

There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

v2:
[AS]: rename dax_layout_is_idle_page func to dax_page_unused

Did you even compile test this?

Thanks for catching this Dave, my mistake, I was building without CONFIG_FS_DAX, as I was using DEVICE_GENERIC only. I'll send the corrected patch in replay.

Regards,

Alex Sierra

Quoted 37 lines
Quoted 15 lines
---
  fs/dax.c            |  4 ++--
  fs/ext4/inode.c     |  5 +----
  fs/xfs/xfs_file.c   |  4 +---
  include/linux/dax.h | 10 ++++++++++
  4 files changed, 14 insertions(+), 9 deletions(-)
fs/dax.c +0 -0
index 26d5dcd2d69e..321f4ddc6643 100644

fs/dax.c static void dax_disassociate_entry(void *entry, struct address_space *mapping, · line 361

Quoted 5 lines
fs/dax.c +1 -1
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
  	for_each_mapped_pfn(entry, pfn) {
  		struct page *page = pfn_to_page(pfn);
 
-		WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+		WARN_ON_ONCE(trunc && !dax_layout_is_idle_page(page));

Because you still use dax_layout_is_idle_page() here, not dax_page_unused()...

fs/dax.c static struct page *dax_busy_page(void *entry) · line 375

Quoted 9 lines
fs/dax.c +1 -1
  		WARN_ON_ONCE(page->mapping && page->mapping != mapping);
  		page->mapping = NULL;
  		page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
  	for_each_mapped_pfn(entry, pfn) {
  		struct page *page = pfn_to_page(pfn);
 
-		if (page_ref_count(page) > 1)
+		if (!dax_layout_is_idle_page(page))

Here too.

Cheers,

Dave.

#5 lore raw

PATCH v3 1/8 ext4/xfs: add page refcount helper

In reply to Alex Sierra

From: Ralph Campbell <[email protected]>

There are several places where ZONE_DEVICE struct pages assume a reference count == 1 means the page is idle and free. Instead of open coding this, add a helper function to hide this detail.

v3:
[AS]: rename dax_layout_is_idle_page func to dax_page_unused

---
 fs/dax.c            |  4 ++--
 fs/ext4/inode.c     |  5 +----
 fs/xfs/xfs_file.c   |  4 +---
 include/linux/dax.h | 10 ++++++++++
 4 files changed, 14 insertions(+), 9 deletions(-)
fs/dax.c +2 -2
index 26d5dcd2d69e..4820bb511d68 100644
@@ -358,7 +358,7 @@ static void dax_disassociate_entry(void *entry, struct address_space *mapping,
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		WARN_ON_ONCE(trunc && page_ref_count(page) > 1);
+		WARN_ON_ONCE(trunc && !dax_page_unused(page));
 		WARN_ON_ONCE(page->mapping && page->mapping != mapping);
 		page->mapping = NULL;
 		page->index = 0;
@@ -372,7 +372,7 @@ static struct page *dax_busy_page(void *entry)
 	for_each_mapped_pfn(entry, pfn) {
 		struct page *page = pfn_to_page(pfn);
 
-		if (page_ref_count(page) > 1)
+		if (!dax_page_unused(page))
 			return page;
 	}
 	return NULL;
fs/ext4/inode.c +1 -4
index c173c8405856..9ee00186412f 100644
@@ -3972,10 +3972,7 @@ int ext4_break_layouts(struct inode *inode)
 		if (!page)
 			return 0;
 
-		error = ___wait_var_event(&page->_refcount,
-				atomic_read(&page->_refcount) == 1,
-				TASK_INTERRUPTIBLE, 0, 0,
-				ext4_wait_dax_page(ei));
+		error = dax_wait_page(ei, page, ext4_wait_dax_page);
 	} while (error == 0);
 
 	return error;
fs/xfs/xfs_file.c +1 -3
index 5b0f93f73837..39565fe5f817 100644
@@ -782,9 +782,7 @@ xfs_break_dax_layouts(
 		return 0;
 
 	*retry = true;
-	return ___wait_var_event(&page->_refcount,
-			atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
-			0, 0, xfs_wait_dax_page(inode));
+	return dax_wait_page(inode, page, xfs_wait_dax_page);
 }
 
 int
include/linux/dax.h +10 -0
index b52f084aa643..8b5da1d60dbc 100644
@@ -243,6 +243,16 @@ static inline bool dax_mapping(struct address_space *mapping)
 	return mapping->host && IS_DAX(mapping->host);
 }
 
+static inline bool dax_page_unused(struct page *page)
+{
+	return page_ref_count(page) == 1;
+}
+
+#define dax_wait_page(_inode, _page, _wait_cb)				\
+	___wait_var_event(&(_page)->_refcount,				\
+		dax_page_unused(_page),				\
+		TASK_INTERRUPTIBLE, 0, 0, _wait_cb(_inode))
+
 #ifdef CONFIG_DEV_DAX_HMEM_DEVICES
 void hmem_register_device(int target_nid, struct resource *r);
 #else
Signature · 6 lines
2.32.0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#6 lore raw

PATCH v3 2/8 mm: remove extra ZONE_DEVICE struct page refcount

In reply to Alex Sierra

From: Ralph Campbell <[email protected]>

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.

v2:
AS: merged this patch in linux 5.11 version

---
 arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
 drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
 fs/dax.c                               |  4 +-
 include/linux/dax.h                    |  2 +-
 include/linux/memremap.h               |  7 +--
 include/linux/mm.h                     | 44 -----------------
 lib/test_hmm.c                         |  2 +-
 mm/internal.h                          |  8 +++
 mm/memremap.c                          | 68 +++++++-------------------
 mm/migrate.c                           |  5 --
 mm/page_alloc.c                        |  3 ++
 mm/swap.c                              | 45 ++---------------
 12 files changed, 45 insertions(+), 147 deletions(-)
arch/powerpc/kvm/book3s_hv_uvmem.c +1 -1
index 84e5a2dc8be5..acee67710620 100644
@@ -711,7 +711,7 @@ static struct page *kvmppc_uvmem_get_page(unsigned long gpa, struct kvm *kvm)
 
 	dpage = pfn_to_page(uvmem_pfn);
 	dpage->zone_device_data = pvt;
-	get_page(dpage);
+	init_page_count(dpage);
 	lock_page(dpage);
 	return dpage;
 out_clear:
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -1
index 92987daa5e17..8bc7120e1216 100644
@@ -324,7 +324,7 @@ nouveau_dmem_page_alloc_locked(struct nouveau_drm *drm)
 			return NULL;
 	}
 
-	get_page(page);
+	init_page_count(page);
 	lock_page(page);
 	return page;
 }
fs/dax.c +2 -2
index 321f4ddc6643..7b4c6b35b098 100644
@@ -560,14 +560,14 @@ static void *grab_mapping_entry(struct xa_state *xas,
 
 /**
  * dax_layout_busy_page_range - find first pinned page in @mapping
- * @mapping: address space to scan for a page with ref count > 1
+ * @mapping: address space to scan for a page with ref count > 0
  * @start: Starting offset. Page containing 'start' is included.
  * @end: End offset. Page containing 'end' is included. If 'end' is LLONG_MAX,
  *       pages from 'start' till the end of file are included.
  *
  * DAX requires ZONE_DEVICE mapped pages. These pages are never
  * 'onlined' to the page allocator so they are considered idle when
- * page->count == 1. A filesystem uses this interface to determine if
+ * page->count == 0. A filesystem uses this interface to determine if
  * any page in the mapping is busy, i.e. for DMA, or other
  * get_user_pages() usages.
  *
include/linux/dax.h +1 -1
index 8b5da1d60dbc..05fc982ce153 100644
@@ -245,7 +245,7 @@ static inline bool dax_mapping(struct address_space *mapping)
 
 static inline bool dax_page_unused(struct page *page)
 {
-	return page_ref_count(page) == 1;
+	return page_ref_count(page) == 0;
 }
 
 #define dax_wait_page(_inode, _page, _wait_cb)				\
include/linux/memremap.h +4 -3
index 79c49e7f5c30..327f32427d21 100644
@@ -66,9 +66,10 @@ enum memory_type {
 
 struct dev_pagemap_ops {
 	/*
-	 * Called once the page refcount reaches 1.  (ZONE_DEVICE pages never
-	 * reach 0 refcount unless there is a refcount bug. This allows the
-	 * device driver to implement its own memory management.)
+	 * Called once the page refcount reaches 0. The reference count
+	 * should be reset to one with init_page_count(page) before reusing
+	 * the page. This allows the device driver to implement its own
+	 * memory management.
 	 */
 	void (*page_free)(struct page *page);
 
include/linux/mm.h +0 -44

56 lines not on this page · raw message

lib/test_hmm.c +1 -1
index 80a78877bd93..6998f10350ea 100644
@@ -561,7 +561,7 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice)
 	}
 
 	dpage->zone_device_data = rpage;
-	get_page(dpage);
+	init_page_count(dpage);
 	lock_page(dpage);
 	return dpage;
 
mm/internal.h +8 -0
index 25d2b2439f19..d3e58746f2d2 100644
@@ -623,4 +623,12 @@ struct migration_target_control {
 	gfp_t gfp_mask;
 };
 
+#ifdef CONFIG_DEV_PAGEMAP_OPS
+void free_zone_device_page(struct page *page);
+#else
+static inline void free_zone_device_page(struct page *page)
+{
+}
+#endif
+
 #endif	/* __MM_INTERNAL_H */
mm/memremap.c +19 -49

121 lines not on this page · raw message

mm/migrate.c +0 -5
index 20ca887ea769..8c2430d3e77b 100644
@@ -376,11 +376,6 @@ static int expected_page_refs(struct address_space *mapping, struct page *page)
 {
 	int expected_count = 1;
 
-	/*
-	 * Device private pages have an extra refcount as they are
-	 * ZONE_DEVICE pages.
-	 */
-	expected_count += is_device_private_page(page);
 	if (mapping)
 		expected_count += thp_nr_pages(page) + page_has_private(page);
 
mm/page_alloc.c +3 -0
index 519a60d5b6f7..4612c457d0b0 100644
@@ -6210,6 +6210,9 @@ void __ref memmap_init_zone_device(struct zone *zone,
 
 		__init_single_page(page, pfn, zone_idx, nid);
 
+		/* ZONE_DEVICE pages start with a zero reference count. */
+		set_page_count(page, 0);
+
 		/*
 		 * Mark page reserved as it will need to wait for onlining
 		 * phase for it to be fully associated with a zone.
mm/swap.c +5 -40

72 lines not on this page · raw message

Signature · 1 line
2.17.1
#7 lore raw

Re: [PATCH v3 2/8] mm: remove extra ZONE_DEVICE struct page refcount

Quoted 13 lines · Alex Sierra

From: Ralph Campbell <[email protected]>

ZONE_DEVICE struct pages have an extra reference count that complicates the code for put_page() and several places in the kernel that need to check the reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to be treated specially for ZONE_DEVICE.

v2:
AS: merged this patch in linux 5.11 version

Quoted 15 lines
---
  arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
  drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
  fs/dax.c                               |  4 +-
  include/linux/dax.h                    |  2 +-
  include/linux/memremap.h               |  7 +--
  include/linux/mm.h                     | 44 -----------------
  lib/test_hmm.c                         |  2 +-
  mm/internal.h                          |  8 +++
  mm/memremap.c                          | 68 +++++++-------------------
  mm/migrate.c                           |  5 --
  mm/page_alloc.c                        |  3 ++
  mm/swap.c                              | 45 ++---------------
  12 files changed, 45 insertions(+), 147 deletions(-)

I think it is great that you are picking this up and trying to revive it.

However, I have a number of concerns about how it affects existing ZONE_DEVICE MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this addressing them. For example, dev_dax_probe() allocates MEMORY_DEVICE_GENERIC struct pages and then:
dev_dax_fault()

    dev_dax_huge_fault()
      __dev_dax_pte_fault()
        vmf_insert_mixed()

which just inserts the PFN into the CPU page tables without increasing the page refcount so it is zero (whereas it was one before). But using get_page() will trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current notion of free verses allocated for these struct pages. I suppose init_page_count() could be called on all the struct pages in dev_dax_probe() to fix that though.

I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File systems have clear allocate and free states for backing storage but there are the complications with the page cache references, etc. to consider. The >1 to 1 reference count seems to be used to tell when a page is idle (no I/O, reclaim scanners) rather than free (not allocated to any file) but I'm not 100% sure about that since I don't really understand all the issues around why a file system needs to have a DAX mount option besides knowing that the storage block size has to be a multiple of the page size.

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#8 lore raw

Re: [PATCH v3 2/8] mm: remove extra ZONE_DEVICE struct page refcount

Am 2021-06-17 um 3:16 p.m. schrieb Ralph Campbell:

Quoted 46 lines · Alex Sierra
Quoted 29 lines

From: Ralph Campbell <[email protected]>

ZONE_DEVICE struct pages have an extra reference count that complicates the
code for put_page() and several places in the kernel that need to check the
reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to
be treated specially for ZONE_DEVICE.

v2:
AS: merged this patch in linux 5.11 version

---
  arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +- drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
  fs/dax.c                               |  4 +- include/linux/dax.h                    |  2 +- include/linux/memremap.h               |  7 +-- include/linux/mm.h                     | 44 ----------------- lib/test_hmm.c                         |  2 +- mm/internal.h                          |  8 +++ mm/memremap.c                          | 68 +++++++------------------- mm/migrate.c                           |  5 -- mm/page_alloc.c                        |  3 ++
  mm/swap.c                              | 45 ++---------------

Quoted 2 lines
  12 files changed, 45 insertions(+), 147 deletions(-)

I think it is great that you are picking this up and trying to revive it.

However, I have a number of concerns about how it affects existing ZONE_DEVICE
MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this addressing them. For example, dev_dax_probe() allocates MEMORY_DEVICE_GENERIC
struct pages and then:
  dev_dax_fault()
    dev_dax_huge_fault()
      __dev_dax_pte_fault()
        vmf_insert_mixed()
which just inserts the PFN into the CPU page tables without increasing

Quoted 8 lines

the page
refcount so it is zero (whereas it was one before). But using get_page() will
trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current notion of
free verses allocated for these struct pages. I suppose init_page_count() could be called on all the struct pages in dev_dax_probe() to fix that though.

Hi Ralph,

For DEVICE_GENERIC pages free_zone_device_page doesn't do anything. So I'm not sure what the reference counting is good for in this case.

Alex is going to add free_zone_device_page support for DEVICE_GENERIC pages (patch 8 of this series). However, even then, it only does anything if there is an actual call to put_page. Where would that call come from in the dev_dax driver case?

Quoted 15 lines

I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File systems have clear
allocate and free states for backing storage but there are the complications with
the page cache references, etc. to consider. The >1 to 1 reference count seems to
be used to tell when a page is idle (no I/O, reclaim scanners) rather than free
(not allocated to any file) but I'm not 100% sure about that since I don't really
understand all the issues around why a file system needs to have a DAX mount option
besides knowing that the storage block size has to be a multiple of the page size.

The only thing that happens in free_zone_device_page for FS_DAX pages is wake_up_var(&page->_refcount). I guess, whoever is waiting for this wake-up will need to be prepared to see a refcount 0 instead of 1 now. I see these callers that would need to be updated:

./fs/ext4/inode.c:        error = ___wait_var_event(&page->_refcount, ./fs/ext4/inode.c-                atomic_read(&page->_refcount) == 1, ./fs/ext4/inode.c-                TASK_INTERRUPTIBLE, 0, 0, ./fs/ext4/inode.c-                ext4_wait_dax_page(ei)); --
./fs/fuse/dax.c:    return ___wait_var_event(&page->_refcount, ./fs/fuse/dax.c-            atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
./fs/fuse/dax.c-            0, 0, fuse_wait_dax_page(inode)); --
./fs/xfs/xfs_file.c:    return ___wait_var_event(&page->_refcount, ./fs/xfs/xfs_file.c-            atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
./fs/xfs/xfs_file.c-            0, 0, xfs_wait_dax_page(inode));

Regarding your page-cache comment, doesn't DAX mean that those file pages are not in the page cache?

Regards,
  Felix

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#9 lore raw

Re: [PATCH v3 2/8] mm: remove extra ZONE_DEVICE struct page refcount

Quoted 56 lines · Felix Kuehling

Am 2021-06-17 um 3:16 p.m. schrieb Ralph Campbell:

Quoted 45 lines · Alex Sierra
Quoted 16 lines

From: Ralph Campbell <[email protected]>

ZONE_DEVICE struct pages have an extra reference count that complicates the
code for put_page() and several places in the kernel that need to check the
reference count to see that a page is not being used (gup, compaction, migration, etc.). Clean up the code so the reference count doesn't need to
be treated specially for ZONE_DEVICE.

v2:
AS: merged this patch in linux 5.11 version

Quoted 15 lines
---
   arch/powerpc/kvm/book3s_hv_uvmem.c     |  2 +-
   drivers/gpu/drm/nouveau/nouveau_dmem.c |  2 +-
   fs/dax.c                               |  4 +-
   include/linux/dax.h                    |  2 +-
   include/linux/memremap.h               |  7 +--
   include/linux/mm.h                     | 44 -----------------
   lib/test_hmm.c                         |  2 +-
   mm/internal.h                          |  8 +++
   mm/memremap.c                          | 68 +++++++-------------------
   mm/migrate.c                           |  5 --
   mm/page_alloc.c                        |  3 ++
   mm/swap.c                              | 45 ++---------------
   12 files changed, 45 insertions(+), 147 deletions(-)

I think it is great that you are picking this up and trying to revive it.

However, I have a number of concerns about how it affects existing ZONE_DEVICE
MEMORY_DEVICE_GENERIC and MEMORY_DEVICE_FS_DAX users and I don't see this addressing them. For example, dev_dax_probe() allocates MEMORY_DEVICE_GENERIC
struct pages and then:
  dev_dax_fault()
    dev_dax_huge_fault()
      __dev_dax_pte_fault()
        vmf_insert_mixed()
which just inserts the PFN into the CPU page tables without increasing

Quoted 8 lines

the page
refcount so it is zero (whereas it was one before). But using get_page() will
trigger VM_BUG_ON_PAGE() if it is enabled. There isn't any current notion of
free verses allocated for these struct pages. I suppose init_page_count() could be called on all the struct pages in dev_dax_probe() to fix that though.

Hi Ralph,

Quoted 7 lines

For DEVICE_GENERIC pages free_zone_device_page doesn't do anything. So I'm not sure what the reference counting is good for in this case.

Alex is going to add free_zone_device_page support for DEVICE_GENERIC pages (patch 8 of this series). However, even then, it only does anything if there is an actual call to put_page. Where would that call come from in the dev_dax driver case?

Correct, the drivers/dax/device.c driver allocates MEMORY_DEVICE_GENERIC struct pages and doesn't seem to allocate/free the page nor increment/decrement the reference count but it does call vmf_insert_mixed() if the /dev/file is mmap()'ed into a user process' address space. If devm_memremap_pages() returns the array of ZONE_DEVICE struct pages initialized with a reference count of zero, then the CPU page tables will have a PTE/PFN that points to a struct page with a zero reference count. This goes against the normal expectation in the rest of the mm code that assumes a page mapped by a CPU has a non-zero reference count.
So yes, nothing "bad" happens because put_page() isn't called but the reference count will differ from other drivers that call vmf_insert_mixed() or vm_insert_page() where the page was allocated with alloc_pages() or similar.

Quoted 34 lines
Quoted 6 lines

I'm even less clear about how to fix MEMORY_DEVICE_FS_DAX. File systems have clear
allocate and free states for backing storage but there are the complications with
the page cache references, etc. to consider. The >1 to 1 reference count seems to

Quoted 8 lines

be used to tell when a page is idle (no I/O, reclaim scanners) rather than free
(not allocated to any file) but I'm not 100% sure about that since I don't really
understand all the issues around why a file system needs to have a DAX mount option
besides knowing that the storage block size has to be a multiple of the page size.

The only thing that happens in free_zone_device_page for FS_DAX pages is wake_up_var(&page->_refcount). I guess, whoever is waiting for this wake-up will need to be prepared to see a refcount 0 instead of 1 now. I see these callers that would need to be updated:

./fs/ext4/inode.c:        error = ___wait_var_event(&page->_refcount, ./fs/ext4/inode.c-                atomic_read(&page->_refcount) == 1, ./fs/ext4/inode.c-                TASK_INTERRUPTIBLE, 0, 0, ./fs/ext4/inode.c-                ext4_wait_dax_page(ei)); --
./fs/fuse/dax.c:    return ___wait_var_event(&page->_refcount, ./fs/fuse/dax.c-            atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
./fs/fuse/dax.c-            0, 0, fuse_wait_dax_page(inode)); --
./fs/xfs/xfs_file.c:    return ___wait_var_event(&page->_refcount, ./fs/xfs/xfs_file.c-            atomic_read(&page->_refcount) == 1, TASK_INTERRUPTIBLE,
./fs/xfs/xfs_file.c-            0, 0, xfs_wait_dax_page(inode));

Quoted 6 lines

Regarding your page-cache comment, doesn't DAX mean that those file pages are not in the page cache?

Regards,
  Felix

I don't really understand the FS_DAX code. I can see the __wait_var_event() is being used when truncating or punching holes in files but I'm not quite sure if it is using the >1 to 1 reference count to know when a page has no "extra" references or if it means the page is actually "free" and no longer assigned to a file.
I really think some FS_DAX expert needs to weigh in on these reference count changes.

#10 lore raw

PATCH v3 3/8 kernel: resource: lookup_resource as exported symbol

In reply to Alex Sierra

The AMD architecture for the Frontier supercomputer will have device memory which can be coherently accessed by the CPU. The system BIOS advertises this memory as SPM (special purpose memory) in the UEFI system address map.

The AMDGPU driver needs to be able to lookup this resource in order to claim it as MEMORY_DEVICE_GENERIC using devm_memremap_pages.

---
 kernel/resource.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/resource.c +1 -1
index 627e61b0c124..269489bb7097 100644
@@ -783,7 +783,7 @@ struct resource *lookup_resource(struct resource *root, resource_size_t start)
 
 	return res;
 }
-
+EXPORT_SYMBOL_GPL(lookup_resource);
 /*
  * Insert a resource into the resource tree. If successful, return NULL,
  * otherwise return the conflicting resource (compare to __request_resource())
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#11 lore raw

PATCH v3 4/8 drm/amdkfd: add SPM support for SVM

In reply to Alex Sierra

When CPU is connected throug XGMI, it has coherent access to VRAM resource. In this case that resource is taken from a table in the device gmc aperture base. This resource is used along with the device type, which could be DEVICE_PRIVATE or DEVICE_GENERIC to create the device page map region.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +9 -3
index c8ca3252cbc2..f5939449a99f 100644
@@ -895,6 +895,7 @@ int svm_migrate_init(struct amdgpu_device *adev)
 	struct resource *res;
 	unsigned long size;
 	void *r;
+	bool xgmi_connected_to_cpu = adev->gmc.xgmi.connected_to_cpu;
 
 	/* Page migration works on Vega10 or newer */
 	if (kfddev->device_info->asic_family < CHIP_VEGA10)
@@ -907,17 +908,22 @@ int svm_migrate_init(struct amdgpu_device *adev)
 	 * should remove reserved size
 	 */
 	size = ALIGN(adev->gmc.real_vram_size, 2ULL << 20);
-	res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
+	if (xgmi_connected_to_cpu)
+		res = lookup_resource(&iomem_resource, adev->gmc.aper_base);
+	else
+		res = devm_request_free_mem_region(adev->dev, &iomem_resource, size);
+
 	if (IS_ERR(res))
 		return -ENOMEM;
 
-	pgmap->type = MEMORY_DEVICE_PRIVATE;
 	pgmap->nr_range = 1;
 	pgmap->range.start = res->start;
 	pgmap->range.end = res->end;
+	pgmap->type = xgmi_connected_to_cpu ?
+				MEMORY_DEVICE_GENERIC : MEMORY_DEVICE_PRIVATE;
 	pgmap->ops = &svm_migrate_pgmap_ops;
 	pgmap->owner = SVM_ADEV_PGMAP_OWNER(adev);
-	pgmap->flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE;
+	pgmap->flags = 0;
 	r = devm_memremap_pages(adev->dev, pgmap);
 	if (IS_ERR(r)) {
 		pr_err("failed to register HMM device memory\n");
Signature · 1 line
2.17.1
#12 lore raw

PATCH v3 5/8 drm/amdkfd: generic type as sys mem on migration to ram

In reply to Alex Sierra

Generic device type memory on VRAM to RAM migration, has similar access as System RAM from the CPU. This flag sets the source from the sender. Which in Generic type case, should be set as SYSTEM.

---
 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +2 -1
index f5939449a99f..7b41006c1164 100644
@@ -653,8 +653,9 @@ svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
 	migrate.vma = vma;
 	migrate.start = start;
 	migrate.end = end;
-	migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE;
 	migrate.pgmap_owner = SVM_ADEV_PGMAP_OWNER(adev);
+	migrate.flags = adev->gmc.xgmi.connected_to_cpu ?
+			MIGRATE_VMA_SELECT_SYSTEM : MIGRATE_VMA_SELECT_DEVICE_PRIVATE;
 
 	size = 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t);
 	size *= npages;
Signature · 1 line
2.17.1
#13 lore raw

PATCH v3 6/8 include/linux/mm.h: helpers to check zone device generic type

In reply to Alex Sierra

Two helpers added. One checks if zone device page is generic type. The other if page is either private or generic type.

---
 include/linux/mm.h | 8 ++++++++
 1 file changed, 8 insertions(+)
include/linux/mm.h +8 -0
index d8d79bb94be8..f5b247a63044 100644
@@ -1125,6 +1125,14 @@ static inline bool is_device_private_page(const struct page *page)
 		page->pgmap->type == MEMORY_DEVICE_PRIVATE;
 }
 
+static inline bool is_device_page(const struct page *page)
+{
+	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
+		is_zone_device_page(page) &&
+		(page->pgmap->type == MEMORY_DEVICE_PRIVATE ||
+		 page->pgmap->type == MEMORY_DEVICE_GENERIC);
+}
+
 static inline bool is_pci_p2pdma_page(const struct page *page)
 {
 	return IS_ENABLED(CONFIG_DEV_PAGEMAP_OPS) &&
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#14 lore raw

PATCH v3 7/8 mm: add generic type support to migrate_vma helpers

In reply to Alex Sierra

Device generic type case added for migrate_vma_pages and migrate_vma_check_page helpers.
Both, generic and private device types have the same conditions to decide to migrate pages from/to device memory.

---
 mm/migrate.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/migrate.c +4 -4
index 8c2430d3e77b..3b6aaba96fe6 100644
@@ -2602,7 +2602,7 @@ static bool migrate_vma_check_page(struct page *page)
 		 * FIXME proper solution is to rework migration_entry_wait() so
 		 * it does not need to take a reference on page.
 		 */
-		return is_device_private_page(page);
+		return is_device_page(page);
 	}
 
 	/* For file back page */
@@ -3064,10 +3064,10 @@ void migrate_vma_pages(struct migrate_vma *migrate)
 		mapping = page_mapping(page);
 
 		if (is_zone_device_page(newpage)) {
-			if (is_device_private_page(newpage)) {
+			if (is_device_page(newpage)) {
 				/*
-				 * For now only support private anonymous when
-				 * migrating to un-addressable device memory.
+				 * For now only support private and generic
+				 * anonymous when migrating to device memory.
 				 */
 				if (mapping) {
 					migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#15 lore raw

PATCH v3 8/8 mm: call pgmap->ops->page_free for DEVICE_GENERIC pages

In reply to Alex Sierra

Add MEMORY_DEVICE_GENERIC case to free_zone_device_page callback.
Device generic type memory case is now able to free its pages properly.

---
 mm/memremap.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/memremap.c +3 -2
index 614b3d600e95..6c884e2542a9 100644
@@ -438,7 +438,7 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn,
 EXPORT_SYMBOL_GPL(get_dev_pagemap);
 
 #ifdef CONFIG_DEV_PAGEMAP_OPS
-static void free_device_private_page(struct page *page)
+static void free_device_page(struct page *page)
 {
 
 	__ClearPageWaiters(page);
@@ -477,7 +477,8 @@ void free_zone_device_page(struct page *page)
 		wake_up_var(&page->_refcount);
 		return;
 	case MEMORY_DEVICE_PRIVATE:
-		free_device_private_page(page);
+	case MEMORY_DEVICE_GENERIC:
+		free_device_page(page);
 		return;
 	default:
 		return;
Signature · 1 line
2.17.1
#16 lore raw
In reply to Alex Sierra
Quoted 22 lines · Alex Sierra

v1:
AMD is building a system architecture for the Frontier supercomputer with a coherent interconnect between CPUs and GPUs. This hardware architecture allows the CPUs to coherently access GPU device memory. We have hardware in our labs and we are working with our partner HPE on the BIOS, firmware and software for delivery to the DOE.

The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver looks it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC using devm_memremap_pages.

Now we're trying to migrate data to and from that memory using the migrate_vma_* helpers so we can support page-based migration in our unified memory allocations, while also supporting CPU access to those pages.

This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave correctly in the migrate_vma_* helpers. We are looking for feedback about this approach. If we're close, what's needed to make our patches acceptable upstream? If we're not close, any suggestions how else to achieve what we are trying to do (i.e. page migration and coherent CPU access to VRAM)?

Quoted 3 lines

This work is based on HMM and our SVM memory manager that was recently upstreamed to Dave Airlie's drm-next branch
https://lore.kernel.org/dri-devel/[email protected]/T/#r996356015e295780eb50453e7dbd5d0d68b47cbc

Corrected link:

https://cgit.freedesktop.org/drm/drm/log/?h=drm-next

Regards,
Alex Sierra

Quoted 4 lines

On top of that we did some rework of our VRAM management for migrations to remove some incorrect assumptions, allow partially successful migrations and GPU memory mappings that mix pages in VRAM and system memory.
https://patchwork.kernel.org/project/dri-devel/list/?series=489811

Corrected link:

https://lore.kernel.org/dri-devel/[email protected]/T/#r996356015e295780eb50453e7dbd5d0d68b47cbc

Regards,
Alex Sierra

Quoted 47 lines

v2:
This patch series version has merged "[RFC PATCH v3 0/2]
mm: remove extra ZONE_DEVICE struct page refcount" patch series made by Ralph Campbell. It also applies at the top of these series, our changes to support device generic type in migration_vma helpers. This has been tested in systems with device memory that has coherent access by CPU.

Also addresses the following feedback made in v1:
- Isolate in one patch kernel/resource.c modification, based on Christoph's feedback.
- Add helpers check for generic and private type to avoid duplicated long lines.

v3:
- Include cover letter from v1
- Rename dax_layout_is_idle_page func to dax_page_unused in patch ext4/xfs: add page refcount helper

Patches 1-2 Rebased Ralph Campbell's ZONE_DEVICE page refcounting patches Patches 4-5 are for context to show how we are looking up the SPM memory and registering it with devmap.
Patches 3,6-8 are the changes we are trying to upstream or rework to make them acceptable upstream.

Alex Sierra (6):
kernel: resource: lookup_resource as exported symbol drm/amdkfd: add SPM support for SVM
drm/amdkfd: generic type as sys mem on migration to ram include/linux/mm.h: helpers to check zone device generic type
mm: add generic type support to migrate_vma helpers
mm: call pgmap->ops->page_free for DEVICE_GENERIC pages

Ralph Campbell (2):
ext4/xfs: add page refcount helper
mm: remove extra ZONE_DEVICE struct page refcount

  arch/powerpc/kvm/book3s_hv_uvmem.c       |  2 +-
  drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 ++++--
  drivers/gpu/drm/nouveau/nouveau_dmem.c   |  2 +-
  fs/dax.c                                 |  8 +--
  fs/ext4/inode.c                          |  5 +-
  fs/xfs/xfs_file.c                        |  4 +-
  include/linux/dax.h                      | 10 ++++
  include/linux/memremap.h                 |  7 +--
  include/linux/mm.h                       | 52 +++---------------
Quoted 9 lines
  kernel/resource.c                        |  2 +-
  lib/test_hmm.c                           |  2 +-
  mm/internal.h                            |  8 +++
  mm/memremap.c                            | 69 +++++++-----------------
  mm/migrate.c                             | 13 ++---
  mm/page_alloc.c                          |  3 ++
  mm/swap.c                                | 45 ++--------------
  16 files changed, 83 insertions(+), 164 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

maintainer
#17 lore raw
In reply to Alex Sierra
Quoted 16 lines · Alex Sierra

v1:
AMD is building a system architecture for the Frontier supercomputer with a coherent interconnect between CPUs and GPUs. This hardware architecture allows the CPUs to coherently access GPU device memory. We have hardware in our labs and we are working with our partner HPE on the BIOS, firmware and software for delivery to the DOE.

The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver looks it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC using devm_memremap_pages.

Now we're trying to migrate data to and from that memory using the migrate_vma_* helpers so we can support page-based migration in our unified memory allocations, while also supporting CPU access to those pages.

Quoted 5 lines

This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave correctly in the migrate_vma_* helpers. We are looking for feedback about this approach. If we're close, what's needed to make our patches acceptable upstream? If we're not close, any suggestions how else to achieve what we are trying to do (i.e. page migration and coherent CPU access to VRAM)?

Is there a way we can test the codepaths touched by this patchset? It doesn't have to be via a complete qemu simulation of the GPU device memory, but some way of creating MEMORY_DEVICE_GENERIC subject to migrate_vma_* helpers so we can test for regressions moving forward.

Thanks,

- Ted

maintainer
#18 lore raw
Quoted 22 lines · Theodore Ts'o
Quoted 16 lines · Alex Sierra

v1:
AMD is building a system architecture for the Frontier supercomputer with a coherent interconnect between CPUs and GPUs. This hardware architecture allows the CPUs to coherently access GPU device memory. We have hardware in our labs and we are working with our partner HPE on the BIOS, firmware and software for delivery to the DOE.

The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver looks it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC using devm_memremap_pages.

Now we're trying to migrate data to and from that memory using the migrate_vma_* helpers so we can support page-based migration in our unified memory allocations, while also supporting CPU access to those pages.

Quoted 5 lines

This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave correctly in the migrate_vma_* helpers. We are looking for feedback about this approach. If we're close, what's needed to make our patches acceptable upstream? If we're not close, any suggestions how else to achieve what we are trying to do (i.e. page migration and coherent CPU access to VRAM)?

Quoted 4 lines

Is there a way we can test the codepaths touched by this patchset? It doesn't have to be via a complete qemu simulation of the GPU device memory, but some way of creating MEMORY_DEVICE_GENERIC subject to migrate_vma_* helpers so we can test for regressions moving forward.

Hi Theodore,

I can think of two ways to test the changes for MEMORY_DEVICE_GENERIC in this patch series in a way that is reproducible without special hardware and firmware:

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

For the migration helper changes we could modify or parametrize lib/hmm_test.c to create DEVICE_GENERIC pages instead of DEVICE_PRIVATE. Then run tools/testing/selftests/vm/hmm-tests.c.

Regards,
  Felix

Quoted 4 lines

Thanks,

- Ted

#19 lore raw
Quoted 8 lines · Felix Kuehling

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

For the migration helper changes we could modify or parametrize lib/hmm_test.c to create DEVICE_GENERIC pages instead of DEVICE_PRIVATE. Then run tools/testing/selftests/vm/hmm-tests.c.

We'll also need a real in-tree user of the enhanced DEVICE_GENERIC memory. So while the refcounting cleanups early in the series are something I'd really like to see upstream as soon as everything is sorted out, the actual bits that can't only be used by your updated driver should wait for that.

maintainer
#20 lore raw

Am 2021-06-24 um 1:30 a.m. schrieb Christoph Hellwig:

Quoted 6 lines · Felix Kuehling
Quoted 5 lines

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

Quoted 8 lines
Quoted 3 lines

For the migration helper changes we could modify or parametrize lib/hmm_test.c to create DEVICE_GENERIC pages instead of DEVICE_PRIVATE. Then run tools/testing/selftests/vm/hmm-tests.c.

We'll also need a real in-tree user of the enhanced DEVICE_GENERIC memory. So while the refcounting cleanups early in the series are something I'd really like to see upstream as soon as everything is sorted out, the actual bits that can't only be used by your updated driver should wait for that.

The driver changes are pretty much ready to go.

But we have a bit of a chicken-egg problem because those changes likely go through different trees. The GPU driver changes will go through drm-next, but we can't merge them there until our dependencies have been merged there from upstream. Unless we protect everything with some #ifdef.

Regards,
  Felix

maintainer
#21 lore raw
In reply to Felix Kuehling
Quoted 9 lines · Felix Kuehling

I can think of two ways to test the changes for MEMORY_DEVICE_GENERIC in this patch series in a way that is reproducible without special hardware and firmware:

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

Sorry for the thread necromancy, but now that the merge window is past....

Today I test ext4's dax support, without having any $$$ DAX hardware, by using the kernel command line "memmap=4G!9G:memmap=9G!14G" which reserves memory so that creates two pmem device and then I run xfstests with DAX enabled using qemu or using a Google Compute Engine VM, using TEST_DEV=/dev/pmem0 and SCRATCH_DEV=/dev/pmem1.

If you can give me a recipe for what kernel configs I should enable, and what magic kernel command line arguments to use, then I'd be able to test your patch set with ext4.

Cheers,

- Ted

maintainer
#22 lore raw

Am 2021-07-16 um 11:07 a.m. schrieb Theodore Y. Ts'o:

Quoted 11 lines · Felix Kuehling
Quoted 8 lines

I can think of two ways to test the changes for MEMORY_DEVICE_GENERIC in this patch series in a way that is reproducible without special hardware and firmware:

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

Sorry for the thread necromancy, but now that the merge window is past....

No worries. Alejandro should have a new version of this series in a few days, with updates to hmm_test and some fixes.

Quoted 10 lines

Today I test ext4's dax support, without having any $$$ DAX hardware, by using the kernel command line "memmap=4G!9G:memmap=9G!14G" which reserves memory so that creates two pmem device and then I run xfstests with DAX enabled using qemu or using a Google Compute Engine VM, using TEST_DEV=/dev/pmem0 and SCRATCH_DEV=/dev/pmem1.

If you can give me a recipe for what kernel configs I should enable, and what magic kernel command line arguments to use, then I'd be able to test your patch set with ext4.

That would be great!

Regarding kernel config options, it should be the same as what you're using for DAX testing today. We're not changing or adding any Kconfig options. But let me take a stab:

ZONE_DEVICE
HMM_MIRROR
MMU_NOTIFIER
DEVICE_PRIVATE (maybe not needed for your test)
FS_DAX

I'm not sure what you're looking for in terms of kernel command line, other than the memmap options you already found. There are some more options to run hmm_test with fake SPM (DEVICE_GENERIC) memory, but we're already running that ourselves. That will also be in the next revision of this patch series.

If you can run your xfstests with DAX on top of this patch series, that would be very helpful. That's to make sure the ZONE_DEVICE page refcount changes don't break DAX.

Regards,
  Felix

Quoted 4 lines

Cheers,

- Ted

#23 lore raw
Quoted 6 lines · Felix Kuehling

Am 2021-07-16 um 11:07 a.m. schrieb Theodore Y. Ts'o:

Quoted 5 lines · Felix Kuehling
Quoted 4 lines

I can think of two ways to test the changes for MEMORY_DEVICE_GENERIC in this patch series in a way that is reproducible without special hardware and firmware:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

Sorry for the thread necromancy, but now that the merge window is past....

No worries. Alejandro should have a new version of this series in a few days, with updates to hmm_test and some fixes.

V4 patch series have been sent for review.
https://marc.info/?l=dri-devel&m=162654971618911&w=2

Regards,
Alex Sierra

Quoted 23 lines
Quoted 11 lines

Today I test ext4's dax support, without having any $$$ DAX hardware, by using the kernel command line "memmap=4G!9G:memmap=9G!14G" which reserves memory so that creates two pmem device and then I run xfstests with DAX enabled using qemu or using a Google Compute Engine VM, using TEST_DEV=/dev/pmem0 and SCRATCH_DEV=/dev/pmem1.

If you can give me a recipe for what kernel configs I should enable, and what magic kernel command line arguments to use, then I'd be able to test your patch set with ext4.

That would be great!

Regarding kernel config options, it should be the same as what you're using for DAX testing today. We're not changing or adding any Kconfig options. But let me take a stab:

ZONE_DEVICE
HMM_MIRROR
MMU_NOTIFIER
DEVICE_PRIVATE (maybe not needed for your test)
FS_DAX

Quoted 5 lines

I'm not sure what you're looking for in terms of kernel command line, other than the memmap options you already found. There are some more options to run hmm_test with fake SPM (DEVICE_GENERIC) memory, but we're already running that ourselves. That will also be in the next revision of this patch series.

In order to run hmm test with generic device type enabled, set the following:

kernel config:
EFI_FAKE_MEMMAP
RUNTIME_TESTING_MENU
TEST_HMM=m

Kernel parameters to fake SP memory. The addresses set here are based on my system's usable memory enumerated by BIOS-e820 at boot. The test requires two SP devices of at least 256MB. efi_fake_mem=1G@0x100000000:0x40000,1G@0x140000000:0x40000

To run the hmm_test in generic device type pass the SP addresses to the sh script. sudo ./test_hmm.sh smoke 0x100000000 0x140000000

Quoted 12 lines

If you can run your xfstests with DAX on top of this patch series, that would be very helpful. That's to make sure the ZONE_DEVICE page refcount changes don't break DAX.

Regards,
  Felix

Quoted 3 lines

Cheers,

- Ted

#24 lore raw
Quoted 14 lines · Alejandro (Alex)
Quoted 12 lines · Felix Kuehling

Am 2021-07-16 um 11:07 a.m. schrieb Theodore Y. Ts'o:

Quoted 11 lines · Felix Kuehling
Quoted 8 lines

I can think of two ways to test the changes for MEMORY_DEVICE_GENERIC in this patch series in a way that is reproducible without special hardware and firmware:

For the reference counting changes we could use the dax driver with hmem and use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC pages. I'm open to suggestions for good user mode tests to exercise dax functionality on this type of memory.

Sorry for the thread necromancy, but now that the merge window is past....

Quoted 31 lines
Quoted 2 lines

No worries. Alejandro should have a new version of this series in a few days, with updates to hmm_test and some fixes.

V4 patch series have been sent for review.
https://marc.info/?l=dri-devel&m=162654971618911&w=2

Regards,
Alex Sierra

Quoted 22 lines
Quoted 11 lines

Today I test ext4's dax support, without having any $$$ DAX hardware, by using the kernel command line "memmap=4G!9G:memmap=9G!14G" which reserves memory so that creates two pmem device and then I run xfstests with DAX enabled using qemu or using a Google Compute Engine VM, using TEST_DEV=/dev/pmem0 and SCRATCH_DEV=/dev/pmem1.

If you can give me a recipe for what kernel configs I should enable, and what magic kernel command line arguments to use, then I'd be able to test your patch set with ext4.

That would be great!

Regarding kernel config options, it should be the same as what you're using for DAX testing today. We're not changing or adding any Kconfig options. But let me take a stab:

ZONE_DEVICE
HMM_MIRROR
MMU_NOTIFIER
DEVICE_PRIVATE (maybe not needed for your test)
FS_DAX

Hi Theodore,
I wonder if you had chance to set the kernel configs from Felix to enable DAX in xfstests.

I've been trying to test FS DAX on my side using virtio-fs + QEMU. Unfortunately, Im having some problems setting up the environment with the guest kernel. Could you share your VM (QEMU or GCE) setup to run it with xfstests?

Regards,
Alex S.

Quoted 32 lines
Quoted 6 lines

I'm not sure what you're looking for in terms of kernel command line, other than the memmap options you already found. There are some more options to run hmm_test with fake SPM (DEVICE_GENERIC) memory, but we're already running that ourselves. That will also be in the next revision of this patch series.

In order to run hmm test with generic device type enabled, set the following:

kernel config:
EFI_FAKE_MEMMAP
RUNTIME_TESTING_MENU
TEST_HMM=m

Kernel parameters to fake SP memory. The addresses set here are based on my system's usable memory enumerated by BIOS-e820 at boot. The test requires two SP devices of at least 256MB. efi_fake_mem=1G@0x100000000:0x40000,1G@0x140000000:0x40000

To run the hmm_test in generic device type pass the SP addresses to the sh script. sudo ./test_hmm.sh smoke 0x100000000 0x140000000

Quoted 12 lines

If you can run your xfstests with DAX on top of this patch series, that would be very helpful. That's to make sure the ZONE_DEVICE page refcount changes don't break DAX.

Regards,
   Felix

Quoted 3 lines

Cheers,

                        - Ted

maintainer
#25 lore raw

Am 2021-07-23 um 6:46 p.m. schrieb Sierra Guiza, Alejandro (Alex):

Quoted 56 lines · Alejandro (Alex)
Quoted 19 lines · Felix Kuehling
Quoted 17 lines

Am 2021-07-16 um 11:07 a.m. schrieb Theodore Y. Ts'o:

Quoted 16 lines · Felix Kuehling
Quoted 6 lines

I can think of two ways to test the changes for MEMORY_DEVICE_GENERIC in
this patch series in a way that is reproducible without special hardware and
firmware:

Quoted 7 lines

For the reference counting changes we could use the dax driver with hmem and
use efi_fake_mem on the kernel command line to create some DEVICE_GENERIC
pages. I'm open to suggestions for good user mode tests to exercise dax
functionality on this type of memory.

Sorry for the thread necromancy, but now that the merge window is past....

Quoted 31 lines
Quoted 2 lines

No worries. Alejandro should have a new version of this series in a few days, with updates to hmm_test and some fixes.

V4 patch series have been sent for review.
https://marc.info/?l=dri-devel&m=162654971618911&w=2

Regards,
Alex Sierra

Quoted 22 lines
Quoted 11 lines

Today I test ext4's dax support, without having any $$$ DAX hardware, by using the kernel command line "memmap=4G!9G:memmap=9G!14G" which reserves memory so that creates two pmem device and then I run xfstests with DAX enabled using qemu or using a Google Compute Engine VM, using TEST_DEV=/dev/pmem0 and SCRATCH_DEV=/dev/pmem1.

If you can give me a recipe for what kernel configs I should enable, and what magic kernel command line arguments to use, then I'd be able to test your patch set with ext4.

That would be great!

Regarding kernel config options, it should be the same as what you're using for DAX testing today. We're not changing or adding any Kconfig options. But let me take a stab:

ZONE_DEVICE
HMM_MIRROR
MMU_NOTIFIER
DEVICE_PRIVATE (maybe not needed for your test)
FS_DAX

Hi Theodore,
I wonder if you had chance to set the kernel configs from Felix to enable DAX in xfstests.

Quoted 7 lines

I've been trying to test FS DAX on my side using virtio-fs + QEMU. Unfortunately, Im having some problems setting up the environment with the guest kernel. Could you share your VM (QEMU or GCE) setup to run it with xfstests?

Regards,
Alex S.

Hi Theodore,

Sorry to keep bugging you. I'm wondering if you've had a chance to test FS DAX with Alex's last patch series? ([PATCH v4 00/13] Support DEVICE_GENERIC memory in migrate_vma_*) I think other than minor cosmetic fixes, this should be ready to merge, if it passes your tests.

Thanks,
  Felix

Quoted 39 lines
Quoted 39 lines
Quoted 8 lines

I'm not sure what you're looking for in terms of kernel command line, other than the memmap options you already found. There are some more options to run hmm_test with fake SPM (DEVICE_GENERIC) memory, but we're
already running that ourselves. That will also be in the next revision of this patch series.

In order to run hmm test with generic device type enabled, set the following:

kernel config:
EFI_FAKE_MEMMAP
RUNTIME_TESTING_MENU
TEST_HMM=m

Kernel parameters to fake SP memory. The addresses set here are based on my system's usable memory enumerated by BIOS-e820 at boot. The test requires two SP devices of at least 256MB. efi_fake_mem=1G@0x100000000:0x40000,1G@0x140000000:0x40000

To run the hmm_test in generic device type pass the SP addresses to the sh script.
sudo ./test_hmm.sh smoke 0x100000000 0x140000000

Quoted 13 lines

If you can run your xfstests with DAX on top of this patch series, that would be very helpful. That's to make sure the ZONE_DEVICE page refcount
changes don't break DAX.

Regards,
   Felix

Quoted 3 lines

Cheers,

                        - Ted