Skip to contentSkip to conversation

Migrate on fault for device pages

PATCH Superseded [email protected] 6 participants 16 messages Feb 11 – Mar 23

Subsystems (7)

v5 of 17 Superseded Latest is v17 →

lore
Share & tools
lore
#0 lore raw

v5 cover Migrate on fault for device pages

From: Mika Penttilä <[email protected]>

Currently, the way device page faulting and migration works is not optimal, if you want to do both fault handling and migration at once.

Being able to migrate not present pages (or pages mapped with incorrect permissions, eg. COW) to the GPU requires doing either of the following sequences:

1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages

Or:

1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
a) call hmm_range_fault() to fault pages in
b) call migrate_vma_*() again to migrate now present pages

The problem with the first sequence is that you always have to do two page walks even when most of the time the pages are present or zero page mappings so the common case takes a performance hit.

The second sequence is better for the common case, but far worse if pages aren't present because now you have to walk the page tables three times (once to find the page is not present, once so hmm_range_fault() can find a non-present page to fault in and once again to setup the migration). It is also tricky to code correctly. One page table walk could costs over 1000 cpu cycles on X86-64, which is a significant hit.

We should be able to walk the page table once, faulting pages in as required and replacing them with migration entries if requested.

Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling. Also, for the migrate_vma_setup() call paths, a flag, MIGRATE_VMA_FAULT, is added to tell to add fault handling to migrate.

One extra benefit of migrating with hmm_range_fault() path is the migrate_vma.vma gets populated, so no need to retrieve that separataly.

Tested in X86-64 VM with HMM test device, passing the selftests. For performance, the migrate throughput tests from the selftests show similar numbers (within error margin) as unmodified kernel. Tested also rebased on the
"Remove device private pages from physical address space" series:
https://lore.kernel.org/linux-mm/[email protected]/
plus a small patch to adjust with no problems.

Changes v4-v5
- rebase on 6.19
- fixed David's email address
- fixed link issue without CONFIG_TRANSPARENT_HUGEPAGE
- refactored into smaller commits
- added more comments to code

Changes v3-v4:
- rebase on 6.19-rc8
- fixed issues found by kernel test robot with random configs
- fixed typos

Changes v2-v3:
- rebase on 6.19-rc7
- fixed issues found by kernel test robot
- fixed smatch issues reported by Dan Carpenter <[email protected]>
- fixes to lock handling (pmd/pte) on errors
- added assertions for pmd/pte lock states
- other issues discovered by Matthew, thanks!

Changes v1-v2:
- rebase on 6.19-rc6
- fixed issues found by kernel test robot
- fixed locking (pmd/ptl) to cover handle_ and prepare_ regions parts if migrating
- other issues discovered by Matthew, thanks!

Changes RFC-v1:
- rebase on 6.19-rc5
- adjust for the device THP
- changes from feedback

Revisions:
- RFC https://lore.kernel.org/linux-mm/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/
- v2: https://lore.kernel.org/all/[email protected]/
- v3: https://lore.kernel.org/all/[email protected]/
- v4: https://lore.kernel.org/all/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>

Mika Penttilä (6):
mm:/Kconfig changes for migrate on fault for device pages
mm: Add helper to convert HMM pfn to migrate pfn mm/hmm: do the plumbing for HMM to participate in migration
mm: implement device page migration for HMM pagewalk
mm: add new testcase for the migrate on fault case mm:/migrate_device.c: remove migrate_vma_collect_*()

 include/linux/hmm.h                    |  18 +-
 include/linux/migrate.h                |  27 +-
 lib/test_hmm.c                         | 100 ++-
 lib/test_hmm_uapi.h                    |  19 +-
 mm/Kconfig                             |   2 +
 mm/hmm.c                               | 814 +++++++++++++++++++++++--
 mm/migrate_device.c                    | 602 +++---------------
 tools/testing/selftests/mm/hmm-tests.c |  54 ++
 8 files changed, 1053 insertions(+), 583 deletions(-)

base-commit: 05f7e89ab9731565d8a62e3b5d1ec206485eeb0b

Signature · 1 line
2.50.0
#1 lore raw

PATCH v5 1/6 mm:/Kconfig changes for migrate on fault for device pages

From: Mika Penttilä <[email protected]>

With the unified HMM/migrate_device page table walk migrate_device needs HMM enabled and HMM needs MMU notifiers. Enable them explicitly to avoid breaking random configs.

Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc "Liam R. Howlett" <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Michal Hocko <[email protected]>
---
 mm/Kconfig | 2 ++
 1 file changed, 2 insertions(+)
mm/Kconfig +2 -0
index a992f2203eb9..1b8778f34922 100644
@@ -661,6 +661,7 @@ config MIGRATION
 
 config DEVICE_MIGRATION
 	def_bool MIGRATION && ZONE_DEVICE
+	select HMM_MIRROR
 
 config ARCH_ENABLE_HUGEPAGE_MIGRATION
 	bool
@@ -1236,6 +1237,7 @@ config ZONE_DEVICE
 config HMM_MIRROR
 	bool
 	depends on MMU
+	select MMU_NOTIFIER
 
 config GET_FREE_REGION
 	bool
Signature · 1 line
2.50.0
maintainer Acked-by on v5 1/6
#2 lore raw

Re: [PATCH v5 1/6] mm:/Kconfig changes for migrate on fault for device pages

Quoted 16 lines · On 2/11/26 09:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

With the unified HMM/migrate_device page table walk migrate_device needs HMM enabled and HMM needs MMU notifiers. Enable them explicitly to avoid breaking random configs.

Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc "Liam R. Howlett" <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Michal Hocko <[email protected]>
Quoted 1 line

---

Signature · 3 lines
Cheers,

David
#3 lore raw

PATCH v5 2/6 mm: Add helper to convert HMM pfn to migrate pfn

In reply to [email protected]

From: Mika Penttilä <[email protected]>

The unified HMM/migrate_device pagewalk does the "collecting" in HMM side, so need a helper to transfer pfns to migrate_vma world.

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 include/linux/hmm.h     | 18 +++++++++--
 include/linux/migrate.h |  3 +-
 mm/hmm.c                |  6 ----
 mm/migrate_device.c     | 69 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 87 insertions(+), 9 deletions(-)
include/linux/hmm.h +16 -2
index db75ffc949a7..b5418e318782 100644
@@ -12,7 +12,7 @@
 #include <linux/mm.h>
 
 struct mmu_interval_notifier;
-
+struct migrate_vma;
 /*
  * On output:
  * 0             - The page is faultable and a future call with 
@@ -27,6 +27,7 @@ struct mmu_interval_notifier;
  * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer
  * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
  *                      to mark that page is already DMA mapped
+ * HMM_PFN_MIGRATE    - Migrate PTE installed
  *
  * On input:
  * 0                 - Return the current state of the page, do not fault it.
@@ -34,6 +35,7 @@ struct mmu_interval_notifier;
  *                     will fail
  * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fault()
  *                     will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * HMM_PFN_REQ_MIGRATE - For default_flags, request to migrate to device
  */
 enum hmm_pfn_flags {
 	/* Output fields and flags */
@@ -48,15 +50,25 @@ enum hmm_pfn_flags {
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
 
-	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+	/* Migrate request */
+	HMM_PFN_MIGRATE    = 1UL << (BITS_PER_LONG - 7),
+	HMM_PFN_COMPOUND   = 1UL << (BITS_PER_LONG - 8),
+	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),
 
 	/* Input flags */
 	HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
 	HMM_PFN_REQ_WRITE = HMM_PFN_WRITE,
+	HMM_PFN_REQ_MIGRATE = HMM_PFN_MIGRATE,
 
 	HMM_PFN_FLAGS = ~((1UL << HMM_PFN_ORDER_SHIFT) - 1),
 };
 
+enum {
+	/* These flags are carried from input-to-output */
+	HMM_PFN_INOUT_FLAGS = HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA |
+		HMM_PFN_P2PDMA_BUS,
+};
+
 /*
  * hmm_pfn_to_page() - return struct page pointed to by a device entry
  *
@@ -107,6 +119,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
  * @default_flags: default flags for the range (write, read, ... see hmm doc)
  * @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter
  * @dev_private_owner: owner of device private pages
+ * @migrate: structure for migrating the associated vma
  */
 struct hmm_range {
 	struct mmu_interval_notifier *notifier;
@@ -117,6 +130,7 @@ struct hmm_range {
 	unsigned long		default_flags;
 	unsigned long		pfn_flags_mask;
 	void			*dev_private_owner;
+	struct migrate_vma      *migrate;
 };
 
 /*
include/linux/migrate.h +2 -1
index 26ca00c325d9..8e6c28efd4f8 100644
@@ -3,6 +3,7 @@
 #define _LINUX_MIGRATE_H
 
 #include <linux/mm.h>
+#include <linux/hmm.h>
 #include <linux/mempolicy.h>
 #include <linux/migrate_mode.h>
 #include <linux/hugetlb.h>
@@ -192,7 +193,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
 			unsigned long *dst_pfns, unsigned long npages);
-
+void migrate_hmm_range_setup(struct hmm_range *range);
 #endif /* CONFIG_MIGRATION */
 
 #endif /* _LINUX_MIGRATE_H */
mm/hmm.c +0 -6
index 4ec74c18bef6..21ff99379836 100644
@@ -41,12 +41,6 @@ enum {
 	HMM_NEED_ALL_BITS = HMM_NEED_FAULT | HMM_NEED_WRITE_FAULT,
 };
 
-enum {
-	/* These flags are carried from input-to-output */
-	HMM_PFN_INOUT_FLAGS = HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA |
-			      HMM_PFN_P2PDMA_BUS,
-};
-
 static int hmm_pfns_fill(unsigned long addr, unsigned long end,
 			 struct hmm_range *range, unsigned long cpu_flags)
 {
mm/migrate_device.c +69 -0

72 lines not on this page · raw message

Signature · 1 line
2.50.0
#4 lore raw

Re: [PATCH v5 2/6] mm: Add helper to convert HMM pfn to migrate pfn

Quoted 4 lines · On 2/11/26 09:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

The unified HMM/migrate_device pagewalk does the "collecting" in HMM side, so need a helper to transfer pfns to migrate_vma

s/in/on the/ ?

"so we" ?

"to the" ?

Quoted 23 lines

world.

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 include/linux/hmm.h     | 18 +++++++++--
 include/linux/migrate.h |  3 +-
 mm/hmm.c                |  6 ----
 mm/migrate_device.c     | 69 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 87 insertions(+), 9 deletions(-)
include/linux/hmm.h +0 -0
index db75ffc949a7..b5418e318782 100644

include/linux/hmm.h line 15

Quoted 5 lines
include/linux/hmm.h +1 -1
@@ -12,7 +12,7 @@
 #include <linux/mm.h>
 
 struct mmu_interval_notifier;
-
+struct migrate_vma;

Keep the empty line, please.

include/linux/hmm.h struct mmu_interval_notifier; · line 30

Quoted 8 lines
include/linux/hmm.h +1 -0
 /*
  * On output:
  * 0             - The page is faultable and a future call with 
@@ -27,6 +27,7 @@ struct mmu_interval_notifier;
  * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer
  * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
  *                      to mark that page is already DMA mapped
+ * HMM_PFN_MIGRATE    - Migrate PTE installed

I have no idea what exactly that means. Can we a bit clearer? Especially, what does it mean when this is set alongside HMM_PFN_VALID? What if it is not set alongside HMM_PFN_VALID.

Shouldn't we document what HMM_PFN_COMPOUND and HMM_PFN_MIGRATE mean as well somewhere?

Quoted 24 lines
include/linux/hmm.h +5 -1
  *
  * On input:
  * 0                 - Return the current state of the page, do not fault it.
@@ -34,6 +35,7 @@ struct mmu_interval_notifier;
  *                     will fail
  * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fault()
  *                     will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * HMM_PFN_REQ_MIGRATE - For default_flags, request to migrate to device
  */
 enum hmm_pfn_flags {
 	/* Output fields and flags */
@@ -48,15 +50,25 @@ enum hmm_pfn_flags {
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
 
-	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+	/* Migrate request */
+	HMM_PFN_MIGRATE    = 1UL << (BITS_PER_LONG - 7),
+	HMM_PFN_COMPOUND   = 1UL << (BITS_PER_LONG - 8),
+	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),
 
 	/* Input flags */
 	HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
 	HMM_PFN_REQ_WRITE = HMM_PFN_WRITE,

include/linux/hmm.h enum hmm_pfn_flags { · line 70

Quoted 10 lines
include/linux/hmm.h +6 -0
+	HMM_PFN_REQ_MIGRATE = HMM_PFN_MIGRATE,
 
 	HMM_PFN_FLAGS = ~((1UL << HMM_PFN_ORDER_SHIFT) - 1),
 };
 
+enum {
+	/* These flags are carried from input-to-output */
+	HMM_PFN_INOUT_FLAGS = HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA |
+		HMM_PFN_P2PDMA_BUS,
+};

Why is that an anonymous enum with a single value? Ah, you are moving that. This should be spelled out in the patch description (and why you are doing it).

Is there any reason we can't move that into hmm_pfn_flags?

include/linux/hmm.h static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn) · line 122

Quoted 9 lines
include/linux/hmm.h +2 -0
+
 /*
  * hmm_pfn_to_page() - return struct page pointed to by a device entry
  *
@@ -107,6 +119,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
  * @default_flags: default flags for the range (write, read, ... see hmm doc)
  * @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter
  * @dev_private_owner: owner of device private pages
+ * @migrate: structure for migrating the associated vma

Is it really for "migrating a vma"? I assume it's for migrating a range of a VMA.

Quoted 54 lines
include/linux/hmm.h +1 -0
  */
 struct hmm_range {
 	struct mmu_interval_notifier *notifier;
@@ -117,6 +130,7 @@ struct hmm_range {
 	unsigned long		default_flags;
 	unsigned long		pfn_flags_mask;
 	void			*dev_private_owner;
+	struct migrate_vma      *migrate;
 };
 
 /*
include/linux/migrate.h +2 -1
index 26ca00c325d9..8e6c28efd4f8 100644
@@ -3,6 +3,7 @@
 #define _LINUX_MIGRATE_H
 
 #include <linux/mm.h>
+#include <linux/hmm.h>
 #include <linux/mempolicy.h>
 #include <linux/migrate_mode.h>
 #include <linux/hugetlb.h>
@@ -192,7 +193,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
 			unsigned long *dst_pfns, unsigned long npages);
-
+void migrate_hmm_range_setup(struct hmm_range *range);
 #endif /* CONFIG_MIGRATION */
 
 #endif /* _LINUX_MIGRATE_H */
mm/hmm.c +0 -6
index 4ec74c18bef6..21ff99379836 100644
@@ -41,12 +41,6 @@ enum {
 	HMM_NEED_ALL_BITS = HMM_NEED_FAULT | HMM_NEED_WRITE_FAULT,
 };
 
-enum {
-	/* These flags are carried from input-to-output */
-	HMM_PFN_INOUT_FLAGS = HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA |
-			      HMM_PFN_P2PDMA_BUS,
-};
-
 static int hmm_pfns_fill(unsigned long addr, unsigned long end,
 			 struct hmm_range *range, unsigned long cpu_flags)
 {
mm/migrate_device.c +0 -0
index 23379663b1e1..c773a82ea1ed 100644

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1495

Quoted 7 lines
mm/migrate_device.c +4 -0
@@ -1489,3 +1489,72 @@ int migrate_device_coherent_folio(struct folio *folio)
 		return 0;
 	return -EBUSY;
 }
+
+/**
+ * migrate_hmm_range_setup() - prepare to migrate a range of memory
+ * @range: contains pointer to migrate_vma to be populated

"pointer to hmm_range to be migrated" ?

Why can't we just pass the migrate_vma? Or is there another use case for adding the migrate_vma to hmm_range?

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1499

Quoted 4 lines
mm/migrate_device.c +4 -0
+ *
+ * When collecting happens by hmm_range_fault(), this populates
+ * the migrate->src[] and migrate->dst[] using range->hmm_pfns[].
+ * Also, migrate->cpages and migrate->npages get initialized.

Can you please document what exactly this function does with the magical HMM_PFN_VALID | HMM_PFN_MIGRATE, and why it skips everything else?

What is the caller supposed to setup?

I am getting the feeling that this is severely under-documented :)

Quoted 10 lines
mm/migrate_device.c +10 -0
+ */
+void migrate_hmm_range_setup(struct hmm_range *range)
+{
+
+	struct migrate_vma *migrate = range->migrate;
+
+	if (!migrate)
+		return;
+
+	migrate->npages = (migrate->end - migrate->start) >> PAGE_SHIFT;

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1519

Quoted 10 lines
mm/migrate_device.c +10 -0
+	migrate->cpages = 0;
+
+	for (unsigned long i = 0; i < migrate->npages; i++) {
+
+		unsigned long pfn = range->hmm_pfns[i];
+
+		pfn &= ~HMM_PFN_INOUT_FLAGS;
+
+		/*
+		 *

Please don't add unnecessary empty comment lines above/below the text.

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1522

Quoted 3 lines
mm/migrate_device.c +3 -0
+		 *  Don't do migration if valid and migrate flags are not both set.
+		 *
+		 */

/*
* There is nothing to migrate if the entry is not valid
* migration entry.
*/

Which raises the question:

When would HMM_PFN_MIGRATE be set without HMM_PFN_VALID

Quoted 6 lines
mm/migrate_device.c +6 -0
+		if ((pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) !=
+		    (HMM_PFN_VALID | HMM_PFN_MIGRATE)) {
+			migrate->src[i] = 0;
+			migrate->dst[i] = 0;
+			continue;
+		}

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1538

Quoted 10 lines
mm/migrate_device.c +10 -0
+
+		migrate->cpages++;
+
+		/*
+		 *
+		 * The zero page is encoded in a special way, valid and migrate is
+		 * set, and pfn part is zero. Encode specially for migrate also.
+		 *
+		 */
+		if (pfn == (HMM_PFN_VALID|HMM_PFN_MIGRATE)) {

I think you can simplify all that by doing a

if (pfn & ~(HMM_PFN_VALID|HMM_PFN_COMPOUND|HMM_PFN_MIGRATE))

	migrate->src[i] = migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn)));
else
	migrate->src[i] = 0;

migrate->src[i] |= (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0; migrate->src[i] |= (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0; migrate->dst[i] = 0;

And I wonder whether you could replace the (HMM_PFN_VALID|HMM_PFN_COMPOUND|HMM_PFN_MIGRATE) by HMM_PFN_FLAGS, to only check for non-zero PFNs?

Signature · 3 lines
Cheers,

David
#5 lore raw

Re: [PATCH v5 2/6] mm: Add helper to convert HMM pfn to migrate pfn

Hi,

Quoted 10 lines · David Hildenbrand (Arm)
Quoted 4 lines · On 2/11/26 09:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

The unified HMM/migrate_device pagewalk does the "collecting" in HMM side, so need a helper to transfer pfns to migrate_vma

s/in/on the/ ?

"so we" ?

"to the" ?

Yes will correct.

Quoted 25 lines
Quoted 25 lines

world.

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 include/linux/hmm.h     | 18 +++++++++--
 include/linux/migrate.h |  3 +-
 mm/hmm.c                |  6 ----
 mm/migrate_device.c     | 69 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 87 insertions(+), 9 deletions(-)
include/linux/hmm.h +0 -0
index db75ffc949a7..b5418e318782 100644
@@ -12,7 +12,7 @@
 #include <linux/mm.h>
 
Quoted 4 lines

include/linux/hmm.h line 15

Quoted 3 lines
include/linux/hmm.h +1 -1
 struct mmu_interval_notifier;
-
+struct migrate_vma;

Keep the empty line, please.

ack

Quoted 13 lines

include/linux/hmm.h struct mmu_interval_notifier; · line 30

Quoted 9 lines
include/linux/hmm.h +1 -0
 /*
  * On output:
  * 0             - The page is faultable and a future call with 
@@ -27,6 +27,7 @@ struct mmu_interval_notifier;
  * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer
  * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
  *                      to mark that page is already DMA mapped
+ * HMM_PFN_MIGRATE    - Migrate PTE installed

I have no idea what exactly that means. Can we a bit clearer? Especially, what does it mean when this is set alongside HMM_PFN_VALID? What if it is not set alongside HMM_PFN_VALID.

Agreed needs more documentation. Look below for explanation also.

Quoted 1 line

Shouldn't we document what HMM_PFN_COMPOUND and HMM_PFN_MIGRATE mean as well somewhere?

I think they deserve some words as well.

Quoted 30 lines
Quoted 30 lines
include/linux/hmm.h +6 -1
  *
  * On input:
  * 0                 - Return the current state of the page, do not fault it.
@@ -34,6 +35,7 @@ struct mmu_interval_notifier;
  *                     will fail
  * HMM_PFN_REQ_WRITE - The output must have HMM_PFN_WRITE or hmm_range_fault()
  *                     will fail. Must be combined with HMM_PFN_REQ_FAULT.
+ * HMM_PFN_REQ_MIGRATE - For default_flags, request to migrate to device
  */
 enum hmm_pfn_flags {
 	/* Output fields and flags */
@@ -48,15 +50,25 @@ enum hmm_pfn_flags {
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
 
-	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+	/* Migrate request */
+	HMM_PFN_MIGRATE    = 1UL << (BITS_PER_LONG - 7),
+	HMM_PFN_COMPOUND   = 1UL << (BITS_PER_LONG - 8),
+	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),
 
 	/* Input flags */
 	HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
 	HMM_PFN_REQ_WRITE = HMM_PFN_WRITE,
+	HMM_PFN_REQ_MIGRATE = HMM_PFN_MIGRATE,
 
 	HMM_PFN_FLAGS = ~((1UL << HMM_PFN_ORDER_SHIFT) - 1),
 };
 
Quoted 8 lines

include/linux/hmm.h enum hmm_pfn_flags { · line 70

Quoted 5 lines
include/linux/hmm.h +5 -0
+enum {
+	/* These flags are carried from input-to-output */
+	HMM_PFN_INOUT_FLAGS = HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA |
+		HMM_PFN_P2PDMA_BUS,
+};

Why is that an anonymous enum with a single value? Ah, you are moving that. This should be spelled out in the patch description (and why you are doing it).

Yes I moved for visibility to migrate_device.

Quoted 1 line

Is there any reason we can't move that into hmm_pfn_flags?

Can't think of any, I think that would be good to do.

Quoted 12 lines

include/linux/hmm.h static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn) · line 122

Quoted 10 lines
include/linux/hmm.h +2 -0
+
 /*
  * hmm_pfn_to_page() - return struct page pointed to by a device entry
  *
@@ -107,6 +119,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
  * @default_flags: default flags for the range (write, read, ... see hmm doc)
  * @pfn_flags_mask: allows to mask pfn flags so that only default_flags matter
  * @dev_private_owner: owner of device private pages
+ * @migrate: structure for migrating the associated vma

Is it really for "migrating a vma"? I assume it's for migrating a range of a VMA.

Yes, a range. Will fix.

Quoted 55 lines
Quoted 55 lines
include/linux/hmm.h +1 -0
  */
 struct hmm_range {
 	struct mmu_interval_notifier *notifier;
@@ -117,6 +130,7 @@ struct hmm_range {
 	unsigned long		default_flags;
 	unsigned long		pfn_flags_mask;
 	void			*dev_private_owner;
+	struct migrate_vma      *migrate;
 };
 
 /*
include/linux/migrate.h +2 -1
index 26ca00c325d9..8e6c28efd4f8 100644
@@ -3,6 +3,7 @@
 #define _LINUX_MIGRATE_H
 
 #include <linux/mm.h>
+#include <linux/hmm.h>
 #include <linux/mempolicy.h>
 #include <linux/migrate_mode.h>
 #include <linux/hugetlb.h>
@@ -192,7 +193,7 @@ void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns,
 			unsigned long npages);
 void migrate_device_finalize(unsigned long *src_pfns,
 			unsigned long *dst_pfns, unsigned long npages);
-
+void migrate_hmm_range_setup(struct hmm_range *range);
 #endif /* CONFIG_MIGRATION */
 
 #endif /* _LINUX_MIGRATE_H */
mm/hmm.c +0 -6
index 4ec74c18bef6..21ff99379836 100644
@@ -41,12 +41,6 @@ enum {
 	HMM_NEED_ALL_BITS = HMM_NEED_FAULT | HMM_NEED_WRITE_FAULT,
 };
 
-enum {
-	/* These flags are carried from input-to-output */
-	HMM_PFN_INOUT_FLAGS = HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA |
-			      HMM_PFN_P2PDMA_BUS,
-};
-
 static int hmm_pfns_fill(unsigned long addr, unsigned long end,
 			 struct hmm_range *range, unsigned long cpu_flags)
 {
mm/migrate_device.c +0 -0
index 23379663b1e1..c773a82ea1ed 100644
@@ -1489,3 +1489,72 @@ int migrate_device_coherent_folio(struct folio *folio)
Quoted 8 lines

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1495

Quoted 7 lines
mm/migrate_device.c +4 -0
 		return 0;
 	return -EBUSY;
 }
+
+/**
+ * migrate_hmm_range_setup() - prepare to migrate a range of memory
+ * @range: contains pointer to migrate_vma to be populated

"pointer to hmm_range to be migrated" ?

Tried to say range->migrate's src and dst arrays, cpages and npages get populated.

Quoted 3 lines

Why can't we just pass the migrate_vma? Or is there another use case for adding the migrate_vma to hmm_range?

We have to pass hmm_range anyway because of range->hmm_pfns. migrate_hmm_range_setup() is supposed to be called after hmm_range_fault(), which populates migrate_vma's .start .end and .vma. So passing a hmm_range also emphasizes the intended usage.

Quoted 11 lines

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1499

Quoted 4 lines
mm/migrate_device.c +4 -0
+ *
+ * When collecting happens by hmm_range_fault(), this populates
+ * the migrate->src[] and migrate->dst[] using range->hmm_pfns[].
+ * Also, migrate->cpages and migrate->npages get initialized.

Can you please document what exactly this function does with the magical HMM_PFN_VALID | HMM_PFN_MIGRATE, and why it skips everything else?

What is the caller supposed to setup?

I am getting the feeling that this is severely under-documented :)

Yes will add documentation. The caller of hmm_range_fault() is supposed to
fill migrate_vma's pgmap_owner, src and dst array pointers, and flags for migrate direction. (there is minor change in v6 https://lore.kernel.org/linux-mm/[email protected]/)
for the flags requirement.

Quoted 14 lines
Quoted 14 lines
mm/migrate_device.c +13 -0
+ */
+void migrate_hmm_range_setup(struct hmm_range *range)
+{
+
+	struct migrate_vma *migrate = range->migrate;
+
+	if (!migrate)
+		return;
+
+	migrate->npages = (migrate->end - migrate->start) >> PAGE_SHIFT;
+	migrate->cpages = 0;
+
+	for (unsigned long i = 0; i < migrate->npages; i++) {
Quoted 8 lines

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1519

Quoted 7 lines
mm/migrate_device.c +7 -0
+
+		unsigned long pfn = range->hmm_pfns[i];
+
+		pfn &= ~HMM_PFN_INOUT_FLAGS;
+
+		/*
+		 *

Please don't add unnecessary empty comment lines above/below the text.

ack

Quoted 10 lines

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1522

Quoted 4 lines
mm/migrate_device.c +3 -0
+		 *  Don't do migration if valid and migrate flags are not both set.
+		 *
+		 */

/*
* There is nothing to migrate if the entry is not valid
* migration entry.
*/

Quoted 3 lines

Which raises the question:

When would HMM_PFN_MIGRATE be set without HMM_PFN_VALID

As is HMM_PFN_MIGRATE is not supposed to be set without HMM_PFN_VALID... But, I just realized that in order to allow hmm pfn 0 we have to make HMM_PFN_MIGRATE alone mean the "empty page" (hole, pte_none, zero page)

Quoted 25 lines
Quoted 7 lines
mm/migrate_device.c +6 -0
+		if ((pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) !=
+		    (HMM_PFN_VALID | HMM_PFN_MIGRATE)) {
+			migrate->src[i] = 0;
+			migrate->dst[i] = 0;
+			continue;
+		}

mm/migrate_device.c int migrate_device_coherent_folio(struct folio *folio) · line 1538

Quoted 10 lines
mm/migrate_device.c +10 -0
+
+		migrate->cpages++;
+
+		/*
+		 *
+		 * The zero page is encoded in a special way, valid and migrate is
+		 * set, and pfn part is zero. Encode specially for migrate also.
+		 *
+		 */
+		if (pfn == (HMM_PFN_VALID|HMM_PFN_MIGRATE)) {

I think you can simplify all that by doing a

if (pfn & ~(HMM_PFN_VALID|HMM_PFN_COMPOUND|HMM_PFN_MIGRATE))

	migrate->src[i] = migrate_pfn(page_to_pfn(hmm_pfn_to_page(pfn)));
else
	migrate->src[i] = 0;
Quoted 7 lines

migrate->src[i] |= (pfn & HMM_PFN_WRITE) ? MIGRATE_PFN_WRITE : 0; migrate->src[i] |= (pfn & HMM_PFN_COMPOUND) ? MIGRATE_PFN_COMPOUND : 0; migrate->dst[i] = 0;

And I wonder whether you could replace the (HMM_PFN_VALID|HMM_PFN_COMPOUND|HMM_PFN_MIGRATE) by HMM_PFN_FLAGS, to only check for non-zero PFNs?

For current code the above would be ok. With HMM_PFN_MIGRATE alone for empty page have to adjust. Will fix.

Quoted 1 line

Thanks for the review!

--Mika

maintainer
#6 lore raw

Re: [PATCH v5 2/6] mm: Add helper to convert HMM pfn to migrate pfn

Quoted 7 lines · Mika Penttilä

Hi,

Quoted 3 lines · David Hildenbrand (Arm)
Quoted 2 lines · On 2/11/26 09:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

<...>

Quoted 11 lines
Quoted 11 lines

include/linux/hmm.h enum hmm_pfn_flags { · line 56

Quoted 11 lines

enum hmm_pfn_flags {

include/linux/hmm.h +4 -1
 	/* Output fields and flags */
@@ -48,15 +50,25 @@ enum hmm_pfn_flags {
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
 
-	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+	/* Migrate request */
+	HMM_PFN_MIGRATE    = 1UL << (BITS_PER_LONG - 7),
+	HMM_PFN_COMPOUND   = 1UL << (BITS_PER_LONG - 8),
+	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),

Please don't add fields which are not flags to enum hmm_pfn_flags.

Thanks

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
include/linux/hmm.h +1 -0
 	/* Input flags */
 	HMM_PFN_REQ_FAULT = HMM_PFN_VALID,
 	HMM_PFN_REQ_WRITE = HMM_PFN_WRITE,
+	HMM_PFN_REQ_MIGRATE = HMM_PFN_MIGRATE,
 
 	HMM_PFN_FLAGS = ~((1UL << HMM_PFN_ORDER_SHIFT) - 1),
 };
 
maintainer
#7 lore raw

Re: [PATCH v5 2/6] mm: Add helper to convert HMM pfn to migrate pfn

Quoted 17 lines · Leon Romanovsky
Quoted 7 lines · Mika Penttilä

Hi,

Quoted 3 lines · David Hildenbrand (Arm)
Quoted 2 lines · On 2/11/26 09:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

<...>

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

enum hmm_pfn_flags {

include/linux/hmm.h +0 -0
 	/* Output fields and flags */
@@ -48,15 +50,25 @@ enum hmm_pfn_flags {
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
 
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

include/linux/hmm.h enum hmm_pfn_flags { · line 56

Quoted 5 lines
include/linux/hmm.h +4 -1
-	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
+	/* Migrate request */
+	HMM_PFN_MIGRATE    = 1UL << (BITS_PER_LONG - 7),
+	HMM_PFN_COMPOUND   = 1UL << (BITS_PER_LONG - 8),
+	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 13),

Please don't add fields which are not flags to enum hmm_pfn_flags.

If you are adding a multi-bit field then use GENMASK not shift

Jason

#8 lore raw

PATCH v5 3/6 mm/hmm: do the plumbing for HMM to participate in migration

In reply to [email protected]

From: Mika Penttilä <[email protected]>

Do the preparations in hmm_range_fault() and pagewalk callbacks to do the "collecting" part of migration, needed for migration on fault.

These steps include locking for pmd/pte if migrating, capturing the vma for further migrate actions, and calling the still dummy hmm_vma_handle_migrate_prepare_pmd() and hmm_vma_handle_migrate_prepare() functions in the pagewalk.

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 include/linux/migrate.h |  18 +-
 lib/test_hmm.c          |   2 +-
 mm/hmm.c                | 419 +++++++++++++++++++++++++++++++++++-----
 3 files changed, 386 insertions(+), 53 deletions(-)
include/linux/migrate.h +17 -1
index 8e6c28efd4f8..818272b2a7b5 100644
@@ -98,6 +98,16 @@ static inline int set_movable_ops(const struct movable_operations *ops, enum pag
 	return -ENOSYS;
 }
 
+enum migrate_vma_info {
+	MIGRATE_VMA_SELECT_NONE = 0,
+	MIGRATE_VMA_SELECT_COMPOUND = MIGRATE_VMA_SELECT_NONE,
+};
+
+static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
+{
+	return MIGRATE_VMA_SELECT_NONE;
+}
+
 #endif /* CONFIG_MIGRATION */
 
 #ifdef CONFIG_NUMA_BALANCING
@@ -141,7 +151,7 @@ static inline unsigned long migrate_pfn(unsigned long pfn)
 	return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID;
 }
 
-enum migrate_vma_direction {
+enum migrate_vma_info {
 	MIGRATE_VMA_SELECT_SYSTEM = 1 << 0,
 	MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
 	MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
@@ -183,6 +193,12 @@ struct migrate_vma {
 	struct page		*fault_page;
 };
 
+// TODO: enable migration
+static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
+{
+	return 0;
+}
+
 int migrate_vma_setup(struct migrate_vma *args);
 void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
lib/test_hmm.c +1 -1
index 455a6862ae50..94f1f4cff8b1 100644
@@ -145,7 +145,7 @@ static bool dmirror_is_private_zone(struct dmirror_device *mdevice)
 		HMM_DMIRROR_MEMORY_DEVICE_PRIVATE);
 }
 
-static enum migrate_vma_direction
+static enum migrate_vma_info
 dmirror_select_device(struct dmirror *dmirror)
 {
 	return (dmirror->mdevice->zone_device_type ==
mm/hmm.c +368 -51

592 lines not on this page · raw message

Signature · 1 line
2.50.0
#9 lore raw

PATCH v5 4/6 mm: setup device page migration in HMM pagewalk

In reply to [email protected]

From: Mika Penttilä <[email protected]>

Implement the needed hmm_vma_handle_migrate_prepare_pmd() and hmm_vma_handle_migrate_prepare() functions which are mostly carried over from migrate_device.c, as well as the needed split functions.

Make migrate_device take use of HMM pagewalk for collecting part of migration.

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 include/linux/migrate.h |  10 +-
 mm/hmm.c                | 403 +++++++++++++++++++++++++++++++++++++++-
 mm/migrate_device.c     |  25 ++-
 3 files changed, 425 insertions(+), 13 deletions(-)
include/linux/migrate.h +8 -2
index 818272b2a7b5..104eda2dd881 100644
@@ -156,6 +156,7 @@ enum migrate_vma_info {
 	MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1,
 	MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2,
 	MIGRATE_VMA_SELECT_COMPOUND = 1 << 3,
+	MIGRATE_VMA_FAULT = 1 << 4,
 };
 
 struct migrate_vma {
@@ -193,10 +194,15 @@ struct migrate_vma {
 	struct page		*fault_page;
 };
 
-// TODO: enable migration
 static inline enum migrate_vma_info hmm_select_migrate(struct hmm_range *range)
 {
-	return 0;
+	enum migrate_vma_info minfo;
+
+	minfo = range->migrate ? range->migrate->flags : 0;
+	minfo |= (range->default_flags & HMM_PFN_REQ_MIGRATE) ?
+		MIGRATE_VMA_SELECT_SYSTEM : 0;
+
+	return minfo;
 }
 
 int migrate_vma_setup(struct migrate_vma *args);
mm/hmm.c +396 -7

430 lines not on this page · raw message

mm/migrate_device.c +21 -4
index c773a82ea1ed..222cce2e934d 100644
@@ -734,7 +734,16 @@ static void migrate_vma_unmap(struct migrate_vma *migrate)
  */
 int migrate_vma_setup(struct migrate_vma *args)
 {
+	int ret;
 	long nr_pages = (args->end - args->start) >> PAGE_SHIFT;
+	struct hmm_range range = {
+		.notifier = NULL,
+		.start = args->start,
+		.end = args->end,
+		.hmm_pfns = args->src,
+		.dev_private_owner = args->pgmap_owner,
+		.migrate = args
+	};
 
 	args->start &= PAGE_MASK;
 	args->end &= PAGE_MASK;
@@ -759,17 +768,25 @@ int migrate_vma_setup(struct migrate_vma *args)
 	args->cpages = 0;
 	args->npages = 0;
 
-	migrate_vma_collect(args);
+	if (args->flags & MIGRATE_VMA_FAULT)
+		range.default_flags |= HMM_PFN_REQ_FAULT;
+
+	ret = hmm_range_fault(&range);
 
-	if (args->cpages)
-		migrate_vma_unmap(args);
+	migrate_hmm_range_setup(&range);
+
+	/* Remove migration PTEs */
+	if (ret) {
+		migrate_vma_pages(args);
+		migrate_vma_finalize(args);
+	}
 
 	/*
 	 * At this point pages are locked and unmapped, and thus they have
 	 * stable content and can safely be copied to destination memory that
 	 * is allocated by the drivers.
 	 */
-	return 0;
+	return ret;
 
 }
 EXPORT_SYMBOL(migrate_vma_setup);
Signature · 1 line
2.50.0
#10 lore raw

PATCH v5 5/6 mm: add new testcase for the migrate on fault case

In reply to [email protected]

From: Mika Penttilä <[email protected]>

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 lib/test_hmm.c                         | 98 ++++++++++++++++++++++++++
 lib/test_hmm_uapi.h                    | 19 ++---
 tools/testing/selftests/mm/hmm-tests.c | 54 ++++++++++++++
 3 files changed, 162 insertions(+), 9 deletions(-)
lib/test_hmm.c +98 -0
index 94f1f4cff8b1..8e8c8970d1f7 100644
@@ -36,6 +36,7 @@
 #define DMIRROR_RANGE_FAULT_TIMEOUT	1000
 #define DEVMEM_CHUNK_SIZE		(256 * 1024 * 1024U)
 #define DEVMEM_CHUNKS_RESERVE		16
+#define PFNS_ARRAY_SIZE			64
 
 /*
  * For device_private pages, dpage is just a dummy struct page
@@ -1196,6 +1197,99 @@ static int dmirror_migrate_to_device(struct dmirror *dmirror,
 	return ret;
 }
 
+static int do_fault_and_migrate(struct dmirror *dmirror, struct hmm_range *range)
+{
+	struct migrate_vma *migrate = range->migrate;
+	int ret;
+
+	mmap_read_lock(dmirror->notifier.mm);
+
+	/* Fault-in pages for migration and update device page table */
+	ret = dmirror_range_fault(dmirror, range);
+
+	pr_debug("Migrating from sys mem to device mem\n");
+	migrate_hmm_range_setup(range);
+
+	dmirror_migrate_alloc_and_copy(migrate, dmirror);
+	migrate_vma_pages(migrate);
+	dmirror_migrate_finalize_and_map(migrate, dmirror);
+	migrate_vma_finalize(migrate);
+
+	mmap_read_unlock(dmirror->notifier.mm);
+	return ret;
+}
+
+static int dmirror_fault_and_migrate_to_device(struct dmirror *dmirror,
+					       struct hmm_dmirror_cmd *cmd)
+{
+	unsigned long start, size, end, next;
+	unsigned long src_pfns[PFNS_ARRAY_SIZE] = { 0 };
+	unsigned long dst_pfns[PFNS_ARRAY_SIZE] = { 0 };
+	struct migrate_vma migrate = { 0 };
+	struct hmm_range range = { 0 };
+	struct dmirror_bounce bounce;
+	int ret = 0;
+
+	/* Whole range */
+	start = cmd->addr;
+	size = cmd->npages << PAGE_SHIFT;
+	end = start + size;
+
+	if (!mmget_not_zero(dmirror->notifier.mm)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	migrate.pgmap_owner = dmirror->mdevice;
+	migrate.src = src_pfns;
+	migrate.dst = dst_pfns;
+
+	range.migrate = &migrate;
+	range.hmm_pfns = src_pfns;
+	range.pfn_flags_mask = 0;
+	range.default_flags = HMM_PFN_REQ_FAULT | HMM_PFN_REQ_MIGRATE;
+	range.dev_private_owner = dmirror->mdevice;
+	range.notifier = &dmirror->notifier;
+
+	for (next = start; next < end; next = range.end) {
+		range.start = next;
+		range.end = min(end, next + (PFNS_ARRAY_SIZE << PAGE_SHIFT));
+
+		pr_debug("Fault and migrate range start:%#lx end:%#lx\n",
+			 range.start, range.end);
+
+		ret = do_fault_and_migrate(dmirror, &range);
+		if (ret)
+			goto out_mmput;
+	}
+
+	/*
+	 * Return the migrated data for verification.
+	 * Only for pages in device zone
+	 */
+	ret = dmirror_bounce_init(&bounce, start, size);
+	if (ret)
+		goto out_mmput;
+
+	mutex_lock(&dmirror->mutex);
+	ret = dmirror_do_read(dmirror, start, end, &bounce);
+	mutex_unlock(&dmirror->mutex);
+	if (ret == 0) {
+		ret = copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr, bounce.size);
+		if (ret)
+			ret = -EFAULT;
+	}
+
+	cmd->cpages = bounce.cpages;
+	dmirror_bounce_fini(&bounce);
+
+
+out_mmput:
+	mmput(dmirror->notifier.mm);
+out:
+	return ret;
+}
+
 static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *range,
 			    unsigned char *perm, unsigned long entry)
 {
@@ -1512,6 +1606,10 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
 		ret = dmirror_migrate_to_device(dmirror, &cmd);
 		break;
 
+	case HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV:
+		ret = dmirror_fault_and_migrate_to_device(dmirror, &cmd);
+		break;
+
 	case HMM_DMIRROR_MIGRATE_TO_SYS:
 		ret = dmirror_migrate_to_system(dmirror, &cmd);
 		break;
lib/test_hmm_uapi.h +10 -9
index f94c6d457338..0b6e7a419e36 100644
@@ -29,15 +29,16 @@ struct hmm_dmirror_cmd {
 };
 
 /* Expose the address space of the calling process through hmm device file */
-#define HMM_DMIRROR_READ		_IOWR('H', 0x00, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_WRITE		_IOWR('H', 0x01, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_MIGRATE_TO_DEV	_IOWR('H', 0x02, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_MIGRATE_TO_SYS	_IOWR('H', 0x03, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_SNAPSHOT		_IOWR('H', 0x04, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_EXCLUSIVE		_IOWR('H', 0x05, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_CHECK_EXCLUSIVE	_IOWR('H', 0x06, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_RELEASE		_IOWR('H', 0x07, struct hmm_dmirror_cmd)
-#define HMM_DMIRROR_FLAGS		_IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ			_IOWR('H', 0x00, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_WRITE			_IOWR('H', 0x01, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_TO_DEV		_IOWR('H', 0x02, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_ON_FAULT_TO_DEV	_IOWR('H', 0x03, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_MIGRATE_TO_SYS		_IOWR('H', 0x04, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_SNAPSHOT			_IOWR('H', 0x05, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_EXCLUSIVE			_IOWR('H', 0x06, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_CHECK_EXCLUSIVE		_IOWR('H', 0x07, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_RELEASE			_IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_FLAGS			_IOWR('H', 0x09, struct hmm_dmirror_cmd)
 
 #define HMM_DMIRROR_FLAG_FAIL_ALLOC	(1ULL << 0)
 
tools/testing/selftests/mm/hmm-tests.c +54 -0

66 lines not on this page · raw message

Signature · 1 line
2.50.0
#11 lore raw

PATCH v5 6/6 mm:/migrate_device.c: remove migrate_vma_collect_*()

In reply to [email protected]

From: Mika Penttilä <[email protected]>

With the unified fault handling and migrate path, the migrate_vma_collect_*() functions are unused, let's remove them.

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>
---
 mm/migrate_device.c | 508 --------------------------------------------
 1 file changed, 508 deletions(-)
mm/migrate_device.c +0 -508
index 222cce2e934d..f130ddbdd12c 100644
@@ -18,514 +18,6 @@
 #include <asm/tlbflush.h>
 #include "internal.h"
 
-static int migrate_vma_collect_skip(unsigned long start,
-				    unsigned long end,
-				    struct mm_walk *walk)
-{
-	struct migrate_vma *migrate = walk->private;
-	unsigned long addr;
-
-	for (addr = start; addr < end; addr += PAGE_SIZE) {
-		migrate->dst[migrate->npages] = 0;
-		migrate->src[migrate->npages++] = 0;
-	}
-
-	return 0;
-}
-
-static int migrate_vma_collect_hole(unsigned long start,
-				    unsigned long end,
-				    __always_unused int depth,
-				    struct mm_walk *walk)
-{
-	struct migrate_vma *migrate = walk->private;
-	unsigned long addr;
-
-	/* Only allow populating anonymous memory. */
-	if (!vma_is_anonymous(walk->vma))
-		return migrate_vma_collect_skip(start, end, walk);
-
-	if (thp_migration_supported() &&
-		(migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
-		(IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
-		 IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
-		migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE |
-						MIGRATE_PFN_COMPOUND;
-		migrate->dst[migrate->npages] = 0;
-		migrate->npages++;
-		migrate->cpages++;
-
-		/*
-		 * Collect the remaining entries as holes, in case we
-		 * need to split later
-		 */
-		return migrate_vma_collect_skip(start + PAGE_SIZE, end, walk);
-	}
-
-	for (addr = start; addr < end; addr += PAGE_SIZE) {
-		migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE;
-		migrate->dst[migrate->npages] = 0;
-		migrate->npages++;
-		migrate->cpages++;
-	}
-
-	return 0;
-}
-
-/**
- * migrate_vma_split_folio() - Helper function to split a THP folio
- * @folio: the folio to split
- * @fault_page: struct page associated with the fault if any
- *
- * Returns 0 on success
- */
-static int migrate_vma_split_folio(struct folio *folio,
-				   struct page *fault_page)
-{
-	int ret;
-	struct folio *fault_folio = fault_page ? page_folio(fault_page) : NULL;
-	struct folio *new_fault_folio = NULL;
-
-	if (folio != fault_folio) {
-		folio_get(folio);
-		folio_lock(folio);
-	}
-
-	ret = split_folio(folio);
-	if (ret) {
-		if (folio != fault_folio) {
-			folio_unlock(folio);
-			folio_put(folio);
-		}
-		return ret;
-	}
-
-	new_fault_folio = fault_page ? page_folio(fault_page) : NULL;
-
-	/*
-	 * Ensure the lock is held on the correct
-	 * folio after the split
-	 */
-	if (!new_fault_folio) {
-		folio_unlock(folio);
-		folio_put(folio);
-	} else if (folio != new_fault_folio) {
-		if (new_fault_folio != fault_folio) {
-			folio_get(new_fault_folio);
-			folio_lock(new_fault_folio);
-		}
-		folio_unlock(folio);
-		folio_put(folio);
-	}
-
-	return 0;
-}
-
-/** migrate_vma_collect_huge_pmd - collect THP pages without splitting the
- * folio for device private pages.
- * @pmdp: pointer to pmd entry
- * @start: start address of the range for migration
- * @end: end address of the range for migration
- * @walk: mm_walk callback structure
- * @fault_folio: folio associated with the fault if any
- *
- * Collect the huge pmd entry at @pmdp for migration and set the
- * MIGRATE_PFN_COMPOUND flag in the migrate src entry to indicate that
- * migration will occur at HPAGE_PMD granularity
- */
-static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start,
-					unsigned long end, struct mm_walk *walk,
-					struct folio *fault_folio)
-{
-	struct mm_struct *mm = walk->mm;
-	struct folio *folio;
-	struct migrate_vma *migrate = walk->private;
-	spinlock_t *ptl;
-	int ret;
-	unsigned long write = 0;
-
-	ptl = pmd_lock(mm, pmdp);
-	if (pmd_none(*pmdp)) {
-		spin_unlock(ptl);
-		return migrate_vma_collect_hole(start, end, -1, walk);
-	}
-
-	if (pmd_trans_huge(*pmdp)) {
-		if (!(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_skip(start, end, walk);
-		}
-
-		folio = pmd_folio(*pmdp);
-		if (is_huge_zero_folio(folio)) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_hole(start, end, -1, walk);
-		}
-		if (pmd_write(*pmdp))
-			write = MIGRATE_PFN_WRITE;
-	} else if (!pmd_present(*pmdp)) {
-		const softleaf_t entry = softleaf_from_pmd(*pmdp);
-
-		folio = softleaf_to_folio(entry);
-
-		if (!softleaf_is_device_private(entry) ||
-			!(migrate->flags & MIGRATE_VMA_SELECT_DEVICE_PRIVATE) ||
-			(folio->pgmap->owner != migrate->pgmap_owner)) {
-			spin_unlock(ptl);
-			return migrate_vma_collect_skip(start, end, walk);
-		}
-
-		if (softleaf_is_migration(entry)) {
-			migration_entry_wait_on_locked(entry, ptl);
-			spin_unlock(ptl);
-			return -EAGAIN;
-		}
-
-		if (softleaf_is_device_private_write(entry))
-			write = MIGRATE_PFN_WRITE;
-	} else {
-		spin_unlock(ptl);
-		return -EAGAIN;
-	}
-
-	folio_get(folio);
-	if (folio != fault_folio && unlikely(!folio_trylock(folio))) {
-		spin_unlock(ptl);
-		folio_put(folio);
-		return migrate_vma_collect_skip(start, end, walk);
-	}
-
-	if (thp_migration_supported() &&
-		(migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
-		(IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
-		 IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
-
-		struct page_vma_mapped_walk pvmw = {
-			.ptl = ptl,
-			.address = start,
-			.pmd = pmdp,
-			.vma = walk->vma,
-		};
-
-		unsigned long pfn = page_to_pfn(folio_page(folio, 0));
-
-		migrate->src[migrate->npages] = migrate_pfn(pfn) | write
-						| MIGRATE_PFN_MIGRATE
-						| MIGRATE_PFN_COMPOUND;
-		migrate->dst[migrate->npages++] = 0;
-		migrate->cpages++;
-		ret = set_pmd_migration_entry(&pvmw, folio_page(folio, 0));
-		if (ret) {
-			migrate->npages--;
-			migrate->cpages--;
-			migrate->src[migrate->npages] = 0;
-			migrate->dst[migrate->npages] = 0;
-			goto fallback;
-		}
-		migrate_vma_collect_skip(start + PAGE_SIZE, end, walk);
-		spin_unlock(ptl);
-		return 0;
-	}
-
-fallback:
-	spin_unlock(ptl);
-	if (!folio_test_large(folio))
-		goto done;
-	ret = split_folio(folio);
-	if (fault_folio != folio)
-		folio_unlock(folio);
-	folio_put(folio);
-	if (ret)
-		return migrate_vma_collect_skip(start, end, walk);
-	if (pmd_none(pmdp_get_lockless(pmdp)))
-		return migrate_vma_collect_hole(start, end, -1, walk);
-
-done:
-	return -ENOENT;
-}
-
-static int migrate_vma_collect_pmd(pmd_t *pmdp,
-				   unsigned long start,
-				   unsigned long end,
-				   struct mm_walk *walk)
-{
-	struct migrate_vma *migrate = walk->private;
-	struct vm_area_struct *vma = walk->vma;
-	struct mm_struct *mm = vma->vm_mm;
-	unsigned long addr = start, unmapped = 0;
-	spinlock_t *ptl;
-	struct folio *fault_folio = migrate->fault_page ?
-		page_folio(migrate->fault_page) : NULL;
-	pte_t *ptep;
-
-again:
-	if (pmd_trans_huge(*pmdp) || !pmd_present(*pmdp)) {
-		int ret = migrate_vma_collect_huge_pmd(pmdp, start, end, walk, fault_folio);
-
-		if (ret == -EAGAIN)
-			goto again;
-		if (ret == 0)
-			return 0;
-	}
-
-	ptep = pte_offset_map_lock(mm, pmdp, start, &ptl);
-	if (!ptep)
-		goto again;
-	arch_enter_lazy_mmu_mode();
-	ptep += (addr - start) / PAGE_SIZE;
-
-	for (; addr < end; addr += PAGE_SIZE, ptep++) {
-		struct dev_pagemap *pgmap;
-		unsigned long mpfn = 0, pfn;
-		struct folio *folio;
-		struct page *page;
-		softleaf_t entry;
-		pte_t pte;
-
-		pte = ptep_get(ptep);
-
-		if (pte_none(pte)) {
-			if (vma_is_anonymous(vma)) {
-				mpfn = MIGRATE_PFN_MIGRATE;
-				migrate->cpages++;
-			}
-			goto next;
-		}
-
-		if (!pte_present(pte)) {
-			/*
-			 * Only care about unaddressable device page special
-			 * page table entry. Other special swap entries are not
-			 * migratable, and we ignore regular swapped page.
-			 */
-			entry = softleaf_from_pte(pte);
-			if (!softleaf_is_device_private(entry))
-				goto next;
-
-			page = softleaf_to_page(entry);
-			pgmap = page_pgmap(page);
-			if (!(migrate->flags &
-				MIGRATE_VMA_SELECT_DEVICE_PRIVATE) ||
-			    pgmap->owner != migrate->pgmap_owner)
-				goto next;
-
-			folio = page_folio(page);
-			if (folio_test_large(folio)) {
-				int ret;
-
-				arch_leave_lazy_mmu_mode();
-				pte_unmap_unlock(ptep, ptl);
-				ret = migrate_vma_split_folio(folio,
-							  migrate->fault_page);
-
-				if (ret) {
-					if (unmapped)
-						flush_tlb_range(walk->vma, start, end);
-
-					return migrate_vma_collect_skip(addr, end, walk);
-				}
-
-				goto again;
-			}
-
-			mpfn = migrate_pfn(page_to_pfn(page)) |
-					MIGRATE_PFN_MIGRATE;
-			if (softleaf_is_device_private_write(entry))
-				mpfn |= MIGRATE_PFN_WRITE;
-		} else {
-			pfn = pte_pfn(pte);
-			if (is_zero_pfn(pfn) &&
-			    (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
-				mpfn = MIGRATE_PFN_MIGRATE;
-				migrate->cpages++;
-				goto next;
-			}
-			page = vm_normal_page(migrate->vma, addr, pte);
-			if (page && !is_zone_device_page(page) &&
-			    !(migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
-				goto next;
-			} else if (page && is_device_coherent_page(page)) {
-				pgmap = page_pgmap(page);
-
-				if (!(migrate->flags &
-					MIGRATE_VMA_SELECT_DEVICE_COHERENT) ||
-					pgmap->owner != migrate->pgmap_owner)
-					goto next;
-			}
-			folio = page ? page_folio(page) : NULL;
-			if (folio && folio_test_large(folio)) {
-				int ret;
-
-				arch_leave_lazy_mmu_mode();
-				pte_unmap_unlock(ptep, ptl);
-				ret = migrate_vma_split_folio(folio,
-							  migrate->fault_page);
-
-				if (ret) {
-					if (unmapped)
-						flush_tlb_range(walk->vma, start, end);
-
-					return migrate_vma_collect_skip(addr, end, walk);
-				}
-
-				goto again;
-			}
-			mpfn = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE;
-			mpfn |= pte_write(pte) ? MIGRATE_PFN_WRITE : 0;
-		}
-
-		if (!page || !page->mapping) {
-			mpfn = 0;
-			goto next;
-		}
-
-		/*
-		 * By getting a reference on the folio we pin it and that blocks
-		 * any kind of migration. Side effect is that it "freezes" the
-		 * pte.
-		 *
-		 * We drop this reference after isolating the folio from the lru
-		 * for non device folio (device folio are not on the lru and thus
-		 * can't be dropped from it).
-		 */
-		folio = page_folio(page);
-		folio_get(folio);
-
-		/*
-		 * We rely on folio_trylock() to avoid deadlock between
-		 * concurrent migrations where each is waiting on the others
-		 * folio lock. If we can't immediately lock the folio we fail this
-		 * migration as it is only best effort anyway.
-		 *
-		 * If we can lock the folio it's safe to set up a migration entry
-		 * now. In the common case where the folio is mapped once in a
-		 * single process setting up the migration entry now is an
-		 * optimisation to avoid walking the rmap later with
-		 * try_to_migrate().
-		 */
-		if (fault_folio == folio || folio_trylock(folio)) {
-			bool anon_exclusive;
-			pte_t swp_pte;
-
-			flush_cache_page(vma, addr, pte_pfn(pte));
-			anon_exclusive = folio_test_anon(folio) &&
-					  PageAnonExclusive(page);
-			if (anon_exclusive) {
-				pte = ptep_clear_flush(vma, addr, ptep);
-
-				if (folio_try_share_anon_rmap_pte(folio, page)) {
-					set_pte_at(mm, addr, ptep, pte);
-					if (fault_folio != folio)
-						folio_unlock(folio);
-					folio_put(folio);
-					mpfn = 0;
-					goto next;
-				}
-			} else {
-				pte = ptep_get_and_clear(mm, addr, ptep);
-			}
-
-			migrate->cpages++;
-
-			/* Set the dirty flag on the folio now the pte is gone. */
-			if (pte_dirty(pte))
-				folio_mark_dirty(folio);
-
-			/* Setup special migration page table entry */
-			if (mpfn & MIGRATE_PFN_WRITE)
-				entry = make_writable_migration_entry(
-							page_to_pfn(page));
-			else if (anon_exclusive)
-				entry = make_readable_exclusive_migration_entry(
-							page_to_pfn(page));
-			else
-				entry = make_readable_migration_entry(
-							page_to_pfn(page));
-			if (pte_present(pte)) {
-				if (pte_young(pte))
-					entry = make_migration_entry_young(entry);
-				if (pte_dirty(pte))
-					entry = make_migration_entry_dirty(entry);
-			}
-			swp_pte = swp_entry_to_pte(entry);
-			if (pte_present(pte)) {
-				if (pte_soft_dirty(pte))
-					swp_pte = pte_swp_mksoft_dirty(swp_pte);
-				if (pte_uffd_wp(pte))
-					swp_pte = pte_swp_mkuffd_wp(swp_pte);
-			} else {
-				if (pte_swp_soft_dirty(pte))
-					swp_pte = pte_swp_mksoft_dirty(swp_pte);
-				if (pte_swp_uffd_wp(pte))
-					swp_pte = pte_swp_mkuffd_wp(swp_pte);
-			}
-			set_pte_at(mm, addr, ptep, swp_pte);
-
-			/*
-			 * This is like regular unmap: we remove the rmap and
-			 * drop the folio refcount. The folio won't be freed, as
-			 * we took a reference just above.
-			 */
-			folio_remove_rmap_pte(folio, page, vma);
-			folio_put(folio);
-
-			if (pte_present(pte))
-				unmapped++;
-		} else {
-			folio_put(folio);
-			mpfn = 0;
-		}
-
-next:
-		migrate->dst[migrate->npages] = 0;
-		migrate->src[migrate->npages++] = mpfn;
-	}
-
-	/* Only flush the TLB if we actually modified any entries */
-	if (unmapped)
-		flush_tlb_range(walk->vma, start, end);
-
-	arch_leave_lazy_mmu_mode();
-	pte_unmap_unlock(ptep - 1, ptl);
-
-	return 0;
-}
-
-static const struct mm_walk_ops migrate_vma_walk_ops = {
-	.pmd_entry		= migrate_vma_collect_pmd,
-	.pte_hole		= migrate_vma_collect_hole,
-	.walk_lock		= PGWALK_RDLOCK,
-};
-
-/*
- * migrate_vma_collect() - collect pages over a range of virtual addresses
- * @migrate: migrate struct containing all migration information
- *
- * This will walk the CPU page table. For each virtual address backed by a
- * valid page, it updates the src array and takes a reference on the page, in
- * order to pin the page until we lock it and unmap it.
- */
-static void migrate_vma_collect(struct migrate_vma *migrate)
-{
-	struct mmu_notifier_range range;
-
-	/*
-	 * Note that the pgmap_owner is passed to the mmu notifier callback so
-	 * that the registered device driver can skip invalidating device
-	 * private page mappings that won't be migrated.
-	 */
-	mmu_notifier_range_init_owner(&range, MMU_NOTIFY_MIGRATE, 0,
-		migrate->vma->vm_mm, migrate->start, migrate->end,
-		migrate->pgmap_owner);
-	mmu_notifier_invalidate_range_start(&range);
-
-	walk_page_range(migrate->vma->vm_mm, migrate->start, migrate->end,
-			&migrate_vma_walk_ops, migrate);
-
-	mmu_notifier_invalidate_range_end(&range);
-	migrate->end = migrate->start + (migrate->npages << PAGE_SHIFT);
-}
-
 /*
  * migrate_vma_check_page() - check if page is pinned or not
  * @page: struct page to check
Signature · 1 line
2.50.0
#12 lore raw
In reply to [email protected]
Quoted 44 lines · On 2/11/26 09:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

Currently, the way device page faulting and migration works is not optimal, if you want to do both fault handling and migration at once.

Being able to migrate not present pages (or pages mapped with incorrect permissions, eg. COW) to the GPU requires doing either of the following sequences:

1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages

Or:

1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
a) call hmm_range_fault() to fault pages in
b) call migrate_vma_*() again to migrate now present pages

The problem with the first sequence is that you always have to do two page walks even when most of the time the pages are present or zero page mappings so the common case takes a performance hit.

The second sequence is better for the common case, but far worse if pages aren't present because now you have to walk the page tables three times (once to find the page is not present, once so hmm_range_fault() can find a non-present page to fault in and once again to setup the migration). It is also tricky to code correctly. One page table walk could costs over 1000 cpu cycles on X86-64, which is a significant hit.

We should be able to walk the page table once, faulting pages in as required and replacing them with migration entries if requested.

Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling. Also, for the migrate_vma_setup() call paths, a flag, MIGRATE_VMA_FAULT, is added to tell to add fault handling to migrate.

One extra benefit of migrating with hmm_range_fault() path is the migrate_vma.vma gets populated, so no need to retrieve that separataly.

Quoted 7 lines

Tested in X86-64 VM with HMM test device, passing the selftests. For performance, the migrate throughput tests from the selftests show similar numbers (within error margin) as unmodified kernel. Tested also rebased on the
"Remove device private pages from physical address space" series:
https://lore.kernel.org/linux-mm/[email protected]/
plus a small patch to adjust with no problems.

Stumbling over this in my backlog, so far we have to RB tags.

I'm going to take a look at the core MM bits (soon I hope), but it would be great if other people could review the HMM bits and provide proper tags.

Signature · 3 lines
Cheers,

David
#13 lore raw

On 2026-03-13 at 06:58 +1100, "David Hildenbrand (Arm)" <[email protected]> wrote...

Quoted 53 lines · On 2/11/26 09:12, [email protected] wrote:
Quoted 44 lines

From: Mika Penttilä <[email protected]>

Currently, the way device page faulting and migration works is not optimal, if you want to do both fault handling and migration at once.

Being able to migrate not present pages (or pages mapped with incorrect permissions, eg. COW) to the GPU requires doing either of the following sequences:

1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages

Or:

1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
a) call hmm_range_fault() to fault pages in
b) call migrate_vma_*() again to migrate now present pages

The problem with the first sequence is that you always have to do two page walks even when most of the time the pages are present or zero page mappings so the common case takes a performance hit.

The second sequence is better for the common case, but far worse if pages aren't present because now you have to walk the page tables three times (once to find the page is not present, once so hmm_range_fault() can find a non-present page to fault in and once again to setup the migration). It is also tricky to code correctly. One page table walk could costs over 1000 cpu cycles on X86-64, which is a significant hit.

We should be able to walk the page table once, faulting pages in as required and replacing them with migration entries if requested.

Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling. Also, for the migrate_vma_setup() call paths, a flag, MIGRATE_VMA_FAULT, is added to tell to add fault handling to migrate.

One extra benefit of migrating with hmm_range_fault() path is the migrate_vma.vma gets populated, so no need to retrieve that separataly.

Quoted 7 lines

Tested in X86-64 VM with HMM test device, passing the selftests. For performance, the migrate throughput tests from the selftests show similar numbers (within error margin) as unmodified kernel. Tested also rebased on the
"Remove device private pages from physical address space" series:
https://lore.kernel.org/linux-mm/[email protected]/
plus a small patch to adjust with no problems.

Quoted 4 lines

Stumbling over this in my backlog, so far we have to RB tags.

I'm going to take a look at the core MM bits (soon I hope), but it would be great if other people could review the HMM bits and provide proper tags.

Yes, sorry I've been meaning to review this for a while but have been backlogged for a while. I should be able to get to it early next week though.

Quoted 4 lines
Signature · 3 lines
Cheers,

David
#14 lore raw
In reply to [email protected]
Quoted 118 lines · On 2/11/26 19:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

Currently, the way device page faulting and migration works is not optimal, if you want to do both fault handling and migration at once.

Being able to migrate not present pages (or pages mapped with incorrect permissions, eg. COW) to the GPU requires doing either of the following sequences:

1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages

Or:

1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
a) call hmm_range_fault() to fault pages in
b) call migrate_vma_*() again to migrate now present pages

The problem with the first sequence is that you always have to do two page walks even when most of the time the pages are present or zero page mappings so the common case takes a performance hit.

The second sequence is better for the common case, but far worse if pages aren't present because now you have to walk the page tables three times (once to find the page is not present, once so hmm_range_fault() can find a non-present page to fault in and once again to setup the migration). It is also tricky to code correctly. One page table walk could costs over 1000 cpu cycles on X86-64, which is a significant hit.

We should be able to walk the page table once, faulting pages in as required and replacing them with migration entries if requested.

Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling. Also, for the migrate_vma_setup() call paths, a flag, MIGRATE_VMA_FAULT, is added to tell to add fault handling to migrate.

One extra benefit of migrating with hmm_range_fault() path is the migrate_vma.vma gets populated, so no need to retrieve that separataly.

Tested in X86-64 VM with HMM test device, passing the selftests. For performance, the migrate throughput tests from the selftests show similar numbers (within error margin) as unmodified kernel. Tested also rebased on the
"Remove device private pages from physical address space" series:
https://lore.kernel.org/linux-mm/[email protected]/
plus a small patch to adjust with no problems.

Changes v4-v5
- rebase on 6.19
- fixed David's email address
- fixed link issue without CONFIG_TRANSPARENT_HUGEPAGE
- refactored into smaller commits
- added more comments to code

Changes v3-v4:
- rebase on 6.19-rc8
- fixed issues found by kernel test robot with random configs
- fixed typos

Changes v2-v3:
- rebase on 6.19-rc7
- fixed issues found by kernel test robot
- fixed smatch issues reported by Dan Carpenter <[email protected]>
- fixes to lock handling (pmd/pte) on errors
- added assertions for pmd/pte lock states
- other issues discovered by Matthew, thanks!

Changes v1-v2:
- rebase on 6.19-rc6
- fixed issues found by kernel test robot
- fixed locking (pmd/ptl) to cover handle_ and prepare_ regions parts if migrating
- other issues discovered by Matthew, thanks!

Changes RFC-v1:
- rebase on 6.19-rc5
- adjust for the device THP
- changes from feedback

Revisions:
- RFC https://lore.kernel.org/linux-mm/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/
- v2: https://lore.kernel.org/all/[email protected]/
- v3: https://lore.kernel.org/all/[email protected]/
- v4: https://lore.kernel.org/all/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>

Mika Penttilä (6):
mm:/Kconfig changes for migrate on fault for device pages
mm: Add helper to convert HMM pfn to migrate pfn mm/hmm: do the plumbing for HMM to participate in migration
mm: implement device page migration for HMM pagewalk
mm: add new testcase for the migrate on fault case mm:/migrate_device.c: remove migrate_vma_collect_*()

 include/linux/hmm.h                    |  18 +-
 include/linux/migrate.h                |  27 +-
 lib/test_hmm.c                         | 100 ++-
 lib/test_hmm_uapi.h                    |  19 +-
 mm/Kconfig                             |   2 +
 mm/hmm.c                               | 814 +++++++++++++++++++++++--
 mm/migrate_device.c                    | 602 +++---------------
 tools/testing/selftests/mm/hmm-tests.c |  54 ++
 8 files changed, 1053 insertions(+), 583 deletions(-)
Quoted 1 line

base-commit: 05f7e89ab9731565d8a62e3b5d1ec206485eeb0b

Thanks for your patience, I am just trying out the patches and reviewing them in parallel

Balbir

#15 lore raw

Hi,

Quoted 119 lines · Balbir Singh
Quoted 118 lines · On 2/11/26 19:12, [email protected] wrote:

From: Mika Penttilä <[email protected]>

Currently, the way device page faulting and migration works is not optimal, if you want to do both fault handling and migration at once.

Being able to migrate not present pages (or pages mapped with incorrect permissions, eg. COW) to the GPU requires doing either of the following sequences:

1. hmm_range_fault() - fault in non-present pages with correct permissions, etc.
2. migrate_vma_*() - migrate the pages

Or:

1. migrate_vma_*() - migrate present pages
2. If non-present pages detected by migrate_vma_*():
a) call hmm_range_fault() to fault pages in
b) call migrate_vma_*() again to migrate now present pages

The problem with the first sequence is that you always have to do two page walks even when most of the time the pages are present or zero page mappings so the common case takes a performance hit.

The second sequence is better for the common case, but far worse if pages aren't present because now you have to walk the page tables three times (once to find the page is not present, once so hmm_range_fault() can find a non-present page to fault in and once again to setup the migration). It is also tricky to code correctly. One page table walk could costs over 1000 cpu cycles on X86-64, which is a significant hit.

We should be able to walk the page table once, faulting pages in as required and replacing them with migration entries if requested.

Add a new flag to HMM APIs, HMM_PFN_REQ_MIGRATE,
which tells to prepare for migration also during fault handling. Also, for the migrate_vma_setup() call paths, a flag, MIGRATE_VMA_FAULT, is added to tell to add fault handling to migrate.

One extra benefit of migrating with hmm_range_fault() path is the migrate_vma.vma gets populated, so no need to retrieve that separataly.

Tested in X86-64 VM with HMM test device, passing the selftests. For performance, the migrate throughput tests from the selftests show similar numbers (within error margin) as unmodified kernel. Tested also rebased on the
"Remove device private pages from physical address space" series:
https://lore.kernel.org/linux-mm/[email protected]/
plus a small patch to adjust with no problems.

Changes v4-v5
- rebase on 6.19
- fixed David's email address
- fixed link issue without CONFIG_TRANSPARENT_HUGEPAGE
- refactored into smaller commits
- added more comments to code

Changes v3-v4:
- rebase on 6.19-rc8
- fixed issues found by kernel test robot with random configs
- fixed typos

Changes v2-v3:
- rebase on 6.19-rc7
- fixed issues found by kernel test robot
- fixed smatch issues reported by Dan Carpenter <[email protected]>
- fixes to lock handling (pmd/pte) on errors
- added assertions for pmd/pte lock states
- other issues discovered by Matthew, thanks!

Changes v1-v2:
- rebase on 6.19-rc6
- fixed issues found by kernel test robot
- fixed locking (pmd/ptl) to cover handle_ and prepare_ regions parts if migrating
- other issues discovered by Matthew, thanks!

Changes RFC-v1:
- rebase on 6.19-rc5
- adjust for the device THP
- changes from feedback

Revisions:
- RFC https://lore.kernel.org/linux-mm/[email protected]/
- v1: https://lore.kernel.org/all/[email protected]/
- v2: https://lore.kernel.org/all/[email protected]/
- v3: https://lore.kernel.org/all/[email protected]/
- v4: https://lore.kernel.org/all/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Leon Romanovsky <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Balbir Singh <[email protected]>
Cc Zi Yan <[email protected]>
Cc Matthew Brost <[email protected]>

Mika Penttilä (6):
mm:/Kconfig changes for migrate on fault for device pages
mm: Add helper to convert HMM pfn to migrate pfn mm/hmm: do the plumbing for HMM to participate in migration
mm: implement device page migration for HMM pagewalk
mm: add new testcase for the migrate on fault case mm:/migrate_device.c: remove migrate_vma_collect_*()

 include/linux/hmm.h                    |  18 +-
 include/linux/migrate.h                |  27 +-
 lib/test_hmm.c                         | 100 ++-
 lib/test_hmm_uapi.h                    |  19 +-
 mm/Kconfig                             |   2 +
 mm/hmm.c                               | 814 +++++++++++++++++++++++--
 mm/migrate_device.c                    | 602 +++---------------
 tools/testing/selftests/mm/hmm-tests.c |  54 ++
 8 files changed, 1053 insertions(+), 583 deletions(-)
Quoted 6 lines
Quoted 1 line

base-commit: 05f7e89ab9731565d8a62e3b5d1ec206485eeb0b

Thanks for your patience, I am just trying out the patches and reviewing them in parallel

Balbir

No problem, and thanks! btw there's also v6 it has only two minor changes:
https://lore.kernel.org/linux-mm/[email protected]/

--Mika