Skip to contentSkip to conversation

migrate_pages(): batch TLB flushing

RFC In review Huang Ying 9 participants 50 messages Sep 21, 2022 – Nov 2, 2022

Subsystems (3)
lore
Share & tools
lore

Huang Ying

author
#0 lore raw

cover migrate_pages(): batch TLB flushing

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

This patchset is based on v6.0-rc5 and the following patchset,

[PATCH -V3 0/8] migrate_pages(): fix several bugs in error path
https://lore.kernel.org/lkml/[email protected]/

The migrate_pages() related code is converting to folio now. So this patchset cannot apply recent akpm/mm-unstable branch. This patchset is used to check the basic idea. If it is OK, I will rebase the patchset on top of folio changes.

Best Regards,
Huang, Ying

author
#1 lore raw

RFC 1/6 mm/migrate_pages: separate huge page and normal pages migration

This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying.

In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 9 deletions(-)
mm/migrate.c +64 -9
index 571d8c9fd5bc..117134f1c6dc 100644
@@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 
 	trace_mm_migrate_pages_start(mode, reason);
 
+	for (pass = 0; pass < 10 && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(page, page2, from, lru) {
+			nr_subpages = compound_nr(page);
+			cond_resched();
+
+			if (!PageHuge(page))
+				continue;
+
+			rc = unmap_and_move_huge_page(get_new_page,
+						put_new_page, private, page,
+						pass > 2, mode, reason,
+						&ret_pages);
+			/*
+			 * The rules are:
+			 *	Success: hugetlb page will be put back
+			 *	-EAGAIN: stay on the from list
+			 *	-ENOMEM: stay on the from list
+			 *	-ENOSYS: stay on the from list
+			 *	Other errno: put on ret_pages list then splice to
+			 *		     from list
+			 */
+			switch(rc) {
+			case -ENOSYS:
+				/* Hugetlb migration is unsupported */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				list_move_tail(&page->lru, &ret_pages);
+				break;
+			case -ENOMEM:
+				/*
+				 * When memory is low, don't bother to try to migrate
+				 * other pages, just exit.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages + nr_retry_pages;
+				goto out;
+			case -EAGAIN:
+				retry++;
+				nr_retry_pages += nr_subpages;
+				break;
+			case MIGRATEPAGE_SUCCESS:
+				nr_succeeded += nr_subpages;
+				break;
+			default:
+				/*
+				 * Permanent failure (-EBUSY, etc.):
+				 * unlike -EAGAIN case, the failed page is
+				 * removed from migration page list and not
+				 * retried in the next outer loop.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				break;
+			}
+		}
+	}
+	nr_failed += retry;
+	retry = 1;
 thp_subpage_migration:
 	for (pass = 0; pass < 10 && (retry || thp_retry); pass++) {
 		retry = 0;
@@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 			cond_resched();
 
 			if (PageHuge(page))
-				rc = unmap_and_move_huge_page(get_new_page,
-						put_new_page, private, page,
-						pass > 2, mode, reason,
-						&ret_pages);
-			else
-				rc = unmap_and_move(get_new_page, put_new_page,
+				continue;
+
+			rc = unmap_and_move(get_new_page, put_new_page,
 						private, page, pass > 2, mode,
 						reason, &ret_pages);
 			/*
 			 * The rules are:
-			 *	Success: non hugetlb page will be freed, hugetlb
-			 *		 page will be put back
+			 *	Success: page will be freed
 			 *	-EAGAIN: stay on the from list
 			 *	-ENOMEM: stay on the from list
 			 *	-ENOSYS: stay on the from list
@@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 						nr_thp_split++;
 						break;
 					}
-				/* Hugetlb migration is unsupported */
 				} else if (!no_subpage_counting) {
 					nr_failed++;
 				}
Signature · 1 line
2.35.1
reviewer
#2 lore raw

Re: [RFC 1/6] mm/migrate_pages: separate huge page and normal pages migration

Quoted 15 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying.

In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
Quoted 3 lines
---
 mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 9 deletions(-)

Maybe it would be better to have two subroutines for hugetlb migration and normal page migration respectively. migrate_pages() becomes very large at this point.

Quoted 105 lines
mm/migrate.c +64 -9
index 571d8c9fd5bc..117134f1c6dc 100644
@@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 
 	trace_mm_migrate_pages_start(mode, reason);
 
+	for (pass = 0; pass < 10 && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(page, page2, from, lru) {
+			nr_subpages = compound_nr(page);
+			cond_resched();
+
+			if (!PageHuge(page))
+				continue;
+
+			rc = unmap_and_move_huge_page(get_new_page,
+						put_new_page, private, page,
+						pass > 2, mode, reason,
+						&ret_pages);
+			/*
+			 * The rules are:
+			 *	Success: hugetlb page will be put back
+			 *	-EAGAIN: stay on the from list
+			 *	-ENOMEM: stay on the from list
+			 *	-ENOSYS: stay on the from list
+			 *	Other errno: put on ret_pages list then splice to
+			 *		     from list
+			 */
+			switch(rc) {
+			case -ENOSYS:
+				/* Hugetlb migration is unsupported */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				list_move_tail(&page->lru, &ret_pages);
+				break;
+			case -ENOMEM:
+				/*
+				 * When memory is low, don't bother to try to migrate
+				 * other pages, just exit.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages + nr_retry_pages;
+				goto out;
+			case -EAGAIN:
+				retry++;
+				nr_retry_pages += nr_subpages;
+				break;
+			case MIGRATEPAGE_SUCCESS:
+				nr_succeeded += nr_subpages;
+				break;
+			default:
+				/*
+				 * Permanent failure (-EBUSY, etc.):
+				 * unlike -EAGAIN case, the failed page is
+				 * removed from migration page list and not
+				 * retried in the next outer loop.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				break;
+			}
+		}
+	}
+	nr_failed += retry;
+	retry = 1;
 thp_subpage_migration:
 	for (pass = 0; pass < 10 && (retry || thp_retry); pass++) {
 		retry = 0;
@@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 			cond_resched();
 
 			if (PageHuge(page))
-				rc = unmap_and_move_huge_page(get_new_page,
-						put_new_page, private, page,
-						pass > 2, mode, reason,
-						&ret_pages);
-			else
-				rc = unmap_and_move(get_new_page, put_new_page,
+				continue;
+
+			rc = unmap_and_move(get_new_page, put_new_page,
 						private, page, pass > 2, mode,
 						reason, &ret_pages);
 			/*
 			 * The rules are:
-			 *	Success: non hugetlb page will be freed, hugetlb
-			 *		 page will be put back
+			 *	Success: page will be freed
 			 *	-EAGAIN: stay on the from list
 			 *	-ENOMEM: stay on the from list
 			 *	-ENOSYS: stay on the from list
@@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 						nr_thp_split++;
 						break;
 					}
-				/* Hugetlb migration is unsupported */
 				} else if (!no_subpage_counting) {
 					nr_failed++;
 				}
Signature · 1 line
2.35.1

--
Best Regards,
Yan, Zi

#3 lore raw

Re: [RFC 1/6] mm/migrate_pages: separate huge page and normal pages migration

Hi, Zi,

Thank you for comments!

Quoted 21 lines · Zi Yan
Quoted 15 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying.

In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
Quoted 3 lines
---
 mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 64 insertions(+), 9 deletions(-)
Quoted 3 lines

Maybe it would be better to have two subroutines for hugetlb migration and normal page migration respectively. migrate_pages() becomes very large at this point.

Yes. migrate_pages() becomes even larger with this patchset. I will consider more about how to deal with that. I will try the method in your comments in [3/6] for that too.

Best Regards,
Huang, Ying

Quoted 110 lines
Quoted 105 lines
mm/migrate.c +64 -9
index 571d8c9fd5bc..117134f1c6dc 100644
@@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 
 	trace_mm_migrate_pages_start(mode, reason);
 
+	for (pass = 0; pass < 10 && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(page, page2, from, lru) {
+			nr_subpages = compound_nr(page);
+			cond_resched();
+
+			if (!PageHuge(page))
+				continue;
+
+			rc = unmap_and_move_huge_page(get_new_page,
+						put_new_page, private, page,
+						pass > 2, mode, reason,
+						&ret_pages);
+			/*
+			 * The rules are:
+			 *	Success: hugetlb page will be put back
+			 *	-EAGAIN: stay on the from list
+			 *	-ENOMEM: stay on the from list
+			 *	-ENOSYS: stay on the from list
+			 *	Other errno: put on ret_pages list then splice to
+			 *		     from list
+			 */
+			switch(rc) {
+			case -ENOSYS:
+				/* Hugetlb migration is unsupported */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				list_move_tail(&page->lru, &ret_pages);
+				break;
+			case -ENOMEM:
+				/*
+				 * When memory is low, don't bother to try to migrate
+				 * other pages, just exit.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages + nr_retry_pages;
+				goto out;
+			case -EAGAIN:
+				retry++;
+				nr_retry_pages += nr_subpages;
+				break;
+			case MIGRATEPAGE_SUCCESS:
+				nr_succeeded += nr_subpages;
+				break;
+			default:
+				/*
+				 * Permanent failure (-EBUSY, etc.):
+				 * unlike -EAGAIN case, the failed page is
+				 * removed from migration page list and not
+				 * retried in the next outer loop.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				break;
+			}
+		}
+	}
+	nr_failed += retry;
+	retry = 1;
 thp_subpage_migration:
 	for (pass = 0; pass < 10 && (retry || thp_retry); pass++) {
 		retry = 0;
@@ -1431,18 +1491,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 			cond_resched();
 
 			if (PageHuge(page))
-				rc = unmap_and_move_huge_page(get_new_page,
-						put_new_page, private, page,
-						pass > 2, mode, reason,
-						&ret_pages);
-			else
-				rc = unmap_and_move(get_new_page, put_new_page,
+				continue;
+
+			rc = unmap_and_move(get_new_page, put_new_page,
 						private, page, pass > 2, mode,
 						reason, &ret_pages);
 			/*
 			 * The rules are:
-			 *	Success: non hugetlb page will be freed, hugetlb
-			 *		 page will be put back
+			 *	Success: page will be freed
 			 *	-EAGAIN: stay on the from list
 			 *	-ENOMEM: stay on the from list
 			 *	-ENOSYS: stay on the from list
@@ -1468,7 +1524,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 						nr_thp_split++;
 						break;
 					}
-				/* Hugetlb migration is unsupported */
 				} else if (!no_subpage_counting) {
 					nr_failed++;
 				}
Signature · 1 line
2.35.1

--
Best Regards,
Yan, Zi

#4 lore raw

Re: [RFC 1/6] mm/migrate_pages: separate huge page and normal pages migration

In reply to Huang Ying
Quoted 76 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying.

In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
  mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++-------
  1 file changed, 64 insertions(+), 9 deletions(-)
mm/migrate.c +49 -0
index 571d8c9fd5bc..117134f1c6dc 100644
@@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 
  	trace_mm_migrate_pages_start(mode, reason);
 
+	for (pass = 0; pass < 10 && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(page, page2, from, lru) {
+			nr_subpages = compound_nr(page);
+			cond_resched();
+
+			if (!PageHuge(page))
+				continue;
+
+			rc = unmap_and_move_huge_page(get_new_page,
+						put_new_page, private, page,
+						pass > 2, mode, reason,
+						&ret_pages);
+			/*
+			 * The rules are:
+			 *	Success: hugetlb page will be put back
+			 *	-EAGAIN: stay on the from list
+			 *	-ENOMEM: stay on the from list
+			 *	-ENOSYS: stay on the from list
+			 *	Other errno: put on ret_pages list then splice to
+			 *		     from list
+			 */
+			switch(rc) {
+			case -ENOSYS:
+				/* Hugetlb migration is unsupported */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				list_move_tail(&page->lru, &ret_pages);
+				break;
+			case -ENOMEM:
+				/*
+				 * When memory is low, don't bother to try to migrate
+				 * other pages, just exit.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages + nr_retry_pages;
+				goto out;
+			case -EAGAIN:
+				retry++;
+				nr_retry_pages += nr_subpages;
+				break;
+			case MIGRATEPAGE_SUCCESS:
+				nr_succeeded += nr_subpages;
+				break;
+			default:
+				/*
+				 * Permanent failure (-EBUSY, etc.):
+				 * unlike -EAGAIN case, the failed page is

mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1475

Quoted 10 lines
mm/migrate.c +10 -0
+				 * removed from migration page list and not
+				 * retried in the next outer loop.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				break;
+			}
+		}
+	}
+	nr_failed += retry;

Seems we should also record the nr_retry_pages? since the second loop will reset the nr_retry_pages.

nr_failed_pages += nr_retry_pages;

Besides, I also agree with Zi Yan's comment to simplify this larger function.

#5 lore raw

Re: [RFC 1/6] mm/migrate_pages: separate huge page and normal pages migration

Quoted 86 lines · Baolin Wang
Quoted 71 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THPs. Based on that we can batch the TLB shootdown during the page migration and make it possible to use some hardware accelerator for the page copying.
In this patch the huge page (PageHuge()) and normal page and THP migration is separated in migrate_pages() to make it easy to change the normal page and THP migration implementation.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
  mm/migrate.c | 73 +++++++++++++++++++++++++++++++++++++++++++++-------
  1 file changed, 64 insertions(+), 9 deletions(-)
mm/migrate.c +48 -0
index 571d8c9fd5bc..117134f1c6dc 100644
@@ -1414,6 +1414,66 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
    	trace_mm_migrate_pages_start(mode, reason);
  +	for (pass = 0; pass < 10 && retry; pass++) {
+		retry = 0;
+
+		list_for_each_entry_safe(page, page2, from, lru) {
+			nr_subpages = compound_nr(page);
+			cond_resched();
+
+			if (!PageHuge(page))
+				continue;
+
+			rc = unmap_and_move_huge_page(get_new_page,
+						put_new_page, private, page,
+						pass > 2, mode, reason,
+						&ret_pages);
+			/*
+			 * The rules are:
+			 *	Success: hugetlb page will be put back
+			 *	-EAGAIN: stay on the from list
+			 *	-ENOMEM: stay on the from list
+			 *	-ENOSYS: stay on the from list
+			 *	Other errno: put on ret_pages list then splice to
+			 *		     from list
+			 */
+			switch(rc) {
+			case -ENOSYS:
+				/* Hugetlb migration is unsupported */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				list_move_tail(&page->lru, &ret_pages);
+				break;
+			case -ENOMEM:
+				/*
+				 * When memory is low, don't bother to try to migrate
+				 * other pages, just exit.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages + nr_retry_pages;
+				goto out;
+			case -EAGAIN:
+				retry++;
+				nr_retry_pages += nr_subpages;
+				break;
+			case MIGRATEPAGE_SUCCESS:
+				nr_succeeded += nr_subpages;
+				break;
+			default:
+				/*
+				 * Permanent failure (-EBUSY, etc.):
+				 * unlike -EAGAIN case, the failed page is

mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1473

Quoted 10 lines
mm/migrate.c +10 -0
+				 * removed from migration page list and not
+				 * retried in the next outer loop.
+				 */
+				nr_failed++;
+				nr_failed_pages += nr_subpages;
+				break;
+			}
+		}
+	}
+	nr_failed += retry;

Seems we should also record the nr_retry_pages? since the second loop will reset the nr_retry_pages.

Quoted 1 line

nr_failed_pages += nr_retry_pages;

Good catch! Will do that in the next version.

Quoted 2 lines

Besides, I also agree with Zi Yan's comment to simplify this larger function.

Yes. I think so too.

Best Regards,
Huang, Ying

author
#6 lore raw

RFC 2/6 mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

In reply to Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +122 -42
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+				  int page_was_mapped,
+				  struct anon_vma *anon_vma)
+{
+	newpage->mapping = (struct address_space *)anon_vma;
+	newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,
+				   int *page_was_mappedp,
+				   struct anon_vma **anon_vmap)
+{
+	*anon_vmap = (struct anon_vma *)newpage->mapping;
+	*page_was_mappedp = newpage->private;
+	newpage->mapping = NULL;
+	newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP		1
+
+static int __migrate_page_unmap(struct page *page, struct page *newpage,
 				int force, enum migrate_mode mode)
 {
 	struct folio *folio = page_folio(page);
-	struct folio *dst = page_folio(newpage);
 	int rc = -EAGAIN;
-	bool page_was_mapped = false;
+	int page_was_mapped = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool is_lru = !__PageMovable(page);
 
@@ -1058,8 +1077,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		goto out_unlock;
 
 	if (unlikely(!is_lru)) {
-		rc = move_to_new_folio(dst, folio, mode);
-		goto out_unlock_both;
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return MIGRATEPAGE_UNMAP;
 	}
 
 	/*
@@ -1085,11 +1104,41 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
 				page);
 		try_to_migrate(folio, 0);
-		page_was_mapped = true;
+		page_was_mapped = 1;
+	}
+
+	if (!page_mapped(page)) {
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return MIGRATEPAGE_UNMAP;
 	}
 
-	if (!page_mapped(page))
-		rc = move_to_new_folio(dst, folio, mode);
+	if (page_was_mapped)
+		remove_migration_ptes(folio, folio, false);
+
+out_unlock_both:
+	unlock_page(newpage);
+out_unlock:
+	/* Drop an anon_vma reference if we took one */
+	if (anon_vma)
+		put_anon_vma(anon_vma);
+	unlock_page(page);
+out:
+
+	return rc;
+}
+
+static int __migrate_page_move(struct page *page, struct page *newpage,
+			       enum migrate_mode mode)
+{
+	struct folio *folio = page_folio(page);
+	struct folio *dst = page_folio(newpage);
+	int rc;
+	int page_was_mapped = 0;
+	struct anon_vma *anon_vma = NULL;
+
+	__migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+
+	rc = move_to_new_folio(dst, folio, mode);
 
 	/*
 	 * When successful, push newpage to LRU immediately: so that if it
@@ -1110,14 +1159,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		remove_migration_ptes(folio,
 			rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
 
-out_unlock_both:
 	unlock_page(newpage);
-out_unlock:
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
 		put_anon_vma(anon_vma);
 	unlock_page(page);
-out:
 	/*
 	 * If migration is successful, decrease refcount of the newpage,
 	 * which will not free the page because new page owner increased
@@ -1129,18 +1175,31 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 	return rc;
 }
 
-/*
- * Obtain the lock on page, remove all ptes and migrate the page
- * to the newly allocated page in newpage.
- */
-static int unmap_and_move(new_page_t get_new_page,
-				   free_page_t put_new_page,
-				   unsigned long private, struct page *page,
-				   int force, enum migrate_mode mode,
-				   enum migrate_reason reason,
-				   struct list_head *ret)
+static void migrate_page_done(struct page *page,
+			      enum migrate_reason reason)
+{
+	/*
+	 * Compaction can migrate also non-LRU pages which are
+	 * not accounted to NR_ISOLATED_*. They can be recognized
+	 * as __PageMovable
+	 */
+	if (likely(!__PageMovable(page)))
+		mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+				    page_is_file_lru(page), -thp_nr_pages(page));
+
+	if (reason != MR_MEMORY_FAILURE)
+		/* We release the page in page_handle_poison. */
+		put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+			      unsigned long private, struct page *page,
+			      struct page **newpagep, int force,
+			      enum migrate_mode mode, enum migrate_reason reason,
+			      struct list_head *ret)
 {
-	int rc = MIGRATEPAGE_SUCCESS;
+	int rc = MIGRATEPAGE_UNMAP;
 	struct page *newpage = NULL;
 
 	if (!thp_migration_supported() && PageTransHuge(page))
@@ -1151,19 +1210,48 @@ static int unmap_and_move(new_page_t get_new_page,
 		ClearPageActive(page);
 		ClearPageUnevictable(page);
 		/* free_pages_prepare() will clear PG_isolated. */
-		goto out;
+		list_del(&page->lru);
+		migrate_page_done(page, reason);
+		return MIGRATEPAGE_SUCCESS;
 	}
 
 	newpage = get_new_page(page, private);
 	if (!newpage)
 		return -ENOMEM;
+	*newpagep = newpage;
 
-	newpage->private = 0;
-	rc = __unmap_and_move(page, newpage, force, mode);
+	rc = __migrate_page_unmap(page, newpage, force, mode);
+	if (rc == MIGRATEPAGE_UNMAP)
+		return rc;
+
+	/*
+	 * A page that has not been migrated will have kept its
+	 * references and be restored.
+	 */
+	/* restore the page to right list. */
+	if (rc != -EAGAIN)
+		list_move_tail(&page->lru, ret);
+
+	if (put_new_page)
+		put_new_page(newpage, private);
+	else
+		put_page(newpage);
+
+	return rc;
+}
+
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+			     struct page *page, struct page *newpage,
+			     enum migrate_mode mode, enum migrate_reason reason,
+			     struct list_head *ret)
+{
+	int rc;
+
+	rc = __migrate_page_move(page, newpage, mode);
 	if (rc == MIGRATEPAGE_SUCCESS)
 		set_page_owner_migrate_reason(newpage, reason);
 
-out:
 	if (rc != -EAGAIN) {
 		/*
 		 * A page that has been migrated has all references
@@ -1179,20 +1267,7 @@ static int unmap_and_move(new_page_t get_new_page,
 	 * we want to retry.
 	 */
 	if (rc == MIGRATEPAGE_SUCCESS) {
-		/*
-		 * Compaction can migrate also non-LRU pages which are
-		 * not accounted to NR_ISOLATED_*. They can be recognized
-		 * as __PageMovable
-		 */
-		if (likely(!__PageMovable(page)))
-			mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-					page_is_file_lru(page), -thp_nr_pages(page));
-
-		if (reason != MR_MEMORY_FAILURE)
-			/*
-			 * We release the page in page_handle_poison.
-			 */
-			put_page(page);
+		migrate_page_done(page, reason);
 	} else {
 		if (rc != -EAGAIN)
 			list_add_tail(&page->lru, ret);
@@ -1405,6 +1480,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	int pass = 0;
 	bool is_thp = false;
 	struct page *page;
+	struct page *newpage = NULL;
 	struct page *page2;
 	int rc, nr_subpages;
 	LIST_HEAD(ret_pages);
@@ -1493,9 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 			if (PageHuge(page))
 				continue;
 
-			rc = unmap_and_move(get_new_page, put_new_page,
-						private, page, pass > 2, mode,
+			rc = migrate_page_unmap(get_new_page, put_new_page, private,
+						page, &newpage, pass > 2, mode,
 						reason, &ret_pages);
+			if (rc == MIGRATEPAGE_UNMAP)
+				rc = migrate_page_move(put_new_page, private,
+						       page, newpage, mode,
+						       reason, &ret_pages);
 			/*
 			 * The rules are:
 			 *	Success: page will be freed
Signature · 1 line
2.35.1
reviewer
#7 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 37 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +9 -1
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+				  int page_was_mapped,
+				  struct anon_vma *anon_vma)
+{
+	newpage->mapping = (struct address_space *)anon_vma;
+	newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 997

Quoted 10 lines
mm/migrate.c +10 -0
+				   int *page_was_mappedp,
+				   struct anon_vma **anon_vmap)
+{
+	*anon_vmap = (struct anon_vma *)newpage->mapping;
+	*page_was_mappedp = newpage->private;
+	newpage->mapping = NULL;
+	newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP		1

It is better to move this to migrate.h with MIGRATEPAGE_SUCCESS and make them an enum.

Quoted 227 lines
mm/migrate.c +103 -41
+
+static int __migrate_page_unmap(struct page *page, struct page *newpage,
 				int force, enum migrate_mode mode)
 {
 	struct folio *folio = page_folio(page);
-	struct folio *dst = page_folio(newpage);
 	int rc = -EAGAIN;
-	bool page_was_mapped = false;
+	int page_was_mapped = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool is_lru = !__PageMovable(page);
 
@@ -1058,8 +1077,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		goto out_unlock;
 
 	if (unlikely(!is_lru)) {
-		rc = move_to_new_folio(dst, folio, mode);
-		goto out_unlock_both;
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return MIGRATEPAGE_UNMAP;
 	}
 
 	/*
@@ -1085,11 +1104,41 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
 				page);
 		try_to_migrate(folio, 0);
-		page_was_mapped = true;
+		page_was_mapped = 1;
+	}
+
+	if (!page_mapped(page)) {
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return MIGRATEPAGE_UNMAP;
 	}
 
-	if (!page_mapped(page))
-		rc = move_to_new_folio(dst, folio, mode);
+	if (page_was_mapped)
+		remove_migration_ptes(folio, folio, false);
+
+out_unlock_both:
+	unlock_page(newpage);
+out_unlock:
+	/* Drop an anon_vma reference if we took one */
+	if (anon_vma)
+		put_anon_vma(anon_vma);
+	unlock_page(page);
+out:
+
+	return rc;
+}
+
+static int __migrate_page_move(struct page *page, struct page *newpage,
+			       enum migrate_mode mode)
+{
+	struct folio *folio = page_folio(page);
+	struct folio *dst = page_folio(newpage);
+	int rc;
+	int page_was_mapped = 0;
+	struct anon_vma *anon_vma = NULL;
+
+	__migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+
+	rc = move_to_new_folio(dst, folio, mode);
 
 	/*
 	 * When successful, push newpage to LRU immediately: so that if it
@@ -1110,14 +1159,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		remove_migration_ptes(folio,
 			rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
 
-out_unlock_both:
 	unlock_page(newpage);
-out_unlock:
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
 		put_anon_vma(anon_vma);
 	unlock_page(page);
-out:
 	/*
 	 * If migration is successful, decrease refcount of the newpage,
 	 * which will not free the page because new page owner increased
@@ -1129,18 +1175,31 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 	return rc;
 }
 
-/*
- * Obtain the lock on page, remove all ptes and migrate the page
- * to the newly allocated page in newpage.
- */
-static int unmap_and_move(new_page_t get_new_page,
-				   free_page_t put_new_page,
-				   unsigned long private, struct page *page,
-				   int force, enum migrate_mode mode,
-				   enum migrate_reason reason,
-				   struct list_head *ret)
+static void migrate_page_done(struct page *page,
+			      enum migrate_reason reason)
+{
+	/*
+	 * Compaction can migrate also non-LRU pages which are
+	 * not accounted to NR_ISOLATED_*. They can be recognized
+	 * as __PageMovable
+	 */
+	if (likely(!__PageMovable(page)))
+		mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+				    page_is_file_lru(page), -thp_nr_pages(page));
+
+	if (reason != MR_MEMORY_FAILURE)
+		/* We release the page in page_handle_poison. */
+		put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+			      unsigned long private, struct page *page,
+			      struct page **newpagep, int force,
+			      enum migrate_mode mode, enum migrate_reason reason,
+			      struct list_head *ret)
 {
-	int rc = MIGRATEPAGE_SUCCESS;
+	int rc = MIGRATEPAGE_UNMAP;
 	struct page *newpage = NULL;
 
 	if (!thp_migration_supported() && PageTransHuge(page))
@@ -1151,19 +1210,48 @@ static int unmap_and_move(new_page_t get_new_page,
 		ClearPageActive(page);
 		ClearPageUnevictable(page);
 		/* free_pages_prepare() will clear PG_isolated. */
-		goto out;
+		list_del(&page->lru);
+		migrate_page_done(page, reason);
+		return MIGRATEPAGE_SUCCESS;
 	}
 
 	newpage = get_new_page(page, private);
 	if (!newpage)
 		return -ENOMEM;
+	*newpagep = newpage;
 
-	newpage->private = 0;
-	rc = __unmap_and_move(page, newpage, force, mode);
+	rc = __migrate_page_unmap(page, newpage, force, mode);
+	if (rc == MIGRATEPAGE_UNMAP)
+		return rc;
+
+	/*
+	 * A page that has not been migrated will have kept its
+	 * references and be restored.
+	 */
+	/* restore the page to right list. */
+	if (rc != -EAGAIN)
+		list_move_tail(&page->lru, ret);
+
+	if (put_new_page)
+		put_new_page(newpage, private);
+	else
+		put_page(newpage);
+
+	return rc;
+}
+
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+			     struct page *page, struct page *newpage,
+			     enum migrate_mode mode, enum migrate_reason reason,
+			     struct list_head *ret)
+{
+	int rc;
+
+	rc = __migrate_page_move(page, newpage, mode);
 	if (rc == MIGRATEPAGE_SUCCESS)
 		set_page_owner_migrate_reason(newpage, reason);
 
-out:
 	if (rc != -EAGAIN) {
 		/*
 		 * A page that has been migrated has all references
@@ -1179,20 +1267,7 @@ static int unmap_and_move(new_page_t get_new_page,
 	 * we want to retry.
 	 */
 	if (rc == MIGRATEPAGE_SUCCESS) {
-		/*
-		 * Compaction can migrate also non-LRU pages which are
-		 * not accounted to NR_ISOLATED_*. They can be recognized
-		 * as __PageMovable
-		 */
-		if (likely(!__PageMovable(page)))
-			mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-					page_is_file_lru(page), -thp_nr_pages(page));
-
-		if (reason != MR_MEMORY_FAILURE)
-			/*
-			 * We release the page in page_handle_poison.
-			 */
-			put_page(page);
+		migrate_page_done(page, reason);
 	} else {
 		if (rc != -EAGAIN)
 			list_add_tail(&page->lru, ret);
@@ -1405,6 +1480,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	int pass = 0;
 	bool is_thp = false;
 	struct page *page;
+	struct page *newpage = NULL;
 	struct page *page2;
 	int rc, nr_subpages;
 	LIST_HEAD(ret_pages);
@@ -1493,9 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 			if (PageHuge(page))
 				continue;
 
-			rc = unmap_and_move(get_new_page, put_new_page,
-						private, page, pass > 2, mode,
+			rc = migrate_page_unmap(get_new_page, put_new_page, private,
+						page, &newpage, pass > 2, mode,
 						reason, &ret_pages);
+			if (rc == MIGRATEPAGE_UNMAP)
+				rc = migrate_page_move(put_new_page, private,
+						       page, newpage, mode,
+						       reason, &ret_pages);
 			/*
 			 * The rules are:
 			 *	Success: page will be freed
Signature · 1 line
2.35.1

--
Best Regards,
Yan, Zi

#8 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 44 lines · Zi Yan
Quoted 42 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +14 -1
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+				  int page_was_mapped,
+				  struct anon_vma *anon_vma)
+{
+	newpage->mapping = (struct address_space *)anon_vma;
+	newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,
+				   int *page_was_mappedp,
+				   struct anon_vma **anon_vmap)
+{
+	*anon_vmap = (struct anon_vma *)newpage->mapping;
+	*page_was_mappedp = newpage->private;
Quoted 8 lines

mm/migrate.c static int move_to_new_folio(struct folio *dst, struct folio *src, · line 997

Quoted 5 lines
mm/migrate.c +5 -0
+	newpage->mapping = NULL;
+	newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP		1

It is better to move this to migrate.h with MIGRATEPAGE_SUCCESS and make them an enum.

Make sense! Will do this in the next version.

Best Regards,
Huang, Ying

Reviewed-by on v1 2/6
#9 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

In reply to Huang Ying
Quoted 9 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Quoted 6 lines

---

Looks good to me.

#10 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

In reply to Huang Ying
Quoted 8 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

This looks like it could cause a deadlock between two threads migrating the same pages if force == true && mode != MIGRATE_ASYNC as migrate_page_unmap() will call lock_page() while holding the lock on other pages in the list. Therefore the two threads could deadlock if the pages are in a different order.

Quoted 263 lines
Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +122 -42
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
 	return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+				  int page_was_mapped,
+				  struct anon_vma *anon_vma)
+{
+	newpage->mapping = (struct address_space *)anon_vma;
+	newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,
+				   int *page_was_mappedp,
+				   struct anon_vma **anon_vmap)
+{
+	*anon_vmap = (struct anon_vma *)newpage->mapping;
+	*page_was_mappedp = newpage->private;
+	newpage->mapping = NULL;
+	newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP		1
+
+static int __migrate_page_unmap(struct page *page, struct page *newpage,
 				int force, enum migrate_mode mode)
 {
 	struct folio *folio = page_folio(page);
-	struct folio *dst = page_folio(newpage);
 	int rc = -EAGAIN;
-	bool page_was_mapped = false;
+	int page_was_mapped = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool is_lru = !__PageMovable(page);
 
@@ -1058,8 +1077,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		goto out_unlock;
 
 	if (unlikely(!is_lru)) {
-		rc = move_to_new_folio(dst, folio, mode);
-		goto out_unlock_both;
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return MIGRATEPAGE_UNMAP;
 	}
 
 	/*
@@ -1085,11 +1104,41 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
 				page);
 		try_to_migrate(folio, 0);
-		page_was_mapped = true;
+		page_was_mapped = 1;
+	}
+
+	if (!page_mapped(page)) {
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return MIGRATEPAGE_UNMAP;
 	}
 
-	if (!page_mapped(page))
-		rc = move_to_new_folio(dst, folio, mode);
+	if (page_was_mapped)
+		remove_migration_ptes(folio, folio, false);
+
+out_unlock_both:
+	unlock_page(newpage);
+out_unlock:
+	/* Drop an anon_vma reference if we took one */
+	if (anon_vma)
+		put_anon_vma(anon_vma);
+	unlock_page(page);
+out:
+
+	return rc;
+}
+
+static int __migrate_page_move(struct page *page, struct page *newpage,
+			       enum migrate_mode mode)
+{
+	struct folio *folio = page_folio(page);
+	struct folio *dst = page_folio(newpage);
+	int rc;
+	int page_was_mapped = 0;
+	struct anon_vma *anon_vma = NULL;
+
+	__migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+
+	rc = move_to_new_folio(dst, folio, mode);
 
 	/*
 	 * When successful, push newpage to LRU immediately: so that if it
@@ -1110,14 +1159,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 		remove_migration_ptes(folio,
 			rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
 
-out_unlock_both:
 	unlock_page(newpage);
-out_unlock:
 	/* Drop an anon_vma reference if we took one */
 	if (anon_vma)
 		put_anon_vma(anon_vma);
 	unlock_page(page);
-out:
 	/*
 	 * If migration is successful, decrease refcount of the newpage,
 	 * which will not free the page because new page owner increased
@@ -1129,18 +1175,31 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 	return rc;
 }
 
-/*
- * Obtain the lock on page, remove all ptes and migrate the page
- * to the newly allocated page in newpage.
- */
-static int unmap_and_move(new_page_t get_new_page,
-				   free_page_t put_new_page,
-				   unsigned long private, struct page *page,
-				   int force, enum migrate_mode mode,
-				   enum migrate_reason reason,
-				   struct list_head *ret)
+static void migrate_page_done(struct page *page,
+			      enum migrate_reason reason)
+{
+	/*
+	 * Compaction can migrate also non-LRU pages which are
+	 * not accounted to NR_ISOLATED_*. They can be recognized
+	 * as __PageMovable
+	 */
+	if (likely(!__PageMovable(page)))
+		mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+				    page_is_file_lru(page), -thp_nr_pages(page));
+
+	if (reason != MR_MEMORY_FAILURE)
+		/* We release the page in page_handle_poison. */
+		put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+			      unsigned long private, struct page *page,
+			      struct page **newpagep, int force,
+			      enum migrate_mode mode, enum migrate_reason reason,
+			      struct list_head *ret)
 {
-	int rc = MIGRATEPAGE_SUCCESS;
+	int rc = MIGRATEPAGE_UNMAP;
 	struct page *newpage = NULL;
 
 	if (!thp_migration_supported() && PageTransHuge(page))
@@ -1151,19 +1210,48 @@ static int unmap_and_move(new_page_t get_new_page,
 		ClearPageActive(page);
 		ClearPageUnevictable(page);
 		/* free_pages_prepare() will clear PG_isolated. */
-		goto out;
+		list_del(&page->lru);
+		migrate_page_done(page, reason);
+		return MIGRATEPAGE_SUCCESS;
 	}
 
 	newpage = get_new_page(page, private);
 	if (!newpage)
 		return -ENOMEM;
+	*newpagep = newpage;
 
-	newpage->private = 0;
-	rc = __unmap_and_move(page, newpage, force, mode);
+	rc = __migrate_page_unmap(page, newpage, force, mode);
+	if (rc == MIGRATEPAGE_UNMAP)
+		return rc;
+
+	/*
+	 * A page that has not been migrated will have kept its
+	 * references and be restored.
+	 */
+	/* restore the page to right list. */
+	if (rc != -EAGAIN)
+		list_move_tail(&page->lru, ret);
+
+	if (put_new_page)
+		put_new_page(newpage, private);
+	else
+		put_page(newpage);
+
+	return rc;
+}
+
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+			     struct page *page, struct page *newpage,
+			     enum migrate_mode mode, enum migrate_reason reason,
+			     struct list_head *ret)
+{
+	int rc;
+
+	rc = __migrate_page_move(page, newpage, mode);
 	if (rc == MIGRATEPAGE_SUCCESS)
 		set_page_owner_migrate_reason(newpage, reason);
 
-out:
 	if (rc != -EAGAIN) {
 		/*
 		 * A page that has been migrated has all references
@@ -1179,20 +1267,7 @@ static int unmap_and_move(new_page_t get_new_page,
 	 * we want to retry.
 	 */
 	if (rc == MIGRATEPAGE_SUCCESS) {
-		/*
-		 * Compaction can migrate also non-LRU pages which are
-		 * not accounted to NR_ISOLATED_*. They can be recognized
-		 * as __PageMovable
-		 */
-		if (likely(!__PageMovable(page)))
-			mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-					page_is_file_lru(page), -thp_nr_pages(page));
-
-		if (reason != MR_MEMORY_FAILURE)
-			/*
-			 * We release the page in page_handle_poison.
-			 */
-			put_page(page);
+		migrate_page_done(page, reason);
 	} else {
 		if (rc != -EAGAIN)
 			list_add_tail(&page->lru, ret);
@@ -1405,6 +1480,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	int pass = 0;
 	bool is_thp = false;
 	struct page *page;
+	struct page *newpage = NULL;
 	struct page *page2;
 	int rc, nr_subpages;
 	LIST_HEAD(ret_pages);
@@ -1493,9 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 			if (PageHuge(page))
 				continue;
 
-			rc = unmap_and_move(get_new_page, put_new_page,
-						private, page, pass > 2, mode,
+			rc = migrate_page_unmap(get_new_page, put_new_page, private,
+						page, &newpage, pass > 2, mode,
 						reason, &ret_pages);
+			if (rc == MIGRATEPAGE_UNMAP)
+				rc = migrate_page_move(put_new_page, private,
+						       page, newpage, mode,
+						       reason, &ret_pages);
 			/*
 			 * The rules are:
 			 *	Success: page will be freed
#11 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 13 lines · Alistair Popple
Quoted 8 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Quoted 5 lines

This looks like it could cause a deadlock between two threads migrating the same pages if force == true && mode != MIGRATE_ASYNC as migrate_page_unmap() will call lock_page() while holding the lock on other pages in the list. Therefore the two threads could deadlock if the pages are in a different order.

It seems unlikely to me since the page has to be isolated from lru before migration. The isolating from lru is atomic, so the two threads unlikely see the same pages on both lists.

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

Quoted 264 lines
Quoted 264 lines
Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +122 -42
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
      return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+                               int page_was_mapped,
+                               struct anon_vma *anon_vma)
+{
+     newpage->mapping = (struct address_space *)anon_vma;
+     newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,
+                                int *page_was_mappedp,
+                                struct anon_vma **anon_vmap)
+{
+     *anon_vmap = (struct anon_vma *)newpage->mapping;
+     *page_was_mappedp = newpage->private;
+     newpage->mapping = NULL;
+     newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP            1
+
+static int __migrate_page_unmap(struct page *page, struct page *newpage,
                              int force, enum migrate_mode mode)
 {
      struct folio *folio = page_folio(page);
-     struct folio *dst = page_folio(newpage);
      int rc = -EAGAIN;
-     bool page_was_mapped = false;
+     int page_was_mapped = 0;
      struct anon_vma *anon_vma = NULL;
      bool is_lru = !__PageMovable(page);
 
@@ -1058,8 +1077,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              goto out_unlock;
 
      if (unlikely(!is_lru)) {
-             rc = move_to_new_folio(dst, folio, mode);
-             goto out_unlock_both;
+             __migrate_page_record(newpage, page_was_mapped, anon_vma);
+             return MIGRATEPAGE_UNMAP;
      }
 
      /*
@@ -1085,11 +1104,41 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
                              page);
              try_to_migrate(folio, 0);
-             page_was_mapped = true;
+             page_was_mapped = 1;
+     }
+
+     if (!page_mapped(page)) {
+             __migrate_page_record(newpage, page_was_mapped, anon_vma);
+             return MIGRATEPAGE_UNMAP;
      }
 
-     if (!page_mapped(page))
-             rc = move_to_new_folio(dst, folio, mode);
+     if (page_was_mapped)
+             remove_migration_ptes(folio, folio, false);
+
+out_unlock_both:
+     unlock_page(newpage);
+out_unlock:
+     /* Drop an anon_vma reference if we took one */
+     if (anon_vma)
+             put_anon_vma(anon_vma);
+     unlock_page(page);
+out:
+
+     return rc;
+}
+
+static int __migrate_page_move(struct page *page, struct page *newpage,
+                            enum migrate_mode mode)
+{
+     struct folio *folio = page_folio(page);
+     struct folio *dst = page_folio(newpage);
+     int rc;
+     int page_was_mapped = 0;
+     struct anon_vma *anon_vma = NULL;
+
+     __migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+
+     rc = move_to_new_folio(dst, folio, mode);
 
      /*
       * When successful, push newpage to LRU immediately: so that if it
@@ -1110,14 +1159,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              remove_migration_ptes(folio,
                      rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
 
-out_unlock_both:
      unlock_page(newpage);
-out_unlock:
      /* Drop an anon_vma reference if we took one */
      if (anon_vma)
              put_anon_vma(anon_vma);
      unlock_page(page);
-out:
      /*
       * If migration is successful, decrease refcount of the newpage,
       * which will not free the page because new page owner increased
@@ -1129,18 +1175,31 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
      return rc;
 }
 
-/*
- * Obtain the lock on page, remove all ptes and migrate the page
- * to the newly allocated page in newpage.
- */
-static int unmap_and_move(new_page_t get_new_page,
-                                free_page_t put_new_page,
-                                unsigned long private, struct page *page,
-                                int force, enum migrate_mode mode,
-                                enum migrate_reason reason,
-                                struct list_head *ret)
+static void migrate_page_done(struct page *page,
+                           enum migrate_reason reason)
+{
+     /*
+      * Compaction can migrate also non-LRU pages which are
+      * not accounted to NR_ISOLATED_*. They can be recognized
+      * as __PageMovable
+      */
+     if (likely(!__PageMovable(page)))
+             mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+                                 page_is_file_lru(page), -thp_nr_pages(page));
+
+     if (reason != MR_MEMORY_FAILURE)
+             /* We release the page in page_handle_poison. */
+             put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+                           unsigned long private, struct page *page,
+                           struct page **newpagep, int force,
+                           enum migrate_mode mode, enum migrate_reason reason,
+                           struct list_head *ret)
 {
-     int rc = MIGRATEPAGE_SUCCESS;
+     int rc = MIGRATEPAGE_UNMAP;
      struct page *newpage = NULL;
 
      if (!thp_migration_supported() && PageTransHuge(page))
@@ -1151,19 +1210,48 @@ static int unmap_and_move(new_page_t get_new_page,
              ClearPageActive(page);
              ClearPageUnevictable(page);
              /* free_pages_prepare() will clear PG_isolated. */
-             goto out;
+             list_del(&page->lru);
+             migrate_page_done(page, reason);
+             return MIGRATEPAGE_SUCCESS;
      }
 
      newpage = get_new_page(page, private);
      if (!newpage)
              return -ENOMEM;
+     *newpagep = newpage;
 
-     newpage->private = 0;
-     rc = __unmap_and_move(page, newpage, force, mode);
+     rc = __migrate_page_unmap(page, newpage, force, mode);
+     if (rc == MIGRATEPAGE_UNMAP)
+             return rc;
+
+     /*
+      * A page that has not been migrated will have kept its
+      * references and be restored.
+      */
+     /* restore the page to right list. */
+     if (rc != -EAGAIN)
+             list_move_tail(&page->lru, ret);
+
+     if (put_new_page)
+             put_new_page(newpage, private);
+     else
+             put_page(newpage);
+
+     return rc;
+}
+
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+                          struct page *page, struct page *newpage,
+                          enum migrate_mode mode, enum migrate_reason reason,
+                          struct list_head *ret)
+{
+     int rc;
+
+     rc = __migrate_page_move(page, newpage, mode);
      if (rc == MIGRATEPAGE_SUCCESS)
              set_page_owner_migrate_reason(newpage, reason);
 
-out:
      if (rc != -EAGAIN) {
              /*
               * A page that has been migrated has all references
@@ -1179,20 +1267,7 @@ static int unmap_and_move(new_page_t get_new_page,
       * we want to retry.
       */
      if (rc == MIGRATEPAGE_SUCCESS) {
-             /*
-              * Compaction can migrate also non-LRU pages which are
-              * not accounted to NR_ISOLATED_*. They can be recognized
-              * as __PageMovable
-              */
-             if (likely(!__PageMovable(page)))
-                     mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-                                     page_is_file_lru(page), -thp_nr_pages(page));
-
-             if (reason != MR_MEMORY_FAILURE)
-                     /*
-                      * We release the page in page_handle_poison.
-                      */
-                     put_page(page);
+             migrate_page_done(page, reason);
      } else {
              if (rc != -EAGAIN)
                      list_add_tail(&page->lru, ret);
@@ -1405,6 +1480,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
      int pass = 0;
      bool is_thp = false;
      struct page *page;
+     struct page *newpage = NULL;
      struct page *page2;
      int rc, nr_subpages;
      LIST_HEAD(ret_pages);
@@ -1493,9 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
                      if (PageHuge(page))
                              continue;
 
-                     rc = unmap_and_move(get_new_page, put_new_page,
-                                             private, page, pass > 2, mode,
+                     rc = migrate_page_unmap(get_new_page, put_new_page, private,
+                                             page, &newpage, pass > 2, mode,
                                              reason, &ret_pages);
+                     if (rc == MIGRATEPAGE_UNMAP)
+                             rc = migrate_page_move(put_new_page, private,
+                                                    page, newpage, mode,
+                                                    reason, &ret_pages);
                      /*
                       * The rules are:
                       *      Success: page will be freed
#12 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 20 lines · Yang Shi
Quoted 13 lines · Alistair Popple
Quoted 8 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Quoted 5 lines

This looks like it could cause a deadlock between two threads migrating the same pages if force == true && mode != MIGRATE_ASYNC as migrate_page_unmap() will call lock_page() while holding the lock on other pages in the list. Therefore the two threads could deadlock if the pages are in a different order.

Quoted 3 lines

It seems unlikely to me since the page has to be isolated from lru before migration. The isolating from lru is atomic, so the two threads unlikely see the same pages on both lists.

Oh thanks! That is a good point and I agree since lru isolation is atomic the two threads won't see the same pages. migrate_vma_setup() does LRU isolation after locking the page which is why the potential exists there. We could potentially switch that around but given ZONE_DEVICE pages aren't on an lru it wouldn't help much.

Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Quoted 264 lines
Quoted 264 lines
Quoted 264 lines
Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +122 -42
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
      return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+                               int page_was_mapped,
+                               struct anon_vma *anon_vma)
+{
+     newpage->mapping = (struct address_space *)anon_vma;
+     newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,
+                                int *page_was_mappedp,
+                                struct anon_vma **anon_vmap)
+{
+     *anon_vmap = (struct anon_vma *)newpage->mapping;
+     *page_was_mappedp = newpage->private;
+     newpage->mapping = NULL;
+     newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP            1
+
+static int __migrate_page_unmap(struct page *page, struct page *newpage,
                              int force, enum migrate_mode mode)
 {
      struct folio *folio = page_folio(page);
-     struct folio *dst = page_folio(newpage);
      int rc = -EAGAIN;
-     bool page_was_mapped = false;
+     int page_was_mapped = 0;
      struct anon_vma *anon_vma = NULL;
      bool is_lru = !__PageMovable(page);
 
@@ -1058,8 +1077,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              goto out_unlock;
 
      if (unlikely(!is_lru)) {
-             rc = move_to_new_folio(dst, folio, mode);
-             goto out_unlock_both;
+             __migrate_page_record(newpage, page_was_mapped, anon_vma);
+             return MIGRATEPAGE_UNMAP;
      }
 
      /*
@@ -1085,11 +1104,41 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
                              page);
              try_to_migrate(folio, 0);
-             page_was_mapped = true;
+             page_was_mapped = 1;
+     }
+
+     if (!page_mapped(page)) {
+             __migrate_page_record(newpage, page_was_mapped, anon_vma);
+             return MIGRATEPAGE_UNMAP;
      }
 
-     if (!page_mapped(page))
-             rc = move_to_new_folio(dst, folio, mode);
+     if (page_was_mapped)
+             remove_migration_ptes(folio, folio, false);
+
+out_unlock_both:
+     unlock_page(newpage);
+out_unlock:
+     /* Drop an anon_vma reference if we took one */
+     if (anon_vma)
+             put_anon_vma(anon_vma);
+     unlock_page(page);
+out:
+
+     return rc;
+}
+
+static int __migrate_page_move(struct page *page, struct page *newpage,
+                            enum migrate_mode mode)
+{
+     struct folio *folio = page_folio(page);
+     struct folio *dst = page_folio(newpage);
+     int rc;
+     int page_was_mapped = 0;
+     struct anon_vma *anon_vma = NULL;
+
+     __migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+
+     rc = move_to_new_folio(dst, folio, mode);
 
      /*
       * When successful, push newpage to LRU immediately: so that if it
@@ -1110,14 +1159,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              remove_migration_ptes(folio,
                      rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
 
-out_unlock_both:
      unlock_page(newpage);
-out_unlock:
      /* Drop an anon_vma reference if we took one */
      if (anon_vma)
              put_anon_vma(anon_vma);
      unlock_page(page);
-out:
      /*
       * If migration is successful, decrease refcount of the newpage,
       * which will not free the page because new page owner increased
@@ -1129,18 +1175,31 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
      return rc;
 }
 
-/*
- * Obtain the lock on page, remove all ptes and migrate the page
- * to the newly allocated page in newpage.
- */
-static int unmap_and_move(new_page_t get_new_page,
-                                free_page_t put_new_page,
-                                unsigned long private, struct page *page,
-                                int force, enum migrate_mode mode,
-                                enum migrate_reason reason,
-                                struct list_head *ret)
+static void migrate_page_done(struct page *page,
+                           enum migrate_reason reason)
+{
+     /*
+      * Compaction can migrate also non-LRU pages which are
+      * not accounted to NR_ISOLATED_*. They can be recognized
+      * as __PageMovable
+      */
+     if (likely(!__PageMovable(page)))
+             mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+                                 page_is_file_lru(page), -thp_nr_pages(page));
+
+     if (reason != MR_MEMORY_FAILURE)
+             /* We release the page in page_handle_poison. */
+             put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+                           unsigned long private, struct page *page,
+                           struct page **newpagep, int force,
+                           enum migrate_mode mode, enum migrate_reason reason,
+                           struct list_head *ret)
 {
-     int rc = MIGRATEPAGE_SUCCESS;
+     int rc = MIGRATEPAGE_UNMAP;
      struct page *newpage = NULL;
 
      if (!thp_migration_supported() && PageTransHuge(page))
@@ -1151,19 +1210,48 @@ static int unmap_and_move(new_page_t get_new_page,
              ClearPageActive(page);
              ClearPageUnevictable(page);
              /* free_pages_prepare() will clear PG_isolated. */
-             goto out;
+             list_del(&page->lru);
+             migrate_page_done(page, reason);
+             return MIGRATEPAGE_SUCCESS;
      }
 
      newpage = get_new_page(page, private);
      if (!newpage)
              return -ENOMEM;
+     *newpagep = newpage;
 
-     newpage->private = 0;
-     rc = __unmap_and_move(page, newpage, force, mode);
+     rc = __migrate_page_unmap(page, newpage, force, mode);
+     if (rc == MIGRATEPAGE_UNMAP)
+             return rc;
+
+     /*
+      * A page that has not been migrated will have kept its
+      * references and be restored.
+      */
+     /* restore the page to right list. */
+     if (rc != -EAGAIN)
+             list_move_tail(&page->lru, ret);
+
+     if (put_new_page)
+             put_new_page(newpage, private);
+     else
+             put_page(newpage);
+
+     return rc;
+}
+
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+                          struct page *page, struct page *newpage,
+                          enum migrate_mode mode, enum migrate_reason reason,
+                          struct list_head *ret)
+{
+     int rc;
+
+     rc = __migrate_page_move(page, newpage, mode);
      if (rc == MIGRATEPAGE_SUCCESS)
              set_page_owner_migrate_reason(newpage, reason);
 
-out:
      if (rc != -EAGAIN) {
              /*
               * A page that has been migrated has all references
@@ -1179,20 +1267,7 @@ static int unmap_and_move(new_page_t get_new_page,
       * we want to retry.
       */
      if (rc == MIGRATEPAGE_SUCCESS) {
-             /*
-              * Compaction can migrate also non-LRU pages which are
-              * not accounted to NR_ISOLATED_*. They can be recognized
-              * as __PageMovable
-              */
-             if (likely(!__PageMovable(page)))
-                     mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-                                     page_is_file_lru(page), -thp_nr_pages(page));
-
-             if (reason != MR_MEMORY_FAILURE)
-                     /*
-                      * We release the page in page_handle_poison.
-                      */
-                     put_page(page);
+             migrate_page_done(page, reason);
      } else {
              if (rc != -EAGAIN)
                      list_add_tail(&page->lru, ret);
@@ -1405,6 +1480,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
      int pass = 0;
      bool is_thp = false;
      struct page *page;
+     struct page *newpage = NULL;
      struct page *page2;
      int rc, nr_subpages;
      LIST_HEAD(ret_pages);
@@ -1493,9 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
                      if (PageHuge(page))
                              continue;
 
-                     rc = unmap_and_move(get_new_page, put_new_page,
-                                             private, page, pass > 2, mode,
+                     rc = migrate_page_unmap(get_new_page, put_new_page, private,
+                                             page, &newpage, pass > 2, mode,
                                              reason, &ret_pages);
+                     if (rc == MIGRATEPAGE_UNMAP)
+                             rc = migrate_page_move(put_new_page, private,
+                                                    page, newpage, mode,
+                                                    reason, &ret_pages);
                      /*
                       * The rules are:
                       *      Success: page will be freed
#13 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 39 lines · Alistair Popple
Quoted 20 lines · Yang Shi
Quoted 13 lines · Alistair Popple
Quoted 8 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Quoted 5 lines

This looks like it could cause a deadlock between two threads migrating the same pages if force == true && mode != MIGRATE_ASYNC as migrate_page_unmap() will call lock_page() while holding the lock on other pages in the list. Therefore the two threads could deadlock if the pages are in a different order.

Quoted 3 lines

It seems unlikely to me since the page has to be isolated from lru before migration. The isolating from lru is atomic, so the two threads unlikely see the same pages on both lists.

Oh thanks! That is a good point and I agree since lru isolation is atomic the two threads won't see the same pages. migrate_vma_setup() does LRU isolation after locking the page which is why the potential exists there. We could potentially switch that around but given ZONE_DEVICE pages aren't on an lru it wouldn't help much.

Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

Quoted 2 lines

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Thank you very much for pointing this out. I think the deadlock is a real issue. Anyway, we shouldn't forbid other places in kernel to lock 2 pages at the same time.

The simplest solution is to batch page migration only if mode == MIGRATE_ASYNC. Then we may consider to fall back to non-batch mode if mode != MIGRATE_ASYNC and trylock page fails.

Best Regards,
Huang, Ying

[snip]

#14 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 7 lines · Ying
Quoted 7 lines
Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

Quoted 7 lines
Quoted 2 lines

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Thank you very much for pointing this out. I think the deadlock is a real issue. Anyway, we shouldn't forbid other places in kernel to lock 2 pages at the same time.

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

The only deadlock prevention convention that I see is the convention of locking the pages in order of ascending address. That only helps if everything does it that way, and migrate code definitely does not. However...I thought that up until now, at least, the migrate code relied on trylock (which can fail, and so migration can fail, too), to avoid deadlock. Is that changing somehow, I didn't see it?

[1] https://elixir.bootlin.com/linux/latest/source/mm/page-writeback.c#L2296

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
Quoted 4 lines

The simplest solution is to batch page migration only if mode == MIGRATE_ASYNC. Then we may consider to fall back to non-batch mode if mode != MIGRATE_ASYNC and trylock page fails.

#15 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 22 lines · John Hubbard
Quoted 7 lines · Ying
Quoted 7 lines
Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

Quoted 7 lines
Quoted 2 lines

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Thank you very much for pointing this out. I think the deadlock is a real issue. Anyway, we shouldn't forbid other places in kernel to lock 2 pages at the same time.

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

Quoted 6 lines

The only deadlock prevention convention that I see is the convention of locking the pages in order of ascending address. That only helps if everything does it that way, and migrate code definitely does not. However...I thought that up until now, at least, the migrate code relied on trylock (which can fail, and so migration can fail, too), to avoid deadlock. Is that changing somehow, I didn't see it?

The trylock is used by async mode which does try to avoid blocking. But sync mode does use lock. The current implementation of migration does migrate one page at a time, so it is not a problem.

Quoted 16 lines

[1] https://elixir.bootlin.com/linux/latest/source/mm/page-writeback.c#L2296

thanks,

--
John Hubbard
NVIDIA

Quoted 4 lines

The simplest solution is to batch page migration only if mode == MIGRATE_ASYNC. Then we may consider to fall back to non-batch mode if mode != MIGRATE_ASYNC and trylock page fails.

#16 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 18 lines · Yang Shi
Quoted 17 lines · John Hubbard
Quoted 14 lines · Ying
Quoted 9 lines
Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Thank you very much for pointing this out. I think the deadlock is a real issue. Anyway, we shouldn't forbid other places in kernel to lock 2 pages at the same time.

Quoted 4 lines
Quoted 4 lines

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

That's not really the case though. The inner loop of write_cache_page() only ever locks one page at a time, either directly via the unlock_page() on L2338 (those goto's are amazing) or indirectly via (*writepage)() on L2359.

So there's no deadlock potential there because unlocking any previously locked page(s) doesn't depend on obtaining the lock for another page. Unless I've missed something?

Quoted 27 lines
Quoted 6 lines

The only deadlock prevention convention that I see is the convention of locking the pages in order of ascending address. That only helps if everything does it that way, and migrate code definitely does not. However...I thought that up until now, at least, the migrate code relied on trylock (which can fail, and so migration can fail, too), to avoid deadlock. Is that changing somehow, I didn't see it?

The trylock is used by async mode which does try to avoid blocking. But sync mode does use lock. The current implementation of migration does migrate one page at a time, so it is not a problem.

Quoted 16 lines

[1] https://elixir.bootlin.com/linux/latest/source/mm/page-writeback.c#L2296

thanks,

--
John Hubbard
NVIDIA

Quoted 4 lines

The simplest solution is to batch page migration only if mode == MIGRATE_ASYNC. Then we may consider to fall back to non-batch mode if mode != MIGRATE_ASYNC and trylock page fails.

#17 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 25 lines · Alistair Popple
Quoted 18 lines · Yang Shi
Quoted 17 lines · John Hubbard
Quoted 14 lines · Ying
Quoted 9 lines
Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Thank you very much for pointing this out. I think the deadlock is a real issue. Anyway, we shouldn't forbid other places in kernel to lock 2 pages at the same time.

Quoted 4 lines
Quoted 4 lines

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

Quoted 8 lines

That's not really the case though. The inner loop of write_cache_page() only ever locks one page at a time, either directly via the unlock_page() on L2338 (those goto's are amazing) or indirectly via (*writepage)() on L2359.

So there's no deadlock potential there because unlocking any previously locked page(s) doesn't depend on obtaining the lock for another page. Unless I've missed something?

Yes. This is my understanding too after checking ext4_writepage().

Best Regards,
Huang, Ying

Quoted 27 lines
Quoted 27 lines
Quoted 6 lines

The only deadlock prevention convention that I see is the convention of locking the pages in order of ascending address. That only helps if everything does it that way, and migrate code definitely does not. However...I thought that up until now, at least, the migrate code relied on trylock (which can fail, and so migration can fail, too), to avoid deadlock. Is that changing somehow, I didn't see it?

The trylock is used by async mode which does try to avoid blocking. But sync mode does use lock. The current implementation of migration does migrate one page at a time, so it is not a problem.

Quoted 16 lines

[1] https://elixir.bootlin.com/linux/latest/source/mm/page-writeback.c#L2296

thanks,

--
John Hubbard
NVIDIA

Quoted 4 lines

The simplest solution is to batch page migration only if mode == MIGRATE_ASYNC. Then we may consider to fall back to non-batch mode if mode != MIGRATE_ASYNC and trylock page fails.

#18 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 10 lines · Ying
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

That's not really the case though. The inner loop of write_cache_page() only ever locks one page at a time, either directly via the unlock_page() on L2338 (those goto's are amazing) or indirectly via (*writepage)() on L2359.

Quoted 6 lines
Quoted 3 lines

So there's no deadlock potential there because unlocking any previously locked page(s) doesn't depend on obtaining the lock for another page. Unless I've missed something?

Yes. This is my understanding too after checking ext4_writepage().

Yes, I missed the ".writepage() shall unlock the page" design point. Now it seems much more reasonable and safer. :)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#19 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 16 lines · John Hubbard
Quoted 14 lines · Ying
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

That's not really the case though. The inner loop of write_cache_page() only ever locks one page at a time, either directly via the unlock_page() on L2338 (those goto's are amazing) or indirectly via (*writepage)() on L2359.

So there's no deadlock potential there because unlocking any previously locked page(s) doesn't depend on obtaining the lock for another page. Unless I've missed something?

Quoted 5 lines
Quoted 2 lines

Yes. This is my understanding too after checking ext4_writepage().

Yes, I missed the ".writepage() shall unlock the page" design point. Now it seems much more reasonable and safer. :)

.writepage is deprecated (see
https://lore.kernel.org/linux-fsdevel/[email protected]/),
write back actually uses .writepages.

Quoted 7 lines

thanks,

--
John Hubbard
NVIDIA

#20 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 20 lines · Yang Shi
Quoted 19 lines · John Hubbard
Quoted 16 lines · Ying
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

Quoted 8 lines

That's not really the case though. The inner loop of write_cache_page() only ever locks one page at a time, either directly via the unlock_page() on L2338 (those goto's are amazing) or indirectly via (*writepage)() on L2359.

So there's no deadlock potential there because unlocking any previously locked page(s) doesn't depend on obtaining the lock for another page. Unless I've missed something?

Yes. This is my understanding too after checking ext4_writepage().

Quoted 6 lines
Quoted 2 lines

Yes, I missed the ".writepage() shall unlock the page" design point. Now it seems much more reasonable and safer. :)

.writepage is deprecated (see
https://lore.kernel.org/linux-fsdevel/[email protected]/),
write back actually uses .writepages.

write_cache_pages() seems to directly call it, though:

generic_writepages()
write_cache_pages(__writepage)

    __writepage()
      mapping->a_ops->writepage(page, wbc)

So it seems like it's still alive and well. And in any case, it is definitely passing one page at a time from write_cache_pages(), right?

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#21 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 31 lines · John Hubbard
Quoted 20 lines · Yang Shi
Quoted 19 lines · John Hubbard
Quoted 16 lines · Ying
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

I also agree that we cannot make any rules such as "do not lock > 1 page at the same time, elsewhere in the kernel", because it is already happening, for example in page-writeback.c, which locks PAGEVEC_SIZE
(15) pages per batch [1].

Quoted 8 lines

That's not really the case though. The inner loop of write_cache_page() only ever locks one page at a time, either directly via the unlock_page() on L2338 (those goto's are amazing) or indirectly via (*writepage)() on L2359.

So there's no deadlock potential there because unlocking any previously locked page(s) doesn't depend on obtaining the lock for another page. Unless I've missed something?

Yes. This is my understanding too after checking ext4_writepage().

Quoted 6 lines
Quoted 2 lines

Yes, I missed the ".writepage() shall unlock the page" design point. Now it seems much more reasonable and safer. :)

.writepage is deprecated (see
https://lore.kernel.org/linux-fsdevel/[email protected]/),
write back actually uses .writepages.

write_cache_pages() seems to directly call it, though:

Quoted 7 lines

generic_writepages()
write_cache_pages(__writepage)

    __writepage()
      mapping->a_ops->writepage(page, wbc)

So it seems like it's still alive and well. And in any case, it is definitely passing one page at a time from write_cache_pages(), right?

IIRC, the writeback may not call generic_writepages. On my ext4 filesystem, the writeback call stack looks like:

@[

    ext4_writepages+1
    do_writepages+191
    __writeback_single_inode+65
    writeback_sb_inodes+477
    __writeback_inodes_wb+76
    wb_writeback+457
    wb_workfn+680
    process_one_work+485
    worker_thread+80
    kthread+231
    ret_from_fork+34

]: 2

Quoted 8 lines

thanks,

--
John Hubbard
NVIDIA

#22 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 15 lines · Yang Shi

IIRC, the writeback may not call generic_writepages. On my ext4 filesystem, the writeback call stack looks like:

@[

    ext4_writepages+1
    do_writepages+191
    __writeback_single_inode+65
    writeback_sb_inodes+477
    __writeback_inodes_wb+76
    wb_writeback+457
    wb_workfn+680
    process_one_work+485
    worker_thread+80
    kthread+231
    ret_from_fork+34
Quoted 2 lines

]: 2

Sure, that's fine for ext4, in that particular case, but

a) not all filesystems have .writepages hooked up. That's why do_writepages() checks for .writepages(), and falls back to .writepage():

	if (mapping->a_ops->writepages)
		ret = mapping->a_ops->writepages(mapping, wbc);
	else
		ret = generic_writepages(mapping, wbc);

, and

b) there are also a lot of places and ways to invoke writebacks. There are periodic writebacks, and there are data integrity (WB_SYNC_ALL) writebacks, and other places where a page needs to be cleaned--so, a lot of call sites. Some of which will land on a .writepage() sometimes, even now.

For just one example, I see migrate.c's writeout() function directly calling writepage():

	rc = mapping->a_ops->writepage(&folio->page, &wbc);

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#23 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 39 lines · John Hubbard
Quoted 15 lines · Yang Shi

IIRC, the writeback may not call generic_writepages. On my ext4 filesystem, the writeback call stack looks like:

@[

    ext4_writepages+1
    do_writepages+191
    __writeback_single_inode+65
    writeback_sb_inodes+477
    __writeback_inodes_wb+76
    wb_writeback+457
    wb_workfn+680
    process_one_work+485
    worker_thread+80
    kthread+231
    ret_from_fork+34
Quoted 2 lines

]: 2

Sure, that's fine for ext4, in that particular case, but

a) not all filesystems have .writepages hooked up. That's why do_writepages() checks for .writepages(), and falls back to .writepage():

        if (mapping->a_ops->writepages)
                ret = mapping->a_ops->writepages(mapping, wbc);
        else
                ret = generic_writepages(mapping, wbc);

, and

b) there are also a lot of places and ways to invoke writebacks. There are periodic writebacks, and there are data integrity (WB_SYNC_ALL) writebacks, and other places where a page needs to be cleaned--so, a lot of call sites. Some of which will land on a .writepage() sometimes, even now.

Quoted 4 lines

For just one example, I see migrate.c's writeout() function directly calling writepage():

        rc = mapping->a_ops->writepage(&folio->page, &wbc);

Thanks, John. You are right. I think "deprecated" is inaccurate, .writepage is actually no longer used in memory reclaim context, but it is still used for other contexts.

Anyway I think what we tried to figure out in the first place is whether it is possible that filesystem writeback have dead lock with the new batch migration or not. I think the conclusion didn't change.

Quoted 8 lines

thanks,

--
John Hubbard
NVIDIA

#24 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

Quoted 41 lines · Yang Shi
Quoted 39 lines · John Hubbard
Quoted 15 lines · Yang Shi

IIRC, the writeback may not call generic_writepages. On my ext4 filesystem, the writeback call stack looks like:

@[

    ext4_writepages+1
    do_writepages+191
    __writeback_single_inode+65
    writeback_sb_inodes+477
    __writeback_inodes_wb+76
    wb_writeback+457
    wb_workfn+680
    process_one_work+485
    worker_thread+80
    kthread+231
    ret_from_fork+34
Quoted 2 lines

]: 2

Sure, that's fine for ext4, in that particular case, but

a) not all filesystems have .writepages hooked up. That's why do_writepages() checks for .writepages(), and falls back to .writepage():

        if (mapping->a_ops->writepages)
                ret = mapping->a_ops->writepages(mapping, wbc);
        else
                ret = generic_writepages(mapping, wbc);

, and

b) there are also a lot of places and ways to invoke writebacks. There are periodic writebacks, and there are data integrity (WB_SYNC_ALL) writebacks, and other places where a page needs to be cleaned--so, a lot of call sites. Some of which will land on a .writepage() sometimes, even now.

Quoted 8 lines
Quoted 4 lines

For just one example, I see migrate.c's writeout() function directly calling writepage():

        rc = mapping->a_ops->writepage(&folio->page, &wbc);

Thanks, John. You are right. I think "deprecated" is inaccurate, .writepage is actually no longer used in memory reclaim context, but it is still used for other contexts.

Hmm.. it is definitely used currently, but it seems like the plan is to deprecate ->writepage finally IIUC. See
https://lore.kernel.org/linux-mm/[email protected]/

Quoted 13 lines

Anyway I think what we tried to figure out in the first place is whether it is possible that filesystem writeback have dead lock with the new batch migration or not. I think the conclusion didn't change.

Quoted 8 lines

thanks,

--
John Hubbard
NVIDIA

#25 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

In reply to Huang, Ying
Quoted 45 lines · Ying
Quoted 39 lines · Alistair Popple
Quoted 20 lines · Yang Shi
Quoted 13 lines · Alistair Popple
Quoted 8 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Quoted 5 lines

This looks like it could cause a deadlock between two threads migrating the same pages if force == true && mode != MIGRATE_ASYNC as migrate_page_unmap() will call lock_page() while holding the lock on other pages in the list. Therefore the two threads could deadlock if the pages are in a different order.

Quoted 3 lines

It seems unlikely to me since the page has to be isolated from lru before migration. The isolating from lru is atomic, so the two threads unlikely see the same pages on both lists.

Oh thanks! That is a good point and I agree since lru isolation is atomic the two threads won't see the same pages. migrate_vma_setup() does LRU isolation after locking the page which is why the potential exists there. We could potentially switch that around but given ZONE_DEVICE pages aren't on an lru it wouldn't help much.

Quoted 6 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

Quoted 2 lines

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

Quoted 7 lines

Thank you very much for pointing this out. I think the deadlock is a real issue. Anyway, we shouldn't forbid other places in kernel to lock 2 pages at the same time.

The simplest solution is to batch page migration only if mode == MIGRATE_ASYNC. Then we may consider to fall back to non-batch mode if mode != MIGRATE_ASYNC and trylock page fails.

Seems like so...

Quoted 5 lines

Best Regards,
Huang, Ying

[snip]

#26 lore raw

Re: [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move()

In reply to Alistair Popple
Quoted 28 lines · Alistair Popple
Quoted 20 lines · Yang Shi
Quoted 13 lines · Alistair Popple
Quoted 8 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

In this patch, unmap_and_move() is split to migrate_page_unmap() and migrate_page_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused newpage->mapping and newpage->private are used.

Quoted 5 lines

This looks like it could cause a deadlock between two threads migrating the same pages if force == true && mode != MIGRATE_ASYNC as migrate_page_unmap() will call lock_page() while holding the lock on other pages in the list. Therefore the two threads could deadlock if the pages are in a different order.

Quoted 3 lines

It seems unlikely to me since the page has to be isolated from lru before migration. The isolating from lru is atomic, so the two threads unlikely see the same pages on both lists.

Quoted 5 lines

Oh thanks! That is a good point and I agree since lru isolation is atomic the two threads won't see the same pages. migrate_vma_setup() does LRU isolation after locking the page which is why the potential exists there. We could potentially switch that around but given ZONE_DEVICE pages aren't on an lru it wouldn't help much.

Aha, I see. It has a different lock - isolation order from regular pages.

Quoted 10 lines
Quoted 7 lines

But there might be other cases which may incur deadlock, for example, filesystem writeback IIUC. Some filesystems may lock a bunch of pages then write them back in a batch. The same pages may be on the migration list and they are also dirty and seen by writeback. I'm not sure whether I miss something that could prevent such a deadlock from happening.

I'm not overly familiar with that area but I would assume any filesystem code doing this would already have to deal with deadlock potential.

AFAIK, actually not IIUC. For example, write back just simply look up page cache and lock them one by one.

Quoted 265 lines
Quoted 265 lines
Quoted 265 lines
Quoted 265 lines
Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 164 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 122 insertions(+), 42 deletions(-)
mm/migrate.c +122 -42
index 117134f1c6dc..4a81e0bfdbcd 100644
@@ -976,13 +976,32 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
      return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
+static void __migrate_page_record(struct page *newpage,
+                               int page_was_mapped,
+                               struct anon_vma *anon_vma)
+{
+     newpage->mapping = (struct address_space *)anon_vma;
+     newpage->private = page_was_mapped;
+}
+
+static void __migrate_page_extract(struct page *newpage,
+                                int *page_was_mappedp,
+                                struct anon_vma **anon_vmap)
+{
+     *anon_vmap = (struct anon_vma *)newpage->mapping;
+     *page_was_mappedp = newpage->private;
+     newpage->mapping = NULL;
+     newpage->private = 0;
+}
+
+#define MIGRATEPAGE_UNMAP            1
+
+static int __migrate_page_unmap(struct page *page, struct page *newpage,
                              int force, enum migrate_mode mode)
 {
      struct folio *folio = page_folio(page);
-     struct folio *dst = page_folio(newpage);
      int rc = -EAGAIN;
-     bool page_was_mapped = false;
+     int page_was_mapped = 0;
      struct anon_vma *anon_vma = NULL;
      bool is_lru = !__PageMovable(page);
 
@@ -1058,8 +1077,8 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              goto out_unlock;
 
      if (unlikely(!is_lru)) {
-             rc = move_to_new_folio(dst, folio, mode);
-             goto out_unlock_both;
+             __migrate_page_record(newpage, page_was_mapped, anon_vma);
+             return MIGRATEPAGE_UNMAP;
      }
 
      /*
@@ -1085,11 +1104,41 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
                              page);
              try_to_migrate(folio, 0);
-             page_was_mapped = true;
+             page_was_mapped = 1;
+     }
+
+     if (!page_mapped(page)) {
+             __migrate_page_record(newpage, page_was_mapped, anon_vma);
+             return MIGRATEPAGE_UNMAP;
      }
 
-     if (!page_mapped(page))
-             rc = move_to_new_folio(dst, folio, mode);
+     if (page_was_mapped)
+             remove_migration_ptes(folio, folio, false);
+
+out_unlock_both:
+     unlock_page(newpage);
+out_unlock:
+     /* Drop an anon_vma reference if we took one */
+     if (anon_vma)
+             put_anon_vma(anon_vma);
+     unlock_page(page);
+out:
+
+     return rc;
+}
+
+static int __migrate_page_move(struct page *page, struct page *newpage,
+                            enum migrate_mode mode)
+{
+     struct folio *folio = page_folio(page);
+     struct folio *dst = page_folio(newpage);
+     int rc;
+     int page_was_mapped = 0;
+     struct anon_vma *anon_vma = NULL;
+
+     __migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+
+     rc = move_to_new_folio(dst, folio, mode);
 
      /*
       * When successful, push newpage to LRU immediately: so that if it
@@ -1110,14 +1159,11 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
              remove_migration_ptes(folio,
                      rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
 
-out_unlock_both:
      unlock_page(newpage);
-out_unlock:
      /* Drop an anon_vma reference if we took one */
      if (anon_vma)
              put_anon_vma(anon_vma);
      unlock_page(page);
-out:
      /*
       * If migration is successful, decrease refcount of the newpage,
       * which will not free the page because new page owner increased
@@ -1129,18 +1175,31 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
      return rc;
 }
 
-/*
- * Obtain the lock on page, remove all ptes and migrate the page
- * to the newly allocated page in newpage.
- */
-static int unmap_and_move(new_page_t get_new_page,
-                                free_page_t put_new_page,
-                                unsigned long private, struct page *page,
-                                int force, enum migrate_mode mode,
-                                enum migrate_reason reason,
-                                struct list_head *ret)
+static void migrate_page_done(struct page *page,
+                           enum migrate_reason reason)
+{
+     /*
+      * Compaction can migrate also non-LRU pages which are
+      * not accounted to NR_ISOLATED_*. They can be recognized
+      * as __PageMovable
+      */
+     if (likely(!__PageMovable(page)))
+             mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+                                 page_is_file_lru(page), -thp_nr_pages(page));
+
+     if (reason != MR_MEMORY_FAILURE)
+             /* We release the page in page_handle_poison. */
+             put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+                           unsigned long private, struct page *page,
+                           struct page **newpagep, int force,
+                           enum migrate_mode mode, enum migrate_reason reason,
+                           struct list_head *ret)
 {
-     int rc = MIGRATEPAGE_SUCCESS;
+     int rc = MIGRATEPAGE_UNMAP;
      struct page *newpage = NULL;
 
      if (!thp_migration_supported() && PageTransHuge(page))
@@ -1151,19 +1210,48 @@ static int unmap_and_move(new_page_t get_new_page,
              ClearPageActive(page);
              ClearPageUnevictable(page);
              /* free_pages_prepare() will clear PG_isolated. */
-             goto out;
+             list_del(&page->lru);
+             migrate_page_done(page, reason);
+             return MIGRATEPAGE_SUCCESS;
      }
 
      newpage = get_new_page(page, private);
      if (!newpage)
              return -ENOMEM;
+     *newpagep = newpage;
 
-     newpage->private = 0;
-     rc = __unmap_and_move(page, newpage, force, mode);
+     rc = __migrate_page_unmap(page, newpage, force, mode);
+     if (rc == MIGRATEPAGE_UNMAP)
+             return rc;
+
+     /*
+      * A page that has not been migrated will have kept its
+      * references and be restored.
+      */
+     /* restore the page to right list. */
+     if (rc != -EAGAIN)
+             list_move_tail(&page->lru, ret);
+
+     if (put_new_page)
+             put_new_page(newpage, private);
+     else
+             put_page(newpage);
+
+     return rc;
+}
+
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+                          struct page *page, struct page *newpage,
+                          enum migrate_mode mode, enum migrate_reason reason,
+                          struct list_head *ret)
+{
+     int rc;
+
+     rc = __migrate_page_move(page, newpage, mode);
      if (rc == MIGRATEPAGE_SUCCESS)
              set_page_owner_migrate_reason(newpage, reason);
 
-out:
      if (rc != -EAGAIN) {
              /*
               * A page that has been migrated has all references
@@ -1179,20 +1267,7 @@ static int unmap_and_move(new_page_t get_new_page,
       * we want to retry.
       */
      if (rc == MIGRATEPAGE_SUCCESS) {
-             /*
-              * Compaction can migrate also non-LRU pages which are
-              * not accounted to NR_ISOLATED_*. They can be recognized
-              * as __PageMovable
-              */
-             if (likely(!__PageMovable(page)))
-                     mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-                                     page_is_file_lru(page), -thp_nr_pages(page));
-
-             if (reason != MR_MEMORY_FAILURE)
-                     /*
-                      * We release the page in page_handle_poison.
-                      */
-                     put_page(page);
+             migrate_page_done(page, reason);
      } else {
              if (rc != -EAGAIN)
                      list_add_tail(&page->lru, ret);
@@ -1405,6 +1480,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
      int pass = 0;
      bool is_thp = false;
      struct page *page;
+     struct page *newpage = NULL;
      struct page *page2;
      int rc, nr_subpages;
      LIST_HEAD(ret_pages);
@@ -1493,9 +1569,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
                      if (PageHuge(page))
                              continue;
 
-                     rc = unmap_and_move(get_new_page, put_new_page,
-                                             private, page, pass > 2, mode,
+                     rc = migrate_page_unmap(get_new_page, put_new_page, private,
+                                             page, &newpage, pass > 2, mode,
                                              reason, &ret_pages);
+                     if (rc == MIGRATEPAGE_UNMAP)
+                             rc = migrate_page_move(put_new_page, private,
+                                                    page, newpage, mode,
+                                                    reason, &ret_pages);
                      /*
                       * The rules are:
                       *      Success: page will be freed
author
#27 lore raw

RFC 3/6 mm/migrate_pages: restrict number of pages to migrate in batch

In reply to Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

If we had batched the page unmapping, all pages to be migrated would be unmapped before copying the contents and flags of the pages. If the number of pages that were passed to migrate_pages() was too large, too many pages would be unmapped. Then, the execution of their processes would be stopped for too long time. For example, migrate_pages() syscall will call migrate_pages() with all pages of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 93 +++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 67 insertions(+), 26 deletions(-)
mm/migrate.c +67 -26
index 4a81e0bfdbcd..1077af858e36 100644
@@ -1439,32 +1439,7 @@ static inline int try_split_thp(struct page *page, struct list_head *split_pages
 	return rc;
 }
 
-/*
- * migrate_pages - migrate the pages specified in a list, to the free pages
- *		   supplied as the target for the page migration
- *
- * @from:		The list of pages to be migrated.
- * @get_new_page:	The function used to allocate free pages to be used
- *			as the target of the page migration.
- * @put_new_page:	The function used to free target pages if migration
- *			fails, or NULL if no special handling is necessary.
- * @private:		Private data to be passed on to get_new_page()
- * @mode:		The migration mode that specifies the constraints for
- *			page migration, if any.
- * @reason:		The reason for page migration.
- * @ret_succeeded:	Set to the number of normal pages migrated successfully if
- *			the caller passes a non-NULL pointer.
- *
- * The function returns after 10 attempts or if no pages are movable any more
- * because the list has become empty or no retryable pages exist any more.
- * It is caller's responsibility to call putback_movable_pages() to return pages
- * to the LRU or free list only if ret != 0.
- *
- * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
- * an error code. The number of THP splits will be considered as the number of
- * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
- */
-int migrate_pages(struct list_head *from, new_page_t get_new_page,
+static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 		free_page_t put_new_page, unsigned long private,
 		enum migrate_mode mode, int reason, unsigned int *ret_succeeded)
 {
@@ -1709,6 +1684,72 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	return rc;
 }
 
+/*
+ * migrate_pages - migrate the pages specified in a list, to the free pages
+ *		   supplied as the target for the page migration
+ *
+ * @from:		The list of pages to be migrated.
+ * @get_new_page:	The function used to allocate free pages to be used
+ *			as the target of the page migration.
+ * @put_new_page:	The function used to free target pages if migration
+ *			fails, or NULL if no special handling is necessary.
+ * @private:		Private data to be passed on to get_new_page()
+ * @mode:		The migration mode that specifies the constraints for
+ *			page migration, if any.
+ * @reason:		The reason for page migration.
+ * @ret_succeeded:	Set to the number of normal pages migrated successfully if
+ *			the caller passes a non-NULL pointer.
+ *
+ * The function returns after 10 attempts or if no pages are movable any more
+ * because the list has become empty or no retryable pages exist any more.
+ * It is caller's responsibility to call putback_movable_pages() to return pages
+ * to the LRU or free list only if ret != 0.
+ *
+ * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
+ * an error code. The number of THP splits will be considered as the number of
+ * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
+ */
+int migrate_pages(struct list_head *from, new_page_t get_new_page,
+		free_page_t put_new_page, unsigned long private,
+		enum migrate_mode mode, int reason, unsigned int *pret_succeeded)
+{
+	int rc, rc_gether = 0;
+	int ret_succeeded, ret_succeeded_gether = 0;
+	int nr_pages;
+	struct page *page;
+	LIST_HEAD(pagelist);
+	LIST_HEAD(ret_pages);
+
+again:
+	nr_pages = 0;
+	list_for_each_entry(page, from, lru) {
+		nr_pages += compound_nr(page);
+		if (nr_pages > HPAGE_PMD_NR)
+			break;
+	}
+	if (nr_pages > HPAGE_PMD_NR)
+		list_cut_before(&pagelist, from, &page->lru);
+	else
+		list_splice_init(from, &pagelist);
+	rc = migrate_pages_batch(&pagelist, get_new_page, put_new_page, private,
+				 mode, reason, &ret_succeeded);
+	ret_succeeded_gether += ret_succeeded;
+	list_splice_tail_init(&pagelist, &ret_pages);
+	if (rc == -ENOMEM) {
+		rc_gether = rc;
+		goto out;
+	}
+	rc_gether += rc;
+	if (!list_empty(from))
+		goto again;
+out:
+	if (pret_succeeded)
+		*pret_succeeded = ret_succeeded_gether;
+	list_splice(&ret_pages, from);
+
+	return rc_gether;
+}
+
 struct page *alloc_migration_target(struct page *page, unsigned long private)
 {
 	struct folio *folio = page_folio(page);
Signature · 1 line
2.35.1
reviewer
#28 lore raw

Re: [RFC 3/6] mm/migrate_pages: restrict number of pages to migrate in batch

Quoted 96 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

If we had batched the page unmapping, all pages to be migrated would be unmapped before copying the contents and flags of the pages. If the number of pages that were passed to migrate_pages() was too large, too many pages would be unmapped. Then, the execution of their processes would be stopped for too long time. For example, migrate_pages() syscall will call migrate_pages() with all pages of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 93 +++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 67 insertions(+), 26 deletions(-)
mm/migrate.c +32 -26
index 4a81e0bfdbcd..1077af858e36 100644
@@ -1439,32 +1439,7 @@ static inline int try_split_thp(struct page *page, struct list_head *split_pages
 	return rc;
 }
 
-/*
- * migrate_pages - migrate the pages specified in a list, to the free pages
- *		   supplied as the target for the page migration
- *
- * @from:		The list of pages to be migrated.
- * @get_new_page:	The function used to allocate free pages to be used
- *			as the target of the page migration.
- * @put_new_page:	The function used to free target pages if migration
- *			fails, or NULL if no special handling is necessary.
- * @private:		Private data to be passed on to get_new_page()
- * @mode:		The migration mode that specifies the constraints for
- *			page migration, if any.
- * @reason:		The reason for page migration.
- * @ret_succeeded:	Set to the number of normal pages migrated successfully if
- *			the caller passes a non-NULL pointer.
- *
- * The function returns after 10 attempts or if no pages are movable any more
- * because the list has become empty or no retryable pages exist any more.
- * It is caller's responsibility to call putback_movable_pages() to return pages
- * to the LRU or free list only if ret != 0.
- *
- * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
- * an error code. The number of THP splits will be considered as the number of
- * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
- */
-int migrate_pages(struct list_head *from, new_page_t get_new_page,
+static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 		free_page_t put_new_page, unsigned long private,
 		enum migrate_mode mode, int reason, unsigned int *ret_succeeded)
 {
@@ -1709,6 +1684,72 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	return rc;
 }
 
+/*
+ * migrate_pages - migrate the pages specified in a list, to the free pages
+ *		   supplied as the target for the page migration
+ *
+ * @from:		The list of pages to be migrated.
+ * @get_new_page:	The function used to allocate free pages to be used
+ *			as the target of the page migration.
+ * @put_new_page:	The function used to free target pages if migration
+ *			fails, or NULL if no special handling is necessary.
+ * @private:		Private data to be passed on to get_new_page()
+ * @mode:		The migration mode that specifies the constraints for
+ *			page migration, if any.
+ * @reason:		The reason for page migration.
+ * @ret_succeeded:	Set to the number of normal pages migrated successfully if
+ *			the caller passes a non-NULL pointer.
+ *
+ * The function returns after 10 attempts or if no pages are movable any more
+ * because the list has become empty or no retryable pages exist any more.
+ * It is caller's responsibility to call putback_movable_pages() to return pages
+ * to the LRU or free list only if ret != 0.
+ *
+ * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
+ * an error code. The number of THP splits will be considered as the number of
+ * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
+ */
+int migrate_pages(struct list_head *from, new_page_t get_new_page,
+		free_page_t put_new_page, unsigned long private,
+		enum migrate_mode mode, int reason, unsigned int *pret_succeeded)
+{
+	int rc, rc_gether = 0;
+	int ret_succeeded, ret_succeeded_gether = 0;

mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1727

Quoted 10 lines
mm/migrate.c +10 -0
+	int nr_pages;
+	struct page *page;
+	LIST_HEAD(pagelist);
+	LIST_HEAD(ret_pages);
+
+again:
+	nr_pages = 0;
+	list_for_each_entry(page, from, lru) {
+		nr_pages += compound_nr(page);
+		if (nr_pages > HPAGE_PMD_NR)

It is better to define a new MACRO like NR_MAX_BATCHED_MIGRATION to be HPAGE_PMD_NR. It makes code easier to understand and change.

Quoted 30 lines
mm/migrate.c +25 -0
+			break;
+	}
+	if (nr_pages > HPAGE_PMD_NR)
+		list_cut_before(&pagelist, from, &page->lru);
+	else
+		list_splice_init(from, &pagelist);
+	rc = migrate_pages_batch(&pagelist, get_new_page, put_new_page, private,
+				 mode, reason, &ret_succeeded);
+	ret_succeeded_gether += ret_succeeded;
+	list_splice_tail_init(&pagelist, &ret_pages);
+	if (rc == -ENOMEM) {
+		rc_gether = rc;
+		goto out;
+	}
+	rc_gether += rc;
+	if (!list_empty(from))
+		goto again;
+out:
+	if (pret_succeeded)
+		*pret_succeeded = ret_succeeded_gether;
+	list_splice(&ret_pages, from);
+
+	return rc_gether;
+}
+
 struct page *alloc_migration_target(struct page *page, unsigned long private)
 {
 	struct folio *folio = page_folio(page);
Signature · 1 line
2.35.1

--
Best Regards,
Yan, Zi

reviewer
#29 lore raw

Re: [RFC 3/6] mm/migrate_pages: restrict number of pages to migrate in batch

Quoted 54 lines · Zi Yan
Quoted 52 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

If we had batched the page unmapping, all pages to be migrated would be unmapped before copying the contents and flags of the pages. If the number of pages that were passed to migrate_pages() was too large, too many pages would be unmapped. Then, the execution of their processes would be stopped for too long time. For example, migrate_pages() syscall will call migrate_pages() with all pages of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 93 +++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 67 insertions(+), 26 deletions(-)
mm/migrate.c +0 -21
index 4a81e0bfdbcd..1077af858e36 100644
@@ -1439,32 +1439,7 @@ static inline int try_split_thp(struct page *page, struct list_head *split_pages
 	return rc;
 }
 
-/*
- * migrate_pages - migrate the pages specified in a list, to the free pages
- *		   supplied as the target for the page migration
- *
- * @from:		The list of pages to be migrated.
- * @get_new_page:	The function used to allocate free pages to be used
- *			as the target of the page migration.
- * @put_new_page:	The function used to free target pages if migration
- *			fails, or NULL if no special handling is necessary.
- * @private:		Private data to be passed on to get_new_page()
- * @mode:		The migration mode that specifies the constraints for
- *			page migration, if any.
- * @reason:		The reason for page migration.
- * @ret_succeeded:	Set to the number of normal pages migrated successfully if
- *			the caller passes a non-NULL pointer.
- *
- * The function returns after 10 attempts or if no pages are movable any more
- * because the list has become empty or no retryable pages exist any more.
- * It is caller's responsibility to call putback_movable_pages() to return pages
- * to the LRU or free list only if ret != 0.
- *
Quoted 8 lines

mm/migrate.c static inline int try_split_thp(struct page *page, struct list_head *split_pages · line 1444

Quoted 8 lines
mm/migrate.c +1 -5
- * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
- * an error code. The number of THP splits will be considered as the number of
- * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
- */
-int migrate_pages(struct list_head *from, new_page_t get_new_page,
+static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 		free_page_t put_new_page, unsigned long private,
 		enum migrate_mode mode, int reason, unsigned int *ret_succeeded)

We are not batching hugetlb page migration, right? migrate_pages_batch() should not include the hugetlb page migration code. migrate_pages() should look like:

migrate_pages()

{
        migrate hugetlb pages if they exist;
        migrate_pages_batch();
}
Quoted 77 lines
Quoted 36 lines
mm/migrate.c +31 -0
 {
@@ -1709,6 +1684,72 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	return rc;
 }
 
+/*
+ * migrate_pages - migrate the pages specified in a list, to the free pages
+ *		   supplied as the target for the page migration
+ *
+ * @from:		The list of pages to be migrated.
+ * @get_new_page:	The function used to allocate free pages to be used
+ *			as the target of the page migration.
+ * @put_new_page:	The function used to free target pages if migration
+ *			fails, or NULL if no special handling is necessary.
+ * @private:		Private data to be passed on to get_new_page()
+ * @mode:		The migration mode that specifies the constraints for
+ *			page migration, if any.
+ * @reason:		The reason for page migration.
+ * @ret_succeeded:	Set to the number of normal pages migrated successfully if
+ *			the caller passes a non-NULL pointer.
+ *
+ * The function returns after 10 attempts or if no pages are movable any more
+ * because the list has become empty or no retryable pages exist any more.
+ * It is caller's responsibility to call putback_movable_pages() to return pages
+ * to the LRU or free list only if ret != 0.
+ *
+ * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
+ * an error code. The number of THP splits will be considered as the number of
+ * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
+ */
+int migrate_pages(struct list_head *from, new_page_t get_new_page,
+		free_page_t put_new_page, unsigned long private,
+		enum migrate_mode mode, int reason, unsigned int *pret_succeeded)
+{
+	int rc, rc_gether = 0;
+	int ret_succeeded, ret_succeeded_gether = 0;

mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1727

Quoted 10 lines
mm/migrate.c +10 -0
+	int nr_pages;
+	struct page *page;
+	LIST_HEAD(pagelist);
+	LIST_HEAD(ret_pages);
+
+again:
+	nr_pages = 0;
+	list_for_each_entry(page, from, lru) {
+		nr_pages += compound_nr(page);
+		if (nr_pages > HPAGE_PMD_NR)

It is better to define a new MACRO like NR_MAX_BATCHED_MIGRATION to be HPAGE_PMD_NR. It makes code easier to understand and change.

Quoted 27 lines
mm/migrate.c +25 -0
+			break;
+	}
+	if (nr_pages > HPAGE_PMD_NR)
+		list_cut_before(&pagelist, from, &page->lru);
+	else
+		list_splice_init(from, &pagelist);
+	rc = migrate_pages_batch(&pagelist, get_new_page, put_new_page, private,
+				 mode, reason, &ret_succeeded);
+	ret_succeeded_gether += ret_succeeded;
+	list_splice_tail_init(&pagelist, &ret_pages);
+	if (rc == -ENOMEM) {
+		rc_gether = rc;
+		goto out;
+	}
+	rc_gether += rc;
+	if (!list_empty(from))
+		goto again;
+out:
+	if (pret_succeeded)
+		*pret_succeeded = ret_succeeded_gether;
+	list_splice(&ret_pages, from);
+
+	return rc_gether;
+}
+
 struct page *alloc_migration_target(struct page *page, unsigned long private)
 {
Quoted 8 lines
Quoted 3 lines
mm/migrate.c +0 -0
 	struct folio *folio = page_folio(page);
Signature · 1 line
2.35.1

--
Best Regards,
Yan, Zi

--
Best Regards,
Yan, Zi

#30 lore raw

Re: [RFC 3/6] mm/migrate_pages: restrict number of pages to migrate in batch

In reply to Zi Yan
Quoted 103 lines · Zi Yan
Quoted 101 lines · Huang Ying

This is a preparation patch to batch the page unmapping and moving for the normal pages and THP.

If we had batched the page unmapping, all pages to be migrated would be unmapped before copying the contents and flags of the pages. If the number of pages that were passed to migrate_pages() was too large, too many pages would be unmapped. Then, the execution of their processes would be stopped for too long time. For example, migrate_pages() syscall will call migrate_pages() with all pages of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 93 +++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 67 insertions(+), 26 deletions(-)
mm/migrate.c +37 -26
index 4a81e0bfdbcd..1077af858e36 100644
@@ -1439,32 +1439,7 @@ static inline int try_split_thp(struct page *page, struct list_head *split_pages
 	return rc;
 }
 
-/*
- * migrate_pages - migrate the pages specified in a list, to the free pages
- *		   supplied as the target for the page migration
- *
- * @from:		The list of pages to be migrated.
- * @get_new_page:	The function used to allocate free pages to be used
- *			as the target of the page migration.
- * @put_new_page:	The function used to free target pages if migration
- *			fails, or NULL if no special handling is necessary.
- * @private:		Private data to be passed on to get_new_page()
- * @mode:		The migration mode that specifies the constraints for
- *			page migration, if any.
- * @reason:		The reason for page migration.
- * @ret_succeeded:	Set to the number of normal pages migrated successfully if
- *			the caller passes a non-NULL pointer.
- *
- * The function returns after 10 attempts or if no pages are movable any more
- * because the list has become empty or no retryable pages exist any more.
- * It is caller's responsibility to call putback_movable_pages() to return pages
- * to the LRU or free list only if ret != 0.
- *
- * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
- * an error code. The number of THP splits will be considered as the number of
- * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
- */
-int migrate_pages(struct list_head *from, new_page_t get_new_page,
+static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 		free_page_t put_new_page, unsigned long private,
 		enum migrate_mode mode, int reason, unsigned int *ret_succeeded)
 {
@@ -1709,6 +1684,72 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,
 	return rc;
 }
 
+/*
+ * migrate_pages - migrate the pages specified in a list, to the free pages
+ *		   supplied as the target for the page migration
+ *
+ * @from:		The list of pages to be migrated.
+ * @get_new_page:	The function used to allocate free pages to be used
+ *			as the target of the page migration.
+ * @put_new_page:	The function used to free target pages if migration
+ *			fails, or NULL if no special handling is necessary.
+ * @private:		Private data to be passed on to get_new_page()
+ * @mode:		The migration mode that specifies the constraints for
+ *			page migration, if any.
+ * @reason:		The reason for page migration.
+ * @ret_succeeded:	Set to the number of normal pages migrated successfully if
+ *			the caller passes a non-NULL pointer.
+ *
+ * The function returns after 10 attempts or if no pages are movable any more
+ * because the list has become empty or no retryable pages exist any more.
+ * It is caller's responsibility to call putback_movable_pages() to return pages
+ * to the LRU or free list only if ret != 0.
+ *
+ * Returns the number of {normal page, THP, hugetlb} that were not migrated, or
+ * an error code. The number of THP splits will be considered as the number of
+ * non-migrated THP, no matter how many subpages of the THP are migrated successfully.
+ */
+int migrate_pages(struct list_head *from, new_page_t get_new_page,
+		free_page_t put_new_page, unsigned long private,
+		enum migrate_mode mode, int reason, unsigned int *pret_succeeded)
+{
+	int rc, rc_gether = 0;
+	int ret_succeeded, ret_succeeded_gether = 0;
+	int nr_pages;
+	struct page *page;
+	LIST_HEAD(pagelist);
+	LIST_HEAD(ret_pages);
+
Quoted 8 lines

mm/migrate.c int migrate_pages(struct list_head *from, new_page_t get_new_page, · line 1727

Quoted 5 lines
mm/migrate.c +5 -0
+again:
+	nr_pages = 0;
+	list_for_each_entry(page, from, lru) {
+		nr_pages += compound_nr(page);
+		if (nr_pages > HPAGE_PMD_NR)

It is better to define a new MACRO like NR_MAX_BATCHED_MIGRATION to be HPAGE_PMD_NR. It makes code easier to understand and change.

OK. Will do that.

Best Regards,
Huang, Ying

Quoted 35 lines
Quoted 30 lines
mm/migrate.c +25 -0
+			break;
+	}
+	if (nr_pages > HPAGE_PMD_NR)
+		list_cut_before(&pagelist, from, &page->lru);
+	else
+		list_splice_init(from, &pagelist);
+	rc = migrate_pages_batch(&pagelist, get_new_page, put_new_page, private,
+				 mode, reason, &ret_succeeded);
+	ret_succeeded_gether += ret_succeeded;
+	list_splice_tail_init(&pagelist, &ret_pages);
+	if (rc == -ENOMEM) {
+		rc_gether = rc;
+		goto out;
+	}
+	rc_gether += rc;
+	if (!list_empty(from))
+		goto again;
+out:
+	if (pret_succeeded)
+		*pret_succeeded = ret_succeeded_gether;
+	list_splice(&ret_pages, from);
+
+	return rc_gether;
+}
+
 struct page *alloc_migration_target(struct page *page, unsigned long private)
 {
 	struct folio *folio = page_folio(page);
Signature · 1 line
2.35.1

--
Best Regards,
Yan, Zi

author
#31 lore raw

RFC 4/6 mm/migrate_pages: batch _unmap and _move

In reply to Huang Ying

In this patch the _unmap and _move stage of the page migration is batched. That for, previously, it is,

for each page

    _unmap()
    _move()

Now, it is,

for each page

    _unmap()
  for each page
    _move()

Based on this, we can batch the TLB flushing and use some hardware accelerator to copy pages between batched _unmap and batched _move stages.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 155 +++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 139 insertions(+), 16 deletions(-)
mm/migrate.c +139 -16
index 1077af858e36..165cbbc834e2 100644
@@ -996,6 +996,32 @@ static void __migrate_page_extract(struct page *newpage,
 
 #define MIGRATEPAGE_UNMAP		1
 
+static void migrate_page_undo_page(struct page *page,
+				   int page_was_mapped,
+				   struct anon_vma *anon_vma,
+				   struct list_head *ret)
+{
+	struct folio *folio = page_folio(page);
+
+	if (page_was_mapped)
+		remove_migration_ptes(folio, folio, false);
+	if (anon_vma)
+		put_anon_vma(anon_vma);
+	unlock_page(page);
+	list_move_tail(&page->lru, ret);
+}
+
+static void migrate_page_undo_newpage(struct page *newpage,
+				      free_page_t put_new_page,
+				      unsigned long private)
+{
+	unlock_page(newpage);
+	if (put_new_page)
+		put_new_page(newpage, private);
+	else
+		put_page(newpage);
+}
+
 static int __migrate_page_unmap(struct page *page, struct page *newpage,
 				int force, enum migrate_mode mode)
 {
@@ -1140,6 +1166,8 @@ static int __migrate_page_move(struct page *page, struct page *newpage,
 
 	rc = move_to_new_folio(dst, folio, mode);
 
+	if (rc != -EAGAIN)
+		list_del(&newpage->lru);
 	/*
 	 * When successful, push newpage to LRU immediately: so that if it
 	 * turns out to be an mlocked page, remove_migration_ptes() will
@@ -1155,6 +1183,11 @@ static int __migrate_page_move(struct page *page, struct page *newpage,
 			lru_add_drain();
 	}
 
+	if (rc == -EAGAIN) {
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return rc;
+	}
+
 	if (page_was_mapped)
 		remove_migration_ptes(folio,
 			rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
@@ -1220,6 +1253,7 @@ static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
 		return -ENOMEM;
 	*newpagep = newpage;
 
+	newpage->private = 0;
 	rc = __migrate_page_unmap(page, newpage, force, mode);
 	if (rc == MIGRATEPAGE_UNMAP)
 		return rc;
@@ -1258,7 +1292,7 @@ static int migrate_page_move(free_page_t put_new_page, unsigned long private,
 		 * removed and will be freed. A page that has not been
 		 * migrated will have kept its references and be restored.
 		 */
-		list_del(&page->lru);
+		list_del_init(&page->lru);
 	}
 
 	/*
@@ -1268,9 +1302,8 @@ static int migrate_page_move(free_page_t put_new_page, unsigned long private,
 	 */
 	if (rc == MIGRATEPAGE_SUCCESS) {
 		migrate_page_done(page, reason);
-	} else {
-		if (rc != -EAGAIN)
-			list_add_tail(&page->lru, ret);
+	} else if (rc != -EAGAIN) {
+		list_add_tail(&page->lru, ret);
 
 		if (put_new_page)
 			put_new_page(newpage, private);
@@ -1455,11 +1488,13 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 	int pass = 0;
 	bool is_thp = false;
 	struct page *page;
-	struct page *newpage = NULL;
+	struct page *newpage = NULL, *newpage2;
 	struct page *page2;
 	int rc, nr_subpages;
 	LIST_HEAD(ret_pages);
 	LIST_HEAD(thp_split_pages);
+	LIST_HEAD(unmap_pages);
+	LIST_HEAD(new_pages);
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
 	bool no_subpage_counting = false;
 
@@ -1541,19 +1576,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 			nr_subpages = compound_nr(page);
 			cond_resched();
 
-			if (PageHuge(page))
+			if (PageHuge(page)) {
+				list_move_tail(&page->lru, &ret_pages);
 				continue;
+			}
 
 			rc = migrate_page_unmap(get_new_page, put_new_page, private,
 						page, &newpage, pass > 2, mode,
 						reason, &ret_pages);
-			if (rc == MIGRATEPAGE_UNMAP)
-				rc = migrate_page_move(put_new_page, private,
-						       page, newpage, mode,
-						       reason, &ret_pages);
 			/*
 			 * The rules are:
 			 *	Success: page will be freed
+			 *	Unmap: page will be put on unmap_pages list,
+			 *	       new page put on new_pages list
 			 *	-EAGAIN: stay on the from list
 			 *	-ENOMEM: stay on the from list
 			 *	-ENOSYS: stay on the from list
@@ -1589,7 +1624,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 			case -ENOMEM:
 				/*
 				 * When memory is low, don't bother to try to migrate
-				 * other pages, just exit.
+				 * other pages, move unmapped pages, then exit.
 				 */
 				if (is_thp) {
 					nr_thp_failed++;
@@ -1610,9 +1645,11 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 				 * the caller otherwise the page refcnt will be leaked.
 				 */
 				list_splice_init(&thp_split_pages, from);
-				/* nr_failed isn't updated for not used */
 				nr_thp_failed += thp_retry;
-				goto out;
+				if (list_empty(&unmap_pages))
+					goto out;
+				else
+					goto move;
 			case -EAGAIN:
 				if (is_thp)
 					thp_retry++;
@@ -1625,6 +1662,10 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 				if (is_thp)
 					nr_thp_succeeded++;
 				break;
+			case MIGRATEPAGE_UNMAP:
+				list_move_tail(&page->lru, &unmap_pages);
+				list_add_tail(&newpage->lru, &new_pages);
+				break;
 			default:
 				/*
 				 * Permanent failure (-EBUSY, etc.):
@@ -1645,12 +1686,96 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 	nr_failed += retry;
 	nr_thp_failed += thp_retry;
 	nr_failed_pages += nr_retry_pages;
+move:
+	retry = 1;
+	thp_retry = 1;
+	for (pass = 0; pass < 10 && (retry || thp_retry); pass++) {
+		retry = 0;
+		thp_retry = 0;
+		nr_retry_pages = 0;
+
+		newpage = list_first_entry(&new_pages, struct page, lru);
+		newpage2 = list_next_entry(newpage, lru);
+		list_for_each_entry_safe(page, page2, &unmap_pages, lru) {
+			/*
+			 * THP statistics is based on the source huge page.
+			 * Capture required information that might get lost
+			 * during migration.
+			 */
+			is_thp = PageTransHuge(page) && !PageHuge(page);
+			nr_subpages = compound_nr(page);
+			cond_resched();
+
+			rc = migrate_page_move(put_new_page, private,
+					       page, newpage, mode,
+					       reason, &ret_pages);
+			/*
+			 * The rules are:
+			 *	Success: page will be freed
+			 *	-EAGAIN: stay on the unmap_pages list
+			 *	Other errno: put on ret_pages list then splice to
+			 *		     from list
+			 */
+			switch(rc) {
+			case -EAGAIN:
+				if (is_thp)
+					thp_retry++;
+				else if (!no_subpage_counting)
+					retry++;
+				nr_retry_pages += nr_subpages;
+				break;
+			case MIGRATEPAGE_SUCCESS:
+				nr_succeeded += nr_subpages;
+				if (is_thp)
+					nr_thp_succeeded++;
+				break;
+			default:
+				/*
+				 * Permanent failure (-EBUSY, etc.):
+				 * unlike -EAGAIN case, the failed page is
+				 * removed from migration page list and not
+				 * retried in the next outer loop.
+				 */
+				if (is_thp)
+					nr_thp_failed++;
+				else if (!no_subpage_counting)
+					nr_failed++;
+
+				nr_failed_pages += nr_subpages;
+				break;
+			}
+			newpage = newpage2;
+			newpage2 = list_next_entry(newpage, lru);
+		}
+	}
+	nr_failed += retry;
+	nr_thp_failed += thp_retry;
+	nr_failed_pages += nr_retry_pages;
+
+	rc = nr_failed + nr_thp_failed;
+out:
+	/* Cleanup remaining pages */
+	newpage = list_first_entry(&new_pages, struct page, lru);
+	newpage2 = list_next_entry(newpage, lru);
+	list_for_each_entry_safe(page, page2, &unmap_pages, lru) {
+		int page_was_mapped = 0;
+		struct anon_vma *anon_vma = NULL;
+
+		__migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
+		migrate_page_undo_page(page, page_was_mapped, anon_vma,
+				       &ret_pages);
+		list_del(&newpage->lru);
+		migrate_page_undo_newpage(newpage, put_new_page, private);
+		newpage = newpage2;
+		newpage2 = list_next_entry(newpage, lru);
+	}
+
 	/*
 	 * Try to migrate subpages of fail-to-migrate THPs, no nr_failed
 	 * counting in this round, since all subpages of a THP is counted
 	 * as 1 failure in the first round.
 	 */
-	if (!list_empty(&thp_split_pages)) {
+	if (rc >= 0 && !list_empty(&thp_split_pages)) {
 		/*
 		 * Move non-migrated pages (after 10 retries) to ret_pages
 		 * to avoid migrating them again.
@@ -1662,8 +1787,6 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 		goto thp_subpage_migration;
 	}
 
-	rc = nr_failed + nr_thp_failed;
-out:
 	/*
 	 * Put the permanent failure page back to migration list, they
 	 * will be put back to the right list by the caller.
Signature · 1 line
2.35.1
author
#32 lore raw

RFC 5/6 mm/migrate_pages: share more code between _unmap and _move

In reply to Huang Ying

This is a code cleanup patch to reduce the duplicated code between the _unmap and _move stages of migrate_pages(). No functionality change is expected.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c | 240 +++++++++++++++++++++------------------------------
 1 file changed, 100 insertions(+), 140 deletions(-)
mm/migrate.c +100 -140
index 165cbbc834e2..042fa147f302 100644
@@ -999,6 +999,7 @@ static void __migrate_page_extract(struct page *newpage,
 static void migrate_page_undo_page(struct page *page,
 				   int page_was_mapped,
 				   struct anon_vma *anon_vma,
+				   bool locked,
 				   struct list_head *ret)
 {
 	struct folio *folio = page_folio(page);
@@ -1007,30 +1008,77 @@ static void migrate_page_undo_page(struct page *page,
 		remove_migration_ptes(folio, folio, false);
 	if (anon_vma)
 		put_anon_vma(anon_vma);
-	unlock_page(page);
-	list_move_tail(&page->lru, ret);
+	if (locked)
+		unlock_page(page);
+	if (ret)
+		list_move_tail(&page->lru, ret);
 }
 
 static void migrate_page_undo_newpage(struct page *newpage,
+				      bool locked,
 				      free_page_t put_new_page,
 				      unsigned long private)
 {
-	unlock_page(newpage);
+	if (locked)
+		unlock_page(newpage);
 	if (put_new_page)
 		put_new_page(newpage, private);
 	else
 		put_page(newpage);
 }
 
-static int __migrate_page_unmap(struct page *page, struct page *newpage,
-				int force, enum migrate_mode mode)
+static void migrate_page_done(struct page *page,
+			      enum migrate_reason reason)
+{
+	/*
+	 * Compaction can migrate also non-LRU pages which are
+	 * not accounted to NR_ISOLATED_*. They can be recognized
+	 * as __PageMovable
+	 */
+	if (likely(!__PageMovable(page)))
+		mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
+				    page_is_file_lru(page), -thp_nr_pages(page));
+
+	if (reason != MR_MEMORY_FAILURE)
+		/* We release the page in page_handle_poison. */
+		put_page(page);
+}
+
+/* Obtain the lock on page, remove all ptes. */
+static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
+			      unsigned long private, struct page *page,
+			      struct page **newpagep, int force,
+			      enum migrate_mode mode, enum migrate_reason reason,
+			      struct list_head *ret)
 {
 	struct folio *folio = page_folio(page);
-	int rc = -EAGAIN;
+	int rc = MIGRATEPAGE_UNMAP;
+	struct page *newpage = NULL;
 	int page_was_mapped = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool is_lru = !__PageMovable(page);
+	bool locked = false;
+	bool newpage_locked = false;
+
+	if (!thp_migration_supported() && PageTransHuge(page))
+		return -ENOSYS;
 
+	if (page_count(page) == 1) {
+		/* Page was freed from under us. So we are done. */
+		ClearPageActive(page);
+		ClearPageUnevictable(page);
+		/* free_pages_prepare() will clear PG_isolated. */
+		list_del(&page->lru);
+		migrate_page_done(page, reason);
+		return MIGRATEPAGE_SUCCESS;
+	}
+
+	newpage = get_new_page(page, private);
+	if (!newpage)
+		return -ENOMEM;
+	*newpagep = newpage;
+
+	rc = -EAGAIN;
 	if (!trylock_page(page)) {
 		if (!force || mode == MIGRATE_ASYNC)
 			goto out;
@@ -1053,6 +1101,7 @@ static int __migrate_page_unmap(struct page *page, struct page *newpage,
 
 		lock_page(page);
 	}
+	locked = true;
 
 	if (PageWriteback(page)) {
 		/*
@@ -1067,10 +1116,10 @@ static int __migrate_page_unmap(struct page *page, struct page *newpage,
 			break;
 		default:
 			rc = -EBUSY;
-			goto out_unlock;
+			goto out;
 		}
 		if (!force)
-			goto out_unlock;
+			goto out;
 		wait_on_page_writeback(page);
 	}
 
@@ -1100,7 +1149,8 @@ static int __migrate_page_unmap(struct page *page, struct page *newpage,
 	 * This is much like races on refcount of oldpage: just don't BUG().
 	 */
 	if (unlikely(!trylock_page(newpage)))
-		goto out_unlock;
+		goto out;
+	newpage_locked = true;
 
 	if (unlikely(!is_lru)) {
 		__migrate_page_record(newpage, page_was_mapped, anon_vma);
@@ -1123,7 +1173,7 @@ static int __migrate_page_unmap(struct page *page, struct page *newpage,
 		VM_BUG_ON_PAGE(PageAnon(page), page);
 		if (page_has_private(page)) {
 			try_to_free_buffers(folio);
-			goto out_unlock_both;
+			goto out;
 		}
 	} else if (page_mapped(page)) {
 		/* Establish migration ptes */
@@ -1141,20 +1191,28 @@ static int __migrate_page_unmap(struct page *page, struct page *newpage,
 	if (page_was_mapped)
 		remove_migration_ptes(folio, folio, false);
 
-out_unlock_both:
-	unlock_page(newpage);
-out_unlock:
-	/* Drop an anon_vma reference if we took one */
-	if (anon_vma)
-		put_anon_vma(anon_vma);
-	unlock_page(page);
 out:
+	/*
+	 * A page that has not been migrated will have kept its
+	 * references and be restored.
+	 */
+	/* restore the page to right list. */
+	if (rc != -EAGAIN)
+                ret = NULL;
+
+	migrate_page_undo_page(page, page_was_mapped, anon_vma, locked, ret);
+	if (newpage)
+		migrate_page_undo_newpage(newpage, newpage_locked,
+					  put_new_page, private);
 
 	return rc;
 }
 
-static int __migrate_page_move(struct page *page, struct page *newpage,
-			       enum migrate_mode mode)
+/* Migrate the page to the newly allocated page in newpage. */
+static int migrate_page_move(free_page_t put_new_page, unsigned long private,
+			     struct page *page, struct page *newpage,
+			     enum migrate_mode mode, enum migrate_reason reason,
+			     struct list_head *ret)
 {
 	struct folio *folio = page_folio(page);
 	struct folio *dst = page_folio(newpage);
@@ -1165,9 +1223,10 @@ static int __migrate_page_move(struct page *page, struct page *newpage,
 	__migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
 
 	rc = move_to_new_folio(dst, folio, mode);
+	if (rc)
+		goto out;
 
-	if (rc != -EAGAIN)
-		list_del(&newpage->lru);
+	list_del(&newpage->lru);
 	/*
 	 * When successful, push newpage to LRU immediately: so that if it
 	 * turns out to be an mlocked page, remove_migration_ptes() will
@@ -1177,139 +1236,40 @@ static int __migrate_page_move(struct page *page, struct page *newpage,
 	 * unsuccessful, and other cases when a page has been temporarily
 	 * isolated from the unevictable LRU: but this case is the easiest.
 	 */
-	if (rc == MIGRATEPAGE_SUCCESS) {
-		lru_cache_add(newpage);
-		if (page_was_mapped)
-			lru_add_drain();
-	}
-
-	if (rc == -EAGAIN) {
-		__migrate_page_record(newpage, page_was_mapped, anon_vma);
-		return rc;
-	}
-
+	lru_cache_add(newpage);
 	if (page_was_mapped)
-		remove_migration_ptes(folio,
-			rc == MIGRATEPAGE_SUCCESS ? dst : folio, false);
+		lru_add_drain();
 
+	if (page_was_mapped)
+		remove_migration_ptes(folio, dst, false);
 	unlock_page(newpage);
-	/* Drop an anon_vma reference if we took one */
-	if (anon_vma)
-		put_anon_vma(anon_vma);
-	unlock_page(page);
+	set_page_owner_migrate_reason(newpage, reason);
 	/*
 	 * If migration is successful, decrease refcount of the newpage,
 	 * which will not free the page because new page owner increased
 	 * refcounter.
 	 */
-	if (rc == MIGRATEPAGE_SUCCESS)
-		put_page(newpage);
-
-	return rc;
-}
+	put_page(newpage);
 
-static void migrate_page_done(struct page *page,
-			      enum migrate_reason reason)
-{
 	/*
-	 * Compaction can migrate also non-LRU pages which are
-	 * not accounted to NR_ISOLATED_*. They can be recognized
-	 * as __PageMovable
+	 * A page that has been migrated has all references removed
+	 * and will be freed.
 	 */
-	if (likely(!__PageMovable(page)))
-		mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON +
-				    page_is_file_lru(page), -thp_nr_pages(page));
-
-	if (reason != MR_MEMORY_FAILURE)
-		/* We release the page in page_handle_poison. */
-		put_page(page);
-}
-
-/* Obtain the lock on page, remove all ptes. */
-static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
-			      unsigned long private, struct page *page,
-			      struct page **newpagep, int force,
-			      enum migrate_mode mode, enum migrate_reason reason,
-			      struct list_head *ret)
-{
-	int rc = MIGRATEPAGE_UNMAP;
-	struct page *newpage = NULL;
-
-	if (!thp_migration_supported() && PageTransHuge(page))
-		return -ENOSYS;
-
-	if (page_count(page) == 1) {
-		/* Page was freed from under us. So we are done. */
-		ClearPageActive(page);
-		ClearPageUnevictable(page);
-		/* free_pages_prepare() will clear PG_isolated. */
-		list_del(&page->lru);
-		migrate_page_done(page, reason);
-		return MIGRATEPAGE_SUCCESS;
-	}
-
-	newpage = get_new_page(page, private);
-	if (!newpage)
-		return -ENOMEM;
-	*newpagep = newpage;
-
-	newpage->private = 0;
-	rc = __migrate_page_unmap(page, newpage, force, mode);
-	if (rc == MIGRATEPAGE_UNMAP)
-		return rc;
-
-	/*
-	 * A page that has not been migrated will have kept its
-	 * references and be restored.
-	 */
-	/* restore the page to right list. */
-	if (rc != -EAGAIN)
-		list_move_tail(&page->lru, ret);
-
-	if (put_new_page)
-		put_new_page(newpage, private);
-	else
-		put_page(newpage);
+	list_del(&page->lru);
+	migrate_page_undo_page(page, 0, anon_vma, true, NULL);
+	migrate_page_done(page, reason);
 
 	return rc;
-}
 
-/* Migrate the page to the newly allocated page in newpage. */
-static int migrate_page_move(free_page_t put_new_page, unsigned long private,
-			     struct page *page, struct page *newpage,
-			     enum migrate_mode mode, enum migrate_reason reason,
-			     struct list_head *ret)
-{
-	int rc;
-
-	rc = __migrate_page_move(page, newpage, mode);
-	if (rc == MIGRATEPAGE_SUCCESS)
-		set_page_owner_migrate_reason(newpage, reason);
-
-	if (rc != -EAGAIN) {
-		/*
-		 * A page that has been migrated has all references
-		 * removed and will be freed. A page that has not been
-		 * migrated will have kept its references and be restored.
-		 */
-		list_del_init(&page->lru);
+out:
+	if (rc == -EAGAIN) {
+		__migrate_page_record(newpage, page_was_mapped, anon_vma);
+		return rc;
 	}
 
-	/*
-	 * If migration is successful, releases reference grabbed during
-	 * isolation. Otherwise, restore the page to right list unless
-	 * we want to retry.
-	 */
-	if (rc == MIGRATEPAGE_SUCCESS) {
-		migrate_page_done(page, reason);
-	} else if (rc != -EAGAIN) {
-		list_add_tail(&page->lru, ret);
-
-		if (put_new_page)
-			put_new_page(newpage, private);
-		else
-			put_page(newpage);
-	}
+	migrate_page_undo_page(page, page_was_mapped, anon_vma, true, ret);
+	list_del(&newpage->lru);
+	migrate_page_undo_newpage(newpage, true, put_new_page, private);
 
 	return rc;
 }
@@ -1763,9 +1723,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 
 		__migrate_page_extract(newpage, &page_was_mapped, &anon_vma);
 		migrate_page_undo_page(page, page_was_mapped, anon_vma,
-				       &ret_pages);
+				       true, &ret_pages);
 		list_del(&newpage->lru);
-		migrate_page_undo_newpage(newpage, put_new_page, private);
+		migrate_page_undo_newpage(newpage, true, put_new_page, private);
 		newpage = newpage2;
 		newpage2 = list_next_entry(newpage, lru);
 	}
Signature · 1 line
2.35.1
author
#33 lore raw

RFC 6/6 mm/migrate_pages: batch flushing TLB

In reply to Huang Ying

The TLB flushing will cost quite some CPU cycles during the page migration in some situations. For example, when migrate a page of a process with multiple active threads that run on multiple CPUs. After batching the _unmap and _move in migrate_pages(), the TLB flushing can be batched easily with the existing TLB flush batching mechanism. This patch implements that.

We use the following test case to test the patch.

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

Cc Zi Yan <[email protected]>
Cc Yang Shi <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Matthew Wilcox <[email protected]>
---
 mm/migrate.c |  4 +++-
 mm/rmap.c    | 24 ++++++++++++++++++++----
 2 files changed, 23 insertions(+), 5 deletions(-)
mm/migrate.c +3 -1
index 042fa147f302..a0de0d9b4d41 100644
@@ -1179,7 +1179,7 @@ static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
 		/* Establish migration ptes */
 		VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
 				page);
-		try_to_migrate(folio, 0);
+		try_to_migrate(folio, TTU_BATCH_FLUSH);
 		page_was_mapped = 1;
 	}
 
@@ -1647,6 +1647,8 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page,
 	nr_thp_failed += thp_retry;
 	nr_failed_pages += nr_retry_pages;
 move:
+	try_to_unmap_flush();
+
 	retry = 1;
 	thp_retry = 1;
 	for (pass = 0; pass < 10 && (retry || thp_retry); pass++) {
mm/rmap.c +20 -4
index 93d5a6f793d2..ab88136720dc 100644
@@ -1960,8 +1960,24 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 			pteval = huge_ptep_clear_flush(vma, address, pvmw.pte);
 		} else {
 			flush_cache_page(vma, address, pte_pfn(*pvmw.pte));
-			/* Nuke the page table entry. */
-			pteval = ptep_clear_flush(vma, address, pvmw.pte);
+			/*
+			 * Nuke the page table entry.
+			 */
+			if (should_defer_flush(mm, flags)) {
+				/*
+				 * We clear the PTE but do not flush so potentially
+				 * a remote CPU could still be writing to the folio.
+				 * If the entry was previously clean then the
+				 * architecture must guarantee that a clear->dirty
+				 * transition on a cached TLB entry is written through
+				 * and traps if the PTE is unmapped.
+				 */
+				pteval = ptep_get_and_clear(mm, address, pvmw.pte);
+
+				set_tlb_ubc_flush_pending(mm, pte_dirty(pteval));
+			} else {
+				pteval = ptep_clear_flush(vma, address, pvmw.pte);
+			}
 		}
 
 		/* Set the dirty flag on the folio now the pte is gone. */
@@ -2128,10 +2144,10 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)
 
 	/*
 	 * Migration always ignores mlock and only supports TTU_RMAP_LOCKED and
-	 * TTU_SPLIT_HUGE_PMD and TTU_SYNC flags.
+	 * TTU_SPLIT_HUGE_PMD, TTU_SYNC and TTU_BATCH_FLUSH flags.
 	 */
 	if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD |
-					TTU_SYNC)))
+					TTU_SYNC | TTU_BATCH_FLUSH)))
 		return;
 
 	if (folio_is_zone_device(folio) &&
Signature · 1 line
2.35.1
reviewer
#34 lore raw
In reply to Huang Ying
Quoted 45 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

Quoted 7 lines

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

Thank you for the patchset. Batching page migration will definitely improve its throughput from my past experiments[1] and starting with TLB flushing is a good first step.

BTW, what is the rationality behind the increased page migration success rate per second?

Quoted 6 lines

This patchset is based on v6.0-rc5 and the following patchset,

[PATCH -V3 0/8] migrate_pages(): fix several bugs in error path
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

The migrate_pages() related code is converting to folio now. So this patchset cannot apply recent akpm/mm-unstable branch. This patchset is used to check the basic idea. If it is OK, I will rebase the patchset on top of folio changes.

Best Regards,
Huang, Ying

[1] https://lwn.net/Articles/784925/

--
Best Regards,
Yan, Zi

#35 lore raw
Quoted 52 lines · Zi Yan
Quoted 50 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

Quoted 6 lines
Quoted 2 lines

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

Thank you for the patchset. Batching page migration will definitely improve its throughput from my past experiments[1] and starting with TLB flushing is a good first step.

Thanks for the pointer, the patch description provides valuable information for me already!

Quoted 2 lines

BTW, what is the rationality behind the increased page migration success rate per second?

From perf profiling data, in the base kernel,

migrate_pages.migrate_to_node.do_migrate_pages.kernel_migrate_pages.__x64_sys_migrate_pages: 2.87
ptep_clear_flush.try_to_migrate_one.rmap_walk_anon.try_to_migrate.__unmap_and_move: 2.39

Because pmbench run in the system too, the CPU cycles of migrate_pages() is about 2.87%. While the CPU cycles for TLB flushing is 2.39%. That is, 2.39/2.87 = 83.3% CPU cycles of migrate_pages() are used for TLB flushing.

After batching the TLB flushing, the perf profiling data becomes,

migrate_pages.migrate_to_node.do_migrate_pages.kernel_migrate_pages.__x64_sys_migrate_pages: 2.77
move_to_new_folio.migrate_pages_batch.migrate_pages.migrate_to_node.do_migrate_pages: 1.68
copy_page.folio_copy.migrate_folio.move_to_new_folio.migrate_pages_batch: 1.21

1.21/2.77 = 43.7% CPU cycles of migrate_pages() are used for page copying now.

try_to_migrate_one: 0.23

The CPU cycles of unmapping and TLB flushing becomes 0.23/2.77 = 8.3% of migrate_pages().

All in all, after the optimization, we do much less TLB flushing, which consumes a lot of CPU cycles before the optimization. So the throughput of migrate_pages() increases greatly.

I will add these data in the next version of patch.

Best Regards,
Huang, Ying

Quoted 20 lines
Quoted 6 lines

This patchset is based on v6.0-rc5 and the following patchset,

[PATCH -V3 0/8] migrate_pages(): fix several bugs in error path
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

The migrate_pages() related code is converting to folio now. So this patchset cannot apply recent akpm/mm-unstable branch. This patchset is used to check the basic idea. If it is OK, I will rebase the patchset on top of folio changes.

Best Regards,
Huang, Ying

[1] https://lwn.net/Articles/784925/

--
Best Regards,
Yan, Zi

#36 lore raw
In reply to Zi Yan

Hi Huang,

    This is an exciting change, but on ARM64 machine the TLB flushing are not through IPI, it depends on 'vale1is'

instruction,so I'm wondering if there's also a  benefit on arm64, and I'm going to test it on an ARM64 machine.

Quoted 79 lines · Zi Yan
Quoted 45 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

Quoted 7 lines

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

Thank you for the patchset. Batching page migration will definitely improve its throughput from my past experiments[1] and starting with TLB flushing is a good first step.

BTW, what is the rationality behind the increased page migration success rate per second?

Quoted 5 lines

This patchset is based on v6.0-rc5 and the following patchset,

[PATCH -V3 0/8] migrate_pages(): fix several bugs in error path
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

The migrate_pages() related code is converting to folio now. So this patchset cannot apply recent akpm/mm-unstable branch. This patchset is used to check the basic idea. If it is OK, I will rebase the patchset on top of folio changes.

Best Regards,
Huang, Ying

[1] https://lwn.net/Articles/784925/

--
Best Regards,
Yan, Zi

#37 lore raw
Quoted 7 lines · haoxin

Hi Huang,

This is an exciting change, but on ARM64 machine the TLB flushing are not through IPI, it depends on 'vale1is'

instruction so I'm wondering if there's also a benefit on arm64, and I'm going to test it on an ARM64 machine.

We have no arm64 machine to test and I know very little about arm64. Thanks for information and testing.

Best Regards,
Huang, Ying

Quoted 81 lines

( 2022/9/21 H11:47, Zi Yan S:

Quoted 79 lines · Huang Ying
Quoted 46 lines

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

Quoted 7 lines

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

Thank you for the patchset. Batching page migration will definitely improve its throughput from my past experiments[1] and starting with TLB flushing is a good first step.

BTW, what is the rationality behind the increased page migration success rate per second?

Quoted 5 lines

This patchset is based on v6.0-rc5 and the following patchset,

[PATCH -V3 0/8] migrate_pages(): fix several bugs in error path
https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines

The migrate_pages() related code is converting to folio now. So this patchset cannot apply recent akpm/mm-unstable branch. This patchset is used to check the basic idea. If it is OK, I will rebase the patchset on top of folio changes.

Best Regards,
Huang, Ying

[1] https://lwn.net/Articles/784925/

--
Best Regards,
Yan, Zi

#38 lore raw
In reply to Huang Ying
Quoted 32 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

Quoted 7 lines

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

Thanks for the patchset. I find it hitting the following BUG() when running mmtests/autonumabench:

kernel BUG at mm/migrate.c:2432!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI

CPU: 7 PID: 7150 Comm: numa01 Not tainted 6.0.0-rc5+ #171
Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.5.6 10/06/2021
RIP: 0010:migrate_misplaced_page+0x670/0x830 
Code: 36 48 8b 3c c5 e0 7a 19 8d e8 dc 10 f7 ff 4c 89 e7 e8 f4 43 f5 ff 8b 55 bc 85 d2 75 6f 48 8b 45 c0 4c 39 e8 0f 84 b0 fb ff ff <0f> 0b 48 8b 7d 90 e9 ec fc ff ff 48 83 e8 01 e9 48 fa ff ff 48 83
RSP: 0000:ffffb1b29ec3fd38 EFLAGS: 00010202
RAX: ffffe946460f8248 RBX: 0000000000000001 RCX: ffffe946460f8248
RDX: 0000000000000000 RSI: ffffe946460f8248 RDI: ffffb1b29ec3fce0
RBP: ffffb1b29ec3fda8 R08: 0000000000000000 R09: 0000000000000005
R10: 0000000000000001 R11: 0000000000000000 R12: ffffe946460f8240
R13: ffffb1b29ec3fd68 R14: 0000000000000001 R15: ffff9698beed5000
FS:  00007fcc31fee640(0000) GS:ffff9697b0000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcc3a3a5000 CR3: 000000016e89c002 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <TASK>
 __handle_mm_fault+0xb87/0xff0
 handle_mm_fault+0x126/0x3c0
 do_user_addr_fault+0x1ed/0x690
 exc_page_fault+0x84/0x2c0
 asm_exc_page_fault+0x27/0x30 
RIP: 0033:0x7fccfa1a1180
Code: 81 fa 80 00 00 00 76 d2 c5 fe 7f 40 40 c5 fe 7f 40 60 48 83 c7 80 48 81 fa 00 01 00 00 76 2b 48 8d 90 80 00 00 00 48 83 e2 c0 <c5> fd 7f 02 c5 fd 7f 42 20 c5 fd 7f 42 40 c5 fd 7f 42 60 48 83 ea
RSP: 002b:00007fcc31fede38 EFLAGS: 00010283
RAX: 00007fcc39fff010 RBX: 000000000000002c RCX: 00007fccfa11ea3d
RDX: 00007fcc3a3a5000 RSI: 0000000000000000 RDI: 00007fccf9ffef90
RBP: 00007fcc39fff010 R08: 00007fcc31fee640 R09: 00007fcc31fee640
R10: 00007ffdecef614f R11: 0000000000000246 R12: 00000000c0000000
R13: 0000000000000000 R14: 00007fccfa094850 R15: 00007ffdecef6190

This is BUG_ON(!list_empty(&migratepages)) in migrate_misplaced_page().

Regards,
Bharata.

#39 lore raw
Quoted 76 lines · Bharata B Rao
Quoted 32 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

Quoted 7 lines

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

Thanks for the patchset. I find it hitting the following BUG() when running mmtests/autonumabench:

kernel BUG at mm/migrate.c:2432!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI

CPU: 7 PID: 7150 Comm: numa01 Not tainted 6.0.0-rc5+ #171
Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.5.6 10/06/2021
RIP: 0010:migrate_misplaced_page+0x670/0x830 
Code: 36 48 8b 3c c5 e0 7a 19 8d e8 dc 10 f7 ff 4c 89 e7 e8 f4 43 f5 ff 8b 55 bc 85 d2 75 6f 48 8b 45 c0 4c 39 e8 0f 84 b0 fb ff ff <0f> 0b 48 8b 7d 90 e9 ec fc ff ff 48 83 e8 01 e9 48 fa ff ff 48 83
RSP: 0000:ffffb1b29ec3fd38 EFLAGS: 00010202
RAX: ffffe946460f8248 RBX: 0000000000000001 RCX: ffffe946460f8248
RDX: 0000000000000000 RSI: ffffe946460f8248 RDI: ffffb1b29ec3fce0
RBP: ffffb1b29ec3fda8 R08: 0000000000000000 R09: 0000000000000005
R10: 0000000000000001 R11: 0000000000000000 R12: ffffe946460f8240
R13: ffffb1b29ec3fd68 R14: 0000000000000001 R15: ffff9698beed5000
FS:  00007fcc31fee640(0000) GS:ffff9697b0000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcc3a3a5000 CR3: 000000016e89c002 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <TASK>
 __handle_mm_fault+0xb87/0xff0
 handle_mm_fault+0x126/0x3c0
 do_user_addr_fault+0x1ed/0x690
 exc_page_fault+0x84/0x2c0
 asm_exc_page_fault+0x27/0x30 
RIP: 0033:0x7fccfa1a1180
Code: 81 fa 80 00 00 00 76 d2 c5 fe 7f 40 40 c5 fe 7f 40 60 48 83 c7 80 48 81 fa 00 01 00 00 76 2b 48 8d 90 80 00 00 00 48 83 e2 c0 <c5> fd 7f 02 c5 fd 7f 42 20 c5 fd 7f 42 40 c5 fd 7f 42 60 48 83 ea
RSP: 002b:00007fcc31fede38 EFLAGS: 00010283
RAX: 00007fcc39fff010 RBX: 000000000000002c RCX: 00007fccfa11ea3d
RDX: 00007fcc3a3a5000 RSI: 0000000000000000 RDI: 00007fccf9ffef90
RBP: 00007fcc39fff010 R08: 00007fcc31fee640 R09: 00007fcc31fee640
R10: 00007ffdecef614f R11: 0000000000000246 R12: 00000000c0000000
R13: 0000000000000000 R14: 00007fccfa094850 R15: 00007ffdecef6190
Quoted 1 line

This is BUG_ON(!list_empty(&migratepages)) in migrate_misplaced_page().

Thank you very much for reporting! I haven't reproduced this yet. But I will pay special attention to this when develop the next version, even if I cannot reproduce this finally.

Best Regards,
Huang, Ying

#40 lore raw
Quoted 5 lines · Ying
Quoted 4 lines · Bharata B Rao

Thanks for the patchset. I find it hitting the following BUG() when running mmtests/autonumabench:

Quoted 7 lines
Quoted 3 lines

kernel BUG at mm/migrate.c:2432!

This is BUG_ON(!list_empty(&migratepages)) in migrate_misplaced_page().

Thank you very much for reporting! I haven't reproduced this yet. But I will pay special attention to this when develop the next version, even if I cannot reproduce this finally.

The following change fixes the above reported BUG_ON().

mm/migrate.c +1 -1
index a0de0d9b4d41..c11dd82245e5 100644
@@ -1197,7 +1197,7 @@ static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
         * references and be restored.
         */
        /* restore the page to right list. */
-       if (rc != -EAGAIN)
+       if (rc == -EAGAIN)
                 ret = NULL;
 
        migrate_page_undo_page(page, page_was_mapped, anon_vma, locked, ret);

The pages that returned from unmapping stage with -EAGAIN used to end up on "ret" list rather than continuing on the "from" list.

Regards,
Bharata.

#41 lore raw
Quoted 30 lines · Bharata B Rao
Quoted 5 lines · Ying
Quoted 4 lines · Bharata B Rao

Thanks for the patchset. I find it hitting the following BUG() when running mmtests/autonumabench:

Quoted 7 lines
Quoted 3 lines

kernel BUG at mm/migrate.c:2432!

This is BUG_ON(!list_empty(&migratepages)) in migrate_misplaced_page().

Thank you very much for reporting! I haven't reproduced this yet. But I will pay special attention to this when develop the next version, even if I cannot reproduce this finally.

The following change fixes the above reported BUG_ON().

mm/migrate.c +1 -1
index a0de0d9b4d41..c11dd82245e5 100644
@@ -1197,7 +1197,7 @@ static int migrate_page_unmap(new_page_t get_new_page, free_page_t put_new_page,
         * references and be restored.
         */
        /* restore the page to right list. */
-       if (rc != -EAGAIN)
+       if (rc == -EAGAIN)
                 ret = NULL;
 
        migrate_page_undo_page(page, page_was_mapped, anon_vma, locked, ret);
Quoted 2 lines

The pages that returned from unmapping stage with -EAGAIN used to end up on "ret" list rather than continuing on the "from" list.

Yes. You are right. Thank you very much!

Digging some history, it is found that the code was correct in previous versions, but became wrong for mistake during code rebasing. Will be more careful in the future and try to organize the patchset better to make it easier to review the changes.

Best Regards,
Huang, Ying

#42 lore raw
In reply to Huang Ying
Quoted 15 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

Quoted 8 lines

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

We use a very similar sequence for the migrate_vma_*() set of calls. It would be good if we could one day consolidate the two. I believe the biggest hindrance to that is migrate_vma_*() operates on arrays of pfns rather than a list of pages. The reason for that is it needs to migrate non-lru pages and hence can't use page->lru to create a list of pages to migrate.

So from my perspective I think this direction is good as it would help with that. One thing to watch out for is deadlocking if locking multiple pages though.

Quoted 41 lines

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%.

This patchset is based on v6.0-rc5 and the following patchset,

[PATCH -V3 0/8] migrate_pages(): fix several bugs in error path
https://lore.kernel.org/lkml/[email protected]/

The migrate_pages() related code is converting to folio now. So this patchset cannot apply recent akpm/mm-unstable branch. This patchset is used to check the basic idea. If it is OK, I will rebase the patchset on top of folio changes.

Best Regards,
Huang, Ying

#43 lore raw
In reply to Huang Ying

Hi,  Huang

Quoted 43 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

Quoted 7 lines

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this: (i am not sure whether it is reasonable, but it seems worked) ./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

o/patch w/patch
real 0m0.035s real 0m0.024s
user 0m0.000s user 0m0.000s
sys 0m0.035s sys 0m0.024s

the migratepages time is reduced above 32%.

But there has a problem, i see the batch flush is called by migrate_pages_batch

	try_to_unmap_flush
		arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/

#44 lore raw
Quoted 65 lines · haoxin

Hi, Huang

( 2022/9/21 H2:06, Huang Ying S:

Quoted 43 lines

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

Quoted 7 lines

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this: (i am not sure whether it is reasonable, but it seems worked) ./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

o/patch w/patch
real 0m0.035s real 0m0.024s
user 0m0.000s user 0m0.000s
sys 0m0.035s sys 0m0.024s

the migratepages time is reduced above 32%.

Quoted 8 lines

But there has a problem, i see the batch flush is called by migrate_pages_batch

	try_to_unmap_flush
		arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Yes. And should_defer_flush() will always return false too. That is, the TLB will still be flushed, but will not be batched.

Quoted 2 lines

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/

Yes. This will bring TLB flush batching to ARM64.

Best Regards,
Huang, Ying

#45 lore raw
Quoted 72 lines · Ying
Quoted 70 lines · haoxin

Hi, Huang

( 2022/9/21 H2:06, Huang Ying S:

Quoted 43 lines

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

Quoted 7 lines

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this: (i am not sure whether it is reasonable, but it seems worked) ./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

o/patch w/patch
real 0m0.035s real 0m0.024s
user 0m0.000s user 0m0.000s
sys 0m0.035s sys 0m0.024s

the migratepages time is reduced above 32%.

But there has a problem, i see the batch flush is called by migrate_pages_batch

	try_to_unmap_flush
		arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.
Quoted 5 lines
Quoted 3 lines

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Yes. And should_defer_flush() will always return false too. That is, the TLB will still be flushed, but will not be batched.

Oh, yes, i  ignore this, thank you.

Quoted 4 lines
Quoted 3 lines

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/

Yes. This will bring TLB flush batching to ARM64.

Next time,  i will combine with this patch, and do some test again, do you have any suggestion about  benchmark ?

Quoted 3 lines

Best Regards,
Huang, Ying

#46 lore raw
Quoted 78 lines · haoxin

( 2022/9/28 H10:01, Huang, Ying S:

Quoted 72 lines
Quoted 70 lines · haoxin

Hi, Huang

( 2022/9/21 H2:06, Huang Ying S:

Quoted 43 lines

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

for each page
unmap
flush TLB
copy
restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

for each page
unmap
for each page
flush TLB
for each page
copy
for each page
restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

Quoted 7 lines

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this: (i am not sure whether it is reasonable, but it seems worked) ./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

o/patch w/patch
real 0m0.035s real 0m0.024s
user 0m0.000s user 0m0.000s
sys 0m0.035s sys 0m0.024s

the migratepages time is reduced above 32%.

But there has a problem, i see the batch flush is called by migrate_pages_batch

	try_to_unmap_flush
		arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.
Quoted 5 lines
Quoted 3 lines

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Yes. And should_defer_flush() will always return false too. That is, the TLB will still be flushed, but will not be batched.

Quoted 7 lines

Oh, yes, i ignore this, thank you.

Quoted 4 lines
Quoted 3 lines

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/

Yes. This will bring TLB flush batching to ARM64.

Next time, i will combine with this patch, and do some test again, do you have any suggestion about benchmark ?

I think your benchmark should be OK. If multiple threads are used, the effect of patchset will be better.

Best Regards,
Huang, Ying

#47 lore raw
In reply to haoxin
Quoted 50 lines · haoxin

Hi, Huang

Quoted 47 lines · Huang Ying

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

   for each page
     unmap
     flush TLB
     copy
     restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying.  That is, we can change the code to something as follows,

   for each page
     unmap
   for each page
     flush TLB
   for each page
     copy
   for each page
     restore map

The total number of TLB flushing IPI can be reduced considerably.  And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching.  Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases.  So the latency may be hurt.  To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR.  That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

  - Run pmbench memory accessing benchmark

Quoted 8 lines
Quoted 3 lines

  - Run migratepages to migrate pages of pmbench between node 0 and node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this:  (i am not sure whether it is reasonable, but it seems worked) ./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

FYI, I have ported pmbench to AArch64 [1]. The project seems to be abandoned on bitbucket,

I wonder if it makes sense to fork it elsewhere and push the pending PRs there.

[1] https://bitbucket.org/jisooy/pmbench/pull-requests/5

Quoted 25 lines

o/patch w/patch
real    0m0.035s      real    0m0.024s
user    0m0.000s      user    0m0.000s
sys    0m0.035s        sys    0m0.024s

the migratepages time is reduced above 32%.

But there has a problem, i see the batch flush is called by migrate_pages_batch
    try_to_unmap_flush
        arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/

#48 lore raw
Quoted 60 lines · Hesham Almatary
Quoted 50 lines · haoxin

Hi, Huang

( 2022/9/21 H2:06, Huang Ying S:

Quoted 47 lines

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

  for each page
   unmap
   flush TLB
   copy
   restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

  for each page
   unmap
  for each page
   flush TLB
  for each page
   copy
  for each page
   restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

Quoted 9 lines
Quoted 3 lines

- Run migratepages to migrate pages of pmbench between node 0 and
  node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this: (i am not sure whether it is reasonable, but it seems worked)
./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

Quoted 7 lines

FYI, I have ported pmbench to AArch64 [1]. The project seems to be abandoned on bitbucket,

I wonder if it makes sense to fork it elsewhere and push the pending PRs there.

[1] https://bitbucket.org/jisooy/pmbench/pull-requests/5

Maybe try to contact the original author with email firstly?

Best Regards,
Huang, Ying

Quoted 25 lines
Quoted 25 lines

o/patch w/patch
real  0m0.035s   real  0m0.024s
user  0m0.000s   user  0m0.000s
sys  0m0.035s    sys  0m0.024s

the migratepages time is reduced above 32%.

But there has a problem, i see the batch flush is called by migrate_pages_batch
  try_to_unmap_flush
    arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/

#49 lore raw
Quoted 62 lines · Ying
Quoted 60 lines · Hesham Almatary
Quoted 59 lines · haoxin

Hi, Huang

( 2022/9/21 H2:06, Huang Ying S:

Quoted 43 lines

From: "Huang, Ying" <[email protected]>

Now, migrate_pages() migrate pages one by one, like the fake code as follows,

? for each page
?? unmap
?? flush TLB
?? copy
?? restore map

If multiple pages are passed to migrate_pages(), there are opportunities to batch the TLB flushing and copying. That is, we can change the code to something as follows,

? for each page
?? unmap
? for each page
?? flush TLB
? for each page
?? copy
? for each page
?? restore map

The total number of TLB flushing IPI can be reduced considerably. And we may use some hardware accelerator such as DSA to accelerate the page copying.

So in this patch, we refactor the migrate_pages() implementation and implement the TLB flushing batching. Base on this, hardware accelerated page copying can be implemented.

If too many pages are passed to migrate_pages(), in the naive batched implementation, we may unmap too many pages at the same time. The possibility for a task to wait for the migrated pages to be mapped again increases. So the latency may be hurt. To deal with this issue, the max number of pages be unmapped in batch is restricted to no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration.

We use the following test to measure the performance impact of the patchset,

Quoted 7 lines

On a 2-socket Intel server,

- Run pmbench memory accessing benchmark

- Run migratepages to migrate pages of pmbench between node 0 and
? node 1 back and forth.

As the pmbench can not run on arm64 machine, so i use lmbench instead. I test case like this: (i am not sure whether it is reasonable, but it seems worked)
./bw_mem -N10000 10000m rd &
time migratepages pid node0 node1

Quoted 8 lines
Quoted 7 lines

FYI, I have ported pmbench to AArch64 [1]. The project seems to be abandoned on bitbucket,

I wonder if it makes sense to fork it elsewhere and push the pending PRs there.

[1] https://bitbucket.org/jisooy/pmbench/pull-requests/5

Maybe try to contact the original author with email firstly?

That's  a good idea. I'm not planning to fork/maintain it myself, but if anyone

is interested in doing so, I am happy to help out and submit PRs there.

Quoted 28 lines

Best Regards,
Huang, Ying

Quoted 25 lines
Quoted 25 lines

o/patch w/patch
real? 0m0.035s?? real? 0m0.024s
user? 0m0.000s?? user? 0m0.000s
sys? 0m0.035s??? sys? 0m0.024s

the migratepages time is reduced above 32%.

But there has a problem, i see the batch flush is called by migrate_pages_batch
??try_to_unmap_flush
??? arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.

But in arm64, the arch_tlbbatch_flush are not supported, becasue it not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.

So, the tlb batch flush means no any flush is did, it is a empty func.

Maybe this patch can help solve this problem.
https://lore.kernel.org/linux-arm-kernel/[email protected]/T/