Accelerate page migration with batching and multi threads
Subsystems (4)
Zi Yan
reviewercover Accelerate page migration with batching and multi threads
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
The motivations are:
1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on x86_64, ARM64 and 64KB on ARM64.
2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system.
Design
===
It is based on Shivank's patchset and revise MIGRATE_SYNC_NO_COPY (renamed to MIGRATE_NO_COPY) to avoid folio copy operation inside migrate_folio_move() and perform them in one shot afterwards. A copy_page_lists_mt() function is added to use multi threads to copy folios from src list to dst list.
Changes compared to Shivank's patchset (mainly rewrote batching folio copy code)
===
1. mig_info is removed, so no memory allocation is needed during batching folio copies. src->private is used to store old page state and anon_vma after folio metadata is copied from src to dst.
2. move_to_new_folio() and migrate_folio_move() are refactored to remove redundant code in migrate_folios_batch_move().
3. folio_mc_copy() is used for the single threaded copy code to keep the original kernel behavior.
Performance
===
I benchmarked move_pages() throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured.
The tables below show move_pages() throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s.
The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32
32 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.80
2MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32
1 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84
TODOs
===
1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority.
2. Eliminate memory allocation during multi-threaded folio copy routine if possible.
3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3].
4. Use non-temporal CPU instructions to avoid cache pollution issues.
5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since kmap_local_page() would be needed for each kernel folio copy work threads and expensive.
6. A better interface than copy_page_lists_mt() to allow DMA data copy to be used as well.
Let me know your thoughts. Thanks.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
Byungchul Park (1):
mm: separate move/undo doing on folio list from migrate_pages_batch()
Zi Yan (4):
mm/migrate: factor out code in move_to_new_folio() and
migrate_folio_move()
mm/migrate: add migrate_folios_batch_move to batch the folio move
operations
mm/migrate: introduce multi-threaded page copy routine
test: add sysctl for folio copy tests and adjust
NR_MAX_BATCHED_MIGRATION
include/linux/migrate.h | 3 + include/linux/migrate_mode.h | 2 + include/linux/mm.h | 4 + include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++- mm/Makefile | 2 +- mm/copy_pages.c | 190 +++++++++++++++ mm/migrate.c | 443 +++++++++++++++++++++++++++-------- 8 files changed, 577 insertions(+), 97 deletions(-) create mode 100644 mm/copy_pages.c
Signature · 1 line
2.45.2
Zi Yan
reviewerRFC PATCH 1/5 mm: separate move/undo doing on folio list from migrate_pages_batch()
From: Byungchul Park <[email protected]>
Functionally, no change. This is a preparatory patch picked from luf (lazy unmap flush) patch series. This patch improve code organization and readability for steps involving migrate_folio_move().
Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch().
--- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-)
mm/migrate.c +83 -51
@@ -1690,6 +1690,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio,return nr_failed; } +static void migrate_folios_move (struct list_head *src_folios, +struct list_head *dst_folios, +free_folio_t put_new_folio,unsigned long private, +enum migrate_mode mode,int reason, +struct list_head *ret_folios, +struct migrate_pages_stats *stats, +int *retry,int *thp_retry,int *nr_failed, +int *nr_retry_pages) +{ +struct folio *folio, *folio2, *dst, *dst2; +bool is_thp; +int nr_pages; +int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); + +cond_resched (); + + rc =migrate_folio_move (put_new_folio, private, + folio, dst, mode, + reason, ret_folios); +/* +* The rules are: +* Success: folio will be freed +* -EAGAIN: stay on the unmap_folios list +* Other errno: put on ret_folios list +*/ +switch (rc) { +case -EAGAIN : + *retry +=1 ; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; +break ; +case MIGRATEPAGE_SUCCESS : + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; +break ; +default : + *nr_failed +=1 ; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; +break ; + } + dst = dst2; + dst2 =list_next_entry (dst, lru); + } +} + +static void migrate_folios_undo (struct list_head *src_folios, +struct list_head *dst_folios, +free_folio_t put_new_folio,unsigned long private, +struct list_head *ret_folios) +{ +struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; + +__migrate_folio_extract (dst, &old_page_state, &anon_vma); +migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , + anon_vma, true, ret_folios); +list_del (&dst->lru); +migrate_folio_undo_dst (dst, true, put_new_folio, private); + dst = dst2; + dst2 =list_next_entry (dst, lru); + } +} +/* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios.
@@ -1712,7 +1787,7 @@ static int migrate_pages_batch(struct list_head *from,int pass =0 ;bool is_thp = false;bool is_large = false; -struct folio *folio, *folio2, *dst =NULL , *dst2; +struct folio *folio, *folio2, *dst =NULL ;int rc, rc_saved =0 , nr_pages;LIST_HEAD (unmap_folios);LIST_HEAD (dst_folios);
@@ -1883,42 +1958,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry =0 ; nr_retry_pages =0 ; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 =list_next_entry (dst, lru); -list_for_each_entry_safe (folio, folio2, &unmap_folios, lru) { - is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); - nr_pages =folio_nr_pages (folio); - -cond_resched (); - - rc =migrate_folio_move (put_new_folio, private, - folio, dst, mode, - reason, ret_folios); -/* -* The rules are: -* Success: folio will be freed -* -EAGAIN: stay on the unmap_folios list -* Other errno: put on ret_folios list -*/ -switch (rc) { -case -EAGAIN : - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; -break ; -case MIGRATEPAGE_SUCCESS : - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; -break ; -default : - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; -break ; - } - dst = dst2; - dst2 =list_next_entry (dst, lru); - } +/* Move the unmapped folios */ +migrate_folios_move (&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry;
@@ -1927,20 +1971,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out:/* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 =list_next_entry (dst, lru); -list_for_each_entry_safe (folio, folio2, &unmap_folios, lru) { -int old_page_state =0 ; -struct anon_vma *anon_vma =NULL ; - -__migrate_folio_extract (dst, &old_page_state, &anon_vma); -migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , - anon_vma, true, ret_folios); -list_del (&dst->lru); -migrate_folio_undo_dst (dst, true, put_new_folio, private); - dst = dst2; - dst2 =list_next_entry (dst, lru); - } +migrate_folios_undo (&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios);return rc; }
Signature · 1 line
2.45.2
Zi Yan
reviewerRFC PATCH 2/5 mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move()
In reply to Zi YanNo function change is intended. The factored out code will be reused in an upcoming batched folio move function.
--- mm/migrate.c | 101 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 36 deletions(-)
mm/migrate.c +65 -36
@@ -1014,18 +1014,7 @@ static int fallback_migrate_folio(struct address_space *mapping,return migrate_folio (mapping, dst, src, mode); } -/* -* Move a page to a newly allocated page -* The page is locked and all ptes have been successfully removed. -* -* The new page will have replaced the old page if this function -* is successful. -* -* Return value: -* < 0 - error code -* MIGRATEPAGE_SUCCESS - success -*/ -static int move_to_new_folio (struct folio *dst,struct folio *src, +static int _move_to_new_folio_prep (struct folio *dst,struct folio *src,enum migrate_mode mode) {int rc = -EAGAIN ;
@@ -1072,7 +1061,13 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,WARN_ON_ONCE (rc ==MIGRATEPAGE_SUCCESS && !folio_test_isolated (src)); } +out: +return rc; +} +static void _move_to_new_folio_finalize (struct folio *dst,struct folio *src, +int rc) +{/* * When successful, old pagecache src->mapping must be cleared before * src is freed; but stats require that PageAnon be left as PageAnon.
@@ -1099,7 +1094,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,if (likely (!folio_is_zone_device (dst)))flush_dcache_folio (dst); } -out: +} + + +/* +* Move a page to a newly allocated page +* The page is locked and all ptes have been successfully removed. +* +* The new page will have replaced the old page if this function +* is successful. +* +* Return value: +* < 0 - error code +* MIGRATEPAGE_SUCCESS - success +*/ +static int move_to_new_folio (struct folio *dst,struct folio *src, +enum migrate_mode mode) +{ +int rc; + + rc =_move_to_new_folio_prep (dst, src, mode); + +_move_to_new_folio_finalize (dst, src, rc); +return rc; }
@@ -1344,29 +1361,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,return rc; } -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move (free_folio_t put_new_folio,unsigned long private, -struct folio *src,struct folio *dst, -enum migrate_mode mode,enum migrate_reason reason, -struct list_head *ret) +static void _migrate_folio_move_finalize1 (struct folio *src,struct folio *dst, +int old_page_state) { -int rc; -int old_page_state =0 ; -struct anon_vma *anon_vma =NULL ; -bool is_lru = !__folio_test_movable (src); -struct list_head *prev; - -__migrate_folio_extract (dst, &old_page_state, &anon_vma); - prev = dst->lru.prev; -list_del (&dst->lru); - - rc =move_to_new_folio (dst, src, mode); -if (rc) - goto out; - -if (unlikely (!is_lru)) - goto out_unlock_both; -/* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will
@@ -1382,8 +1379,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,if (old_page_state &PAGE_WAS_MAPPED )remove_migration_ptes (src, dst,0 ); +} -out_unlock_both: +static void _migrate_folio_move_finalize2 (struct folio *src,struct folio *dst, +enum migrate_reason reason, +struct anon_vma *anon_vma) +{folio_unlock (dst);set_page_owner_migrate_reason (&dst->page, reason);/*
@@ -1403,6 +1404,34 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private,put_anon_vma (anon_vma);folio_unlock (src);migrate_folio_done (src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move (free_folio_t put_new_folio,unsigned long private, +struct folio *src,struct folio *dst, +enum migrate_mode mode,enum migrate_reason reason, +struct list_head *ret) +{ +int rc; +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; +bool is_lru = !__folio_test_movable (src); +struct list_head *prev; + +__migrate_folio_extract (dst, &old_page_state, &anon_vma); + prev = dst->lru.prev; +list_del (&dst->lru); + + rc =move_to_new_folio (dst, src, mode); +if (rc) + goto out; + +if (unlikely (!is_lru)) + goto out_unlock_both; + +_migrate_folio_move_finalize1 (src, dst, old_page_state); +out_unlock_both: +_migrate_folio_move_finalize2 (src, dst, reason, anon_vma);return rc; out:
Signature · 1 line
2.45.2
Zi Yan
reviewerRFC PATCH 3/5 mm/migrate: add migrate_folios_batch_move to batch the folio move operations
In reply to Zi YanThis is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It also adds MIGRATE_NO_COPY back to migrate_mode, so that folio copy will be skipped during metadata copy process and performed in a batch later.
Currently, the folio move operation is performed individually for each folio in sequential manner:for_each_folio() {
Copy folio metadata like flags and mappings
Copy the folio content from src to dst
Update page tables with dst folio
}
With this patch, we transition to a batch processing approach as shown below:for_each_folio() {
Copy folio metadata like flags and mappings
}
Batch copy all src folios to dstfor_each_folio() {
Update page tables with dst folios
}
dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used.
--- include/linux/migrate_mode.h | 2 + mm/migrate.c | 207 +++++++++++++++++++++++++++++++++-- 2 files changed, 201 insertions(+), 8 deletions(-)
include/linux/migrate_mode.h +2 -0
@@ -7,11 +7,13 @@* on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages +* MIGRATE_NO_COPY will not copy page content */ enum migrate_mode {MIGRATE_ASYNC ,MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC , +MIGRATE_NO_COPY , };enum migrate_reason {
mm/migrate.c +199 -8
271 lines not on this page · raw message
Signature · 1 line
2.45.2
kernel test robot
build botRe: [RFC PATCH 3/5] mm/migrate: add migrate_folios_batch_move to batch the folio move operations
kernel test robot noticed the following build warnings:
Hi Zi,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master sysctl/sysctl-next v6.13-rc5 next-20241220] [cannot apply to mcgrof/sysctl-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Zi-Yan/mm-separate-move-undo-doing-on-folio-list-from-migrate_pages_batch/20250104-012955 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250103172419.4148674-4-ziy%40nvidia.com patch subject: [RFC PATCH 3/5] mm/migrate: add migrate_folios_batch_move to batch the folio move operations config: i386-buildonly-randconfig-004-20250104 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250104/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
mm/migrate.c: In function 'migrate_folios_batch_move':
>> mm/migrate.c:1805:14: warning: variable 'is_lru' set but not used [-Wunused-but-set-variable]
1805 | bool is_lru;
| ^~~~~~vim +/is_lru +1805 mm/migrate.c
1791
1792 static void migrate_folios_batch_move(struct list_head *src_folios,
1793 struct list_head *dst_folios,
1794 free_folio_t put_new_folio, unsigned long private,
1795 enum migrate_mode mode, int reason,
1796 struct list_head *ret_folios,
1797 struct migrate_pages_stats *stats,
1798 int *retry, int *thp_retry, int *nr_failed,
1799 int *nr_retry_pages)
1800 {
1801 struct folio *folio, *folio2, *dst, *dst2;
1802 int rc, nr_pages = 0, nr_mig_folios = 0;
1803 int old_page_state = 0;
1804 struct anon_vma *anon_vma = NULL;
> 1805 bool is_lru;
1806 int is_thp = 0;
1807 LIST_HEAD(err_src);
1808 LIST_HEAD(err_dst);
1809
1810 if (mode != MIGRATE_ASYNC) {
1811 *retry += 1;
1812 return;
1813 }
1814
1815 /*
1816 * Iterate over the list of locked src/dst folios to copy the metadata
1817 */
1818 dst = list_first_entry(dst_folios, struct folio, lru);
1819 dst2 = list_next_entry(dst, lru);
1820 list_for_each_entry_safe(folio, folio2, src_folios, lru) {
1821 is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
1822 nr_pages = folio_nr_pages(folio);
1823 is_lru = !__folio_test_movable(folio);
1824
1825 /*
1826 * dst->private is not cleared here. It is cleared and moved to
1827 * src->private in __migrate_folio().
1828 */
1829 __migrate_folio_read(dst, &old_page_state, &anon_vma);
1830
1831 /*
1832 * Use MIGRATE_NO_COPY mode in migrate_folio family functions
1833 * to copy the flags, mapping and some other ancillary information.
1834 * This does everything except the page copy. The actual page copy
1835 * is handled later in a batch manner.
1836 */
1837 rc = _move_to_new_folio_prep(dst, folio, MIGRATE_NO_COPY);
1838
1839 /*
1840 * -EAGAIN: Move src/dst folios to tmp lists for retry
1841 * Other Errno: Put src folio on ret_folios list, remove the dst folio
1842 * Success: Copy the folio bytes, restoring working pte, unlock and
1843 * decrement refcounter
1844 */
1845 if (rc == -EAGAIN) {
1846 *retry += 1;
1847 *thp_retry += is_thp;
1848 *nr_retry_pages += nr_pages;
1849
1850 list_move_tail(&folio->lru, &err_src);
1851 list_move_tail(&dst->lru, &err_dst);
1852 __migrate_folio_record(dst, old_page_state, anon_vma);
1853 } else if (rc != MIGRATEPAGE_SUCCESS) {
1854 *nr_failed += 1;
1855 stats->nr_thp_failed += is_thp;
1856 stats->nr_failed_pages += nr_pages;
1857
1858 list_del(&dst->lru);
1859 migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED,
1860 anon_vma, true, ret_folios);
1861 migrate_folio_undo_dst(dst, true, put_new_folio, private);
1862 } else /* MIGRATEPAGE_SUCCESS */
1863 nr_mig_folios++;
1864
1865 dst = dst2;
1866 dst2 = list_next_entry(dst, lru);
1867 }
1868
1869 /* Exit if folio list for batch migration is empty */
1870 if (!nr_mig_folios)
1871 goto out;
1872
1873 /* Batch copy the folios */
1874 {
1875 dst = list_first_entry(dst_folios, struct folio, lru);
1876 dst2 = list_next_entry(dst, lru);
1877 list_for_each_entry_safe(folio, folio2, src_folios, lru) {
1878 is_thp = folio_test_large(folio) &&
1879 folio_test_pmd_mappable(folio);
1880 nr_pages = folio_nr_pages(folio);
1881 rc = folio_mc_copy(dst, folio);
1882
1883 if (rc) {
1884 int old_page_state = 0;
1885 struct anon_vma *anon_vma = NULL;
1886
1887 /*
1888 * dst->private is moved to src->private in
1889 * __migrate_folio(), so page state and anon_vma
1890 * values can be extracted from (src) folio.
1891 */
1892 __migrate_folio_extract(folio, &old_page_state,
1893 &anon_vma);
1894 migrate_folio_undo_src(folio,
1895 old_page_state & PAGE_WAS_MAPPED,
1896 anon_vma, true, ret_folios);
1897 list_del(&dst->lru);
1898 migrate_folio_undo_dst(dst, true, put_new_folio,
1899 private);
1900 }
1901
1902 switch (rc) {
1903 case MIGRATEPAGE_SUCCESS:
1904 stats->nr_succeeded += nr_pages;
1905 stats->nr_thp_succeeded += is_thp;
1906 break;
1907 default:
1908 *nr_failed += 1;
1909 stats->nr_thp_failed += is_thp;
1910 stats->nr_failed_pages += nr_pages;
1911 break;
1912 }
1913
1914 dst = dst2;
1915 dst2 = list_next_entry(dst, lru);
1916 }
1917 }
1918
1919 /*
1920 * Iterate the folio lists to remove migration pte and restore them
1921 * as working pte. Unlock the folios, add/remove them to LRU lists (if
1922 * applicable) and release the src folios.
1923 */
1924 dst = list_first_entry(dst_folios, struct folio, lru);
1925 dst2 = list_next_entry(dst, lru);
1926 list_for_each_entry_safe(folio, folio2, src_folios, lru) {
1927 is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio);
1928 nr_pages = folio_nr_pages(folio);
1929 /*
1930 * dst->private is moved to src->private in __migrate_folio(),
1931 * so page state and anon_vma values can be extracted from
1932 * (src) folio.
1933 */
1934 __migrate_folio_extract(folio, &old_page_state, &anon_vma);
1935 list_del(&dst->lru);
1936
1937 _move_to_new_folio_finalize(dst, folio, MIGRATEPAGE_SUCCESS);
1938
1939 /*
1940 * Below few steps are only applicable for lru pages which is
1941 * ensured as we have removed the non-lru pages from our list.
1942 */
1943 _migrate_folio_move_finalize1(folio, dst, old_page_state);
1944
1945 _migrate_folio_move_finalize2(folio, dst, reason, anon_vma);
1946
1947 /* Page migration successful, increase stat counter */
1948 stats->nr_succeeded += nr_pages;
1949 stats->nr_thp_succeeded += is_thp;
1950
1951 dst = dst2;
1952 dst2 = list_next_entry(dst, lru);
1953 }
1954 out:
1955 /* Add tmp folios back to the list to let CPU re-attempt migration. */
1956 list_splice(&err_src, src_folios);
1957 list_splice(&err_dst, dst_folios);
1958 }
1959
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH 3/5] mm/migrate: add migrate_folios_batch_move to batch the folio move operations
In reply to Zi YanQuoted 31 lines · Zi Yan
This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It also adds
MIGRATE_NO_COPYback tomigrate_mode, so that folio copy will be skipped during metadata copy process and performed in a batch later.Currently, the folio move operation is performed individually for each folio in sequential manner:
for_each_folio(){
Copy folio metadata like flags and mappings
Copy the folio content from src to dst
Update page tables with dst folio
}With this patch, we transition to a batch processing approach as shown below:
for_each_folio(){
Copy folio metadata like flags and mappings
}
Batch copy all src folios to dstfor_each_folio(){
Update page tables with dst folios
}
dst->privateis used to store page states and possibleanon_vmavalue, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process,src->privateis used to store the data after metadata copy process, since src is no longer used.
Quoted 3 lines
Originally-by Shivank Garg <[email protected]>Signed-off-by Zi Yan <[email protected]>---
Hi Zi,
Please retain my Signed-off-by for future posting of batch page migration patchset.
I think we can separate out the MIGRATE_NO_COPY support into separate patch.
Thanks,
Shivank
Quoted 306 lines
include/linux/migrate_mode.h | 2 + mm/migrate.c | 207 +++++++++++++++++++++++++++++++++-- 2 files changed, 201 insertions(+), 8 deletions(-)include/linux/migrate_mode.h +2
-0@@ -7,11 +7,13 @@* on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages +* MIGRATE_NO_COPY will not copy page content */ enum migrate_mode {MIGRATE_ASYNC ,MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC , +MIGRATE_NO_COPY , };enum migrate_reason {mm/migrate.c +199
-8@@ -51,6 +51,7 @@#include "internal.h" +bool isolate_movable_page (struct page *page,isolate_mode_t mode) {struct folio *folio =folio_get_nontail_page (page);@@ -752,14 +753,19 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, enum migrate_mode mode) {int rc, expected_count =folio_expected_refs (mapping, src); +unsigned long dst_private = (unsigned long )dst->private;/* Check whether src does not have extra refs before we do more work */ if (folio_ref_count (src) != expected_count)return -EAGAIN ; - rc =folio_mc_copy (dst, src); -if (unlikely (rc)) -return rc; +if (mode ==MIGRATE_NO_COPY ) + dst->private =NULL ; +else { + rc =folio_mc_copy (dst, src); +if (unlikely (rc)) +return rc; + } rc =__folio_migrate_mapping (mapping, dst, src, expected_count);if (rc !=MIGRATEPAGE_SUCCESS )@@ -769,6 +775,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,folio_attach_private (dst,folio_detach_private (src));folio_migrate_flags (dst, src); + +if (mode ==MIGRATE_NO_COPY ) + src->private = (void *)dst_private; +return MIGRATEPAGE_SUCCESS ; }@@ -1042,7 +1052,7 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, mode); else rc =fallback_migrate_folio (mapping, dst, src, mode); - }else { + }else if (mode !=MIGRATE_NO_COPY ) {const struct movable_operations *mops;/* @@ -1060,7 +1070,8 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, rc = mops->migrate_page (&dst->page, &src->page, mode);WARN_ON_ONCE (rc ==MIGRATEPAGE_SUCCESS && !folio_test_isolated (src)); - } + }else + rc = -EAGAIN ; out:return rc; }@@ -1138,7 +1149,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private = (void *)anon_vma + old_page_state; } -static void __migrate_folio_extract (struct folio *dst, +static void __migrate_folio_read (struct folio *dst,int *old_page_state,struct anon_vma **anon_vmap) {@@ -1146,6 +1157,13 @@ static void __migrate_folio_extract(struct folio *dst, *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES ); *old_page_state = private &PAGE_OLD_STATES ; +} + +static void __migrate_folio_extract (struct folio *dst, +int *old_page_state, +struct anon_vma **anon_vmap) +{ +__migrate_folio_read (dst, old_page_state, anon_vmap); dst->private =NULL ; }@@ -1771,6 +1789,174 @@ static void migrate_folios_move(struct list_head *src_folios, } } +static void migrate_folios_batch_move (struct list_head *src_folios, +struct list_head *dst_folios, +free_folio_t put_new_folio,unsigned long private, +enum migrate_mode mode,int reason, +struct list_head *ret_folios, +struct migrate_pages_stats *stats, +int *retry,int *thp_retry,int *nr_failed, +int *nr_retry_pages) +{ +struct folio *folio, *folio2, *dst, *dst2; +int rc, nr_pages =0 , nr_mig_folios =0 ; +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; +bool is_lru; +int is_thp =0 ; +LIST_HEAD (err_src); +LIST_HEAD (err_dst); + +if (mode !=MIGRATE_ASYNC ) { + *retry +=1 ; +return ; + } + +/* +* Iterate over the list of locked src/dst folios to copy the metadata +*/ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); + is_lru = !__folio_test_movable (folio); + +/* +* dst->private is not cleared here. It is cleared and moved to +* src->private in __migrate_folio(). +*/ +__migrate_folio_read (dst, &old_page_state, &anon_vma); + +/* +* Use MIGRATE_NO_COPY mode in migrate_folio family functions +* to copy the flags, mapping and some other ancillary information. +* This does everything except the page copy. The actual page copy +* is handled later in a batch manner. +*/ + rc =_move_to_new_folio_prep (dst, folio,MIGRATE_NO_COPY ); + +/* +* -EAGAIN: Move src/dst folios to tmp lists for retry +* Other Errno: Put src folio on ret_folios list, remove the dst folio +* Success: Copy the folio bytes, restoring working pte, unlock and +* decrement refcounter +*/ +if (rc == -EAGAIN ) { + *retry +=1 ; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + +list_move_tail (&folio->lru, &err_src); +list_move_tail (&dst->lru, &err_dst); +__migrate_folio_record (dst, old_page_state, anon_vma); + }else if (rc !=MIGRATEPAGE_SUCCESS ) { + *nr_failed +=1 ; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + +list_del (&dst->lru); +migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , + anon_vma, true, ret_folios); +migrate_folio_undo_dst (dst, true, put_new_folio, private); + }else /* MIGRATEPAGE_SUCCESS */ + nr_mig_folios++; + + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + +/* Exit if folio list for batch migration is empty */ +if (!nr_mig_folios) + goto out; + +/* Batch copy the folios */ + { + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) && +folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); + rc =folio_mc_copy (dst, folio); + +if (rc) { +int old_page_state =0 ; +struct anon_vma *anon_vma =NULL ; + +/* +* dst->private is moved to src->private in +* __migrate_folio(), so page state and anon_vma +* values can be extracted from (src) folio. +*/ +__migrate_folio_extract (folio, &old_page_state, + &anon_vma); +migrate_folio_undo_src (folio, + old_page_state &PAGE_WAS_MAPPED , + anon_vma, true, ret_folios); +list_del (&dst->lru); +migrate_folio_undo_dst (dst, true, put_new_folio, + private); + } + +switch (rc) { +case MIGRATEPAGE_SUCCESS : + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; +break ; +default : + *nr_failed +=1 ; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; +break ; + } + + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + } + +/* +* Iterate the folio lists to remove migration pte and restore them +* as working pte. Unlock the folios, add/remove them to LRU lists (if +* applicable) and release the src folios. +*/ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (folio, folio2, src_folios, lru) { + is_thp =folio_test_large (folio) &&folio_test_pmd_mappable (folio); + nr_pages =folio_nr_pages (folio); +/* +* dst->private is moved to src->private in __migrate_folio(), +* so page state and anon_vma values can be extracted from +* (src) folio. +*/ +__migrate_folio_extract (folio, &old_page_state, &anon_vma); +list_del (&dst->lru); + +_move_to_new_folio_finalize (dst, folio,MIGRATEPAGE_SUCCESS ); + +/* +* Below few steps are only applicable for lru pages which is +* ensured as we have removed the non-lru pages from our list. +*/ +_migrate_folio_move_finalize1 (folio, dst, old_page_state); + +_migrate_folio_move_finalize2 (folio, dst, reason, anon_vma); + +/* Page migration successful, increase stat counter */ + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + + dst = dst2; + dst2 =list_next_entry (dst, lru); + } +out: +/* Add tmp folios back to the list to let CPU re-attempt migration. */ +list_splice (&err_src, src_folios); +list_splice (&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private,@@ -1981,13 +2167,18 @@ static int migrate_pages_batch(struct list_head *from,/* Flush TLBs for all unmapped folios */ try_to_unmap_flush (); - retry =1 ; + retry =0 ; +/* Batch move the unmapped folios */ +migrate_folios_batch_move (&unmap_folios, &dst_folios, put_new_folio, + private, mode, reason, ret_folios, stats, &retry, + &thp_retry, &nr_failed, &nr_retry_pages); +for (pass =0 ; pass < nr_pass && retry; pass++) { retry =0 ; thp_retry =0 ; nr_retry_pages =0 ; -/* Move the unmapped folios */ +/* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry,
Zi Yan
reviewerRe: [RFC PATCH 3/5] mm/migrate: add migrate_folios_batch_move to batch the folio move operations
Quoted 36 lines · Shivank Garg
Quoted 31 lines · Zi Yan
This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It also adds
MIGRATE_NO_COPYback tomigrate_mode, so that folio copy will be skipped during metadata copy process and performed in a batch later.Currently, the folio move operation is performed individually for each folio in sequential manner:
for_each_folio(){
Copy folio metadata like flags and mappings
Copy the folio content from src to dst
Update page tables with dst folio
}With this patch, we transition to a batch processing approach as shown below:
for_each_folio(){
Copy folio metadata like flags and mappings
}
Batch copy all src folios to dstfor_each_folio(){
Update page tables with dst folios
}
dst->privateis used to store page states and possibleanon_vmavalue, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process,src->privateis used to store the data after metadata copy process, since src is no longer used.Quoted 3 lines
Originally-by Shivank Garg <[email protected]>Signed-off-by Zi Yan <[email protected]>---
Quoted 6 lines
Hi Zi,
Please retain my Signed-off-by for future posting of batch page migration patchset.
I think we can separate out the
MIGRATE_NO_COPYsupport into separate patch.
Sure. Will change them in the next version.
Best Regards,
Yan, Zi
Zi Yan
reviewerRFC PATCH 4/5 mm/migrate: introduce multi-threaded page copy routine
In reply to Zi YanNow page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add copy_page_lists_mt() to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold.
--- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mm/copy_pages.c
include/linux/migrate.h +3 -0
@@ -80,6 +80,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio);int folio_migrate_mapping (struct address_space *mapping,struct folio *newfolio,struct folio *folio,int extra_count); +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items); +#else static inline void putback_movable_pages (struct list_head *l) {}
mm/Makefile +1 -1
@@ -92,7 +92,7 @@ obj-$(CONFIG_KMSAN) += kmsan/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o -obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION) += migrate.o copy_pages.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o
mm/copy_pages.c new file +186 -0
186 lines not on this page · raw message
mm/migrate.c +9 -10
@@ -1799,7 +1799,7 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) {struct folio *folio, *folio2, *dst, *dst2; -int rc, nr_pages =0 , nr_mig_folios =0 ; +int rc, nr_pages =0 , total_nr_pages =0 , total_nr_folios =0 ;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;bool is_lru;
@@ -1807,11 +1807,6 @@ static void migrate_folios_batch_move(struct list_head *src_folios,LIST_HEAD (err_src);LIST_HEAD (err_dst); -if (mode !=MIGRATE_ASYNC ) { - *retry +=1 ; -return ; - } -/* * Iterate over the list of locked src/dst folios to copy the metadata */
@@ -1859,19 +1854,23 @@ static void migrate_folios_batch_move(struct list_head *src_folios,migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , anon_vma, true, ret_folios);migrate_folio_undo_dst (dst, true, put_new_folio, private); - }else /* MIGRATEPAGE_SUCCESS */ - nr_mig_folios++; + }else {/* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; + total_nr_folios++; + } dst = dst2; dst2 =list_next_entry (dst, lru); }/* Exit if folio list for batch migration is empty */ -if (!nr_mig_folios) +if (!total_nr_pages) goto out;/* Batch copy the folios */ - { +if (total_nr_pages >32 ) { +copy_page_lists_mt (dst_folios, src_folios, total_nr_folios); + }else { dst = list_first_entry(dst_folios, struct folio, lru); dst2 =list_next_entry (dst, lru);list_for_each_entry_safe (folio, folio2, src_folios, lru) {
Signature · 1 line
2.45.2
Re: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine
Quoted 15 lines · Zi Yan
Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add
copy_page_lists_mt()to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold.Signed-off-by Zi Yan <[email protected]>--- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-)create mode 100644
mm/copy_pages.c
[...snip...]
Quoted 44 lines
+++
b/mm/copy_pages.cmm/copy_pages.c new file +42
-0@@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Parallel page copy routine. +*/ + +#include <linux/sysctl.h> +#include <linux/highmem.h> +#include <linux/workqueue.h> +#include <linux/slab.h> +#include <linux/migrate.h> + + +unsigned int limit_mt_num =4 ; + +struct copy_item { +char *to; +char *from; +unsigned long chunk_size; +}; + +struct copy_page_info { +struct work_struct copy_page_work; +unsigned long num_items; +struct copy_item item_list[]; +}; + +static void copy_page_routine (char *vto,char *vfrom, +unsigned long chunk_size) +{ +memcpy (vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread (struct work_struct *work) +{ +struct copy_page_info *my_work = (struct copy_page_info *)work; +int i; + +for (i =0 ; i < my_work->num_items; ++i) +copy_page_routine (my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +}
mm/copy_pages.c line 52
Quoted 10 lines
mm/copy_pages.c new file +10
-0+ +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items) +{ +int err =0 ; +unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); +int i; +struct copy_page_info *work_items[32 ] = {0 }; +const struct cpumask *per_node_cpumask =cpumask_of_node (to_node);
What happens here if to_node is a NUMA node without CPUs? (e.g. CXL node).
And even with a NUMA node with CPUs I think offloading copies to CPUs of either "from node" or "to node" will end up a CPU touching two pages in two different NUMA nodes anyway, one page in the local node and the other page in the remote node.
In that sense, I don't understand when push_0_pull_1 (introduced in patch 5) should be 0 or 1. Am I missing something?
Quoted 30 lines
mm/copy_pages.c new file +30
-0+int cpu_id_list[32 ] = {0 }; +int cpu; +int max_items_per_thread; +int item_idx; +struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, +cpumask_weight (per_node_cpumask)); + +if (total_mt_num >32 ) + total_mt_num =32 ; + +/* Each threads get part of each page, if nr_items < totla_mt_num */ +if (nr_items < total_mt_num) + max_items_per_thread = nr_items; +else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ?1 :0 ); + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu] =kzalloc (sizeof (struct copy_page_info ) + +sizeof (struct copy_item ) * max_items_per_thread, +GFP_NOWAIT ); + +if (!work_items[cpu]) { + err = -ENOMEM ; + goto free_work_items; + } + }
[...snip...]
mm/copy_pages.c line 185
Quoted 10 lines
mm/copy_pages.c new file +10
-0+ +/* Wait until it finishes */ +for (i =0 ; i < total_mt_num; ++i) +flush_work ((struct work_struct *)work_items[i]); + +free_work_items: +for (cpu =0 ; cpu < total_mt_num; ++cpu) +kfree (work_items[cpu]); + +return err;
Should the kernel re-try migration without multi-threading if it failed to allocate memory?
---
Hyeonggon
Zi Yan
reviewerRe: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine
Quoted 68 lines · Hyeonggon Yoo
Quoted 15 lines · Zi Yan
Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add
copy_page_lists_mt()to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold.Signed-off-by Zi Yan <[email protected]>--- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-)create mode 100644
mm/copy_pages.c[...snip...]
Quoted 49 lines
+++
b/mm/copy_pages.cmm/copy_pages.c new file +47
-0@@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Parallel page copy routine. +*/ + +#include <linux/sysctl.h> +#include <linux/highmem.h> +#include <linux/workqueue.h> +#include <linux/slab.h> +#include <linux/migrate.h> + + +unsigned int limit_mt_num =4 ; + +struct copy_item { +char *to; +char *from; +unsigned long chunk_size; +}; + +struct copy_page_info { +struct work_struct copy_page_work; +unsigned long num_items; +struct copy_item item_list[]; +}; + +static void copy_page_routine (char *vto,char *vfrom, +unsigned long chunk_size) +{ +memcpy (vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread (struct work_struct *work) +{ +struct copy_page_info *my_work = (struct copy_page_info *)work; +int i; + +for (i =0 ; i < my_work->num_items; ++i) +copy_page_routine (my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +} + +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items) +{ +int err =0 ;
Quoted 8 lines
mm/copy_pages.c line 52
Quoted 5 lines
mm/copy_pages.c new file +5
-0+unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); +int i; +struct copy_page_info *work_items[32 ] = {0 }; +const struct cpumask *per_node_cpumask =cpumask_of_node (to_node);What happens here if
to_nodeis a NUMA node without CPUs? (e.g. CXL node).
I did not think about that case. In that case, from_node will be used. If both from and to are CPUless nodes, maybe the node of the executing CPU should be used to select cpumask here.
Quoted 8 lines
And even with a NUMA node with CPUs I think offloading copies to CPUs of either "from node" or "to node" will end up a CPU touching two pages in two different NUMA nodes anyway, one page in the local node and the other page in the remote node.
In that sense, I don't understand when
push_0_pull_1(introduced in patch 5) should be 0 or 1. Am I missing something?
From my experiments, copy throughput differs between pushing data from local CPUs and pulling data from remote CPUs. On NVIDIA Grace CPU, pushing data has higher throughput. Back in 2019, when I tested it on Intel Xeon Broadwell, pulling data has higher throughput. In the final version, a boot time benchmark might be needed to decide whether to push data or pull data.
Quoted 33 lines
Quoted 30 lines
mm/copy_pages.c new file +30
-0+int cpu_id_list[32 ] = {0 }; +int cpu; +int max_items_per_thread; +int item_idx; +struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, +cpumask_weight (per_node_cpumask)); + +if (total_mt_num >32 ) + total_mt_num =32 ; + +/* Each threads get part of each page, if nr_items < totla_mt_num */ +if (nr_items < total_mt_num) + max_items_per_thread = nr_items; +else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ?1 :0 ); + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu] =kzalloc (sizeof (struct copy_page_info ) + +sizeof (struct copy_item ) * max_items_per_thread, +GFP_NOWAIT ); + +if (!work_items[cpu]) { + err = -ENOMEM ; + goto free_work_items; + } + }[...snip...]
Quoted 13 lines
mm/copy_pages.c line 185
Quoted 10 lines
mm/copy_pages.c new file +10
-0+ +/* Wait until it finishes */ +for (i =0 ; i < total_mt_num; ++i) +flush_work ((struct work_struct *)work_items[i]); + +free_work_items: +for (cpu =0 ; cpu < total_mt_num; ++cpu) +kfree (work_items[cpu]); + +return err;Should the kernel re-try migration without multi-threading if it failed to allocate memory?
Sure. Will add it in the next version.
Thank you for the reviews.
--
Best Regards,
Yan, Zi
Byungchul Park
reviewerRe: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine
In reply to Zi YanQuoted 90 lines · Zi Yan
Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add
copy_page_lists_mt()to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold.Signed-off-by Zi Yan <[email protected]>--- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mm/copy_pages.cinclude/linux/migrate.h +3
-0@@ -80,6 +80,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio);int folio_migrate_mapping (struct address_space *mapping,struct folio *newfolio,struct folio *folio,int extra_count); +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items); +#else static inline void putback_movable_pages (struct list_head *l) {}mm/Makefile +1
-1@@ -92,7 +92,7 @@ obj-$(CONFIG_KMSAN) += kmsan/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o -obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION) += migrate.o copy_pages.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.omm/copy_pages.c new file +42
-0@@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Parallel page copy routine. +*/ + +#include <linux/sysctl.h> +#include <linux/highmem.h> +#include <linux/workqueue.h> +#include <linux/slab.h> +#include <linux/migrate.h> + + +unsigned int limit_mt_num =4 ; + +struct copy_item { +char *to; +char *from; +unsigned long chunk_size; +}; + +struct copy_page_info { +struct work_struct copy_page_work; +unsigned long num_items; +struct copy_item item_list[]; +}; + +static void copy_page_routine (char *vto,char *vfrom, +unsigned long chunk_size) +{ +memcpy (vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread (struct work_struct *work) +{ +struct copy_page_info *my_work = (struct copy_page_info *)work; +int i; + +for (i =0 ; i < my_work->num_items; ++i) +copy_page_routine (my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +}
mm/copy_pages.c line 52
Quoted 10 lines
mm/copy_pages.c new file +10
-0+ +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items) +{ +int err =0 ; +unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); +int i; +struct copy_page_info *work_items[32 ] = {0 }; +const struct cpumask *per_node_cpumask =cpumask_of_node (to_node);
Hi,
Why do you use the cpumask of dst's node than src for where queueing the works on? Is it for utilizing CPU cache? Isn't it better to use src's node than dst where nothing has been loaded to CPU cache? Or why don't you avoid specifying cpus to queue on but let system_unbound_wq select the appropriate CPUs e.g. idlest CPUs, when the system is not that idle?
Am I missing something?
Byungchul
Quoted 190 lines
mm/copy_pages.c new file +134
-0+int cpu_id_list[32 ] = {0 }; +int cpu; +int max_items_per_thread; +int item_idx; +struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, +cpumask_weight (per_node_cpumask)); + +if (total_mt_num >32 ) + total_mt_num =32 ; + +/* Each threads get part of each page, if nr_items < totla_mt_num */ +if (nr_items < total_mt_num) + max_items_per_thread = nr_items; +else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ?1 :0 ); + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu] =kzalloc (sizeof (struct copy_page_info ) + +sizeof (struct copy_item ) * max_items_per_thread, +GFP_NOWAIT ); +if (!work_items[cpu]) { + err = -ENOMEM ; + goto free_work_items; + } + } + + i =0 ; +/* TODO: need a better cpu selection method */ +for_each_cpu (cpu, per_node_cpumask) { +if (i >= total_mt_num) +break ; + cpu_id_list[i] = cpu; + ++i; + } + +if (nr_items < total_mt_num) { +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + + item_idx =0 ; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (src, src2, src_folios, lru) { +unsigned long chunk_size =PAGE_SIZE *folio_nr_pages (src) / total_mt_num; +/* XXX: not working in HIGHMEM */ +char *vfrom =page_address (&src->page); +char *vto =page_address (&dst->page); + +VM_WARN_ON (PAGE_SIZE *folio_nr_pages (src) % total_mt_num); +VM_WARN_ON (folio_nr_pages (dst) !=folio_nr_pages (src)); + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + + item_idx++; + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + +for (cpu =0 ; cpu < total_mt_num; ++cpu) +queue_work_on (cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + }else { +int num_xfer_per_thread = nr_items / total_mt_num; +int per_cpu_item_idx; + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu =0 ; + per_cpu_item_idx =0 ; + item_idx =0 ; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (src, src2, src_folios, lru) { +/* XXX: not working in HIGHMEM */ + work_items[cpu]->item_list[per_cpu_item_idx].to = +page_address (&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from = +page_address (&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = +PAGE_SIZE *folio_nr_pages (src); + +VM_WARN_ON (folio_nr_pages (dst) != +folio_nr_pages (src)); + + per_cpu_item_idx++; + item_idx++; + dst = dst2; + dst2 =list_next_entry (dst, lru); + +if (per_cpu_item_idx == work_items[cpu]->num_items) { +queue_work_on (cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx =0 ; + cpu++; + } + } +if (item_idx != nr_items) +pr_warn ("%s: only %d out of %d pages are transferred\n" , + __func__, item_idx -1 , nr_items); + } + +/* Wait until it finishes */ +for (i =0 ; i < total_mt_num; ++i) +flush_work ((struct work_struct *)work_items[i]); + +free_work_items: +for (cpu =0 ; cpu < total_mt_num; ++cpu) +kfree (work_items[cpu]); + +return err; +}mm/migrate.c +9
-10@@ -1799,7 +1799,7 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) {struct folio *folio, *folio2, *dst, *dst2; -int rc, nr_pages =0 , nr_mig_folios =0 ; +int rc, nr_pages =0 , total_nr_pages =0 , total_nr_folios =0 ;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;bool is_lru;@@ -1807,11 +1807,6 @@ static void migrate_folios_batch_move(struct list_head *src_folios,LIST_HEAD (err_src);LIST_HEAD (err_dst); -if (mode !=MIGRATE_ASYNC ) { - *retry +=1 ; -return ; - } -/* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1859,19 +1854,23 @@ static void migrate_folios_batch_move(struct list_head *src_folios,migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , anon_vma, true, ret_folios);migrate_folio_undo_dst (dst, true, put_new_folio, private); - }else /* MIGRATEPAGE_SUCCESS */ - nr_mig_folios++; + }else {/* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; + total_nr_folios++; + } dst = dst2; dst2 =list_next_entry (dst, lru); }/* Exit if folio list for batch migration is empty */ -if (!nr_mig_folios) +if (!total_nr_pages) goto out;/* Batch copy the folios */ - { +if (total_nr_pages >32 ) { +copy_page_lists_mt (dst_folios, src_folios, total_nr_folios); + }else { dst = list_first_entry(dst_folios, struct folio, lru); dst2 =list_next_entry (dst, lru);list_for_each_entry_safe (folio, folio2, src_folios, lru) {Signature · 1 line
2.45.2
Zi Yan
reviewerRe: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine
Quoted 104 lines · Byungchul Park
Quoted 90 lines · Zi Yan
Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add
copy_page_lists_mt()to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold.Signed-off-by Zi Yan <[email protected]>--- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mm/copy_pages.cinclude/linux/migrate.h +3
-0@@ -80,6 +80,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio);int folio_migrate_mapping (struct address_space *mapping,struct folio *newfolio,struct folio *folio,int extra_count); +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items); +#else static inline void putback_movable_pages (struct list_head *l) {}mm/Makefile +1
-1@@ -92,7 +92,7 @@ obj-$(CONFIG_KMSAN) += kmsan/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o -obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION) += migrate.o copy_pages.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.omm/copy_pages.c new file +42
-0@@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Parallel page copy routine. +*/ + +#include <linux/sysctl.h> +#include <linux/highmem.h> +#include <linux/workqueue.h> +#include <linux/slab.h> +#include <linux/migrate.h> + + +unsigned int limit_mt_num =4 ; + +struct copy_item { +char *to; +char *from; +unsigned long chunk_size; +}; + +struct copy_page_info { +struct work_struct copy_page_work; +unsigned long num_items; +struct copy_item item_list[]; +}; + +static void copy_page_routine (char *vto,char *vfrom, +unsigned long chunk_size) +{ +memcpy (vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread (struct work_struct *work) +{ +struct copy_page_info *my_work = (struct copy_page_info *)work; +int i; + +for (i =0 ; i < my_work->num_items; ++i) +copy_page_routine (my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +}mm/copy_pages.c line 52
Quoted 10 lines
mm/copy_pages.c new file +10
-0+ +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items) +{ +int err =0 ; +unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); +int i; +struct copy_page_info *work_items[32 ] = {0 }; +const struct cpumask *per_node_cpumask =cpumask_of_node (to_node);Hi,
Quoted 3 lines
Why do you use the cpumask of dst's node than src for where queueing the works on? Is it for utilizing CPU cache? Isn't it better to use src's node than dst where nothing has been loaded to CPU cache? Or why
Because some vendor’s CPU achieves higher copy throughput by pushing data from src to dst, whereas other vendor’s CPU get higher by pulling data from dst to src. More in [1].
Quoted 3 lines
don't you avoid specifying cpus to queue on but let
system_unbound_wqselect the appropriate CPUs e.g. idlest CPUs, when the system is not that idle?
Based on wq_select_unbound_cpu()[2], a round robin method is used to select target CPUs, not the idlest CPUs. queue_work_node(), which queue jobs on a NUMA node, uses select_numa_node_cpu() and it just chooses the random (or known as first) CPU from the NUMA node[3]. There is no idleness detection in workqueue implementation yet.
In addition, based on CPU topology, not all idle CPUs are equal. For example, AMD CPUs have CCDs and two cores from one CCD would saturate the CCD bandwidth. This means if you want to achieve high copy throughput, even if all cores in a CCD are idle, other idle CPUs from another CCD should be chosen first[4].
I am planning to reach out to scheduling folks to learn more about CPU scheduling and come up with a better workqueue or an alternative for multithreading page migration.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://elixir.bootlin.com/linux/v6.13.2/source/kernel/workqueue.c#L2212
[3] https://elixir.bootlin.com/linux/v6.13.2/source/kernel/workqueue.c#L2408
[4] https://lore.kernel.org/linux-mm/[email protected]/
Quoted 182 lines
Byungchul
Quoted 179 lines
mm/copy_pages.c new file +134
-0+int cpu_id_list[32 ] = {0 }; +int cpu; +int max_items_per_thread; +int item_idx; +struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, +cpumask_weight (per_node_cpumask)); + +if (total_mt_num >32 ) + total_mt_num =32 ; + +/* Each threads get part of each page, if nr_items < totla_mt_num */ +if (nr_items < total_mt_num) + max_items_per_thread = nr_items; +else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ?1 :0 ); + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu] =kzalloc (sizeof (struct copy_page_info ) + +sizeof (struct copy_item ) * max_items_per_thread, +GFP_NOWAIT ); +if (!work_items[cpu]) { + err = -ENOMEM ; + goto free_work_items; + } + } + + i =0 ; +/* TODO: need a better cpu selection method */ +for_each_cpu (cpu, per_node_cpumask) { +if (i >= total_mt_num) +break ; + cpu_id_list[i] = cpu; + ++i; + } + +if (nr_items < total_mt_num) { +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + + item_idx =0 ; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (src, src2, src_folios, lru) { +unsigned long chunk_size =PAGE_SIZE *folio_nr_pages (src) / total_mt_num; +/* XXX: not working in HIGHMEM */ +char *vfrom =page_address (&src->page); +char *vto =page_address (&dst->page); + +VM_WARN_ON (PAGE_SIZE *folio_nr_pages (src) % total_mt_num); +VM_WARN_ON (folio_nr_pages (dst) !=folio_nr_pages (src)); + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + + item_idx++; + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + +for (cpu =0 ; cpu < total_mt_num; ++cpu) +queue_work_on (cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + }else { +int num_xfer_per_thread = nr_items / total_mt_num; +int per_cpu_item_idx; + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu =0 ; + per_cpu_item_idx =0 ; + item_idx =0 ; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (src, src2, src_folios, lru) { +/* XXX: not working in HIGHMEM */ + work_items[cpu]->item_list[per_cpu_item_idx].to = +page_address (&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from = +page_address (&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = +PAGE_SIZE *folio_nr_pages (src); + +VM_WARN_ON (folio_nr_pages (dst) != +folio_nr_pages (src)); + + per_cpu_item_idx++; + item_idx++; + dst = dst2; + dst2 =list_next_entry (dst, lru); + +if (per_cpu_item_idx == work_items[cpu]->num_items) { +queue_work_on (cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx =0 ; + cpu++; + } + } +if (item_idx != nr_items) +pr_warn ("%s: only %d out of %d pages are transferred\n" , + __func__, item_idx -1 , nr_items); + } + +/* Wait until it finishes */ +for (i =0 ; i < total_mt_num; ++i) +flush_work ((struct work_struct *)work_items[i]); + +free_work_items: +for (cpu =0 ; cpu < total_mt_num; ++cpu) +kfree (work_items[cpu]); + +return err; +}mm/migrate.c +6
-9@@ -1799,7 +1799,7 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) {struct folio *folio, *folio2, *dst, *dst2; -int rc, nr_pages =0 , nr_mig_folios =0 ; +int rc, nr_pages =0 , total_nr_pages =0 , total_nr_folios =0 ;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;bool is_lru;@@ -1807,11 +1807,6 @@ static void migrate_folios_batch_move(struct list_head *src_folios,LIST_HEAD (err_src);LIST_HEAD (err_dst); -if (mode !=MIGRATE_ASYNC ) { - *retry +=1 ; -return ; - } -/* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1859,19 +1854,23 @@ static void migrate_folios_batch_move(struct list_head *src_folios,migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , anon_vma, true, ret_folios);migrate_folio_undo_dst (dst, true, put_new_folio, private); - }else /* MIGRATEPAGE_SUCCESS */ - nr_mig_folios++; + }else {/* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; + total_nr_folios++; + } dst = dst2; dst2 =list_next_entry (dst, lru); }/* Exit if folio list for batch migration is empty */ -if (!nr_mig_folios) +if (!total_nr_pages) goto out;
Quoted 11 lines
Quoted 11 lines
mm/migrate.c +3
-1/* Batch copy the folios */ - { +if (total_nr_pages >32 ) { +copy_page_lists_mt (dst_folios, src_folios, total_nr_folios); + }else { dst = list_first_entry(dst_folios, struct folio, lru); dst2 =list_next_entry (dst, lru);list_for_each_entry_safe (folio, folio2, src_folios, lru) {Signature · 1 line
2.45.2
Best Regards,
Yan, Zi
Byungchul Park
reviewerRe: [RFC PATCH 4/5] mm/migrate: introduce multi-threaded page copy routine
Quoted 106 lines · Zi Yan
Quoted 104 lines · Byungchul Park
Quoted 90 lines · Zi Yan
Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Add
copy_page_lists_mt()to copy pages in multi-threaded manners. Empirical data show more than 32 base pages are needed to show the benefit of using multi-threaded page copy, so use 32 as the threshold.Signed-off-by Zi Yan <[email protected]>--- include/linux/migrate.h | 3 + mm/Makefile | 2 +- mm/copy_pages.c | 186 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 19 ++-- 4 files changed, 199 insertions(+), 11 deletions(-) create mode 100644 mm/copy_pages.cinclude/linux/migrate.h +3
-0@@ -80,6 +80,9 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio);int folio_migrate_mapping (struct address_space *mapping,struct folio *newfolio,struct folio *folio,int extra_count); +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items); +#else static inline void putback_movable_pages (struct list_head *l) {}mm/Makefile +1
-1@@ -92,7 +92,7 @@ obj-$(CONFIG_KMSAN) += kmsan/ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o -obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION) += migrate.o copy_pages.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.omm/copy_pages.c new file +42
-0@@ -0,0 +1,186 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Parallel page copy routine. +*/ + +#include <linux/sysctl.h> +#include <linux/highmem.h> +#include <linux/workqueue.h> +#include <linux/slab.h> +#include <linux/migrate.h> + + +unsigned int limit_mt_num =4 ; + +struct copy_item { +char *to; +char *from; +unsigned long chunk_size; +}; + +struct copy_page_info { +struct work_struct copy_page_work; +unsigned long num_items; +struct copy_item item_list[]; +}; + +static void copy_page_routine (char *vto,char *vfrom, +unsigned long chunk_size) +{ +memcpy (vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread (struct work_struct *work) +{ +struct copy_page_info *my_work = (struct copy_page_info *)work; +int i; + +for (i =0 ; i < my_work->num_items; ++i) +copy_page_routine (my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +}mm/copy_pages.c line 52
Quoted 10 lines
mm/copy_pages.c new file +10
-0+ +int copy_page_lists_mt (struct list_head *dst_folios, +struct list_head *src_folios,int nr_items) +{ +int err =0 ; +unsigned int total_mt_num = limit_mt_num; + int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); +int i; +struct copy_page_info *work_items[32 ] = {0 }; +const struct cpumask *per_node_cpumask =cpumask_of_node (to_node);Hi,
Quoted 7 lines
Quoted 3 lines
Why do you use the cpumask of dst's node than src for where queueing the works on? Is it for utilizing CPU cache? Isn't it better to use src's node than dst where nothing has been loaded to CPU cache? Or why
Because some vendor’s CPU achieves higher copy throughput by pushing data from src to dst, whereas other vendor’s CPU get higher by pulling data from dst to src. More in [1].
Ah, okay. You have already added the additional option for it in 5/5.
Quoted 6 lines
Quoted 3 lines
don't you avoid specifying cpus to queue on but let
system_unbound_wqselect the appropriate CPUs e.g. idlest CPUs, when the system is not that idle?Based on wq_select_unbound_cpu()[2], a round robin method is used to select target CPUs, not the idlest CPUs.
queue_work_node(), which queue jobs
It was just an example based on what I think it should be.. but indeed!
Quoted 8 lines
on a NUMA node, uses
select_numa_node_cpu()and it just chooses the random (or known as first) CPU from the NUMA node[3]. There is no idleness detection in workqueue implementation yet.In addition, based on CPU topology, not all idle CPUs are equal. For example, AMD CPUs have CCDs and two cores from one CCD would saturate the CCD bandwidth. This means if you want to achieve high copy throughput, even if all cores in a CCD are idle, other idle CPUs from another CCD should be chosen first[4].
Yeah.. I'd like to think it more what is the best design for it.
Quoted 3 lines
I am planning to reach out to scheduling folks to learn more about CPU scheduling and come up with a better workqueue or an alternative for multithreading page migration.
Good luck.
Byungchul
Quoted 204 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://elixir.bootlin.com/linux/v6.13.2/source/kernel/workqueue.c#L2212
[3] https://elixir.bootlin.com/linux/v6.13.2/source/kernel/workqueue.c#L2408
[4] https://lore.kernel.org/linux-mm/[email protected]/Quoted 182 lines
Byungchul
Quoted 179 lines
mm/copy_pages.c new file +134
-0+int cpu_id_list[32 ] = {0 }; +int cpu; +int max_items_per_thread; +int item_idx; +struct folio *src, *src2, *dst, *dst2; + + total_mt_num = min_t(unsigned int, total_mt_num, +cpumask_weight (per_node_cpumask)); + +if (total_mt_num >32 ) + total_mt_num =32 ; + +/* Each threads get part of each page, if nr_items < totla_mt_num */ +if (nr_items < total_mt_num) + max_items_per_thread = nr_items; +else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ?1 :0 ); + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu] =kzalloc (sizeof (struct copy_page_info ) + +sizeof (struct copy_item ) * max_items_per_thread, +GFP_NOWAIT ); +if (!work_items[cpu]) { + err = -ENOMEM ; + goto free_work_items; + } + } + + i =0 ; +/* TODO: need a better cpu selection method */ +for_each_cpu (cpu, per_node_cpumask) { +if (i >= total_mt_num) +break ; + cpu_id_list[i] = cpu; + ++i; + } + +if (nr_items < total_mt_num) { +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + + item_idx =0 ; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (src, src2, src_folios, lru) { +unsigned long chunk_size =PAGE_SIZE *folio_nr_pages (src) / total_mt_num; +/* XXX: not working in HIGHMEM */ +char *vfrom =page_address (&src->page); +char *vto =page_address (&dst->page); + +VM_WARN_ON (PAGE_SIZE *folio_nr_pages (src) % total_mt_num); +VM_WARN_ON (folio_nr_pages (dst) !=folio_nr_pages (src)); + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + + item_idx++; + dst = dst2; + dst2 =list_next_entry (dst, lru); + } + +for (cpu =0 ; cpu < total_mt_num; ++cpu) +queue_work_on (cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + }else { +int num_xfer_per_thread = nr_items / total_mt_num; +int per_cpu_item_idx; + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu =0 ; + per_cpu_item_idx =0 ; + item_idx =0 ; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 =list_next_entry (dst, lru); +list_for_each_entry_safe (src, src2, src_folios, lru) { +/* XXX: not working in HIGHMEM */ + work_items[cpu]->item_list[per_cpu_item_idx].to = +page_address (&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from = +page_address (&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = +PAGE_SIZE *folio_nr_pages (src); + +VM_WARN_ON (folio_nr_pages (dst) != +folio_nr_pages (src)); + + per_cpu_item_idx++; + item_idx++; + dst = dst2; + dst2 =list_next_entry (dst, lru); + +if (per_cpu_item_idx == work_items[cpu]->num_items) { +queue_work_on (cpu_id_list[cpu], + system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx =0 ; + cpu++; + } + } +if (item_idx != nr_items) +pr_warn ("%s: only %d out of %d pages are transferred\n" , + __func__, item_idx -1 , nr_items); + } + +/* Wait until it finishes */ +for (i =0 ; i < total_mt_num; ++i) +flush_work ((struct work_struct *)work_items[i]); + +free_work_items: +for (cpu =0 ; cpu < total_mt_num; ++cpu) +kfree (work_items[cpu]); + +return err; +}mm/migrate.c +6
-9@@ -1799,7 +1799,7 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) {struct folio *folio, *folio2, *dst, *dst2; -int rc, nr_pages =0 , nr_mig_folios =0 ; +int rc, nr_pages =0 , total_nr_pages =0 , total_nr_folios =0 ;int old_page_state =0 ;struct anon_vma *anon_vma =NULL ;bool is_lru;@@ -1807,11 +1807,6 @@ static void migrate_folios_batch_move(struct list_head *src_folios,LIST_HEAD (err_src);LIST_HEAD (err_dst); -if (mode !=MIGRATE_ASYNC ) { - *retry +=1 ; -return ; - } -/* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1859,19 +1854,23 @@ static void migrate_folios_batch_move(struct list_head *src_folios,migrate_folio_undo_src (folio, old_page_state &PAGE_WAS_MAPPED , anon_vma, true, ret_folios);migrate_folio_undo_dst (dst, true, put_new_folio, private); - }else /* MIGRATEPAGE_SUCCESS */ - nr_mig_folios++; + }else {/* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; + total_nr_folios++; + } dst = dst2; dst2 =list_next_entry (dst, lru); }/* Exit if folio list for batch migration is empty */ -if (!nr_mig_folios) +if (!total_nr_pages) goto out;Quoted 11 lines
Quoted 11 lines
mm/migrate.c +3
-1/* Batch copy the folios */ - { +if (total_nr_pages >32 ) { +copy_page_lists_mt (dst_folios, src_folios, total_nr_folios); + }else { dst = list_first_entry(dst_folios, struct folio, lru); dst2 =list_next_entry (dst, lru);list_for_each_entry_safe (folio, folio2, src_folios, lru) {Signature · 1 line
2.45.2Best Regards,
Yan, Zi
Zi Yan
reviewerRFC PATCH 5/5 test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION
In reply to Zi Yan1. enable multi-threaded copy
2. specify how many CPU threads to use
3. push from local CPUs or pull from remote CPUs
4. change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies.
These are for testing purpose only.
--- include/linux/mm.h | 4 ++++ include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++++++++++++++++++++++++++++- mm/copy_pages.c | 10 +++++++--- mm/migrate.c | 6 ++++-- 5 files changed, 44 insertions(+), 6 deletions(-)
include/linux/mm.h +4 -0
@@ -42,6 +42,10 @@ struct pt_regs;struct folio_batch ;extern int sysctl_page_lock_unfairness; +extern int sysctl_use_mt_copy; +extern unsigned int sysctl_limit_mt_num; +extern unsigned int sysctl_push_0_pull_1; +void mm_core_init (void );void init_mm_internals (void );
include/linux/sysctl.h +1 -0
@@ -52,6 +52,7 @@ struct ctl_dir;/* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */ #define SYSCTL_MAXOLDUID ((void *)&sysctl_vals[10])#define SYSCTL_NEG_ONE ((void *)&sysctl_vals[11]) +#define SYSCTL_32 ((void *)&sysctl_vals[12])extern const int sysctl_vals[];
kernel/sysctl.c +28 -1
@@ -82,7 +82,7 @@#endif /* shared constants to be used in various sysctls */ -const int sysctl_vals[] = {0 ,1 ,2 ,3 ,4 ,100 ,200 ,1000 ,3000 ,INT_MAX ,65535 ,-1 }; +const int sysctl_vals[] = {0 ,1 ,2 ,3 ,4 ,100 ,200 ,1000 ,3000 ,INT_MAX ,65535 ,-1 ,32 };EXPORT_SYMBOL (sysctl_vals);const unsigned long sysctl_long_vals[] = {0 ,1 ,LONG_MAX };
@@ -2091,6 +2091,33 @@ static struct ctl_table vm_table[] = { .extra2 = SYSCTL_ONE, },#endif + { + .procname = "use_mt_copy", + .data = &use_mt_copy, + .maxlen = sizeof(use_mt_copy), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, + { + .procname = "limit_mt_num", + .data = &limit_mt_num, + .maxlen = sizeof(limit_mt_num), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_32, + }, + { + .procname = "push_0_pull_1", + .data = &push_0_pull_1, + .maxlen = sizeof(push_0_pull_1), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "drop_caches", .data = &sysctl_drop_caches,
mm/copy_pages.c +7 -3
@@ -10,7 +10,9 @@#include <linux/migrate.h> -unsigned int limit_mt_num =4 ; +unsigned int sysctl_limit_mt_num =4 ; +/* push by default */ +unsigned int sysctl_push_0_pull_1;struct copy_item {char *to;
@@ -45,11 +47,13 @@ int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, int nr_items) { int err = 0; -unsigned int total_mt_num = limit_mt_num; +unsigned int total_mt_num = sysctl_limit_mt_num; int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); + int from_node = folio_nid(list_first_entry(src_folios, struct folio, lru));int i;struct copy_page_info *work_items[32 ] = {0 }; -const struct cpumask *per_node_cpumask =cpumask_of_node (to_node); +const struct cpumask *per_node_cpumask = +cpumask_of_node (sysctl_push_0_pull_1 ? to_node : from_node);int cpu_id_list[32 ] = {0 };int cpu;int max_items_per_thread;
mm/migrate.c +4 -2
@@ -51,6 +51,7 @@#include "internal.h" +int sysctl_use_mt_copy;bool isolate_movable_page (struct page *page,isolate_mode_t mode) {
@@ -1621,7 +1622,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f }#ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR#else #define NR_MAX_BATCHED_MIGRATION 512#endif
@@ -1868,7 +1869,8 @@ static void migrate_folios_batch_move(struct list_head *src_folios, goto out;/* Batch copy the folios */ -if (total_nr_pages >32 ) { +/* if (total_nr_pages > 32) { */ +if (sysctl_use_mt_copy) {copy_page_lists_mt (dst_folios, src_folios, total_nr_folios); }else { dst = list_first_entry(dst_folios, struct folio, lru);
Signature · 1 line
2.45.2
Gregory Price
reviewerRe: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION
On Fri, Jan 03, 2025 at 12:24:19PM -0500, Zi Yan wrote:
... snip ...
Quoted 20 lines
(quoted diff) +20
-0+ { + .procname = "use_mt_copy", + .data = &use_mt_copy, + .maxlen = sizeof(use_mt_copy), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, + { + .procname = "limit_mt_num", + .data = &limit_mt_num, + .maxlen = sizeof(limit_mt_num), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_32, + }, + { + .procname = "push_0_pull_1",
Review
Quoted 10 lines
(quoted diff) +7
-0+ .data = &push_0_pull_1, + .maxlen = sizeof(push_0_pull_1), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "drop_caches", .data = &sysctl_drop_caches,
Build errors here
~Gregory
---
kernel/sysctl.c +6 -6
@@ -2093,8 +2093,8 @@ static struct ctl_table vm_table[] = {#endif { .procname = "use_mt_copy", - .data = &use_mt_copy, - .maxlen = sizeof(use_mt_copy), + .data = &sysctl_use_mt_copy, + .maxlen = sizeof(sysctl_use_mt_copy), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO,
@@ -2102,8 +2102,8 @@ static struct ctl_table vm_table[] = { }, { .procname = "limit_mt_num", - .data = &limit_mt_num, - .maxlen = sizeof(limit_mt_num), + .data = &sysctl_limit_mt_num, + .maxlen = sizeof(sysctl_limit_mt_num), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE,
@@ -2111,8 +2111,8 @@ static struct ctl_table vm_table[] = { }, { .procname = "push_0_pull_1", - .data = &push_0_pull_1, - .maxlen = sizeof(push_0_pull_1), + .data = &sysctl_push_0_pull_1, + .maxlen = sizeof(sysctl_push_0_pull_1), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO,
Zi Yan
reviewerRe: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION
Quoted 26 lines · Gregory Price
On Fri, Jan 03, 2025 at 12:24:19PM -0500, Zi Yan wrote:
... snip ...Quoted 24 lines
(quoted diff) +24
-0+ { + .procname = "use_mt_copy", + .data = &use_mt_copy, + .maxlen = sizeof(use_mt_copy), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, + { + .procname = "limit_mt_num", + .data = &limit_mt_num, + .maxlen = sizeof(limit_mt_num), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_32, + }, + { + .procname = "push_0_pull_1", + .data = &push_0_pull_1, + .maxlen = sizeof(push_0_pull_1), + .mode = 0644, + .proc_handler = proc_dointvec_minmax,
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +3
-0+ .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "drop_caches", .data = &sysctl_drop_caches,Build errors here
Thanks, these changes must be lost during my patch clean time.
Quoted 32 lines
~Gregory
---
kernel/sysctl.c +4
-4@@ -2093,8 +2093,8 @@ static struct ctl_table vm_table[] = {#endif { .procname = "use_mt_copy", - .data = &use_mt_copy, - .maxlen = sizeof(use_mt_copy), + .data = &sysctl_use_mt_copy, + .maxlen = sizeof(sysctl_use_mt_copy), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO,@@ -2102,8 +2102,8 @@ static struct ctl_table vm_table[] = { }, { .procname = "limit_mt_num", - .data = &limit_mt_num, - .maxlen = sizeof(limit_mt_num), + .data = &sysctl_limit_mt_num, + .maxlen = sizeof(sysctl_limit_mt_num), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ONE,
kernel/sysctl.c static struct ctl_table vm_table[] = { · line 2118
Quoted 10 lines
kernel/sysctl.c +2
-2@@ -2111,8 +2111,8 @@ static struct ctl_table vm_table[] = { }, { .procname = "push_0_pull_1", - .data = &push_0_pull_1, - .maxlen = sizeof(push_0_pull_1), + .data = &sysctl_push_0_pull_1, + .maxlen = sizeof(sysctl_push_0_pull_1), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO,
Best Regards,
Yan, Zi
kernel test robot
build botRe: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION
In reply to Zi Yankernel test robot noticed the following build errors:
Hi Zi,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master sysctl/sysctl-next v6.13-rc5 next-20241220] [cannot apply to mcgrof/sysctl-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Zi-Yan/mm-separate-move-undo-doing-on-folio-list-from-migrate_pages_batch/20250104-012955 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250103172419.4148674-6-ziy%40nvidia.com patch subject: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION config: i386-buildonly-randconfig-001-20250104 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config) compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250104/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> kernel/sysctl.c:2096:13: error: use of undeclared identifier 'use_mt_copy'
2096 | .data = &use_mt_copy,
| ^
kernel/sysctl.c:2097:21: error: use of undeclared identifier 'use_mt_copy'
2097 | .maxlen = sizeof(use_mt_copy),
| ^
>> kernel/sysctl.c:2105:13: error: use of undeclared identifier 'limit_mt_num'
2105 | .data = &limit_mt_num,
| ^
kernel/sysctl.c:2106:21: error: use of undeclared identifier 'limit_mt_num'
2106 | .maxlen = sizeof(limit_mt_num),
| ^
>> kernel/sysctl.c:2114:13: error: use of undeclared identifier 'push_0_pull_1'
2114 | .data = &push_0_pull_1,
| ^
kernel/sysctl.c:2115:21: error: use of undeclared identifier 'push_0_pull_1'
2115 | .maxlen = sizeof(push_0_pull_1),
| ^
>> kernel/sysctl.c:2268:2: error: invalid application of 'sizeof' to an incomplete type 'struct ctl_table[]'
2268 | register_sysctl_init("vm", vm_table);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/sysctl.h:234:46: note: expanded from macro 'register_sysctl_init'
234 | __register_sysctl_init(path, table, #table, ARRAY_SIZE(table))
| ^~~~~~~~~~~~~~~~~
include/linux/array_size.h:11:32: note: expanded from macro 'ARRAY_SIZE'
11 | #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]) + __must_be_array(arr))
| ^~~~~7 errors generated.
vim +/use_mt_copy +2096 kernel/sysctl.c
2032
2033 static struct ctl_table vm_table[] = {
2034 {
2035 .procname = "overcommit_memory",
2036 .data = &sysctl_overcommit_memory,
2037 .maxlen = sizeof(sysctl_overcommit_memory),
2038 .mode = 0644,
2039 .proc_handler = overcommit_policy_handler,
2040 .extra1 = SYSCTL_ZERO,
2041 .extra2 = SYSCTL_TWO,
2042 },
2043 {
2044 .procname = "overcommit_ratio",
2045 .data = &sysctl_overcommit_ratio,
2046 .maxlen = sizeof(sysctl_overcommit_ratio),
2047 .mode = 0644,
2048 .proc_handler = overcommit_ratio_handler,
2049 },
2050 {
2051 .procname = "overcommit_kbytes",
2052 .data = &sysctl_overcommit_kbytes,
2053 .maxlen = sizeof(sysctl_overcommit_kbytes),
2054 .mode = 0644,
2055 .proc_handler = overcommit_kbytes_handler,
2056 },
2057 {
2058 .procname = "page-cluster",
2059 .data = &page_cluster,
2060 .maxlen = sizeof(int),
2061 .mode = 0644,
2062 .proc_handler = proc_dointvec_minmax,
2063 .extra1 = SYSCTL_ZERO,
2064 .extra2 = (void *)&page_cluster_max,
2065 },
2066 {
2067 .procname = "dirtytime_expire_seconds",
2068 .data = &dirtytime_expire_interval,
2069 .maxlen = sizeof(dirtytime_expire_interval),
2070 .mode = 0644,
2071 .proc_handler = dirtytime_interval_handler,
2072 .extra1 = SYSCTL_ZERO,
2073 },
2074 {
2075 .procname = "swappiness",
2076 .data = &vm_swappiness,
2077 .maxlen = sizeof(vm_swappiness),
2078 .mode = 0644,
2079 .proc_handler = proc_dointvec_minmax,
2080 .extra1 = SYSCTL_ZERO,
2081 .extra2 = SYSCTL_TWO_HUNDRED,
2082 },
2083 #ifdef CONFIG_NUMA
2084 {
2085 .procname = "numa_stat",
2086 .data = &sysctl_vm_numa_stat,
2087 .maxlen = sizeof(int),
2088 .mode = 0644,
2089 .proc_handler = sysctl_vm_numa_stat_handler,
2090 .extra1 = SYSCTL_ZERO,
2091 .extra2 = SYSCTL_ONE,
2092 },
2093 #endif
2094 {
2095 .procname = "use_mt_copy",
> 2096 .data = &use_mt_copy,
2097 .maxlen = sizeof(use_mt_copy),
2098 .mode = 0644,
2099 .proc_handler = proc_dointvec_minmax,
2100 .extra1 = SYSCTL_ZERO,
2101 .extra2 = SYSCTL_ONE,
2102 },
2103 {
2104 .procname = "limit_mt_num",
> 2105 .data = &limit_mt_num,
2106 .maxlen = sizeof(limit_mt_num),
2107 .mode = 0644,
2108 .proc_handler = proc_dointvec_minmax,
2109 .extra1 = SYSCTL_ONE,
2110 .extra2 = SYSCTL_32,
2111 },
2112 {
2113 .procname = "push_0_pull_1",
> 2114 .data = &push_0_pull_1,
2115 .maxlen = sizeof(push_0_pull_1),
2116 .mode = 0644,
2117 .proc_handler = proc_dointvec_minmax,
2118 .extra1 = SYSCTL_ZERO,
2119 .extra2 = SYSCTL_ONE,
2120 },
2121 {
2122 .procname = "drop_caches",
2123 .data = &sysctl_drop_caches,
2124 .maxlen = sizeof(int),
2125 .mode = 0200,
2126 .proc_handler = drop_caches_sysctl_handler,
2127 .extra1 = SYSCTL_ONE,
2128 .extra2 = SYSCTL_FOUR,
2129 },
2130 {
2131 .procname = "page_lock_unfairness",
2132 .data = &sysctl_page_lock_unfairness,
2133 .maxlen = sizeof(sysctl_page_lock_unfairness),
2134 .mode = 0644,
2135 .proc_handler = proc_dointvec_minmax,
2136 .extra1 = SYSCTL_ZERO,
2137 },
2138 #ifdef CONFIG_MMU
2139 {
2140 .procname = "max_map_count",
2141 .data = &sysctl_max_map_count,
2142 .maxlen = sizeof(sysctl_max_map_count),
2143 .mode = 0644,
2144 .proc_handler = proc_dointvec_minmax,
2145 .extra1 = SYSCTL_ZERO,
2146 },
2147 #else
2148 {
2149 .procname = "nr_trim_pages",
2150 .data = &sysctl_nr_trim_pages,
2151 .maxlen = sizeof(sysctl_nr_trim_pages),
2152 .mode = 0644,
2153 .proc_handler = proc_dointvec_minmax,
2154 .extra1 = SYSCTL_ZERO,
2155 },
2156 #endif
2157 {
2158 .procname = "vfs_cache_pressure",
2159 .data = &sysctl_vfs_cache_pressure,
2160 .maxlen = sizeof(sysctl_vfs_cache_pressure),
2161 .mode = 0644,
2162 .proc_handler = proc_dointvec_minmax,
2163 .extra1 = SYSCTL_ZERO,
2164 },
2165 #if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
2166 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
2167 {
2168 .procname = "legacy_va_layout",
2169 .data = &sysctl_legacy_va_layout,
2170 .maxlen = sizeof(sysctl_legacy_va_layout),
2171 .mode = 0644,
2172 .proc_handler = proc_dointvec_minmax,
2173 .extra1 = SYSCTL_ZERO,
2174 },
2175 #endif
2176 #ifdef CONFIG_NUMA
2177 {
2178 .procname = "zone_reclaim_mode",
2179 .data = &node_reclaim_mode,
2180 .maxlen = sizeof(node_reclaim_mode),
2181 .mode = 0644,
2182 .proc_handler = proc_dointvec_minmax,
2183 .extra1 = SYSCTL_ZERO,
2184 },
2185 #endif
2186 #ifdef CONFIG_SMP
2187 {
2188 .procname = "stat_interval",
2189 .data = &sysctl_stat_interval,
2190 .maxlen = sizeof(sysctl_stat_interval),
2191 .mode = 0644,
2192 .proc_handler = proc_dointvec_jiffies,
2193 },
2194 {
2195 .procname = "stat_refresh",
2196 .data = NULL,
2197 .maxlen = 0,
2198 .mode = 0600,
2199 .proc_handler = vmstat_refresh,
2200 },
2201 #endif
2202 #ifdef CONFIG_MMU
2203 {
2204 .procname = "mmap_min_addr",
2205 .data = &dac_mmap_min_addr,
2206 .maxlen = sizeof(unsigned long),
2207 .mode = 0644,
2208 .proc_handler = mmap_min_addr_handler,
2209 },
2210 #endif
2211 #if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
2212 (defined(CONFIG_SUPERH) && defined(CONFIG_VSYSCALL))
2213 {
2214 .procname = "vdso_enabled",
2215 #ifdef CONFIG_X86_32
2216 .data = &vdso32_enabled,
2217 .maxlen = sizeof(vdso32_enabled),
2218 #else
2219 .data = &vdso_enabled,
2220 .maxlen = sizeof(vdso_enabled),
2221 #endif
2222 .mode = 0644,
2223 .proc_handler = proc_dointvec,
2224 .extra1 = SYSCTL_ZERO,
2225 },
2226 #endif
2227 {
2228 .procname = "user_reserve_kbytes",
2229 .data = &sysctl_user_reserve_kbytes,
2230 .maxlen = sizeof(sysctl_user_reserve_kbytes),
2231 .mode = 0644,
2232 .proc_handler = proc_doulongvec_minmax,
2233 },
2234 {
2235 .procname = "admin_reserve_kbytes",
2236 .data = &sysctl_admin_reserve_kbytes,
2237 .maxlen = sizeof(sysctl_admin_reserve_kbytes),
2238 .mode = 0644,
2239 .proc_handler = proc_doulongvec_minmax,
2240 },
2241 #ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS
2242 {
2243 .procname = "mmap_rnd_bits",
2244 .data = &mmap_rnd_bits,
2245 .maxlen = sizeof(mmap_rnd_bits),
2246 .mode = 0600,
2247 .proc_handler = proc_dointvec_minmax,
2248 .extra1 = (void *)&mmap_rnd_bits_min,
2249 .extra2 = (void *)&mmap_rnd_bits_max,
2250 },
2251 #endif
2252 #ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS
2253 {
2254 .procname = "mmap_rnd_compat_bits",
2255 .data = &mmap_rnd_compat_bits,
2256 .maxlen = sizeof(mmap_rnd_compat_bits),
2257 .mode = 0600,
2258 .proc_handler = proc_dointvec_minmax,
2259 .extra1 = (void *)&mmap_rnd_compat_bits_min,
2260 .extra2 = (void *)&mmap_rnd_compat_bits_max,
2261 },
2262 #endif
2263 };
2264
2265 int __init sysctl_init_bases(void)
2266 {
2267 register_sysctl_init("kernel", kern_table);
> 2268 register_sysctl_init("vm", vm_table);Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION
In reply to Zi Yankernel test robot noticed the following build errors:
Hi Zi,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master sysctl/sysctl-next v6.13-rc5 next-20241220] [cannot apply to mcgrof/sysctl-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Zi-Yan/mm-separate-move-undo-doing-on-folio-list-from-migrate_pages_batch/20250104-012955 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250103172419.4148674-6-ziy%40nvidia.com patch subject: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION config: i386-buildonly-randconfig-004-20250104 (https://download.01.org/0day-ci/archive/20250104/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250104/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> kernel/sysctl.c:2096:36: error: 'use_mt_copy' undeclared here (not in a function)
2096 | .data = &use_mt_copy,
| ^~~~~~~~~~~
>> kernel/sysctl.c:2105:36: error: 'limit_mt_num' undeclared here (not in a function)
2105 | .data = &limit_mt_num,
| ^~~~~~~~~~~~
>> kernel/sysctl.c:2114:36: error: 'push_0_pull_1' undeclared here (not in a function) 2114 | .data = &push_0_pull_1,
| ^~~~~~~~~~~~~
vim +/use_mt_copy +2096 kernel/sysctl.c
2032
2033 static struct ctl_table vm_table[] = {
2034 {
2035 .procname = "overcommit_memory",
2036 .data = &sysctl_overcommit_memory,
2037 .maxlen = sizeof(sysctl_overcommit_memory),
2038 .mode = 0644,
2039 .proc_handler = overcommit_policy_handler,
2040 .extra1 = SYSCTL_ZERO,
2041 .extra2 = SYSCTL_TWO,
2042 },
2043 {
2044 .procname = "overcommit_ratio",
2045 .data = &sysctl_overcommit_ratio,
2046 .maxlen = sizeof(sysctl_overcommit_ratio),
2047 .mode = 0644,
2048 .proc_handler = overcommit_ratio_handler,
2049 },
2050 {
2051 .procname = "overcommit_kbytes",
2052 .data = &sysctl_overcommit_kbytes,
2053 .maxlen = sizeof(sysctl_overcommit_kbytes),
2054 .mode = 0644,
2055 .proc_handler = overcommit_kbytes_handler,
2056 },
2057 {
2058 .procname = "page-cluster",
2059 .data = &page_cluster,
2060 .maxlen = sizeof(int),
2061 .mode = 0644,
2062 .proc_handler = proc_dointvec_minmax,
2063 .extra1 = SYSCTL_ZERO,
2064 .extra2 = (void *)&page_cluster_max,
2065 },
2066 {
2067 .procname = "dirtytime_expire_seconds",
2068 .data = &dirtytime_expire_interval,
2069 .maxlen = sizeof(dirtytime_expire_interval),
2070 .mode = 0644,
2071 .proc_handler = dirtytime_interval_handler,
2072 .extra1 = SYSCTL_ZERO,
2073 },
2074 {
2075 .procname = "swappiness",
2076 .data = &vm_swappiness,
2077 .maxlen = sizeof(vm_swappiness),
2078 .mode = 0644,
2079 .proc_handler = proc_dointvec_minmax,
2080 .extra1 = SYSCTL_ZERO,
2081 .extra2 = SYSCTL_TWO_HUNDRED,
2082 },
2083 #ifdef CONFIG_NUMA
2084 {
2085 .procname = "numa_stat",
2086 .data = &sysctl_vm_numa_stat,
2087 .maxlen = sizeof(int),
2088 .mode = 0644,
2089 .proc_handler = sysctl_vm_numa_stat_handler,
2090 .extra1 = SYSCTL_ZERO,
2091 .extra2 = SYSCTL_ONE,
2092 },
2093 #endif
2094 {
2095 .procname = "use_mt_copy",
> 2096 .data = &use_mt_copy,
2097 .maxlen = sizeof(use_mt_copy),
2098 .mode = 0644,
2099 .proc_handler = proc_dointvec_minmax,
2100 .extra1 = SYSCTL_ZERO,
2101 .extra2 = SYSCTL_ONE,
2102 },
2103 {
2104 .procname = "limit_mt_num",
> 2105 .data = &limit_mt_num,
2106 .maxlen = sizeof(limit_mt_num),
2107 .mode = 0644,
2108 .proc_handler = proc_dointvec_minmax,
2109 .extra1 = SYSCTL_ONE,
2110 .extra2 = SYSCTL_32,
2111 },
2112 {
2113 .procname = "push_0_pull_1",
> 2114 .data = &push_0_pull_1, 2115 .maxlen = sizeof(push_0_pull_1),
2116 .mode = 0644,
2117 .proc_handler = proc_dointvec_minmax,
2118 .extra1 = SYSCTL_ZERO,
2119 .extra2 = SYSCTL_ONE,
2120 },
2121 {
2122 .procname = "drop_caches",
2123 .data = &sysctl_drop_caches,
2124 .maxlen = sizeof(int),
2125 .mode = 0200,
2126 .proc_handler = drop_caches_sysctl_handler,
2127 .extra1 = SYSCTL_ONE,
2128 .extra2 = SYSCTL_FOUR,
2129 },
2130 {
2131 .procname = "page_lock_unfairness",
2132 .data = &sysctl_page_lock_unfairness,
2133 .maxlen = sizeof(sysctl_page_lock_unfairness),
2134 .mode = 0644,
2135 .proc_handler = proc_dointvec_minmax,
2136 .extra1 = SYSCTL_ZERO,
2137 },
2138 #ifdef CONFIG_MMU
2139 {
2140 .procname = "max_map_count",
2141 .data = &sysctl_max_map_count,
2142 .maxlen = sizeof(sysctl_max_map_count),
2143 .mode = 0644,
2144 .proc_handler = proc_dointvec_minmax,
2145 .extra1 = SYSCTL_ZERO,
2146 },
2147 #else
2148 {
2149 .procname = "nr_trim_pages",
2150 .data = &sysctl_nr_trim_pages,
2151 .maxlen = sizeof(sysctl_nr_trim_pages),
2152 .mode = 0644,
2153 .proc_handler = proc_dointvec_minmax,
2154 .extra1 = SYSCTL_ZERO,
2155 },
2156 #endif
2157 {
2158 .procname = "vfs_cache_pressure",
2159 .data = &sysctl_vfs_cache_pressure,
2160 .maxlen = sizeof(sysctl_vfs_cache_pressure),
2161 .mode = 0644,
2162 .proc_handler = proc_dointvec_minmax,
2163 .extra1 = SYSCTL_ZERO,
2164 },
2165 #if defined(HAVE_ARCH_PICK_MMAP_LAYOUT) || \
2166 defined(CONFIG_ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT)
2167 {
2168 .procname = "legacy_va_layout",
2169 .data = &sysctl_legacy_va_layout,
2170 .maxlen = sizeof(sysctl_legacy_va_layout),
2171 .mode = 0644,
2172 .proc_handler = proc_dointvec_minmax,
2173 .extra1 = SYSCTL_ZERO,
2174 },
2175 #endif
2176 #ifdef CONFIG_NUMA
2177 {
2178 .procname = "zone_reclaim_mode",
2179 .data = &node_reclaim_mode,
2180 .maxlen = sizeof(node_reclaim_mode),
2181 .mode = 0644,
2182 .proc_handler = proc_dointvec_minmax,
2183 .extra1 = SYSCTL_ZERO,
2184 },
2185 #endif
2186 #ifdef CONFIG_SMP
2187 {
2188 .procname = "stat_interval",
2189 .data = &sysctl_stat_interval,
2190 .maxlen = sizeof(sysctl_stat_interval),
2191 .mode = 0644,
2192 .proc_handler = proc_dointvec_jiffies,
2193 },
2194 {
2195 .procname = "stat_refresh",
2196 .data = NULL,
2197 .maxlen = 0,
2198 .mode = 0600,
2199 .proc_handler = vmstat_refresh,
2200 },
2201 #endif
2202 #ifdef CONFIG_MMU
2203 {
2204 .procname = "mmap_min_addr",
2205 .data = &dac_mmap_min_addr,
2206 .maxlen = sizeof(unsigned long),
2207 .mode = 0644,
2208 .proc_handler = mmap_min_addr_handler,
2209 },
2210 #endif
2211 #if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
2212 (defined(CONFIG_SUPERH) && defined(CONFIG_VSYSCALL))
2213 {
2214 .procname = "vdso_enabled",
2215 #ifdef CONFIG_X86_32
2216 .data = &vdso32_enabled,
2217 .maxlen = sizeof(vdso32_enabled),
2218 #else
2219 .data = &vdso_enabled,
2220 .maxlen = sizeof(vdso_enabled),
2221 #endif
2222 .mode = 0644,
2223 .proc_handler = proc_dointvec,
2224 .extra1 = SYSCTL_ZERO,
2225 },
2226 #endif
2227 {
2228 .procname = "user_reserve_kbytes",
2229 .data = &sysctl_user_reserve_kbytes,
2230 .maxlen = sizeof(sysctl_user_reserve_kbytes),
2231 .mode = 0644,
2232 .proc_handler = proc_doulongvec_minmax,
2233 },
2234 {
2235 .procname = "admin_reserve_kbytes",
2236 .data = &sysctl_admin_reserve_kbytes,
2237 .maxlen = sizeof(sysctl_admin_reserve_kbytes),
2238 .mode = 0644,
2239 .proc_handler = proc_doulongvec_minmax,
2240 },
2241 #ifdef CONFIG_HAVE_ARCH_MMAP_RND_BITS
2242 {
2243 .procname = "mmap_rnd_bits",
2244 .data = &mmap_rnd_bits,
2245 .maxlen = sizeof(mmap_rnd_bits),
2246 .mode = 0600,
2247 .proc_handler = proc_dointvec_minmax,
2248 .extra1 = (void *)&mmap_rnd_bits_min,
2249 .extra2 = (void *)&mmap_rnd_bits_max,
2250 },
2251 #endif
2252 #ifdef CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS
2253 {
2254 .procname = "mmap_rnd_compat_bits",
2255 .data = &mmap_rnd_compat_bits,
2256 .maxlen = sizeof(mmap_rnd_compat_bits),
2257 .mode = 0600,
2258 .proc_handler = proc_dointvec_minmax,
2259 .extra1 = (void *)&mmap_rnd_compat_bits_min,
2260 .extra2 = (void *)&mmap_rnd_compat_bits_max,
2261 },
2262 #endif
2263 };
2264
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Gregory Price
reviewerQuoted 8 lines · Zi Yan
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
This is well timed as I've been testing a batch-migration variant of migrate_misplaced_folio for my pagecache promotion work (attached).
I will add this to my pagecache branch and give it a test at some point.
Quick question: is the multi-threaded movement supported in the context of task_work? i.e. in which context is the multi-threaded path safe/unsafe? (inline in a syscall, async only, etc).
~Gregory
---
include/linux/migrate.h +9 -0
@@ -146,6 +146,9 @@ int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node);int migrate_misplaced_folio (struct folio *folio,struct vm_area_struct *vma,int node); +int migrate_misplaced_folio_batch (struct list_head *foliolist, +struct vm_area_struct *vma, +int node);#else static inline int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -157,6 +160,12 @@ static inline int migrate_misplaced_folio(struct folio *folio, {return -EAGAIN ;/* can't migrate now */ } +int migrate_misplaced_folio_batch (struct list_head *foliolist, +struct vm_area_struct *vma, +int node) +{ +return -EAGAIN ;/* can't migrate now */ +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION
mm/migrate.c +22 -0
@@ -2608,5 +2608,27 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,BUG_ON (!list_empty (&migratepages));return nr_remaining ? -EAGAIN :0 ; } + +int migrate_misplaced_folio_batch (struct list_head *folio_list, +struct vm_area_struct *vma, +int node) +{ +pg_data_t *pgdat =NODE_DATA (node); +unsigned int nr_succeeded; +int nr_remaining; + + nr_remaining =migrate_pages (folio_list, alloc_misplaced_dst_folio, +NULL , node,MIGRATE_ASYNC , +MR_NUMA_MISPLACED , &nr_succeeded); +if (nr_remaining) +putback_movable_pages (folio_list); + +if (nr_succeeded) { +count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_succeeded); +mod_node_page_state (pgdat,PGPROMOTE_SUCCESS , nr_succeeded); + } +BUG_ON (!list_empty (folio_list)); +return nr_remaining ? -EAGAIN :0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */
Zi Yan
reviewerQuoted 10 lines · Gregory Price
Quoted 8 lines · Zi Yan
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
Quoted 4 lines
This is well timed as I've been testing a batch-migration variant of
migrate_misplaced_foliofor my pagecache promotion work (attached).I will add this to my pagecache branch and give it a test at some point.
Great. Thanks.
Quoted 4 lines
Quick question: is the multi-threaded movement supported in the context of
task_work? i.e. in which context is the multi-threaded path safe/unsafe? (inline in a syscall, async only, etc).
It should work in any context, like syscall, memory compaction, and so on, since it just distributes memcpy to different CPUs using workqueue.
Quoted 54 lines
~Gregory
---
include/linux/migrate.h +9
-0@@ -146,6 +146,9 @@ int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node);int migrate_misplaced_folio (struct folio *folio,struct vm_area_struct *vma,int node); +int migrate_misplaced_folio_batch (struct list_head *foliolist, +struct vm_area_struct *vma, +int node);#else static inline int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)@@ -157,6 +160,12 @@ static inline int migrate_misplaced_folio(struct folio *folio, {return -EAGAIN ;/* can't migrate now */ } +int migrate_misplaced_folio_batch (struct list_head *foliolist, +struct vm_area_struct *vma, +int node) +{ +return -EAGAIN ;/* can't migrate now */ +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION mm/migrate.c +14
-0@@ -2608,5 +2608,27 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,BUG_ON (!list_empty (&migratepages));return nr_remaining ? -EAGAIN :0 ; } + +int migrate_misplaced_folio_batch (struct list_head *folio_list, +struct vm_area_struct *vma, +int node) +{ +pg_data_t *pgdat =NODE_DATA (node); +unsigned int nr_succeeded; +int nr_remaining; + + nr_remaining =migrate_pages (folio_list, alloc_misplaced_dst_folio, +NULL , node,MIGRATE_ASYNC , +MR_NUMA_MISPLACED , &nr_succeeded); +if (nr_remaining) +putback_movable_pages (folio_list);
mm/migrate.c int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, · line 2634
Quoted 10 lines
mm/migrate.c +8
-0+ +if (nr_succeeded) { +count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_succeeded); +mod_node_page_state (pgdat,PGPROMOTE_SUCCESS , nr_succeeded); + } +BUG_ON (!list_empty (folio_list)); +return nr_remaining ? -EAGAIN :0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */
Best Regards,
Yan, Zi
Quoted 73 lines · Zi Yan
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
The motivations are:
1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on
x86_64, ARM64 and 64KB on ARM64.2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system.
Design
===It is based on Shivank's patchset and revise
MIGRATE_SYNC_NO_COPY(renamed toMIGRATE_NO_COPY) to avoid folio copy operation insidemigrate_folio_move()and perform them in one shot afterwards. Acopy_page_lists_mt()function is added to use multi threads to copy folios from src list to dst list.Changes compared to Shivank's patchset (mainly rewrote batching folio copy code)
===1.
mig_infois removed, so no memory allocation is needed during batching folio copies.src->privateis used to store old page state andanon_vmaafter folio metadata is copied from src to dst.2.
move_to_new_folio()andmigrate_folio_move()are refactored to remove redundant code inmigrate_folios_batch_move().3.
folio_mc_copy()is used for the single threaded copy code to keep the original kernel behavior.Performance
===I benchmarked
move_pages()throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured.The tables below show
move_pages()throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s.The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32 32 5.43 4.90 5.65 7.31 7.60 8.61 6.43 256 6.95 6.89 9.28 14.67 22.41 23.39 23.93 512 7.88 7.26 10.15 17.53 27.82 27.88 33.93 768 7.65 7.42 10.46 18.59 28.65 29.67 30.76 1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
Quoted 13 lines
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32 1 5.94 2.90 6.90 8.56 11.16 8.76 6.41 2 7.67 5.57 7.11 12.48 17.37 15.68 14.10 4 8.01 6.04 10.25 20.14 22.52 27.79 25.28 8 8.42 7.00 11.41 24.73 33.96 32.62 39.55 16 9.41 6.91 12.23 27.51 43.95 49.15 51.38 32 10.23 7.15 13.03 29.52 49.49 69.98 71.51 64 9.40 7.37 13.88 30.38 52.00 76.89 79.41 128 8.59 7.23 14.20 28.39 49.98 78.27 90.18 256 8.43 7.16 14.59 28.14 48.78 76.88 92.28 512 8.31 7.78 14.40 26.20 43.31 63.91 75.21 768 8.30 7.86 14.83 27.41 46.25 69.85 81.31 1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84
Is this done on an idle system or a busy system? For real production workloads, all the CPUs are likely busy. It would be great to have the performance data collected from a busys system too.
Quoted 8 lines
TODOs
===
1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority.
The other potential problem is it is hard to attribute cpu time consumed by the migration work threads to cpu cgroups. In a multi-tenant environment this may result in unfair cpu time counting. However, it is a chronic problem to properly count cpu time for kernel threads. I'm not sure whether it has been solved or not.
Quoted 8 lines
2. Eliminate memory allocation during multi-threaded folio copy routine if possible.
3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3].
4. Use non-temporal CPU instructions to avoid cache pollution issues.
AFAICT, arm64 already uses non-temporal instructions for copy page.
Quoted 41 lines
5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since
kmap_local_page()would be needed for each kernel folio copy work threads and expensive.6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.Let me know your thoughts. Thanks.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Byungchul Park (1):
mm: separate move/undo doing on folio list frommigrate_pages_batch()Zi Yan (4):
mm/migrate: factor out code inmove_to_new_folio()and
migrate_folio_move()
mm/migrate: addmigrate_folios_batch_moveto batch the folio move
operations
mm/migrate: introduce multi-threaded page copy routine
test: add sysctl for folio copy tests and adjust
NR_MAX_BATCHED_MIGRATIONinclude/linux/migrate.h | 3 + include/linux/migrate_mode.h | 2 + include/linux/mm.h | 4 + include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++- mm/Makefile | 2 +- mm/copy_pages.c | 190 +++++++++++++++ mm/migrate.c | 443 +++++++++++++++++++++++++++-------- 8 files changed, 577 insertions(+), 97 deletions(-) create mode 100644 mm/copy_pages.c--
2.45.2
Zi Yan
reviewerQuoted 88 lines · Yang Shi
Quoted 73 lines · Zi Yan
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
The motivations are:
1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on
x86_64, ARM64 and 64KB on ARM64.2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system.
Design
===It is based on Shivank's patchset and revise
MIGRATE_SYNC_NO_COPY(renamed toMIGRATE_NO_COPY) to avoid folio copy operation insidemigrate_folio_move()and perform them in one shot afterwards. Acopy_page_lists_mt()function is added to use multi threads to copy folios from src list to dst list.Changes compared to Shivank's patchset (mainly rewrote batching folio copy code)
===1.
mig_infois removed, so no memory allocation is needed during batching folio copies.src->privateis used to store old page state andanon_vmaafter folio metadata is copied from src to dst.2.
move_to_new_folio()andmigrate_folio_move()are refactored to remove redundant code inmigrate_folios_batch_move().3.
folio_mc_copy()is used for the single threaded copy code to keep the original kernel behavior.Performance
===I benchmarked
move_pages()throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured.The tables below show
move_pages()throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s.The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32 32 5.43 4.90 5.65 7.31 7.60 8.61 6.43 256 6.95 6.89 9.28 14.67 22.41 23.39 23.93 512 7.88 7.26 10.15 17.53 27.82 27.88 33.93 768 7.65 7.42 10.46 18.59 28.65 29.67 30.76 1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
Quoted 13 lines
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32 1 5.94 2.90 6.90 8.56 11.16 8.76 6.41 2 7.67 5.57 7.11 12.48 17.37 15.68 14.10 4 8.01 6.04 10.25 20.14 22.52 27.79 25.28 8 8.42 7.00 11.41 24.73 33.96 32.62 39.55 16 9.41 6.91 12.23 27.51 43.95 49.15 51.38 32 10.23 7.15 13.03 29.52 49.49 69.98 71.51 64 9.40 7.37 13.88 30.38 52.00 76.89 79.41 128 8.59 7.23 14.20 28.39 49.98 78.27 90.18 256 8.43 7.16 14.59 28.14 48.78 76.88 92.28 512 8.31 7.78 14.40 26.20 43.31 63.91 75.21 768 8.30 7.86 14.83 27.41 46.25 69.85 81.31 1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84
Quoted 3 lines
Is this done on an idle system or a busy system? For real production workloads, all the CPUs are likely busy. It would be great to have the performance data collected from a busys system too.
Yes, it was done on an idle system.
I redid the experiments on a busy system by running stress on all CPU cores and the results are as not good, since all CPUs are occupied. Then I switched to system_highpri_wq, the throughput got better, almost on par with the results on an idle machine. The numbers are below.
It becomes a trade-off between page migration throughput vs user application performance on _a busy system_. If a page migration is badly needed, system_highpri_wq can be used to retain high copy throughput. Otherwise, multithreads should not be used.
64KB with system_unbound_wq on a busy system (GB/s):
| ---- | -------- | ---- | ---- | ---- | ---- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | mt_32 | | ---- | -------- | ---- | ---- | ---- | ---- | ----- | ----- | | 32 | 4.05 | 1.51 | 1.32 | 1.20 | 4.31 | 1.05 | 0.02 | | 256 | 6.91 | 3.93 | 4.61 | 0.08 | 4.46 | 4.30 | 3.89 | | 512 | 7.28 | 4.87 | 1.81 | 6.18 | 4.38 | 5.58 | 6.10 | | 768 | 4.57 | 5.72 | 5.35 | 5.24 | 5.94 | 5.66 | 0.20 | | 1024 | 7.88 | 5.73 | 5.81 | 6.52 | 7.29 | 6.06 | 5.62 |
2MB with system_unbound_wq on a busy system (GB/s):
| ---- | ------- | ---- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | mt_32 | | ---- | ------- | ---- | ---- | ---- | ----- | ----- | ----- | | 1 | 1.38 | 0.59 | 1.45 | 1.99 | 1.59 | 2.18 | 1.48 | | 2 | 1.13 | 3.08 | 3.11 | 1.85 | 0.32 | 1.46 | 2.53 | | 4 | 8.31 | 4.02 | 5.68 | 3.22 | 2.96 | 5.77 | 2.91 | | 8 | 8.16 | 5.09 | 1.19 | 4.96 | 4.50 | 3.36 | 4.99 | | 16 | 3.47 | 5.13 | 5.72 | 7.06 | 5.90 | 6.49 | 5.34 | | 32 | 8.42 | 6.97 | 0.13 | 6.77 | 7.69 | 7.56 | 2.87 | | 64 | 7.45 | 8.06 | 7.22 | 8.60 | 8.07 | 7.16 | 0.57 | | 128 | 7.77 | 7.93 | 7.29 | 8.31 | 7.77 | 9.05 | 0.92 | | 256 | 6.91 | 7.20 | 6.80 | 8.56 | 7.81 | 10.13 | 11.21 | | 512 | 6.72 | 7.22 | 7.77 | 9.71 | 10.68 | 10.35 | 10.40 | | 768 | 6.87 | 7.18 | 7.98 | 9.28 | 10.85 | 10.83 | 14.17 | | 1024 | 6.95 | 7.23 | 8.03 | 9.59 | 10.88 | 10.22 | 20.27 |
64KB with system_highpri_wq on a busy system (GB/s):
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | mt_32 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | ----- | | 32 | 4.05 | 2.63 | 1.62 | 1.90 | 3.34 | 3.71 | 3.40 | | 256 | 6.91 | 5.16 | 4.33 | 8.07 | 6.81 | 10.31 | 13.51 | | 512 | 7.28 | 4.89 | 6.43 | 15.72 | 11.31 | 18.03 | 32.69 | | 768 | 4.57 | 6.27 | 6.42 | 11.06 | 8.56 | 14.91 | 9.24 | | 1024 | 7.88 | 6.73 | 0.49 | 17.09 | 19.34 | 23.60 | 18.12 |
2MB with system_highpri_wq on a busy system (GB/s):
| ---- | ------- | ---- | ----- | ----- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | mt_32 | | ---- | ------- | ---- | ----- | ----- | ----- | ----- | ----- | | 1 | 1.38 | 1.18 | 1.17 | 5.00 | 1.68 | 3.86 | 2.46 | | 2 | 1.13 | 1.78 | 1.05 | 0.01 | 3.52 | 1.84 | 1.80 | | 4 | 8.31 | 3.91 | 5.24 | 4.30 | 4.12 | 2.93 | 3.44 | | 8 | 8.16 | 6.09 | 3.67 | 7.81 | 11.10 | 8.47 | 15.21 | | 16 | 3.47 | 6.02 | 8.44 | 11.80 | 9.56 | 12.84 | 9.81 | | 32 | 8.42 | 7.34 | 10.10 | 13.79 | 23.03 | 26.68 | 45.24 | | 64 | 7.45 | 7.90 | 12.27 | 19.99 | 36.08 | 35.11 | 60.26 | | 128 | 7.77 | 7.57 | 13.35 | 24.67 | 35.03 | 41.40 | 51.68 | | 256 | 6.91 | 7.40 | 14.13 | 25.37 | 38.83 | 62.18 | 51.37 | | 512 | 6.72 | 7.26 | 14.72 | 27.37 | 43.99 | 66.84 | 69.63 | | 768 | 6.87 | 7.29 | 14.84 | 26.34 | 47.21 | 67.51 | 80.32 | | 1024 | 6.95 | 7.26 | 14.88 | 26.98 | 47.75 | 74.99 | 85.00 |
Quoted 20 lines
Quoted 9 lines
TODOs
===
1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority.The other potential problem is it is hard to attribute cpu time consumed by the migration work threads to cpu cgroups. In a multi-tenant environment this may result in unfair cpu time counting. However, it is a chronic problem to properly count cpu time for kernel threads. I'm not sure whether it has been solved or not.
Quoted 4 lines
2. Eliminate memory allocation during multi-threaded folio copy routine if possible.
Quoted 6 lines
Quoted 4 lines
3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3].
4. Use non-temporal CPU instructions to avoid cache pollution issues.
AFAICT, arm64 already uses non-temporal instructions for copy page.
Right. My current implementation uses memcpy, which does not use non-temporal on ARM64, since a huge page can be copied by multiple threads. A non-temporal memcpy can be added for this use.
Thank you for the inputs.
Quoted 39 lines
Quoted 39 lines
5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since
kmap_local_page()would be needed for each kernel folio copy work threads and expensive.6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.Let me know your thoughts. Thanks.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Byungchul Park (1):
mm: separate move/undo doing on folio list frommigrate_pages_batch()Zi Yan (4):
mm/migrate: factor out code inmove_to_new_folio()and
migrate_folio_move()
mm/migrate: addmigrate_folios_batch_moveto batch the folio move
operations
mm/migrate: introduce multi-threaded page copy routine
test: add sysctl for folio copy tests and adjust
NR_MAX_BATCHED_MIGRATIONinclude/linux/migrate.h | 3 + include/linux/migrate_mode.h | 2 + include/linux/mm.h | 4 + include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++- mm/Makefile | 2 +- mm/copy_pages.c | 190 +++++++++++++++ mm/migrate.c | 443 +++++++++++++++++++++++++++-------- 8 files changed, 577 insertions(+), 97 deletions(-) create mode 100644 mm/copy_pages.c
Quoted 3 lines
Quoted 3 lines
--
2.45.2
--
Best Regards,
Yan, Zi
On 1/3/2025 10:54 PM, Zi Yan wrote:
Hi Zi,
It's interesting to see my batch page migration patchset evolution with multi-threading support. Thanks for sharing this.
Quoted 64 lines
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
The motivations are:
1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on
x86_64, ARM64 and 64KB on ARM64.2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system.
Design
===It is based on Shivank's patchset and revise
MIGRATE_SYNC_NO_COPY(renamed toMIGRATE_NO_COPY) to avoid folio copy operation insidemigrate_folio_move()and perform them in one shot afterwards. Acopy_page_lists_mt()function is added to use multi threads to copy folios from src list to dst list.Changes compared to Shivank's patchset (mainly rewrote batching folio copy code)
===1.
mig_infois removed, so no memory allocation is needed during batching folio copies.src->privateis used to store old page state andanon_vmaafter folio metadata is copied from src to dst.2.
move_to_new_folio()andmigrate_folio_move()are refactored to remove redundant code inmigrate_folios_batch_move().3.
folio_mc_copy()is used for the single threaded copy code to keep the original kernel behavior.TODOs
===
1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority.2. Eliminate memory allocation during multi-threaded folio copy routine if possible.
3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3].
4. Use non-temporal CPU instructions to avoid cache pollution issues.
Quoted 6 lines
5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since
kmap_local_page()would be needed for each kernel folio copy work threads and expensive.6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.
I think Static Calls can be better option for this.
This will give a flexible copy interface to support both CPU and various DMA-based folio copy. DMA-capable driver can override the default CPU copy path without any additional runtime overheads.
Quoted 28 lines
Performance
===I benchmarked
move_pages()throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured.The tables below show
move_pages()throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s.The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_3232 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32
Quoted 12 lines
1 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84
I'm measuring the throughput(in GB/s) on our AMD EPYC Zen 5 system (2-socket, 64-core per socket with SMT Enabled, 2 NUMA nodes) with base page-size as 4KB and using using mm-everything-2025-01-04-04-41 as base kernel.
Method:
======main() {
...
// code snippet to measure throughput
clock_gettime(CLOCK_MONOTONIC, &t1);
retcode = move_pages(getpid(), num_pages, pages, nodesArray , statusArray, MPOL_MF_MOVE);
clock_gettime(CLOCK_MONOTONIC, &t2);
// tput = num_pages*PAGE_SIZE/(t2-t1)...
}
Measurements:
============
vanilla: base kernel without patchset
mt:0 = MT kernel with use_mt_copy=0
mt:1..mt:32 = MT kernel with use_mt_copy=1 and thread cnt = 1,2,...,32
Measured for both configuration push_0_pull_1=0 and push_0_pull_1=1 and for 4KB migration and THP migration.
--------------------
#1 push_0_pull_1 = 0 (src node CPUs are used)
#1.1 THP=Never, 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.28 1.92 1.80 2.24 2.35 2.22 2.17 4096 2.40 2.40 2.51 2.58 2.83 2.72 2.99 3.25 8192 3.18 2.88 2.83 2.69 3.49 3.46 3.57 3.80 16348 3.17 2.94 2.96 3.17 3.63 3.68 4.06 4.15
#1.2 THP=Always, 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.02 3.39 3.40 3.33 3.51 3.91 4.03 1024 7.13 4.49 3.58 3.56 3.91 3.87 4.39 4.57 2048 5.26 6.47 3.91 4.00 3.71 3.85 4.97 6.83 4096 9.93 7.77 4.58 3.79 3.93 3.53 6.41 4.77 8192 6.47 6.33 4.37 4.67 4.52 4.39 5.30 5.37 16348 7.66 8.00 5.20 5.22 5.24 5.28 6.41 7.02 32768 8.56 8.62 6.34 6.20 6.20 6.19 7.18 8.10 65536 9.41 9.40 7.14 7.15 7.15 7.19 7.96 8.89 262144 10.17 10.19 7.26 7.90 7.98 8.05 9.46 10.30 524288 10.40 9.95 7.25 7.93 8.02 8.76 9.55 10.30
--------------------
#2 push_0_pull_1 = 1 (dst node CPUs are used):
#2.1 THP=Never 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.36 2.01 2.74 2.33 2.31 2.53 2.96 4096 2.40 2.84 2.94 3.04 3.40 3.23 3.31 4.16 8192 3.18 3.27 3.34 3.94 3.77 3.68 4.23 4.76 16348 3.17 3.42 3.66 3.21 3.82 4.40 4.76 4.89
#2.2 THP=Always 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.91 4.03 3.73 4.26 4.13 4.78 3.44 1024 7.13 6.83 4.60 5.13 5.03 5.19 5.94 7.25 2048 5.26 7.09 5.20 5.69 5.83 5.73 6.85 8.13 4096 9.93 9.31 4.90 4.82 4.82 5.26 8.46 8.52 8192 6.47 7.63 5.66 5.85 5.75 6.14 7.45 8.63 16348 7.66 10.00 6.35 6.54 6.66 6.99 8.18 10.21 32768 8.56 9.78 7.06 7.41 7.76 9.02 9.55 11.92 65536 9.41 10.00 8.19 9.20 9.32 8.68 11.00 13.31 262144 10.17 11.17 9.01 9.96 9.99 10.00 11.70 14.27 524288 10.40 11.38 9.07 9.98 10.01 10.09 11.95 14.48
Note:
1. For THP = Never: I'm doing for 16X pages to keep total size same for your experiment with 64KB pagesize)
2. For THP = Always: nr_pages = Number of 4KB pages moved. nr_pages=512 => 512 4KB pages => 1 2MB page)
I'm seeing little (1.5X in some cases) to no benefits. The performance scaling is relatively flat across thread counts.
Is it possible I'm missing something in my testing?
Could the base page size difference (4KB vs 64KB) be playing a role in the scaling behavior? How the performance varies with 4KB pages on your system?
I'd be happy to work with you on investigating this differences. Let me know if you'd like any additional test data or if there are specific configurations I should try.
Quoted 31 lines
Let me know your thoughts. Thanks.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Byungchul Park (1):
mm: separate move/undo doing on folio list frommigrate_pages_batch()Zi Yan (4):
mm/migrate: factor out code inmove_to_new_folio()and
migrate_folio_move()
mm/migrate: addmigrate_folios_batch_moveto batch the folio move
operations
mm/migrate: introduce multi-threaded page copy routine
test: add sysctl for folio copy tests and adjust
NR_MAX_BATCHED_MIGRATIONinclude/linux/migrate.h | 3 + include/linux/migrate_mode.h | 2 + include/linux/mm.h | 4 + include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++- mm/Makefile | 2 +- mm/copy_pages.c | 190 +++++++++++++++ mm/migrate.c | 443 +++++++++++++++++++++++++++-------- 8 files changed, 577 insertions(+), 97 deletions(-) create mode 100644 mm/copy_pages.c
Zi Yan
reviewerQuoted 71 lines · Shivank Garg
On 1/3/2025 10:54 PM, Zi Yan wrote:
Hi Zi,
It's interesting to see my batch page migration patchset evolution with multi-threading support. Thanks for sharing this.
Quoted 64 lines
Hi all,
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
The motivations are:
1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on
x86_64, ARM64 and 64KB on ARM64.2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system.
Design
===It is based on Shivank's patchset and revise
MIGRATE_SYNC_NO_COPY(renamed toMIGRATE_NO_COPY) to avoid folio copy operation insidemigrate_folio_move()and perform them in one shot afterwards. Acopy_page_lists_mt()function is added to use multi threads to copy folios from src list to dst list.Changes compared to Shivank's patchset (mainly rewrote batching folio copy code)
===1.
mig_infois removed, so no memory allocation is needed during batching folio copies.src->privateis used to store old page state andanon_vmaafter folio metadata is copied from src to dst.2.
move_to_new_folio()andmigrate_folio_move()are refactored to remove redundant code inmigrate_folios_batch_move().3.
folio_mc_copy()is used for the single threaded copy code to keep the original kernel behavior.TODOs
===
1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority.2. Eliminate memory allocation during multi-threaded folio copy routine if possible.
3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3].
4. Use non-temporal CPU instructions to avoid cache pollution issues.
Quoted 8 lines
Quoted 6 lines
5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since
kmap_local_page()would be needed for each kernel folio copy work threads and expensive.6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.I think Static Calls can be better option for this.
This is the first time I hear about it. Based on the info I find, I agree it is a great mechanism to switch between two methods globally.
Quoted 4 lines
This will give a flexible copy interface to support both CPU and various DMA-based folio copy. DMA-capable driver can override the default CPU copy path without any additional runtime overheads.
Yes, supporting DMA-based folio copy is also my intention too. I am happy to with you on that. Things to note are:
1. DMA engine should have more copy throughput as a single CPU thread, otherwise the scatter-gather setup overheads will eliminate the benefit of using DMA engine.
2. Unless the DMA engine is really beef and can handle all possible page migration requests, CPU-based migration (single or multi threads) should be a fallback.
In terms of 2, I wonder how much overheads does Static Calls have when switching between functions. Also, a lock might be needed since falling back to CPU might be per migrate_pages(). Considering these two, Static Calls might not work as you intended if switching between CPU and DMA is needed.
Quoted 129 lines
Quoted 30 lines
Performance
===I benchmarked
move_pages()throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured.The tables below show
move_pages()throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s.The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_3232 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32Quoted 12 lines
1 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84I'm measuring the throughput(in GB/s) on our AMD EPYC Zen 5 system (2-socket, 64-core per socket with SMT Enabled, 2 NUMA nodes) with base page-size as 4KB and using using mm-everything-2025-01-04-04-41 as base kernel.
Method:
======main(){
...// code snippet to measure throughput clock_gettime(CLOCK_MONOTONIC, &t1); retcode = move_pages(getpid(), num_pages, pages, nodesArray , statusArray, MPOL_MF_MOVE); clock_gettime(CLOCK_MONOTONIC, &t2); // tput = num_pages*PAGE_SIZE/(t2-t1)...
}Measurements:
============
vanilla: base kernel without patchset
mt:0 = MT kernel with use_mt_copy=0
mt:1..mt:32 = MT kernel with use_mt_copy=1 and thread cnt = 1,2,...,32Measured for both configuration push_0_pull_1=0 and push_0_pull_1=1 and for 4KB migration and THP migration.
--------------------
#1push_0_pull_1= 0 (src node CPUs are used)#1.1 THP=Never, 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.28 1.92 1.80 2.24 2.35 2.22 2.17 4096 2.40 2.40 2.51 2.58 2.83 2.72 2.99 3.25 8192 3.18 2.88 2.83 2.69 3.49 3.46 3.57 3.80 16348 3.17 2.94 2.96 3.17 3.63 3.68 4.06 4.15#1.2 THP=Always, 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.02 3.39 3.40 3.33 3.51 3.91 4.03 1024 7.13 4.49 3.58 3.56 3.91 3.87 4.39 4.57 2048 5.26 6.47 3.91 4.00 3.71 3.85 4.97 6.83 4096 9.93 7.77 4.58 3.79 3.93 3.53 6.41 4.77 8192 6.47 6.33 4.37 4.67 4.52 4.39 5.30 5.37 16348 7.66 8.00 5.20 5.22 5.24 5.28 6.41 7.02 32768 8.56 8.62 6.34 6.20 6.20 6.19 7.18 8.10 65536 9.41 9.40 7.14 7.15 7.15 7.19 7.96 8.89 262144 10.17 10.19 7.26 7.90 7.98 8.05 9.46 10.30 524288 10.40 9.95 7.25 7.93 8.02 8.76 9.55 10.30--------------------
#2push_0_pull_1= 1 (dst node CPUs are used):#2.1 THP=Never 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.36 2.01 2.74 2.33 2.31 2.53 2.96 4096 2.40 2.84 2.94 3.04 3.40 3.23 3.31 4.16 8192 3.18 3.27 3.34 3.94 3.77 3.68 4.23 4.76 16348 3.17 3.42 3.66 3.21 3.82 4.40 4.76 4.89#2.2 THP=Always 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.91 4.03 3.73 4.26 4.13 4.78 3.44 1024 7.13 6.83 4.60 5.13 5.03 5.19 5.94 7.25 2048 5.26 7.09 5.20 5.69 5.83 5.73 6.85 8.13 4096 9.93 9.31 4.90 4.82 4.82 5.26 8.46 8.52 8192 6.47 7.63 5.66 5.85 5.75 6.14 7.45 8.63 16348 7.66 10.00 6.35 6.54 6.66 6.99 8.18 10.21 32768 8.56 9.78 7.06 7.41 7.76 9.02 9.55 11.92 65536 9.41 10.00 8.19 9.20 9.32 8.68 11.00 13.31 262144 10.17 11.17 9.01 9.96 9.99 10.00 11.70 14.27 524288 10.40 11.38 9.07 9.98 10.01 10.09 11.95 14.48Note:
1. For THP = Never: I'm doing for 16X pages to keep total size same for your experiment with 64KB pagesize)
2. For THP = Always:nr_pages= Number of 4KB pages moved. nr_pages=512 => 512 4KB pages => 1 2MB page)I'm seeing little (1.5X in some cases) to no benefits. The performance scaling is relatively flat across thread counts.
Quoted 8 lines
Is it possible I'm missing something in my testing?
Could the base page size difference (4KB vs 64KB) be playing a role in the scaling behavior? How the performance varies with 4KB pages on your system?
I'd be happy to work with you on investigating this differences. Let me know if you'd like any additional test data or if there are specific configurations I should try.
The results surprises me, since I was able to achieve ~9GB/s when migrating 16 2MB THPs with 16 threads on a two socket system with Xeon E5-2650 v3 @ 2.30GHz (a 19.2GB/s bandwidth QPI link between two sockets) back in 2019[1]. These are 10-year-old Haswell CPUs. And your results above show that EPYC 5 can only achieve ~4GB/s when migrating 512 2MB THPs with 16 threads. It just does not make sense.
One thing you might want to try is to set init_on_alloc=0 in your boot parameters to use folio_zero_user() instead of GFP_ZERO to zero pages. That might reduce the time spent on page zeros.
I am also going to rerun the experiments locally on x86_64 boxes to see if your results can be replicated.
Thank you for the review and running these experiments. I really appreciate it.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Best Regards,
Yan, Zi
Quoted 15 lines · Zi Yan
Quoted 9 lines · Shivank Garg
Quoted 3 lines · Zi Yan
6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.I think Static Calls can be better option for this.
This is the first time I hear about it. Based on the info I find, I agree it is a great mechanism to switch between two methods globally.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
This will give a flexible copy interface to support both CPU and various DMA-based folio copy. DMA-capable driver can override the default CPU copy path without any additional runtime overheads.
Yes, supporting DMA-based folio copy is also my intention too. I am happy to with you on that. Things to note are:
1. DMA engine should have more copy throughput as a single CPU thread, otherwise the scatter-gather setup overheads will eliminate the benefit of using DMA engine.
I agree on this.
Quoted 7 lines
2. Unless the DMA engine is really beef and can handle all possible page migration requests, CPU-based migration (single or multi threads) should be a fallback.
In terms of 2, I wonder how much overheads does Static Calls have when switching between functions. Also, a lock might be needed since falling back to CPU might be per
migrate_pages(). Considering these two, Static Calls might not work as you intended if switching between CPU and DMA is needed.
You can check Patch 4/5 and 5/5 for static call implementation for using DMA Driver
https://lore.kernel.org/linux-mm/[email protected]
There are no run-time overheads of this Static call approach as update happens only during DMA driver registration/un-registration - dma_update_migrator() The SRCU synchronization will ensure the safety during updates.
It'll use static_call(_folios_copy)() for the copy path. A wrapper inside the DMA can ensure it fallback to folios_copy().
Does this address your concern regarding the 2?
Quoted 99 lines
Quoted 79 lines
main(){
...// code snippet to measure throughput clock_gettime(CLOCK_MONOTONIC, &t1); retcode = move_pages(getpid(), num_pages, pages, nodesArray , statusArray, MPOL_MF_MOVE); clock_gettime(CLOCK_MONOTONIC, &t2); // tput = num_pages*PAGE_SIZE/(t2-t1)...
}Measurements:
============
vanilla: base kernel without patchset
mt:0 = MT kernel with use_mt_copy=0
mt:1..mt:32 = MT kernel with use_mt_copy=1 and thread cnt = 1,2,...,32Measured for both configuration push_0_pull_1=0 and push_0_pull_1=1 and for 4KB migration and THP migration.
--------------------
#1push_0_pull_1= 0 (src node CPUs are used)#1.1 THP=Never, 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.28 1.92 1.80 2.24 2.35 2.22 2.17 4096 2.40 2.40 2.51 2.58 2.83 2.72 2.99 3.25 8192 3.18 2.88 2.83 2.69 3.49 3.46 3.57 3.80 16348 3.17 2.94 2.96 3.17 3.63 3.68 4.06 4.15#1.2 THP=Always, 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.02 3.39 3.40 3.33 3.51 3.91 4.03 1024 7.13 4.49 3.58 3.56 3.91 3.87 4.39 4.57 2048 5.26 6.47 3.91 4.00 3.71 3.85 4.97 6.83 4096 9.93 7.77 4.58 3.79 3.93 3.53 6.41 4.77 8192 6.47 6.33 4.37 4.67 4.52 4.39 5.30 5.37 16348 7.66 8.00 5.20 5.22 5.24 5.28 6.41 7.02 32768 8.56 8.62 6.34 6.20 6.20 6.19 7.18 8.10 65536 9.41 9.40 7.14 7.15 7.15 7.19 7.96 8.89 262144 10.17 10.19 7.26 7.90 7.98 8.05 9.46 10.30 524288 10.40 9.95 7.25 7.93 8.02 8.76 9.55 10.30--------------------
#2push_0_pull_1= 1 (dst node CPUs are used):#2.1 THP=Never 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.36 2.01 2.74 2.33 2.31 2.53 2.96 4096 2.40 2.84 2.94 3.04 3.40 3.23 3.31 4.16 8192 3.18 3.27 3.34 3.94 3.77 3.68 4.23 4.76 16348 3.17 3.42 3.66 3.21 3.82 4.40 4.76 4.89#2.2 THP=Always 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.91 4.03 3.73 4.26 4.13 4.78 3.44 1024 7.13 6.83 4.60 5.13 5.03 5.19 5.94 7.25 2048 5.26 7.09 5.20 5.69 5.83 5.73 6.85 8.13 4096 9.93 9.31 4.90 4.82 4.82 5.26 8.46 8.52 8192 6.47 7.63 5.66 5.85 5.75 6.14 7.45 8.63 16348 7.66 10.00 6.35 6.54 6.66 6.99 8.18 10.21 32768 8.56 9.78 7.06 7.41 7.76 9.02 9.55 11.92 65536 9.41 10.00 8.19 9.20 9.32 8.68 11.00 13.31 262144 10.17 11.17 9.01 9.96 9.99 10.00 11.70 14.27 524288 10.40 11.38 9.07 9.98 10.01 10.09 11.95 14.48Note:
1. For THP = Never: I'm doing for 16X pages to keep total size same for your experiment with 64KB pagesize)
2. For THP = Always:nr_pages= Number of 4KB pages moved. nr_pages=512 => 512 4KB pages => 1 2MB page)I'm seeing little (1.5X in some cases) to no benefits. The performance scaling is relatively flat across thread counts.
Quoted 8 lines
Is it possible I'm missing something in my testing?
Could the base page size difference (4KB vs 64KB) be playing a role in the scaling behavior? How the performance varies with 4KB pages on your system?
I'd be happy to work with you on investigating this differences. Let me know if you'd like any additional test data or if there are specific configurations I should try.
The results surprises me, since I was able to achieve ~9GB/s when migrating 16 2MB THPs with 16 threads on a two socket system with Xeon E5-2650 v3 @ 2.30GHz (a 19.2GB/s bandwidth QPI link between two sockets) back in 2019[1]. These are 10-year-old Haswell CPUs. And your results above show that EPYC 5 can only achieve ~4GB/s when migrating 512 2MB THPs with 16 threads. It just does not make sense.
One thing you might want to try is to set init_on_alloc=0 in your boot parameters to use
folio_zero_user()instead ofGFP_ZEROto zero pages. That might reduce the time spent on page zeros.
Quoted 8 lines
I am also going to rerun the experiments locally on
x86_64boxes to see if your results can be replicated.Thank you for the review and running these experiments. I really appreciate it.>
Using init_on_alloc=0 gave significant performance gain over the last experiment but I'm still missing the performance scaling you observed.
THP Never
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.40 1.43 2.79 3.48 3.63 3.73 3.63 3.57 4096 2.54 3.32 3.18 4.65 4.83 5.11 5.39 5.78 8192 3.35 4.40 4.39 4.71 3.63 5.04 5.33 6.00 16348 3.76 4.50 4.44 5.33 5.41 5.41 6.47 6.41
THP Always
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 5.21 5.47 5.77 6.92 3.71 2.75 7.54 7.44 1024 6.10 7.65 8.12 8.41 8.87 8.55 9.13 11.36 2048 6.39 6.66 9.58 8.92 10.75 12.99 13.33 12.23 4096 7.33 10.85 8.22 13.57 11.43 10.93 12.53 16.86 8192 7.26 7.46 8.88 11.82 10.55 10.94 13.27 14.11 16348 9.07 8.53 11.82 14.89 12.97 13.22 16.14 18.10 32768 10.45 10.55 11.79 19.19 16.85 17.56 20.58 26.57 65536 11.00 11.12 13.25 18.27 16.18 16.11 19.61 27.73 262144 12.37 12.40 15.65 20.00 19.25 19.38 22.60 31.95 524288 12.44 12.33 15.66 19.78 19.06 18.96 23.31 32.29
Thanks,
Shivank
Quoted 3 lines
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 35 lines · Shivank Garg
Quoted 15 lines · Zi Yan
Quoted 9 lines · Shivank Garg
Quoted 3 lines · Zi Yan
6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.I think Static Calls can be better option for this.
This is the first time I hear about it. Based on the info I find, I agree it is a great mechanism to switch between two methods globally.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
This will give a flexible copy interface to support both CPU and various DMA-based folio copy. DMA-capable driver can override the default CPU copy path without any additional runtime overheads.
Yes, supporting DMA-based folio copy is also my intention too. I am happy to with you on that. Things to note are:
1. DMA engine should have more copy throughput as a single CPU thread, otherwise the scatter-gather setup overheads will eliminate the benefit of using DMA engine.I agree on this.
Quoted 7 lines
2. Unless the DMA engine is really beef and can handle all possible page migration requests, CPU-based migration (single or multi threads) should be a fallback.
In terms of 2, I wonder how much overheads does Static Calls have when switching between functions. Also, a lock might be needed since falling back to CPU might be per
migrate_pages(). Considering these two, Static Calls might not work as you intended if switching between CPU and DMA is needed.
Quoted 6 lines
You can check Patch 4/5 and 5/5 for static call implementation for using DMA Driver
https://lore.kernel.org/linux-mm/[email protected]There are no run-time overheads of this Static call approach as update happens only during DMA driver registration/un-registration -
dma_update_migrator()The SRCU synchronization will ensure the safety during updates.
I understand this part.
Quoted 5 lines
It'll use static_call(_folios_copy)() for the copy path. A wrapper inside the DMA can ensure it fallback to
folios_copy().Does this address your concern regarding the 2?
DMA driver will need to fall back to folios_copy() (using CPU to copy folios), when it thinks DMA engine is overloaded. In my mind, kernel should make the decision whether to use single CPU, multiple CPUs, or DMA engine based on CPU usage and DMA usage. As I am writing it, I realize that might be an overhead we want to avoid, since it takes time to get CPU usage and DMA usage information and should not be on the critical path of page migration. A better approach might be that CPU scheduler and DMA engine can call dma_update_migrator() to change _folios_copy in the static_call, based on CPU usage and DMA usage.
Yes, I think Static Calls should be able to help us choose the right folio copy method, single CPU, multiple CPUs, or DMA engine, to perform folio copies.
BTW, I notice that you called dmaengine_get_dma_device() in folios_copy_dma(), which would incur a huge overhead, based on my past experience using DMA engine for page copy. I know it is needed to make sure DMA is still present, but its cost needs to be minimized to make DMA folio copy usable. Otherwise, the 768MB/s DMA copy throughput from your cover letter cannot convince people to use it for page migration, since single CPU can achieve more than that, as you showed in the table below.
Quoted 104 lines
Quoted 104 lines
Quoted 81 lines
main(){
...// code snippet to measure throughput clock_gettime(CLOCK_MONOTONIC, &t1); retcode = move_pages(getpid(), num_pages, pages, nodesArray , statusArray, MPOL_MF_MOVE); clock_gettime(CLOCK_MONOTONIC, &t2); // tput = num_pages*PAGE_SIZE/(t2-t1)...
}Measurements:
============
vanilla: base kernel without patchset
mt:0 = MT kernel with use_mt_copy=0
mt:1..mt:32 = MT kernel with use_mt_copy=1 and thread cnt = 1,2,...,32Measured for both configuration push_0_pull_1=0 and push_0_pull_1=1 and for 4KB migration and THP migration.
--------------------
#1push_0_pull_1= 0 (src node CPUs are used)#1.1 THP=Never, 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.28 1.92 1.80 2.24 2.35 2.22 2.17 4096 2.40 2.40 2.51 2.58 2.83 2.72 2.99 3.25 8192 3.18 2.88 2.83 2.69 3.49 3.46 3.57 3.80 16348 3.17 2.94 2.96 3.17 3.63 3.68 4.06 4.15#1.2 THP=Always, 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.02 3.39 3.40 3.33 3.51 3.91 4.03 1024 7.13 4.49 3.58 3.56 3.91 3.87 4.39 4.57 2048 5.26 6.47 3.91 4.00 3.71 3.85 4.97 6.83 4096 9.93 7.77 4.58 3.79 3.93 3.53 6.41 4.77 8192 6.47 6.33 4.37 4.67 4.52 4.39 5.30 5.37 16348 7.66 8.00 5.20 5.22 5.24 5.28 6.41 7.02 32768 8.56 8.62 6.34 6.20 6.20 6.19 7.18 8.10 65536 9.41 9.40 7.14 7.15 7.15 7.19 7.96 8.89 262144 10.17 10.19 7.26 7.90 7.98 8.05 9.46 10.30 524288 10.40 9.95 7.25 7.93 8.02 8.76 9.55 10.30--------------------
#2push_0_pull_1= 1 (dst node CPUs are used):#2.1 THP=Never 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.36 2.01 2.74 2.33 2.31 2.53 2.96 4096 2.40 2.84 2.94 3.04 3.40 3.23 3.31 4.16 8192 3.18 3.27 3.34 3.94 3.77 3.68 4.23 4.76 16348 3.17 3.42 3.66 3.21 3.82 4.40 4.76 4.89#2.2 THP=Always 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.91 4.03 3.73 4.26 4.13 4.78 3.44 1024 7.13 6.83 4.60 5.13 5.03 5.19 5.94 7.25 2048 5.26 7.09 5.20 5.69 5.83 5.73 6.85 8.13 4096 9.93 9.31 4.90 4.82 4.82 5.26 8.46 8.52 8192 6.47 7.63 5.66 5.85 5.75 6.14 7.45 8.63 16348 7.66 10.00 6.35 6.54 6.66 6.99 8.18 10.21 32768 8.56 9.78 7.06 7.41 7.76 9.02 9.55 11.92 65536 9.41 10.00 8.19 9.20 9.32 8.68 11.00 13.31 262144 10.17 11.17 9.01 9.96 9.99 10.00 11.70 14.27 524288 10.40 11.38 9.07 9.98 10.01 10.09 11.95 14.48Note:
1. For THP = Never: I'm doing for 16X pages to keep total size same for your experiment with 64KB pagesize)
2. For THP = Always:nr_pages= Number of 4KB pages moved. nr_pages=512 => 512 4KB pages => 1 2MB page)I'm seeing little (1.5X in some cases) to no benefits. The performance scaling is relatively flat across thread counts.
Quoted 8 lines
Is it possible I'm missing something in my testing?
Could the base page size difference (4KB vs 64KB) be playing a role in the scaling behavior? How the performance varies with 4KB pages on your system?
I'd be happy to work with you on investigating this differences. Let me know if you'd like any additional test data or if there are specific configurations I should try.
The results surprises me, since I was able to achieve ~9GB/s when migrating 16 2MB THPs with 16 threads on a two socket system with Xeon E5-2650 v3 @ 2.30GHz (a 19.2GB/s bandwidth QPI link between two sockets) back in 2019[1]. These are 10-year-old Haswell CPUs. And your results above show that EPYC 5 can only achieve ~4GB/s when migrating 512 2MB THPs with 16 threads. It just does not make sense.
One thing you might want to try is to set init_on_alloc=0 in your boot parameters to use
folio_zero_user()instead ofGFP_ZEROto zero pages. That might reduce the time spent on page zeros.I am also going to rerun the experiments locally on
x86_64boxes to see if your results can be replicated.
Quoted 8 lines
Quoted 5 lines
Thank you for the review and running these experiments. I really appreciate it.>
Using init_on_alloc=0 gave significant performance gain over the last experiment but I'm still missing the performance scaling you observed.
It might be the difference between x86 and ARM64, but I am not 100% sure. Based on your data below, 2 or 4 threads seem to the sweep spot for the multi-threaded method on AMD CPUs. BTW, what is the bandwidth between two sockets in your system? From Figure 10 in [1], I see the InfiniteBand between two AMD EPYC 7601 @ 2.2GHz was measured at ~12GB/s unidirectional, ~25GB/s bidirectional. I wonder if your results below are cross-socket link bandwidth limited.
From my results, NVIDIA Grace CPU can achieve high copy throughput with more threads between two sockets, maybe part of the reason is that its cross-socket link theoretical bandwidth is 900GB/s bidirectional.
Quoted 9 lines
THP Never
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.40 1.43 2.79 3.48 3.63 3.73 3.63 3.57 4096 2.54 3.32 3.18 4.65 4.83 5.11 5.39 5.78 8192 3.35 4.40 4.39 4.71 3.63 5.04 5.33 6.00 16348 3.76 4.50 4.44 5.33 5.41 5.41 6.47 6.41THP Always
Quoted 11 lines
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 5.21 5.47 5.77 6.92 3.71 2.75 7.54 7.44 1024 6.10 7.65 8.12 8.41 8.87 8.55 9.13 11.36 2048 6.39 6.66 9.58 8.92 10.75 12.99 13.33 12.23 4096 7.33 10.85 8.22 13.57 11.43 10.93 12.53 16.86 8192 7.26 7.46 8.88 11.82 10.55 10.94 13.27 14.11 16348 9.07 8.53 11.82 14.89 12.97 13.22 16.14 18.10 32768 10.45 10.55 11.79 19.19 16.85 17.56 20.58 26.57 65536 11.00 11.12 13.25 18.27 16.18 16.11 19.61 27.73 262144 12.37 12.40 15.65 20.00 19.25 19.38 22.60 31.95 524288 12.44 12.33 15.66 19.78 19.06 18.96 23.31 32.29
Best Regards,
Yan, Zi
Zi Yan
reviewer<snip>
Quoted 120 lines
Quoted 103 lines
Quoted 103 lines
Quoted 80 lines
main(){
...// code snippet to measure throughput clock_gettime(CLOCK_MONOTONIC, &t1); retcode = move_pages(getpid(), num_pages, pages, nodesArray , statusArray, MPOL_MF_MOVE); clock_gettime(CLOCK_MONOTONIC, &t2); // tput = num_pages*PAGE_SIZE/(t2-t1)...
}Measurements:
============
vanilla: base kernel without patchset
mt:0 = MT kernel with use_mt_copy=0
mt:1..mt:32 = MT kernel with use_mt_copy=1 and thread cnt = 1,2,...,32Measured for both configuration push_0_pull_1=0 and push_0_pull_1=1 and for 4KB migration and THP migration.
--------------------
#1push_0_pull_1= 0 (src node CPUs are used)#1.1 THP=Never, 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.28 1.92 1.80 2.24 2.35 2.22 2.17 4096 2.40 2.40 2.51 2.58 2.83 2.72 2.99 3.25 8192 3.18 2.88 2.83 2.69 3.49 3.46 3.57 3.80 16348 3.17 2.94 2.96 3.17 3.63 3.68 4.06 4.15#1.2 THP=Always, 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.02 3.39 3.40 3.33 3.51 3.91 4.03 1024 7.13 4.49 3.58 3.56 3.91 3.87 4.39 4.57 2048 5.26 6.47 3.91 4.00 3.71 3.85 4.97 6.83 4096 9.93 7.77 4.58 3.79 3.93 3.53 6.41 4.77 8192 6.47 6.33 4.37 4.67 4.52 4.39 5.30 5.37 16348 7.66 8.00 5.20 5.22 5.24 5.28 6.41 7.02 32768 8.56 8.62 6.34 6.20 6.20 6.19 7.18 8.10 65536 9.41 9.40 7.14 7.15 7.15 7.19 7.96 8.89 262144 10.17 10.19 7.26 7.90 7.98 8.05 9.46 10.30 524288 10.40 9.95 7.25 7.93 8.02 8.76 9.55 10.30--------------------
#2push_0_pull_1= 1 (dst node CPUs are used):#2.1 THP=Never 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.36 2.01 2.74 2.33 2.31 2.53 2.96 4096 2.40 2.84 2.94 3.04 3.40 3.23 3.31 4.16 8192 3.18 3.27 3.34 3.94 3.77 3.68 4.23 4.76 16348 3.17 3.42 3.66 3.21 3.82 4.40 4.76 4.89#2.2 THP=Always 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.91 4.03 3.73 4.26 4.13 4.78 3.44 1024 7.13 6.83 4.60 5.13 5.03 5.19 5.94 7.25 2048 5.26 7.09 5.20 5.69 5.83 5.73 6.85 8.13 4096 9.93 9.31 4.90 4.82 4.82 5.26 8.46 8.52 8192 6.47 7.63 5.66 5.85 5.75 6.14 7.45 8.63 16348 7.66 10.00 6.35 6.54 6.66 6.99 8.18 10.21 32768 8.56 9.78 7.06 7.41 7.76 9.02 9.55 11.92 65536 9.41 10.00 8.19 9.20 9.32 8.68 11.00 13.31 262144 10.17 11.17 9.01 9.96 9.99 10.00 11.70 14.27 524288 10.40 11.38 9.07 9.98 10.01 10.09 11.95 14.48Note:
1. For THP = Never: I'm doing for 16X pages to keep total size same for your experiment with 64KB pagesize)
2. For THP = Always:nr_pages= Number of 4KB pages moved. nr_pages=512 => 512 4KB pages => 1 2MB page)I'm seeing little (1.5X in some cases) to no benefits. The performance scaling is relatively flat across thread counts.
Quoted 8 lines
Is it possible I'm missing something in my testing?
Could the base page size difference (4KB vs 64KB) be playing a role in the scaling behavior? How the performance varies with 4KB pages on your system?
I'd be happy to work with you on investigating this differences. Let me know if you'd like any additional test data or if there are specific configurations I should try.
The results surprises me, since I was able to achieve ~9GB/s when migrating 16 2MB THPs with 16 threads on a two socket system with Xeon E5-2650 v3 @ 2.30GHz (a 19.2GB/s bandwidth QPI link between two sockets) back in 2019[1]. These are 10-year-old Haswell CPUs. And your results above show that EPYC 5 can only achieve ~4GB/s when migrating 512 2MB THPs with 16 threads. It just does not make sense.
One thing you might want to try is to set init_on_alloc=0 in your boot parameters to use
folio_zero_user()instead ofGFP_ZEROto zero pages. That might reduce the time spent on page zeros.I am also going to rerun the experiments locally on
x86_64boxes to see if your results can be replicated.Quoted 8 lines
Quoted 5 lines
Thank you for the review and running these experiments. I really appreciate it.>
Using init_on_alloc=0 gave significant performance gain over the last experiment but I'm still missing the performance scaling you observed.
It might be the difference between x86 and ARM64, but I am not 100% sure. Based on your data below, 2 or 4 threads seem to the sweep spot for the multi-threaded method on AMD CPUs. BTW, what is the bandwidth between two sockets in your system? From Figure 10 in [1], I see the InfiniteBand between two AMD EPYC 7601 @ 2.2GHz was measured at ~12GB/s unidirectional, ~25GB/s bidirectional. I wonder if your results below are cross-socket link bandwidth limited.
Quoted 3 lines
From my results, NVIDIA Grace CPU can achieve high copy throughput with more threads between two sockets, maybe part of the reason is that its cross-socket link theoretical bandwidth is 900GB/s bidirectional.
I talked to my colleague about this and he mentioned about CCD architecture on AMD CPUs. IIUC, one or two cores from one CCD can already saturate the CCD’s outgoing bandwidth and all CPUs are enumerated from one CCD to another. This means my naive scheduling algorithm, which use CPUs from 0 to N threads, uses all cores from one CDD first, then move to another CCD. It is not able to saturate the cross-socket bandwidth. Does it make sense to you?
If yes, can you please change the my cpu selection code in mm/copy_pages.c:
+ /* TODO: need a better cpu selection method */
+ for_each_cpu(cpu, per_node_cpumask) {
+ if (i >= total_mt_num)
+ break;
+ cpu_id_list[i] = cpu;
+ ++i;
+ }
to select CPUs from as many CCDs as possible and rerun the tests. That might boost the page migration throughput on AMD CPUs more.
Thanks.
Quoted 21 lines
Quoted 9 lines
THP Never
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.40 1.43 2.79 3.48 3.63 3.73 3.63 3.57 4096 2.54 3.32 3.18 4.65 4.83 5.11 5.39 5.78 8192 3.35 4.40 4.39 4.71 3.63 5.04 5.33 6.00 16348 3.76 4.50 4.44 5.33 5.41 5.41 6.47 6.41THP Always
Quoted 11 lines
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 5.21 5.47 5.77 6.92 3.71 2.75 7.54 7.44 1024 6.10 7.65 8.12 8.41 8.87 8.55 9.13 11.36 2048 6.39 6.66 9.58 8.92 10.75 12.99 13.33 12.23 4096 7.33 10.85 8.22 13.57 11.43 10.93 12.53 16.86 8192 7.26 7.46 8.88 11.82 10.55 10.94 13.27 14.11 16348 9.07 8.53 11.82 14.89 12.97 13.22 16.14 18.10 32768 10.45 10.55 11.79 19.19 16.85 17.56 20.58 26.57 65536 11.00 11.12 13.25 18.27 16.18 16.11 19.61 27.73 262144 12.37 12.40 15.65 20.00 19.25 19.38 22.60 31.95 524288 12.44 12.33 15.66 19.78 19.06 18.96 23.31 32.29
Quoted 1 line
Best Regards,
Yan, Zi
Zi Yan
reviewerQuoted 143 lines · Zi Yan
<snip>
Quoted 120 lines
Quoted 103 lines
Quoted 103 lines
Quoted 80 lines
main(){
...// code snippet to measure throughput clock_gettime(CLOCK_MONOTONIC, &t1); retcode = move_pages(getpid(), num_pages, pages, nodesArray , statusArray, MPOL_MF_MOVE); clock_gettime(CLOCK_MONOTONIC, &t2); // tput = num_pages*PAGE_SIZE/(t2-t1)...
}Measurements:
============
vanilla: base kernel without patchset
mt:0 = MT kernel with use_mt_copy=0
mt:1..mt:32 = MT kernel with use_mt_copy=1 and thread cnt = 1,2,...,32Measured for both configuration push_0_pull_1=0 and push_0_pull_1=1 and for 4KB migration and THP migration.
--------------------
#1push_0_pull_1= 0 (src node CPUs are used)#1.1 THP=Never, 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.28 1.92 1.80 2.24 2.35 2.22 2.17 4096 2.40 2.40 2.51 2.58 2.83 2.72 2.99 3.25 8192 3.18 2.88 2.83 2.69 3.49 3.46 3.57 3.80 16348 3.17 2.94 2.96 3.17 3.63 3.68 4.06 4.15#1.2 THP=Always, 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.02 3.39 3.40 3.33 3.51 3.91 4.03 1024 7.13 4.49 3.58 3.56 3.91 3.87 4.39 4.57 2048 5.26 6.47 3.91 4.00 3.71 3.85 4.97 6.83 4096 9.93 7.77 4.58 3.79 3.93 3.53 6.41 4.77 8192 6.47 6.33 4.37 4.67 4.52 4.39 5.30 5.37 16348 7.66 8.00 5.20 5.22 5.24 5.28 6.41 7.02 32768 8.56 8.62 6.34 6.20 6.20 6.19 7.18 8.10 65536 9.41 9.40 7.14 7.15 7.15 7.19 7.96 8.89 262144 10.17 10.19 7.26 7.90 7.98 8.05 9.46 10.30 524288 10.40 9.95 7.25 7.93 8.02 8.76 9.55 10.30--------------------
#2push_0_pull_1= 1 (dst node CPUs are used):#2.1 THP=Never 4KB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.28 1.36 2.01 2.74 2.33 2.31 2.53 2.96 4096 2.40 2.84 2.94 3.04 3.40 3.23 3.31 4.16 8192 3.18 3.27 3.34 3.94 3.77 3.68 4.23 4.76 16348 3.17 3.42 3.66 3.21 3.82 4.40 4.76 4.89#2.2 THP=Always 2MB (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 4.31 5.91 4.03 3.73 4.26 4.13 4.78 3.44 1024 7.13 6.83 4.60 5.13 5.03 5.19 5.94 7.25 2048 5.26 7.09 5.20 5.69 5.83 5.73 6.85 8.13 4096 9.93 9.31 4.90 4.82 4.82 5.26 8.46 8.52 8192 6.47 7.63 5.66 5.85 5.75 6.14 7.45 8.63 16348 7.66 10.00 6.35 6.54 6.66 6.99 8.18 10.21 32768 8.56 9.78 7.06 7.41 7.76 9.02 9.55 11.92 65536 9.41 10.00 8.19 9.20 9.32 8.68 11.00 13.31 262144 10.17 11.17 9.01 9.96 9.99 10.00 11.70 14.27 524288 10.40 11.38 9.07 9.98 10.01 10.09 11.95 14.48Note:
1. For THP = Never: I'm doing for 16X pages to keep total size same for your experiment with 64KB pagesize)
2. For THP = Always:nr_pages= Number of 4KB pages moved. nr_pages=512 => 512 4KB pages => 1 2MB page)I'm seeing little (1.5X in some cases) to no benefits. The performance scaling is relatively flat across thread counts.
Quoted 8 lines
Is it possible I'm missing something in my testing?
Could the base page size difference (4KB vs 64KB) be playing a role in the scaling behavior? How the performance varies with 4KB pages on your system?
I'd be happy to work with you on investigating this differences. Let me know if you'd like any additional test data or if there are specific configurations I should try.
The results surprises me, since I was able to achieve ~9GB/s when migrating 16 2MB THPs with 16 threads on a two socket system with Xeon E5-2650 v3 @ 2.30GHz (a 19.2GB/s bandwidth QPI link between two sockets) back in 2019[1]. These are 10-year-old Haswell CPUs. And your results above show that EPYC 5 can only achieve ~4GB/s when migrating 512 2MB THPs with 16 threads. It just does not make sense.
One thing you might want to try is to set init_on_alloc=0 in your boot parameters to use
folio_zero_user()instead ofGFP_ZEROto zero pages. That might reduce the time spent on page zeros.I am also going to rerun the experiments locally on
x86_64boxes to see if your results can be replicated.Quoted 8 lines
Quoted 5 lines
Thank you for the review and running these experiments. I really appreciate it.>
Using init_on_alloc=0 gave significant performance gain over the last experiment but I'm still missing the performance scaling you observed.
It might be the difference between x86 and ARM64, but I am not 100% sure. Based on your data below, 2 or 4 threads seem to the sweep spot for the multi-threaded method on AMD CPUs. BTW, what is the bandwidth between two sockets in your system? From Figure 10 in [1], I see the InfiniteBand between two AMD EPYC 7601 @ 2.2GHz was measured at ~12GB/s unidirectional, ~25GB/s bidirectional. I wonder if your results below are cross-socket link bandwidth limited.
Quoted 3 lines
From my results, NVIDIA Grace CPU can achieve high copy throughput with more threads between two sockets, maybe part of the reason is that its cross-socket link theoretical bandwidth is 900GB/s bidirectional.
I talked to my colleague about this and he mentioned about CCD architecture on AMD CPUs. IIUC, one or two cores from one CCD can already saturate the CCD’s outgoing bandwidth and all CPUs are enumerated from one CCD to another. This means my naive scheduling algorithm, which use CPUs from 0 to N threads, uses all cores from one CDD first, then move to another CCD. It is not able to saturate the cross-socket bandwidth. Does it make sense to you?
If yes, can you please change the my cpu selection code in
mm/copy_pages.c:(quoted diff) +7
-0+ /* TODO: need a better cpu selection method */ + for_each_cpu(cpu, per_node_cpumask) { + if (i >= total_mt_num) + break; + cpu_id_list[i] = cpu; + ++i; + }
Quoted 27 lines
to select CPUs from as many CCDs as possible and rerun the tests. That might boost the page migration throughput on AMD CPUs more.
Thanks.
Quoted 22 lines
Quoted 9 lines
THP Never
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.40 1.43 2.79 3.48 3.63 3.73 3.63 3.57 4096 2.54 3.32 3.18 4.65 4.83 5.11 5.39 5.78 8192 3.35 4.40 4.39 4.71 3.63 5.04 5.33 6.00 16348 3.76 4.50 4.44 5.33 5.41 5.41 6.47 6.41THP Always
Quoted 11 lines
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 5.21 5.47 5.77 6.92 3.71 2.75 7.54 7.44 1024 6.10 7.65 8.12 8.41 8.87 8.55 9.13 11.36 2048 6.39 6.66 9.58 8.92 10.75 12.99 13.33 12.23 4096 7.33 10.85 8.22 13.57 11.43 10.93 12.53 16.86 8192 7.26 7.46 8.88 11.82 10.55 10.94 13.27 14.11 16348 9.07 8.53 11.82 14.89 12.97 13.22 16.14 18.10 32768 10.45 10.55 11.79 19.19 16.85 17.56 20.58 26.57 65536 11.00 11.12 13.25 18.27 16.18 16.11 19.61 27.73 262144 12.37 12.40 15.65 20.00 19.25 19.38 22.60 31.95 524288 12.44 12.33 15.66 19.78 19.06 18.96 23.31 32.29
BTW, I rerun the experiments on a two socket Xeon E5-2650 v4 @ 2.20GHz system with pull method. The 4KB is not very impressive, at most 60% more throughput, but 2MB can get ~6.5x of vanilla kernel throughput using 8 or 16 threads.
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |
2MB (GB/s)
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |
Best Regards,
Yan, Zi
On 1/11/2025 1:21 AM, Zi Yan wrote:
<snip>
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
BTW, I notice that you called
dmaengine_get_dma_device()infolios_copy_dma(), which would incur a huge overhead, based on my past experience using DMA engine for page copy. I know it is needed to make sure DMA is still present, but its cost needs to be minimized to make DMA folio copy usable. Otherwise, the 768MB/s DMA copy throughput from your cover letter cannot convince people to use it for page migration, since single CPU can achieve more than that, as you showed in the table below.
Thank you for pointing this.
I'm learning about DMAEngine and will look more into DMA driver part.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Using init_on_alloc=0 gave significant performance gain over the last experiment but I'm still missing the performance scaling you observed.
It might be the difference between x86 and ARM64, but I am not 100% sure. Based on your data below, 2 or 4 threads seem to the sweep spot for the multi-threaded method on AMD CPUs. BTW, what is the bandwidth between two sockets in your system? From Figure 10 in [1], I see the InfiniteBand between two AMD EPYC 7601 @ 2.2GHz was measured at ~12GB/s unidirectional, ~25GB/s bidirectional. I wonder if your results below are cross-socket link bandwidth limited.
I tested the cross-socket bandwidth on my EPYC Zen 5 system and easily getting >10X bandwidth as this. I don't think BW is a issue here.
Quoted 13 lines
Quoted 13 lines
Quoted 4 lines
From my results, NVIDIA Grace CPU can achieve high copy throughput with more threads between two sockets, maybe part of the reason is that its cross-socket link theoretical bandwidth is 900GB/s bidirectional.
I talked to my colleague about this and he mentioned about CCD architecture on AMD CPUs. IIUC, one or two cores from one CCD can already saturate the CCD’s outgoing bandwidth and all CPUs are enumerated from one CCD to another. This means my naive scheduling algorithm, which use CPUs from 0 to N threads, uses all cores from one CDD first, then move to another CCD. It is not able to saturate the cross-socket bandwidth. Does it make sense to you?
Quoted 1 line
Quoted 1 line
If yes, can you please change the my cpu selection code in
mm/copy_pages.c:
This is making sense.
I first tried distributing work threads across different CCDs, which yielded better results.
Also, I switched my system to NPS-2 config (2 Nodes per socket). This was done to eliminate cross-socket connections and variables by focusing on intra-socket page migrations.
Cross-Socket (Node 0 -> Node 2)
THP Always (2 MB pages)
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 262144 12.37 12.52 15.72 24.94 30.40 33.23 34.68 29.67 524288 12.44 12.19 15.70 24.96 32.72 33.40 35.40 29.18
Intra-Socket (Node 0 -> Node 1)
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 262144 12.37 17.10 18.65 26.05 35.56 37.80 33.73 29.29 524288 12.44 16.73 18.87 24.34 35.63 37.49 33.79 29.76
I have temporarily hardcoded the CPU assignments and will work on improving the CPU selection code.
Quoted 57 lines
Quoted 9 lines
(quoted diff) +7
-0+ /* TODO: need a better cpu selection method */ + for_each_cpu(cpu, per_node_cpumask) { + if (i >= total_mt_num) + break; + cpu_id_list[i] = cpu; + ++i; + }Quoted 27 lines
to select CPUs from as many CCDs as possible and rerun the tests. That might boost the page migration throughput on AMD CPUs more.
Thanks.
Quoted 22 lines
Quoted 9 lines
THP Never
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.40 1.43 2.79 3.48 3.63 3.73 3.63 3.57 4096 2.54 3.32 3.18 4.65 4.83 5.11 5.39 5.78 8192 3.35 4.40 4.39 4.71 3.63 5.04 5.33 6.00 16348 3.76 4.50 4.44 5.33 5.41 5.41 6.47 6.41THP Always
Quoted 11 lines
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 5.21 5.47 5.77 6.92 3.71 2.75 7.54 7.44 1024 6.10 7.65 8.12 8.41 8.87 8.55 9.13 11.36 2048 6.39 6.66 9.58 8.92 10.75 12.99 13.33 12.23 4096 7.33 10.85 8.22 13.57 11.43 10.93 12.53 16.86 8192 7.26 7.46 8.88 11.82 10.55 10.94 13.27 14.11 16348 9.07 8.53 11.82 14.89 12.97 13.22 16.14 18.10 32768 10.45 10.55 11.79 19.19 16.85 17.56 20.58 26.57 65536 11.00 11.12 13.25 18.27 16.18 16.11 19.61 27.73 262144 12.37 12.40 15.65 20.00 19.25 19.38 22.60 31.95 524288 12.44 12.33 15.66 19.78 19.06 18.96 23.31 32.29BTW, I rerun the experiments on a two socket Xeon E5-2650 v4 @ 2.20GHz system with pull method. The 4KB is not very impressive, at most 60% more throughput, but 2MB can get ~6.5x of vanilla kernel throughput using 8 or 16 threads.
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |2MB (GB/s)
Quoted 16 lines
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |
I see, thank you for checking.
Meanwhile, I'll continue to explore for performance optimization avenues.
Best Regards,
Shivank
Quoted 5 lines
Best Regards,
Yan, Zi
Quoted 142 lines · Shivank Garg
On 1/11/2025 1:21 AM, Zi Yan wrote:
<snip>Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
BTW, I notice that you called
dmaengine_get_dma_device()infolios_copy_dma(), which would incur a huge overhead, based on my past experience using DMA engine for page copy. I know it is needed to make sure DMA is still present, but its cost needs to be minimized to make DMA folio copy usable. Otherwise, the 768MB/s DMA copy throughput from your cover letter cannot convince people to use it for page migration, since single CPU can achieve more than that, as you showed in the table below.Thank you for pointing this.
I'm learning about DMAEngine and will look more into DMA driver part.Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Using init_on_alloc=0 gave significant performance gain over the last experiment but I'm still missing the performance scaling you observed.
It might be the difference between x86 and ARM64, but I am not 100% sure. Based on your data below, 2 or 4 threads seem to the sweep spot for the multi-threaded method on AMD CPUs. BTW, what is the bandwidth between two sockets in your system? From Figure 10 in [1], I see the InfiniteBand between two AMD EPYC 7601 @ 2.2GHz was measured at ~12GB/s unidirectional, ~25GB/s bidirectional. I wonder if your results below are cross-socket link bandwidth limited.
I tested the cross-socket bandwidth on my EPYC Zen 5 system and easily getting >10X bandwidth as this. I don't think BW is a issue here.
Quoted 13 lines
Quoted 13 lines
Quoted 4 lines
From my results, NVIDIA Grace CPU can achieve high copy throughput with more threads between two sockets, maybe part of the reason is that its cross-socket link theoretical bandwidth is 900GB/s bidirectional.
I talked to my colleague about this and he mentioned about CCD architecture on AMD CPUs. IIUC, one or two cores from one CCD can already saturate the CCD’s outgoing bandwidth and all CPUs are enumerated from one CCD to another. This means my naive scheduling algorithm, which use CPUs from 0 to N threads, uses all cores from one CDD first, then move to another CCD. It is not able to saturate the cross-socket bandwidth. Does it make sense to you?
Quoted 1 line
Quoted 1 line
If yes, can you please change the my cpu selection code in
mm/copy_pages.c:This is making sense.
I first tried distributing work threads across different CCDs, which yielded better results.
Also, I switched my system to NPS-2 config (2 Nodes per socket). This was done to eliminate cross-socket connections and variables by focusing on intra-socket page migrations.
Cross-Socket (Node 0 -> Node 2)
THP Always (2 MB pages)nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 262144 12.37 12.52 15.72 24.94 30.40 33.23 34.68 29.67 524288 12.44 12.19 15.70 24.96 32.72 33.40 35.40 29.18Intra-Socket (Node 0 -> Node 1)
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 262144 12.37 17.10 18.65 26.05 35.56 37.80 33.73 29.29 524288 12.44 16.73 18.87 24.34 35.63 37.49 33.79 29.76I have temporarily hardcoded the CPU assignments and will work on improving the CPU selection code.
Quoted 57 lines
Quoted 9 lines
(quoted diff) +7
-0+ /* TODO: need a better cpu selection method */ + for_each_cpu(cpu, per_node_cpumask) { + if (i >= total_mt_num) + break; + cpu_id_list[i] = cpu; + ++i; + }Quoted 27 lines
to select CPUs from as many CCDs as possible and rerun the tests. That might boost the page migration throughput on AMD CPUs more.
Thanks.
Quoted 22 lines
Quoted 9 lines
THP Never
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 1.40 1.43 2.79 3.48 3.63 3.73 3.63 3.57 4096 2.54 3.32 3.18 4.65 4.83 5.11 5.39 5.78 8192 3.35 4.40 4.39 4.71 3.63 5.04 5.33 6.00 16348 3.76 4.50 4.44 5.33 5.41 5.41 6.47 6.41THP Always
Quoted 11 lines
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 512 5.21 5.47 5.77 6.92 3.71 2.75 7.54 7.44 1024 6.10 7.65 8.12 8.41 8.87 8.55 9.13 11.36 2048 6.39 6.66 9.58 8.92 10.75 12.99 13.33 12.23 4096 7.33 10.85 8.22 13.57 11.43 10.93 12.53 16.86 8192 7.26 7.46 8.88 11.82 10.55 10.94 13.27 14.11 16348 9.07 8.53 11.82 14.89 12.97 13.22 16.14 18.10 32768 10.45 10.55 11.79 19.19 16.85 17.56 20.58 26.57 65536 11.00 11.12 13.25 18.27 16.18 16.11 19.61 27.73 262144 12.37 12.40 15.65 20.00 19.25 19.38 22.60 31.95 524288 12.44 12.33 15.66 19.78 19.06 18.96 23.31 32.29BTW, I rerun the experiments on a two socket Xeon E5-2650 v4 @ 2.20GHz system with pull method. The 4KB is not very impressive, at most 60% more throughput, but 2MB can get ~6.5x of vanilla kernel throughput using 8 or 16 threads.
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |2MB (GB/s)
Quoted 16 lines
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |
Quoted 4 lines
I see, thank you for checking.
Meanwhile, I'll continue to explore for performance optimization avenues.
Hi Zi,
I experimented with your testcase[1] to get 2-2.5X throughput gain than my previous experiment. Multi-threading scaling for 32 threads is ~4X (slightly higher than my previous experiment).
The main difference between our move_pages throughput benchmarks was:
You're explicitly using THP using aligned_alloc and MADV_HUGEPAGE whereas I'm relying on system THP operating on 4KB boundaries.
While both methods use THP and expected similar performance, we saw lower throughput in my case because:
In my test code,
The kernel processes all 512 4KB folios within a 2MB region in the first migration attempt. For subsequent folios, __add_folio_for_migration() returns early with folio_nid(folio) == node, as the pages are already on the target node. This adds extra overheads of vma_lookup and folio_walk_start() in my experiment.
2MB pages (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 1 10.74 11.04 4.68 8.17 6.47 6.09 3.97 6.20 2 12.44 4.90 11.19 14.10 15.33 8.45 10.09 9.97 4 14.82 9.80 11.93 18.35 21.82 17.09 10.53 7.51 8 16.13 9.91 15.26 11.85 26.53 13.09 12.71 13.75 16 15.99 8.81 13.84 22.43 33.89 11.91 12.30 13.26 32 14.03 11.37 17.54 23.96 57.07 18.78 19.51 21.29 64 15.79 9.55 22.19 33.17 57.18 65.51 55.39 62.53 128 18.22 16.65 21.49 30.73 52.99 61.05 58.44 60.38 256 19.78 20.56 24.72 34.94 56.73 71.11 61.83 62.77 512 20.27 21.40 27.47 39.23 65.72 67.97 70.48 71.39 1024 20.48 21.48 27.48 38.30 68.62 77.94 78.00 78.95
[1]: https://github.com/x-y-z/thp-migration-bench/blob/arm64/move_thp.c
Quoted 9 lines
Best Regards,
ShivankQuoted 5 lines
Best Regards,
Yan, Zi
Byungchul Park
reviewerQuoted 1 line · Zi Yan
Hi all,
Hi,
It'd be appreciated to cc me from the next.
Byungchul
Quoted 141 lines
This patchset accelerates page migration by batching folio copy operations and using multiple CPU threads and is based on Shivank's Enhancements to Page Migration with Batch Offloading via DMA patchset[1] and my original accelerate page migration patchset[2]. It is on top of mm-everything-2025-01-03-05-59. The last patch is for testing purpose and should not be considered.
The motivations are:
1. Batching folio copy increases copy throughput. Especially for base page migrations, folio copy throughput is low since there are kernel activities like moving folio metadata and updating page table entries sit between two folio copies. And base page sizes are relatively small, 4KB on
x86_64, ARM64 and 64KB on ARM64.2. Single CPU thread has limited copy throughput. Using multi threads is a natural extension to speed up folio copy, when DMA engine is NOT available in a system.
Design
===It is based on Shivank's patchset and revise
MIGRATE_SYNC_NO_COPY(renamed toMIGRATE_NO_COPY) to avoid folio copy operation insidemigrate_folio_move()and perform them in one shot afterwards. Acopy_page_lists_mt()function is added to use multi threads to copy folios from src list to dst list.Changes compared to Shivank's patchset (mainly rewrote batching folio copy code)
===1.
mig_infois removed, so no memory allocation is needed during batching folio copies.src->privateis used to store old page state andanon_vmaafter folio metadata is copied from src to dst.2.
move_to_new_folio()andmigrate_folio_move()are refactored to remove redundant code inmigrate_folios_batch_move().3.
folio_mc_copy()is used for the single threaded copy code to keep the original kernel behavior.Performance
===I benchmarked
move_pages()throughput on a two socket NUMA system with two NVIDIA Grace CPUs. The base page size is 64KB. Both 64KB page migration and 2MB mTHP page migration are measured.The tables below show
move_pages()throughput with different configurations and different numbers of copied pages. The x-axis is the configurations, from vanilla Linux kernel to using 1, 2, 4, 8, 16, 32 threads with this patchset applied. And the unit is GB/s.The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_3232 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_321 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84TODOs
===
1. Multi-threaded folio copy routine needs to look at CPU scheduler and only use idle CPUs to avoid interfering userspace workloads. Of course more complicated policies can be used based on migration issuing thread priority.2. Eliminate memory allocation during multi-threaded folio copy routine if possible.
3. A runtime check to decide when use multi-threaded folio copy. Something like cache hotness issue mentioned by Matthew[3].
4. Use non-temporal CPU instructions to avoid cache pollution issues.
5. Explicitly make multi-threaded folio copy only available to !HIGHMEM, since
kmap_local_page()would be needed for each kernel folio copy work threads and expensive.6. A better interface than
copy_page_lists_mt()to allow DMA data copy to be used as well.Let me know your thoughts. Thanks.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Byungchul Park (1):
mm: separate move/undo doing on folio list frommigrate_pages_batch()Zi Yan (4):
mm/migrate: factor out code inmove_to_new_folio()and
migrate_folio_move()
mm/migrate: addmigrate_folios_batch_moveto batch the folio move
operations
mm/migrate: introduce multi-threaded page copy routine
test: add sysctl for folio copy tests and adjust
NR_MAX_BATCHED_MIGRATIONinclude/linux/migrate.h | 3 + include/linux/migrate_mode.h | 2 + include/linux/mm.h | 4 + include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++- mm/Makefile | 2 +- mm/copy_pages.c | 190 +++++++++++++++ mm/migrate.c | 443 +++++++++++++++++++++++++++-------- 8 files changed, 577 insertions(+), 97 deletions(-) create mode 100644 mm/copy_pages.cSignature · 1 line
2.45.2
Zi Yan
reviewerQuoted 6 lines · Byungchul Park
Quoted 1 line · Zi Yan
Hi all,
Hi,
It'd be appreciated to cc me from the next.
Sure.
Best Regards,
Yan, Zi