mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
Subsystems (2)
RFC PATCH v2 1/1 mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(), so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs with ERMS) plus a cond_resched().
Introduce copy_mc_highpages(), the page range variant of copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages are in the linear map, copy the whole range with a single copy_mc_to_kernel() call, so the CPU can optimize for the larger extent. Otherwise, fall back to copy_mc_highpage() one page at a time.
On a machine check, queue the poisoned source page for memory_failure() and return -EHWPOISON, as copy_mc_highpage() does.
copy_mc_to_kernel() cannot reschedule, so copying a large folio with one call can run for a long time under cooperative preemption models. Add COPY_MC_PAGES_NON_PREEMPT_BATCH to bound how much is copied in one go:
32MB, the same bound clear_pages() gets from PROCESS_PAGES_NON_PREEMPT_BATCH, or one page on the fallback path.
Switch folio_mc_copy() to copy_mc_highpages(): the whole folio at once under preemptible models, COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time otherwise.
---
On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below). When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
Changes in v2:
- Drop the folio_copy() changes as there is little to gain here. On architectures without copy_mc_to_kernel(), copy_mc_highpages() falls back to the per-page copy_mc_highpage(), which ends up in copy_highpage(). A batched copy_highpages() could help there too, but I haven't tested it on those architectures, so that is left for a follow-up.
- Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied between cond_resched() calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/[email protected]
Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are DRAM, node 2 is CXL), with and without this patch.
move_pages() throughput
-----------------------
A single move_pages() call migrates 1GB of anonymous memory, as 512 2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each), in GiB/s:
2MB THP:
direction base patched speedup DRAM0 -> DRAM1 19.57 ± 0.17 27.51 ± 0.78 1.41x DRAM1 -> DRAM0 19.44 ± 0.12 27.33 ± 0.27 1.41x DRAM0 -> CXL 18.66 ± 0.18 26.90 ± 0.51 1.44x CXL -> DRAM0 15.38 ± 0.59 16.60 ± 0.24 1.08x
1GB hugetlb:
direction base patched speedup DRAM0 -> DRAM1 20.34 ± 0.13 53.88 ± 1.48 2.65x DRAM1 -> DRAM0 19.10 ± 2.65 53.78 ± 0.55 2.82x DRAM0 -> CXL 19.53 ± 0.08 44.10 ± 0.19 2.26x CXL -> DRAM0 16.41 ± 0.07 20.89 ± 0.06 1.27x
kswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A single thread bound to node 0 writes 96GB of anonymous THP memory, so ~80GB has to be demoted to CXL. Median of 10 runs:
base patched kswapd CPU time 4.25 s 3.44 s (-19%) kswapd CPU per GB demoted 63.6 ms 44.1 ms (-31%) demoted by direct reclaim 14.1 GB 0.46 GB allocstall 7427 370
With the patch, kswapd keeps up with the writer, so the writer rarely falls back to direct reclaim.
Other workloads
---------------
- Memory tiering (numa_balancing=2) with Redis, GAPBS PR/BC, a B-tree benchmark and Graph500: copying 2MB folios during demotion is 1.18-1.49x faster.
- A userspace daemon moving 1GB hugetlb pages between DRAM and CXL with move_pages() while GUPS runs on them: demotion is 1.43x faster and promotion 1.28x faster.
--- include/linux/highmem.h | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ mm/util.c | 18 +++++++++++---- 2 files changed, 73 insertions(+), 4 deletions(-)
include/linux/highmem.h +59 -0
@@ -484,6 +484,65 @@ static inline int copy_mc_highpage(struct page *to, struct page *from) }#endif +#if defined(copy_mc_to_kernel) && !defined(CONFIG_HIGHMEM ) && \ + !defined(__HAVE_ARCH_COPY_HIGHPAGE) +/* +* copy_mc_highpages() copies the whole range with a single +* copy_mc_to_kernel() call, which cannot reschedule. That is fine under +* preemptible models, but under cooperatively scheduled models callers +* should copy at most COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time. +* +* With a value of 32MB and assuming a memory bandwidth of ~10GBps, this +* should result in a worst case preemption latency of around 3ms when +* copying pages. +*/ +#define COPY_MC_PAGES_NON_PREEMPT_BATCH (SZ_32M >> PAGE_SHIFT) + +/** +* copy_mc_highpages() - copy a page range with machine check handling +* @to: first destination page +* @from: first source page +* @nr_pages: number of pages +* +* Range variant of copy_mc_highpage(). On a machine check, the poisoned +* source page is queued for memory_failure(). +* +* Does not call cond_resched(). Under cooperatively scheduled models the +* caller is expected to limit @nr_pages to COPY_MC_PAGES_NON_PREEMPT_BATCH. +* +* Return: 0 on success, -EHWPOISON if a machine check was handled. +*/ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long len = nr_pages <<PAGE_SHIFT ; +unsigned long ret, i; + + ret =copy_mc_to_kernel (page_address (to),page_address (from), len); +if (!ret) { +for (i =0 ; i < nr_pages; i++) +kmsan_copy_page_meta (to + i, from + i); +return 0 ; + } +memory_failure_queue (page_to_pfn (from) + ((len - ret) >>PAGE_SHIFT ),0 ); +return -EHWPOISON ; +} +#else + +#define COPY_MC_PAGES_NON_PREEMPT_BATCH 1 + +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long i; + +for (i =0 ; i < nr_pages; i++) +if (copy_mc_highpage (to + i, from + i)) +return -EHWPOISON ; +return 0 ; +} +#endif +static inline void memcpy_page (struct page *dst_page,size_t dst_off,struct page *src_page,size_t src_off,size_t len)
mm/util.c +14 -4
@@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);int folio_mc_copy (struct folio *dst,struct folio *src) { -long nr =folio_nr_pages (src); -long i =0 ; +const unsigned long nr_pages =folio_nr_pages (src); +/* +* Copy the largest extent possible to allow for architecture specific +* extent based optimizations. copy_mc_highpages() does not call +* cond_resched(), so limit the unit size when running under +* non-preemptible scheduling models. +*/ +const unsigned long unit =preempt_model_preemptible () ? + nr_pages :COPY_MC_PAGES_NON_PREEMPT_BATCH ; +unsigned long i =0 , count;for (;;) { -if (copy_mc_highpage (folio_page (dst, i),folio_page (src, i))) + count =min (unit, nr_pages - i); +if (copy_mc_highpages (folio_page (dst, i),folio_page (src, i), count))return -EHWPOISON ; -if (++i == nr) + i += count; +if (i == nr_pages)break ;cond_resched (); }
---
base-commit: 2887e63d2e8e0cf55402612d8996ce955a2c3e55
change-id: 20260714-batpagecopy-5a2b45910e5d
Best regards,
Signature · 1 line
Shivank Garg <[email protected]>
Quoted 11 lines · Shivank Garg
folio_mc_copy()copies a folio one page at a time withcopy_mc_highpage(), so a 2MB folio makes 512 separatecopy_mc_to_kernel()calls. Each page pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs with ERMS) plus acond_resched().Introduce
copy_mc_highpages(), the page range variant ofcopy_mc_highpage(). Whencopy_mc_to_kernel()is available and the pages are in the linear map, copy the whole range with a singlecopy_mc_to_kernel()call, so the CPU can optimize for the larger extent. Otherwise, fall back tocopy_mc_highpage()one page at a time.
Quoted 5 lines
On a machine check, queue the poisoned source page for
memory_failure()and return-EHWPOISON, ascopy_mc_highpage()does.
copy_mc_to_kernel()cannot reschedule, so copying a large folio with one call can run for a long time under cooperative preemption models. Add
Do we care, though? I think most architectures are full preempt/preempt lazy at this point? Is PREEMPT_DYNAMIC the only showstopper?
Quoted 88 lines
COPY_MC_PAGES_NON_PREEMPT_BATCHto bound how much is copied in one go:
32MB, the same boundclear_pages()gets fromPROCESS_PAGES_NON_PREEMPT_BATCH, or one page on the fallback path.Switch
folio_mc_copy()tocopy_mc_highpages(): the whole folio at once under preemptible models,COPY_MC_PAGES_NON_PREEMPT_BATCHpages at a time otherwise.Assisted-by LLMSigned-off-by Shivank Garg <[email protected]>---
On a Zen 6 system with CXL memory,move_pages()gets ~1.4x faster for 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below). When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.Changes in v2:
- Drop thefolio_copy()changes as there is little to gain here. On architectures withoutcopy_mc_to_kernel(),copy_mc_highpages()falls back to the per-pagecopy_mc_highpage(), which ends up incopy_highpage(). A batchedcopy_highpages()could help there too, but I haven't tested it on those architectures, so that is left for a follow-up.
- AddCOPY_MC_PAGES_NON_PREEMPT_BATCHto limit how much is copied betweencond_resched()calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/[email protected]Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are DRAM, node 2 is CXL), with and without this patch.
move_pages()throughput
-----------------------
A singlemove_pages()call migrates 1GB of anonymous memory, as 512 2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each), in GiB/s:2MB THP:
direction base patched speedup DRAM0 -> DRAM1 19.57 ± 0.17 27.51 ± 0.78 1.41x DRAM1 -> DRAM0 19.44 ± 0.12 27.33 ± 0.27 1.41x DRAM0 -> CXL 18.66 ± 0.18 26.90 ± 0.51 1.44x CXL -> DRAM0 15.38 ± 0.59 16.60 ± 0.24 1.08x1GB hugetlb:
direction base patched speedup DRAM0 -> DRAM1 20.34 ± 0.13 53.88 ± 1.48 2.65x DRAM1 -> DRAM0 19.10 ± 2.65 53.78 ± 0.55 2.82x DRAM0 -> CXL 19.53 ± 0.08 44.10 ± 0.19 2.26x CXL -> DRAM0 16.41 ± 0.07 20.89 ± 0.06 1.27xkswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A single thread bound to node 0 writes 96GB of anonymous THP memory, so ~80GB has to be demoted to CXL. Median of 10 runs:base patched kswapd CPU time 4.25 s 3.44 s (-19%) kswapd CPU per GB demoted 63.6 ms 44.1 ms (-31%) demoted by direct reclaim 14.1 GB 0.46 GB allocstall 7427 370With the patch, kswapd keeps up with the writer, so the writer rarely falls back to direct reclaim.
Other workloads
---------------
- Memory tiering (numa_balancing=2) with Redis, GAPBS PR/BC, a B-tree benchmark and Graph500: copying 2MB folios during demotion is 1.18-1.49x faster.
- A userspace daemon moving 1GB hugetlb pages between DRAM and CXL withmove_pages()while GUPS runs on them: demotion is 1.43x faster and promotion 1.28x faster.--- include/linux/highmem.h | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ mm/util.c | 18 +++++++++++---- 2 files changed, 73 insertions(+), 4 deletions(-)include/linux/highmem.h +2
-0@@ -484,6 +484,65 @@ static inline int copy_mc_highpage(struct page *to, struct page *from) }#endif +#if defined(copy_mc_to_kernel) && !defined(CONFIG_HIGHMEM ) && \ + !defined(__HAVE_ARCH_COPY_HIGHPAGE)
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 498
Quoted 10 lines
include/linux/highmem.h +10
-0+/* +* copy_mc_highpages() copies the whole range with a single +* copy_mc_to_kernel() call, which cannot reschedule. That is fine under +* preemptible models, but under cooperatively scheduled models callers +* should copy at most COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time. +* +* With a value of 32MB and assuming a memory bandwidth of ~10GBps, this +* should result in a worst case preemption latency of around 3ms when +* copying pages. +*/
This worst-case is absolutely terrible :)
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 511
Quoted 13 lines
include/linux/highmem.h +13
-0+#define COPY_MC_PAGES_NON_PREEMPT_BATCH (SZ_32M >> PAGE_SHIFT) + +/** +* copy_mc_highpages() - copy a page range with machine check handling +* @to: first destination page +* @from: first source page +* @nr_pages: number of pages +* +* Range variant of copy_mc_highpage(). On a machine check, the poisoned +* source page is queued for memory_failure(). +* +* Does not call cond_resched(). Under cooperatively scheduled models the +* caller is expected to limit @nr_pages to COPY_MC_PAGES_NON_PREEMPT_BATCH.
This is very backwards, callers shouldn't need to check this.
Quoted 7 lines
include/linux/highmem.h +7
-0+* +* Return: 0 on success, -EHWPOISON if a machine check was handled. +*/ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long len = nr_pages <<PAGE_SHIFT ;
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 528
Quoted 10 lines
include/linux/highmem.h +10
-0+unsigned long ret, i; + + ret =copy_mc_to_kernel (page_address (to),page_address (from), len); +if (!ret) { +for (i =0 ; i < nr_pages; i++) +kmsan_copy_page_meta (to + i, from + i); +return 0 ; + } +memory_failure_queue (page_to_pfn (from) + ((len - ret) >>PAGE_SHIFT ),0 ); +return -EHWPOISON ;
These codepaths look strangely inverted, it would be more natural if you did:
if (ret) {
memory_failure_queue(page_to_pfn(from) + ((len - ret) >> PAGE_SHIFT), 0);
return -EWPOISON;
}
for (i = 0; i < nr_pages; i++)
kmsan_copy_page_meta(to + i, from + i);
/* Or, ideally, kmsan_copy_pages_meta(to, from, nr_pages); */
return 0;
}in my opinion.
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 532
Quoted 4 lines
include/linux/highmem.h +4
-0+} +#else + +#define COPY_MC_PAGES_NON_PREEMPT_BATCH 1
I would suggest that perhaps this should be the fallback for non-preemptible models. But it gets a little weird with PREEMPT_DYNAMIC I think?
Quoted 25 lines
include/linux/highmem.h +13
-0+ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long i; + +for (i =0 ; i < nr_pages; i++) +if (copy_mc_highpage (to + i, from + i)) +return -EHWPOISON ; +return 0 ; +} +#endif +static inline void memcpy_page (struct page *dst_page,size_t dst_off,struct page *src_page,size_t src_off,size_t len)mm/util.c +0
-1@@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);int folio_mc_copy (struct folio *dst,struct folio *src) { -long nr =folio_nr_pages (src);
mm/util.c EXPORT_SYMBOL(folio_copy); · line 774
Quoted 10 lines
mm/util.c +9
-1-long i =0 ; +const unsigned long nr_pages =folio_nr_pages (src); +/* +* Copy the largest extent possible to allow for architecture specific +* extent based optimizations. copy_mc_highpages() does not call +* cond_resched(), so limit the unit size when running under +* non-preemptible scheduling models. +*/ +const unsigned long unit =preempt_model_preemptible () ? + nr_pages :COPY_MC_PAGES_NON_PREEMPT_BATCH ;
See? now mm is aware of preemption models! eww!
Signature · 1 line
Pedro
David Hildenbrand (Arm)
maintainerQuoted 7 lines
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 532
Quoted 4 lines
include/linux/highmem.h +4
-0+} +#else + +#define COPY_MC_PAGES_NON_PREEMPT_BATCH 1I would suggest that perhaps this should be the fallback for non-preemptible models. But it gets a little weird with
PREEMPT_DYNAMICI think?
Why can't we just use PROCESS_PAGES_NON_PREEMPT_BATCH where we have similar handling already?
Quoted 23 lines
Quoted 23 lines
include/linux/highmem.h +13
-0+ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long i; + +for (i =0 ; i < nr_pages; i++) +if (copy_mc_highpage (to + i, from + i)) +return -EHWPOISON ; +return 0 ; +} +#endif +static inline void memcpy_page (struct page *dst_page,size_t dst_off,struct page *src_page,size_t src_off,size_t len)mm/util.c +0
-0@@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);
Quoted 15 lines
mm/util.c EXPORT_SYMBOL(folio_copy); · line 774
Quoted 13 lines
mm/util.c +9
-2int folio_mc_copy (struct folio *dst,struct folio *src) { -long nr =folio_nr_pages (src); -long i =0 ; +const unsigned long nr_pages =folio_nr_pages (src); +/* +* Copy the largest extent possible to allow for architecture specific +* extent based optimizations. copy_mc_highpages() does not call +* cond_resched(), so limit the unit size when running under +* non-preemptible scheduling models. +*/ +const unsigned long unit =preempt_model_preemptible () ? + nr_pages :COPY_MC_PAGES_NON_PREEMPT_BATCH ;See? now mm is aware of preemption models! eww!
Yeah, we have that in clear_contig_highpages() already, which is getting replicated here for copying instead of zeroing.
Signature · 3 lines
Cheers, David
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 532
Quoted 4 lines
include/linux/highmem.h +4
-0+} +#else + +#define COPY_MC_PAGES_NON_PREEMPT_BATCH 1I would suggest that perhaps this should be the fallback for non-preemptible models. But it gets a little weird with
PREEMPT_DYNAMICI think?
Quoted 4 lines
Why can't we just use
PROCESS_PAGES_NON_PREEMPT_BATCHwhere we have similar handling already?Quoted 1 line
I did not reuse PROCESS_PAGES_NON_PREEMPT_BATCH because it depends on whether clear_pages is defined. While COPY_MC_PAGES_NON_PREEMPT_BATCH depends on copy_mc_to_kernel and other constraints. So, using a single param for both would make it more messy.
However as Pedro pointed, most architectures are full preempt/preempt lazy, and with this commit 7dadeaa6e851 ("sched: Further restrict the preemption modes"), for PREEMPT_DYNAMIC, preempt=none and preempt=voluntary are rejected at boot. preempt_model_preemptible() is therefore always true for this case.
So, this can be simplfied to drop the COPY_MC_PAGES_NON_PREEMPT_BATCH.
Quoted 41 lines
Quoted 23 lines
Quoted 23 lines
include/linux/highmem.h +13
-0+ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long i; + +for (i =0 ; i < nr_pages; i++) +if (copy_mc_highpage (to + i, from + i)) +return -EHWPOISON ; +return 0 ; +} +#endif +static inline void
memcpy_page(struct page *dst_page, size_t dst_off, struct page *src_page, size_t src_off, size_t len)mm/util.c +0
-0@@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);Quoted 15 lines
mm/util.c EXPORT_SYMBOL(folio_copy); · line 774
Quoted 13 lines
int
folio_mc_copy(struct folio *dst, struct folio *src){mm/util.c +9
-2-long nr =folio_nr_pages (src); -long i =0 ; +const unsigned long nr_pages =folio_nr_pages (src); +/* +* Copy the largest extent possible to allow for architecture specific +* extent based optimizations. copy_mc_highpages() does not call +* cond_resched(), so limit the unit size when running under +* non-preemptible scheduling models. +*/ +const unsigned long unit =preempt_model_preemptible () ? + nr_pages :COPY_MC_PAGES_NON_PREEMPT_BATCH ;See? now mm is aware of preemption models! eww!
Yeah, we have that in
clear_contig_highpages()already, which is getting replicated here for copying instead of zeroing.
David Hildenbrand (Arm)
maintainerQuoted 10 lines · Shivank
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 3 lines
I would suggest that perhaps this should be the fallback for non-preemptible models. But it gets a little weird with
PREEMPT_DYNAMICI think?Why can't we just use
PROCESS_PAGES_NON_PREEMPT_BATCHwhere we have similar handling already?Quoted 1 line
Quoted 4 lines
I did not reuse
PROCESS_PAGES_NON_PREEMPT_BATCHbecause it depends on whetherclear_pagesis defined. WhileCOPY_MC_PAGES_NON_PREEMPT_BATCHdepends oncopy_mc_to_kerneland other constraints. So, using a single param for both would make it more messy.
If it would stay it would all have to be cleaned up to end up with a single one.
Quoted 8 lines
However as Pedro pointed, most architectures are full preempt/preempt lazy, and with this commit
7dadeaa6e851("sched: Further restrict the preemption modes"), forPREEMPT_DYNAMIC, preempt=none and preempt=voluntary are rejected at boot.preempt_model_preemptible()is therefore always true for this case.So, this can be simplfied to drop the
COPY_MC_PAGES_NON_PREEMPT_BATCH.
Can we simplify the zeroing part as well?
Signature · 3 lines
Cheers, David
Quoted 11 lines · David Hildenbrand (Arm)
Quoted 10 lines · Shivank
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 3 lines
I would suggest that perhaps this should be the fallback for non-preemptible models. But it gets a little weird with
PREEMPT_DYNAMICI think?Why can't we just use
PROCESS_PAGES_NON_PREEMPT_BATCHwhere we have similar handling already?Quoted 1 line
Quoted 6 lines
Quoted 4 lines
I did not reuse
PROCESS_PAGES_NON_PREEMPT_BATCHbecause it depends on whetherclear_pagesis defined. WhileCOPY_MC_PAGES_NON_PREEMPT_BATCHdepends oncopy_mc_to_kerneland other constraints. So, using a single param for both would make it more messy.If it would stay it would all have to be cleaned up to end up with a single one.
Agreed.
Quoted 11 lines
Quoted 9 lines
However as Pedro pointed, most architectures are full preempt/preempt lazy, and with this commit
7dadeaa6e851("sched: Further restrict the preemption modes"), forPREEMPT_DYNAMIC, preempt=none and preempt=voluntary are rejected at boot.preempt_model_preemptible()is therefore always true for this case.So, this can be simplfied to drop the
COPY_MC_PAGES_NON_PREEMPT_BATCH.Can we simplify the zeroing part as well?
Yes, I think so that it can be simplified similarly. I'll add a RFC patch on next version.
Thanks,
Shivank
Quoted 12 lines · Pedro Falcato
Quoted 11 lines · Shivank Garg
folio_mc_copy()copies a folio one page at a time withcopy_mc_highpage(), so a 2MB folio makes 512 separatecopy_mc_to_kernel()calls. Each page pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs with ERMS) plus acond_resched().Introduce
copy_mc_highpages(), the page range variant ofcopy_mc_highpage(). Whencopy_mc_to_kernel()is available and the pages are in the linear map, copy the whole range with a singlecopy_mc_to_kernel()call, so the CPU can optimize for the larger extent. Otherwise, fall back tocopy_mc_highpage()one page at a time.
Quoted 8 lines
Quoted 5 lines
On a machine check, queue the poisoned source page for
memory_failure()and return-EHWPOISON, ascopy_mc_highpage()does.
copy_mc_to_kernel()cannot reschedule, so copying a large folio with one call can run for a long time under cooperative preemption models. AddDo we care, though? I think most architectures are full preempt/preempt lazy at this point? Is
PREEMPT_DYNAMICthe only showstopper?
yes, and Since 7dadeaa6e851 ("sched: Further restrict the preemption modes"),PREEMPT_DYNAMIC is not problem either.
Quoted 138 lines
Quoted 89 lines
COPY_MC_PAGES_NON_PREEMPT_BATCHto bound how much is copied in one go:
32MB, the same boundclear_pages()gets fromPROCESS_PAGES_NON_PREEMPT_BATCH, or one page on the fallback path.Switch
folio_mc_copy()tocopy_mc_highpages(): the whole folio at once under preemptible models,COPY_MC_PAGES_NON_PREEMPT_BATCHpages at a time otherwise.Assisted-by LLMSigned-off-by Shivank Garg <[email protected]>---
On a Zen 6 system with CXL memory,move_pages()gets ~1.4x faster for 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below). When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.Changes in v2:
- Drop thefolio_copy()changes as there is little to gain here. On architectures withoutcopy_mc_to_kernel(),copy_mc_highpages()falls back to the per-pagecopy_mc_highpage(), which ends up incopy_highpage(). A batchedcopy_highpages()could help there too, but I haven't tested it on those architectures, so that is left for a follow-up.
- AddCOPY_MC_PAGES_NON_PREEMPT_BATCHto limit how much is copied betweencond_resched()calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/[email protected]Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are DRAM, node 2 is CXL), with and without this patch.
move_pages()throughput
-----------------------
A singlemove_pages()call migrates 1GB of anonymous memory, as 512 2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each), in GiB/s:2MB THP:
direction base patched speedup DRAM0 -> DRAM1 19.57 ± 0.17 27.51 ± 0.78 1.41x DRAM1 -> DRAM0 19.44 ± 0.12 27.33 ± 0.27 1.41x DRAM0 -> CXL 18.66 ± 0.18 26.90 ± 0.51 1.44x CXL -> DRAM0 15.38 ± 0.59 16.60 ± 0.24 1.08x1GB hugetlb:
direction base patched speedup DRAM0 -> DRAM1 20.34 ± 0.13 53.88 ± 1.48 2.65x DRAM1 -> DRAM0 19.10 ± 2.65 53.78 ± 0.55 2.82x DRAM0 -> CXL 19.53 ± 0.08 44.10 ± 0.19 2.26x CXL -> DRAM0 16.41 ± 0.07 20.89 ± 0.06 1.27xkswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A single thread bound to node 0 writes 96GB of anonymous THP memory, so ~80GB has to be demoted to CXL. Median of 10 runs:base patched kswapd CPU time 4.25 s 3.44 s (-19%) kswapd CPU per GB demoted 63.6 ms 44.1 ms (-31%) demoted by direct reclaim 14.1 GB 0.46 GB allocstall 7427 370With the patch, kswapd keeps up with the writer, so the writer rarely falls back to direct reclaim.
Other workloads
---------------
- Memory tiering (numa_balancing=2) with Redis, GAPBS PR/BC, a B-tree benchmark and Graph500: copying 2MB folios during demotion is 1.18-1.49x faster.
- A userspace daemon moving 1GB hugetlb pages between DRAM and CXL withmove_pages()while GUPS runs on them: demotion is 1.43x faster and promotion 1.28x faster.--- include/linux/highmem.h | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ mm/util.c | 18 +++++++++++---- 2 files changed, 73 insertions(+), 4 deletions(-)include/linux/highmem.h +2
-0@@ -484,6 +484,65 @@ static inline int copy_mc_highpage(struct page *to, struct page *from) }#endif +#if defined(copy_mc_to_kernel) && !defined(CONFIG_HIGHMEM ) && \ + !defined(__HAVE_ARCH_COPY_HIGHPAGE)include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 498
Quoted 10 lines
include/linux/highmem.h +10
-0+/* +* copy_mc_highpages() copies the whole range with a single +* copy_mc_to_kernel() call, which cannot reschedule. That is fine under +* preemptible models, but under cooperatively scheduled models callers +* should copy at most COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time. +* +* With a value of 32MB and assuming a memory bandwidth of ~10GBps, this +* should result in a worst case preemption latency of around 3ms when +* copying pages. +*/ This worst-case is absolutely terrible :)
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 511
Quoted 13 lines
include/linux/highmem.h +13
-0+#define COPY_MC_PAGES_NON_PREEMPT_BATCH (SZ_32M >> PAGE_SHIFT) + +/** +* copy_mc_highpages() - copy a page range with machine check handling +* @to: first destination page +* @from: first source page +* @nr_pages: number of pages +* +* Range variant of copy_mc_highpage(). On a machine check, the poisoned +* source page is queued for memory_failure(). +* +* Does not call cond_resched(). Under cooperatively scheduled models the +* caller is expected to limit @nr_pages to COPY_MC_PAGES_NON_PREEMPT_BATCH. This is very backwards, callers shouldn't need to check this.
Quoted 7 lines
include/linux/highmem.h +7
-0+* +* Return: 0 on success, -EHWPOISON if a machine check was handled. +*/ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long len = nr_pages <<PAGE_SHIFT ;include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 528
Quoted 10 lines
include/linux/highmem.h +10
-0+unsigned long ret, i; + + ret =copy_mc_to_kernel (page_address (to),page_address (from), len); +if (!ret) { +for (i =0 ; i < nr_pages; i++) +kmsan_copy_page_meta (to + i, from + i); +return 0 ; + } +memory_failure_queue (page_to_pfn (from) + ((len - ret) >>PAGE_SHIFT ),0 ); +return -EHWPOISON ;These codepaths look strangely inverted, it would be more natural if you did:
Quoted 6 lines
if (ret) {
memory_failure_queue(page_to_pfn(from) + ((len - ret) >> PAGE_SHIFT), 0);
return -EWPOISON;
}
for (i = 0; i < nr_pages; i++)
kmsan_copy_page_meta(to + i, from + i);Will do this way.
Quoted 1 line
/* Or, ideally, kmsan_copy_pages_meta(to, from, nr_pages); */
There is no kmsan_copy_pages_meta, I prefer to keep the loop instead of introducing new wrapper for this single instance.
Quoted 53 lines
return 0; }in my opinion.
include/linux/highmem.h static inline int copy_mc_highpage(struct page *to, struct page *from) · line 532
Quoted 4 lines
include/linux/highmem.h +4
-0+} +#else + +#define COPY_MC_PAGES_NON_PREEMPT_BATCH 1I would suggest that perhaps this should be the fallback for non-preemptible models. But it gets a little weird with
PREEMPT_DYNAMICI think?Quoted 25 lines
include/linux/highmem.h +13
-0+ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long i; + +for (i =0 ; i < nr_pages; i++) +if (copy_mc_highpage (to + i, from + i)) +return -EHWPOISON ; +return 0 ; +} +#endif +static inline void memcpy_page (struct page *dst_page,size_t dst_off,struct page *src_page,size_t src_off,size_t len)mm/util.c +0
-1@@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);int folio_mc_copy (struct folio *dst,struct folio *src) { -long nr =folio_nr_pages (src);mm/util.c EXPORT_SYMBOL(folio_copy); · line 774
Quoted 10 lines
mm/util.c +9
-1-long i =0 ; +const unsigned long nr_pages =folio_nr_pages (src); +/* +* Copy the largest extent possible to allow for architecture specific +* extent based optimizations. copy_mc_highpages() does not call +* cond_resched(), so limit the unit size when running under +* non-preemptible scheduling models. +*/ +const unsigned long unit =preempt_model_preemptible () ? + nr_pages :COPY_MC_PAGES_NON_PREEMPT_BATCH ;See? now mm is aware of preemption models! eww!
--
Pedro
Quoted 4 lines · On Mon, 19 Oct 2026 10:06:49 +0000 Shivank Garg <[email protected]> wrote:
folio_mc_copy()copies a folio one page at a time withcopy_mc_highpage(), so a 2MB folio makes 512 separatecopy_mc_to_kernel()calls. Each page pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs with ERMS) plus acond_resched().
Don't you always want to use 'rep movsq' for these aligned copies (on any cpu anyone cares about).
I'm pretty even on some that don't advertise ERMS.
The fastest a copy loop is ever going to run (on anything but the latest cpu) is one write every clock - so 8 bytes/clock.
A 4k page will then take at least 512 clocks.
Even an old sandy bridge cpu will (IIRC) copy 32 bytes/clock if the destination is 32 byte aligned (not much later cpu double that for 64byte aligned destinations). So the copy part takes at most 128 clocks.
That makes 'rep movsq' faster unless the fixed overhead is in the P4 netburst range.
(I was using lfence when I did my performance measurements, it hides the fixed latency of rep.)
I've never measured zen 1-4.
I also suspect the slow part of a 2M copy is all the D-cache misses (especially on the read side).
OTOH the longer copy is likely to be better...
David
Quoted 198 lines
Introduce
copy_mc_highpages(), the page range variant ofcopy_mc_highpage(). Whencopy_mc_to_kernel()is available and the pages are in the linear map, copy the whole range with a singlecopy_mc_to_kernel()call, so the CPU can optimize for the larger extent. Otherwise, fall back tocopy_mc_highpage()one page at a time.On a machine check, queue the poisoned source page for
memory_failure()and return-EHWPOISON, ascopy_mc_highpage()does.
copy_mc_to_kernel()cannot reschedule, so copying a large folio with one call can run for a long time under cooperative preemption models. AddCOPY_MC_PAGES_NON_PREEMPT_BATCHto bound how much is copied in one go:
32MB, the same boundclear_pages()gets fromPROCESS_PAGES_NON_PREEMPT_BATCH, or one page on the fallback path.Switch
folio_mc_copy()tocopy_mc_highpages(): the whole folio at once under preemptible models,COPY_MC_PAGES_NON_PREEMPT_BATCHpages at a time otherwise.Assisted-by LLMSigned-off-by Shivank Garg <[email protected]>---
On a Zen 6 system with CXL memory,move_pages()gets ~1.4x faster for 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below). When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.Changes in v2:
- Drop thefolio_copy()changes as there is little to gain here. On architectures withoutcopy_mc_to_kernel(),copy_mc_highpages()falls back to the per-pagecopy_mc_highpage(), which ends up incopy_highpage(). A batchedcopy_highpages()could help there too, but I haven't tested it on those architectures, so that is left for a follow-up.
- AddCOPY_MC_PAGES_NON_PREEMPT_BATCHto limit how much is copied betweencond_resched()calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/[email protected]Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are DRAM, node 2 is CXL), with and without this patch.
move_pages()throughput
-----------------------
A singlemove_pages()call migrates 1GB of anonymous memory, as 512 2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each), in GiB/s:2MB THP:
direction base patched speedup DRAM0 -> DRAM1 19.57 ± 0.17 27.51 ± 0.78 1.41x DRAM1 -> DRAM0 19.44 ± 0.12 27.33 ± 0.27 1.41x DRAM0 -> CXL 18.66 ± 0.18 26.90 ± 0.51 1.44x CXL -> DRAM0 15.38 ± 0.59 16.60 ± 0.24 1.08x1GB hugetlb:
direction base patched speedup DRAM0 -> DRAM1 20.34 ± 0.13 53.88 ± 1.48 2.65x DRAM1 -> DRAM0 19.10 ± 2.65 53.78 ± 0.55 2.82x DRAM0 -> CXL 19.53 ± 0.08 44.10 ± 0.19 2.26x CXL -> DRAM0 16.41 ± 0.07 20.89 ± 0.06 1.27xkswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A single thread bound to node 0 writes 96GB of anonymous THP memory, so ~80GB has to be demoted to CXL. Median of 10 runs:base patched kswapd CPU time 4.25 s 3.44 s (-19%) kswapd CPU per GB demoted 63.6 ms 44.1 ms (-31%) demoted by direct reclaim 14.1 GB 0.46 GB allocstall 7427 370With the patch, kswapd keeps up with the writer, so the writer rarely falls back to direct reclaim.
Other workloads
---------------
- Memory tiering (numa_balancing=2) with Redis, GAPBS PR/BC, a B-tree benchmark and Graph500: copying 2MB folios during demotion is 1.18-1.49x faster.
- A userspace daemon moving 1GB hugetlb pages between DRAM and CXL withmove_pages()while GUPS runs on them: demotion is 1.43x faster and promotion 1.28x faster.--- include/linux/highmem.h | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ mm/util.c | 18 +++++++++++---- 2 files changed, 73 insertions(+), 4 deletions(-)include/linux/highmem.h +59
-0@@ -484,6 +484,65 @@ static inline int copy_mc_highpage(struct page *to, struct page *from) }#endif +#if defined(copy_mc_to_kernel) && !defined(CONFIG_HIGHMEM ) && \ + !defined(__HAVE_ARCH_COPY_HIGHPAGE) +/* +* copy_mc_highpages() copies the whole range with a single +* copy_mc_to_kernel() call, which cannot reschedule. That is fine under +* preemptible models, but under cooperatively scheduled models callers +* should copy at most COPY_MC_PAGES_NON_PREEMPT_BATCH pages at a time. +* +* With a value of 32MB and assuming a memory bandwidth of ~10GBps, this +* should result in a worst case preemption latency of around 3ms when +* copying pages. +*/ +#define COPY_MC_PAGES_NON_PREEMPT_BATCH (SZ_32M >> PAGE_SHIFT) + +/** +* copy_mc_highpages() - copy a page range with machine check handling +* @to: first destination page +* @from: first source page +* @nr_pages: number of pages +* +* Range variant of copy_mc_highpage(). On a machine check, the poisoned +* source page is queued for memory_failure(). +* +* Does not call cond_resched(). Under cooperatively scheduled models the +* caller is expected to limit @nr_pages to COPY_MC_PAGES_NON_PREEMPT_BATCH. +* +* Return: 0 on success, -EHWPOISON if a machine check was handled. +*/ +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long len = nr_pages <<PAGE_SHIFT ; +unsigned long ret, i; + + ret =copy_mc_to_kernel (page_address (to),page_address (from), len); +if (!ret) { +for (i =0 ; i < nr_pages; i++) +kmsan_copy_page_meta (to + i, from + i); +return 0 ; + } +memory_failure_queue (page_to_pfn (from) + ((len - ret) >>PAGE_SHIFT ),0 ); +return -EHWPOISON ; +} +#else + +#define COPY_MC_PAGES_NON_PREEMPT_BATCH 1 + +static inline int copy_mc_highpages (struct page *to,struct page *from, +unsigned long nr_pages) +{ +unsigned long i; + +for (i =0 ; i < nr_pages; i++) +if (copy_mc_highpage (to + i, from + i)) +return -EHWPOISON ; +return 0 ; +} +#endif +static inline void memcpy_page (struct page *dst_page,size_t dst_off,struct page *src_page,size_t src_off,size_t len)mm/util.c +14
-5@@ -763,13 +763,23 @@ EXPORT_SYMBOL(folio_copy);int folio_mc_copy (struct folio *dst,struct folio *src) { -long nr =folio_nr_pages (src); -long i =0 ; +const unsigned long nr_pages =folio_nr_pages (src); +/* +* Copy the largest extent possible to allow for architecture specific +* extent based optimizations. copy_mc_highpages() does not call +* cond_resched(), so limit the unit size when running under +* non-preemptible scheduling models. +*/ +const unsigned long unit =preempt_model_preemptible () ? + nr_pages :COPY_MC_PAGES_NON_PREEMPT_BATCH ; +unsigned long i =0 , count;for (;;) { -if (copy_mc_highpage (folio_page (dst, i),folio_page (src, i))) + count =min (unit, nr_pages - i); +if (copy_mc_highpages (folio_page (dst, i),folio_page (src, i), count))return -EHWPOISON ; -if (++i == nr) + i += count; +if (i == nr_pages)break ;cond_resched (); } ---base-commit:
2887e63d2e8e0cf55402612d8996ce955a2c3e55
change-id: 20260714-batpagecopy-5a2b45910e5dBest regards,
Quoted 23 lines · David Laight
Quoted 4 lines · On Mon, 19 Oct 2026 10:06:49 +0000 Shivank Garg <[email protected]> wrote:
folio_mc_copy()copies a folio one page at a time withcopy_mc_highpage(), so a 2MB folio makes 512 separatecopy_mc_to_kernel()calls. Each page pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs with ERMS) plus acond_resched().Don't you always want to use 'rep movsq' for these aligned copies (on any cpu anyone cares about).
I'm pretty even on some that don't advertise ERMS.The fastest a copy loop is ever going to run (on anything but the latest cpu) is one write every clock - so 8 bytes/clock.
A 4k page will then take at least 512 clocks.
Even an old sandy bridge cpu will (IIRC) copy 32 bytes/clock if the destination is 32 byte aligned (not much later cpu double that for 64byte aligned destinations). So the copy part takes at most 128 clocks.
That makes 'rep movsq' faster unless the fixed overhead is in the P4 netburst range.
(I was using lfence when I did my performance measurements, it hides the fixed latency of rep.)
Quoted 7 lines
I've never measured zen 1-4.
I also suspect the slow part of a 2M copy is all the D-cache misses (especially on the read side).
OTOH the longer copy is likely to be better...
Thanks for taking a look and for the detailed explanation.
In my measurements, 'rep movsb' and 'rep movsq' perform about the same for aligned, longer copies.
Since copy_mc_to_kernel() takes an arbitrary length, always using 'rep movsq' would mean adding an alignment/length, etc. or adding a separate copy_mc_pages() (or something similar) for page-aligned copies.
I think that would be worthwhile x86 change on its own, independent of this series. Here, I'm not changing the underlying copy mechanism and behavior other callers.
Would it be okay to have hardware specific changes separately?
Thanks,
Shivank
Quoted 29 lines · Shivank
Quoted 28 lines · David Laight
Quoted 4 lines · On Mon, 19 Oct 2026 10:06:49 +0000 Shivank Garg <[email protected]> wrote:
folio_mc_copy()copies a folio one page at a time withcopy_mc_highpage(), so a 2MB folio makes 512 separatecopy_mc_to_kernel()calls. Each page pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs with ERMS) plus acond_resched().Don't you always want to use 'rep movsq' for these aligned copies (on any cpu anyone cares about).
I'm pretty even on some that don't advertise ERMS.The fastest a copy loop is ever going to run (on anything but the latest cpu) is one write every clock - so 8 bytes/clock.
A 4k page will then take at least 512 clocks.
Even an old sandy bridge cpu will (IIRC) copy 32 bytes/clock if the destination is 32 byte aligned (not much later cpu double that for 64byte aligned destinations). So the copy part takes at most 128 clocks.
That makes 'rep movsq' faster unless the fixed overhead is in the P4 netburst range.
(I was using lfence when I did my performance measurements, it hides the fixed latency of rep.)I've never measured zen 1-4.
I also suspect the slow part of a 2M copy is all the D-cache misses (especially on the read side).
Quoted 7 lines
Quoted 2 lines
OTOH the longer copy is likely to be better...
Thanks for taking a look and for the detailed explanation.
In my measurements, 'rep movsb' and 'rep movsq' perform about the same for aligned, longer copies.
On Intel cpu it makes absolutely no difference which one you use. But some of the older zen do handle them differently.
Quoted 4 lines
Since
copy_mc_to_kernel()takes an arbitrary length, always using 'rep movsq' would mean adding an alignment/length, etc. or adding a separatecopy_mc_pages()(or something similar) for page-aligned copies.
If you know you are copying aligned buffers it really is better to have a copy routine that can rely on that fact.
I suspect it is only x86 'rep movsb' implementations that don't really care. Pretty much all the others are going to have code to check the alignment.
David
Quoted 13 lines
I think that would be worthwhile x86 change on its own, independent of this series. Here, I'm not changing the underlying copy mechanism and behavior other callers.
Would it be okay to have hardware specific changes separately?
Thanks,
Shivank