Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]
By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilization
c. Using Non-temporal instructions for CPU-based memcpy
d. Upscaling/downscaling worker threads based on migration size, CPU
availability (system load), bandwidth saturation, etc.
2. Interface requirements with DMA hardware:
a. Standardizing APIs for DMA drivers and support for different DMA
drivers
b. Enhancing DMA drivers for bulk copying (e.g., SDXi Engine)
3. Resources Accounting:
a. CPU cgroups accounting and fairness [3]
b. Who bears migration cost? - (Migration cost attribution)
References:
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/CAHbLzkpoKP0fVZP5b10wdzAMDLWysDy7oH0qaUssiUXj80R6bw@mail.gmail.com
Best Regards,
Shivank
Quoted 5 lines · Shivank Garg
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
I think this would be a very useful topic to discuss, thanks for proposing it.
Quoted 11 lines
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Indeed, there are multiple use cases for optimizations in this area. With the ramp of memory tiered systems, I think there will be an even greater reliance on memory migration going forward.
Do you have numbers to share on how offloading, even as a proof of concept, moves the needle compared to traditional and sequential memory migration?
Quoted 9 lines
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]
Curious: does memory migration of pages that are actively undergoing DMA with hardware assist fit into any of these?
Quoted 9 lines
By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilization
Why platform specific? I *assume* this means a generic framework that can optimize for scheduling based on the underlying hardware and not specific implementations that can only be used on AMD, for example. Is that the case?
Quoted 16 lines
c. Using Non-temporal instructions for CPU-based memcpy d. Upscaling/downscaling worker threads based on migration size, CPU
availability (system load), bandwidth saturation, etc.
2. Interface requirements with DMA hardware:
a. Standardizing APIs for DMA drivers and support for different DMA
drivers
b. Enhancing DMA drivers for bulk copying (e.g., SDXi Engine)
3. Resources Accounting:
a. CPU cgroups accounting and fairness [3]
b. Who bears migration cost? - (Migration cost attribution)References:
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/CAHbLzkpoKP0fVZP5b10wdzAMDLWysDy7oH0qaUssiUXj80R6bw@mail.gmail.com
Quoted 23 lines · David Rientjes
Quoted 5 lines · Shivank Garg
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
I think this would be a very useful topic to discuss, thanks for proposing it.
Quoted 11 lines
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Quoted 7 lines
Indeed, there are multiple use cases for optimizations in this area. With the ramp of memory tiered systems, I think there will be an even greater reliance on memory migration going forward.
Do you have numbers to share on how offloading, even as a proof of concept, moves the needle compared to traditional and sequential memory migration?
For multithreaded page migration, you can see my RFC patchset[1]:
on NVIDIA Grace:
The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32
32 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.80
2MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_32
1 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84
I also ran it on on a two socket Xeon E5-2650 v4:
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |
2MB (GB/s)
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |
Shivank ran it on AMD EPYC Zen 5, after some tuning (spread threads on different CCDs):
2MB pages (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 1 10.74 11.04 4.68 8.17 6.47 6.09 3.97 6.20 2 12.44 4.90 11.19 14.10 15.33 8.45 10.09 9.97 4 14.82 9.80 11.93 18.35 21.82 17.09 10.53 7.51 8 16.13 9.91 15.26 11.85 26.53 13.09 12.71 13.75 16 15.99 8.81 13.84 22.43 33.89 11.91 12.30 13.26 32 14.03 11.37 17.54 23.96 57.07 18.78 19.51 21.29 64 15.79 9.55 22.19 33.17 57.18 65.51 55.39 62.53 128 18.22 16.65 21.49 30.73 52.99 61.05 58.44 60.38 256 19.78 20.56 24.72 34.94 56.73 71.11 61.83 62.77 512 20.27 21.40 27.47 39.23 65.72 67.97 70.48 71.39 1024 20.48 21.48 27.48 38.30 68.62 77.94 78.00 78.95
Quoted 11 lines
Quoted 10 lines
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]
Quoted 2 lines
Curious: does memory migration of pages that are actively undergoing DMA with hardware assist fit into any of these?
It should be similar to 3, but in this case, DMA is used to copy pages between NUMA nodes, whereas traditional DMA page migration is used to copy pages between host and devices.
Quoted 11 lines
Quoted 10 lines
By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilization
Quoted 4 lines
Why platform specific? I *assume* this means a generic framework that can optimize for scheduling based on the underlying hardware and not specific implementations that can only be used on AMD, for example. Is that the case?
I think the framework will be generic but the CPU scheduling (which core to choose for page copying) will be different from vendor to vendor.
Due to existing CPU structure, like chiplet design, a single CPU scheduling algorithm does not fit for CPUs from different vendors. For example, on NVIDIA Grace, you can use any CPUs to copy pages and always achieve high page copy throughput, but on AMD CPUs with multiple CCDs, spreading copy threads across different CCDs can achieve much higher page copy throughput than putting all threads in a single CCD. I assume Intel CPUs with chiplet design would see the same result.
Quoted 12 lines
Quoted 12 lines
c. Using Non-temporal instructions for CPU-based memcpy d. Upscaling/downscaling worker threads based on migration size, CPU
availability (system load), bandwidth saturation, etc.
2. Interface requirements with DMA hardware:
a. Standardizing APIs for DMA drivers and support for different DMA
drivers
b. Enhancing DMA drivers for bulk copying (e.g., SDXi Engine)
3. Resources Accounting:
a. CPU cgroups accounting and fairness [3]
b. Who bears migration cost? - (Migration cost attribution)
Quoted 5 lines
[1] https://lore.kernel.org/all/[email protected]/
--
Best Regards,
Yan, Zi
Quoted 18 lines · On Mon, 27 Jan 2025 07:37:19 -0500 Zi Yan <[email protected]> wrote:
Quoted 16 lines · David Rientjes
Quoted 5 lines · Shivank Garg
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
I think this would be a very useful topic to discuss, thanks for proposing it.
Quoted 5 lines
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Hi,
With the potential usecase of tiered memory (CXL) migration mentioned above, I'm curious what application scenario is such that we are willing to burn lots of CPU cores at once on that migration? I'm very interested in the DMA offload aspect, but multithreading for that usecase seems like a less good fit as presumably there is something running that is seeing the poor effects of memory latency that is making the move look like a good idea?
Or are we looking at some sort of demotion when the system is idle?
Quoted 134 lines
Quoted 9 lines
Quoted 1 line
Indeed, there are multiple use cases for optimizations in this area. With the ramp of memory tiered systems, I think there will be an even greater reliance on memory migration going forward.
Do you have numbers to share on how offloading, even as a proof of concept, moves the needle compared to traditional and sequential memory migration?
For multithreaded page migration, you can see my RFC patchset[1]:
on NVIDIA Grace:
The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_3232 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_321 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84I also ran it on on a two socket Xeon E5-2650 v4:
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |2MB (GB/s)
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |Shivank ran it on AMD EPYC Zen 5, after some tuning (spread threads on different CCDs):
2MB pages (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 1 10.74 11.04 4.68 8.17 6.47 6.09 3.97 6.20 2 12.44 4.90 11.19 14.10 15.33 8.45 10.09 9.97 4 14.82 9.80 11.93 18.35 21.82 17.09 10.53 7.51 8 16.13 9.91 15.26 11.85 26.53 13.09 12.71 13.75 16 15.99 8.81 13.84 22.43 33.89 11.91 12.30 13.26 32 14.03 11.37 17.54 23.96 57.07 18.78 19.51 21.29 64 15.79 9.55 22.19 33.17 57.18 65.51 55.39 62.53 128 18.22 16.65 21.49 30.73 52.99 61.05 58.44 60.38 256 19.78 20.56 24.72 34.94 56.73 71.11 61.83 62.77 512 20.27 21.40 27.47 39.23 65.72 67.97 70.48 71.39 1024 20.48 21.48 27.48 38.30 68.62 77.94 78.00 78.95Quoted 11 lines
Quoted 10 lines
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]Quoted 2 lines
Curious: does memory migration of pages that are actively undergoing DMA with hardware assist fit into any of these?
It should be similar to 3, but in this case, DMA is used to copy pages between NUMA nodes, whereas traditional DMA page migration is used to copy pages between host and devices.
Quoted 11 lines
Quoted 10 lines
By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilizationQuoted 4 lines
Why platform specific? I *assume* this means a generic framework that can optimize for scheduling based on the underlying hardware and not specific implementations that can only be used on AMD, for example. Is that the case?
I think the framework will be generic but the CPU scheduling (which core to choose for page copying) will be different from vendor to vendor.
Quoted 7 lines
Due to existing CPU structure, like chiplet design, a single CPU scheduling algorithm does not fit for CPUs from different vendors. For example, on NVIDIA Grace, you can use any CPUs to copy pages and always achieve high page copy throughput, but on AMD CPUs with multiple CCDs, spreading copy threads across different CCDs can achieve much higher page copy throughput than putting all threads in a single CCD. I assume Intel CPUs with chiplet design would see the same result.
On this I'd hope we could build something topology aware enough to make the right decisions. All the information should be available to do this without having per uarch specific code.
Jonathan
Quoted 29 lines · Jonathan Cameron
Quoted 18 lines · On Mon, 27 Jan 2025 07:37:19 -0500 Zi Yan <[email protected]> wrote:
Quoted 16 lines · David Rientjes
Quoted 5 lines · Shivank Garg
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
I think this would be a very useful topic to discuss, thanks for proposing it.
Quoted 5 lines
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Hi,
Quoted 7 lines
With the potential usecase of tiered memory (CXL) migration mentioned above, I'm curious what application scenario is such that we are willing to burn lots of CPU cores at once on that migration? I'm very interested in the DMA offload aspect, but multithreading for that usecase seems like a less good fit as presumably there is something running that is seeing the poor effects of memory latency that is making the move look like a good idea?
There are two scenarios: 1) all CPUs are busy and 2) some CPUs are idle.
For the first one, it is going to be a trade off between a) spending CPU cycles on workloads but the workload performance is limited by the long memory latency and low memory bandwidth when accessing data on remote nodes and b) spending CPU cycles on moving hot data much quicker from remote to local to make workloads run quicker. Without multithreading, we are still “burning” one CPU core to migrate data slowly from remote to local; with multithreading, we use more CPU resources to move data quicker to make workloads run faster. Admittedly, not all workloads would benefit from it, but if a workload sees significant performance boost when accessing data from local memory, using more CPU resources to move it over should justify the use.
Quoted 2 lines
Or are we looking at some sort of demotion when the system is idle?
For second scenario, a lot of GPU/accelerator intensive workloads would spend most of its time in GPU/accelerator, leaving CPU idle. It is going to be a great use of these idle CPUs to shuffle hot and cold data to the right place to make sure GPU/accelerator can always access hot data.
Quoted 143 lines
Quoted 135 lines
Quoted 10 lines
Quoted 2 lines
Indeed, there are multiple use cases for optimizations in this area. With the ramp of memory tiered systems, I think there will be an even greater reliance on memory migration going forward.
Do you have numbers to share on how offloading, even as a proof of concept, moves the needle compared to traditional and sequential memory migration?
For multithreaded page migration, you can see my RFC patchset[1]:
on NVIDIA Grace:
The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_3232 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_321 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84I also ran it on on a two socket Xeon E5-2650 v4:
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |2MB (GB/s)
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |Shivank ran it on AMD EPYC Zen 5, after some tuning (spread threads on different CCDs):
2MB pages (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 1 10.74 11.04 4.68 8.17 6.47 6.09 3.97 6.20 2 12.44 4.90 11.19 14.10 15.33 8.45 10.09 9.97 4 14.82 9.80 11.93 18.35 21.82 17.09 10.53 7.51 8 16.13 9.91 15.26 11.85 26.53 13.09 12.71 13.75 16 15.99 8.81 13.84 22.43 33.89 11.91 12.30 13.26 32 14.03 11.37 17.54 23.96 57.07 18.78 19.51 21.29 64 15.79 9.55 22.19 33.17 57.18 65.51 55.39 62.53 128 18.22 16.65 21.49 30.73 52.99 61.05 58.44 60.38 256 19.78 20.56 24.72 34.94 56.73 71.11 61.83 62.77 512 20.27 21.40 27.47 39.23 65.72 67.97 70.48 71.39 1024 20.48 21.48 27.48 38.30 68.62 77.94 78.00 78.95Quoted 11 lines
Quoted 10 lines
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]Quoted 2 lines
Curious: does memory migration of pages that are actively undergoing DMA with hardware assist fit into any of these?
It should be similar to 3, but in this case, DMA is used to copy pages between NUMA nodes, whereas traditional DMA page migration is used to copy pages between host and devices.
Quoted 11 lines
Quoted 10 lines
By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilizationQuoted 4 lines
Why platform specific? I *assume* this means a generic framework that can optimize for scheduling based on the underlying hardware and not specific implementations that can only be used on AMD, for example. Is that the case?
I think the framework will be generic but the CPU scheduling (which core to choose for page copying) will be different from vendor to vendor.
Quoted 7 lines
Due to existing CPU structure, like chiplet design, a single CPU scheduling algorithm does not fit for CPUs from different vendors. For example, on NVIDIA Grace, you can use any CPUs to copy pages and always achieve high page copy throughput, but on AMD CPUs with multiple CCDs, spreading copy threads across different CCDs can achieve much higher page copy throughput than putting all threads in a single CCD. I assume Intel CPUs with chiplet design would see the same result.
Quoted 3 lines
On this I'd hope we could build something topology aware enough to make the right decisions. All the information should be available to do this without having per uarch specific code.
I agree. Currently, I am using workqueue to copy data, but workqueue is not aware of CPU topology or CPU idleness. I think either workqueue needs to be enhanced or something else, which has better CPU resource management and topology awareness, should be used to achieve the optimal multithreading page migration results.
Best Regards,
Yan, Zi
Hi David, Zi,
Quoted 4 lines · Zi Yan
Quoted 2 lines · David Rientjes
On Thu, 23 Jan 2025, Shivank Garg wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
I think this would be a very useful topic to discuss, thanks for proposing it.
Thanks for your interest in our proposal.
Quoted 119 lines
Quoted 13 lines
Quoted 6 lines
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Quoted 6 lines
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Quoted 7 lines
Indeed, there are multiple use cases for optimizations in this area. With the ramp of memory tiered systems, I think there will be an even greater reliance on memory migration going forward.
Do you have numbers to share on how offloading, even as a proof of concept, moves the needle compared to traditional and sequential memory migration?
For multithreaded page migration, you can see my RFC patchset[1]:
on NVIDIA Grace:
The 32-thread copy throughput can be up to 10x of single thread serial folio copy. Batching folio copy not only benefits huge page but also base page.
64KB (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_3232 5.43 4.90 5.65 7.31 7.60 8.61 6.43
256 6.95 6.89 9.28 14.67 22.41 23.39 23.93
512 7.88 7.26 10.15 17.53 27.82 27.88 33.93
768 7.65 7.42 10.46 18.59 28.65 29.67 30.76
1024 7.46 8.01 10.90 17.77 27.04 32.18 38.802MB mTHP (GB/s):
vanilla mt_1 mt_2 mt_4 mt_8 mt_16 mt_321 5.94 2.90 6.90 8.56 11.16 8.76 6.41
2 7.67 5.57 7.11 12.48 17.37 15.68 14.10
4 8.01 6.04 10.25 20.14 22.52 27.79 25.28
8 8.42 7.00 11.41 24.73 33.96 32.62 39.55
16 9.41 6.91 12.23 27.51 43.95 49.15 51.38
32 10.23 7.15 13.03 29.52 49.49 69.98 71.51
64 9.40 7.37 13.88 30.38 52.00 76.89 79.41
128 8.59 7.23 14.20 28.39 49.98 78.27 90.18
256 8.43 7.16 14.59 28.14 48.78 76.88 92.28
512 8.31 7.78 14.40 26.20 43.31 63.91 75.21
768 8.30 7.86 14.83 27.41 46.25 69.85 81.31
1024 8.31 7.90 14.96 27.62 46.75 71.76 83.84I also ran it on on a two socket Xeon E5-2650 v4:
4KB (GB/s)
| ---- | ------- | ---- | ---- | ---- | ---- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ---- | ---- | ----- | | 512 | 1.12 | 1.19 | 1.20 | 1.26 | 1.27 | 1.35 | | 768 | 1.29 | 1.14 | 1.28 | 1.40 | 1.39 | 1.46 | | 1024 | 1.19 | 1.25 | 1.34 | 1.51 | 1.52 | 1.53 | | 2048 | 1.14 | 1.12 | 1.44 | 1.61 | 1.73 | 1.71 | | 4096 | 1.09 | 1.14 | 1.46 | 1.64 | 1.81 | 1.78 |2MB (GB/s)
| ---- | ------- | ---- | ---- | ----- | ----- | ----- | | | vanilla | mt_1 | mt_2 | mt_4 | mt_8 | mt_16 | | ---- | ------- | ---- | ---- | ----- | ----- | ----- | | 1 | 2.03 | 2.21 | 2.69 | 2.93 | 3.17 | 3.14 | | 2 | 2.28 | 2.13 | 3.54 | 4.50 | 4.72 | 4.72 | | 4 | 2.92 | 2.93 | 4.44 | 6.50 | 7.24 | 7.06 | | 8 | 2.29 | 2.37 | 3.21 | 6.86 | 8.83 | 8.44 | | 16 | 2.10 | 2.09 | 4.57 | 8.06 | 8.32 | 9.70 | | 32 | 2.22 | 2.21 | 4.43 | 8.96 | 9.37 | 11.54 | | 64 | 2.35 | 2.35 | 3.15 | 7.77 | 10.77 | 13.61 | | 128 | 2.48 | 2.53 | 5.12 | 8.18 | 11.01 | 15.62 | | 256 | 2.55 | 2.53 | 5.44 | 8.25 | 12.73 | 16.49 | | 512 | 2.61 | 2.52 | 5.73 | 11.26 | 17.18 | 16.97 | | 768 | 2.55 | 2.53 | 5.90 | 11.41 | 14.86 | 17.15 | | 1024 | 2.56 | 2.52 | 5.99 | 11.46 | 16.77 | 17.25 |Shivank ran it on AMD EPYC Zen 5, after some tuning (spread threads on different CCDs):
2MB pages (GB/s):
nr_pages vanilla mt:0 mt:1 mt:2 mt:4 mt:8 mt:16 mt:32 1 10.74 11.04 4.68 8.17 6.47 6.09 3.97 6.20 2 12.44 4.90 11.19 14.10 15.33 8.45 10.09 9.97 4 14.82 9.80 11.93 18.35 21.82 17.09 10.53 7.51 8 16.13 9.91 15.26 11.85 26.53 13.09 12.71 13.75 16 15.99 8.81 13.84 22.43 33.89 11.91 12.30 13.26 32 14.03 11.37 17.54 23.96 57.07 18.78 19.51 21.29 64 15.79 9.55 22.19 33.17 57.18 65.51 55.39 62.53 128 18.22 16.65 21.49 30.73 52.99 61.05 58.44 60.38 256 19.78 20.56 24.72 34.94 56.73 71.11 61.83 62.77 512 20.27 21.40 27.47 39.23 65.72 67.97 70.48 71.39 1024 20.48 21.48 27.48 38.30 68.62 77.94 78.00 78.95Quoted 11 lines
Quoted 10 lines
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]
Quoted 7 lines
Quoted 2 lines
Curious: does memory migration of pages that are actively undergoing DMA with hardware assist fit into any of these?
It should be similar to 3, but in this case, DMA is used to copy pages between NUMA nodes, whereas traditional DMA page migration is used to copy pages between host and devices.
I'm planning to test using SDXi as the DMA engine for offload and it doesn't support migrating pages that are actively undergoing DMA AFAIU.
Quoted 19 lines
Quoted 11 lines
Quoted 10 lines
By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilizationQuoted 4 lines
Why platform specific? I *assume* this means a generic framework that can optimize for scheduling based on the underlying hardware and not specific implementations that can only be used on AMD, for example. Is that the case?
I think the framework will be generic but the CPU scheduling (which core to choose for page copying) will be different from vendor to vendor.
Quoted 7 lines
Due to existing CPU structure, like chiplet design, a single CPU scheduling algorithm does not fit for CPUs from different vendors. For example, on NVIDIA Grace, you can use any CPUs to copy pages and always achieve high page copy throughput, but on AMD CPUs with multiple CCDs, spreading copy threads across different CCDs can achieve much higher page copy throughput than putting all threads in a single CCD. I assume Intel CPUs with chiplet design would see the same result.
Thank you Zi for helping with results and queries.
Quoted 24 lines
Quoted 13 lines
Quoted 13 lines
c. Using Non-temporal instructions for CPU-based memcpy d. Upscaling/downscaling worker threads based on migration size, CPU
availability (system load), bandwidth saturation, etc.
2. Interface requirements with DMA hardware:
a. Standardizing APIs for DMA drivers and support for different DMA
drivers
b. Enhancing DMA drivers for bulk copying (e.g., SDXi Engine)
3. Resources Accounting:
a. CPU cgroups accounting and fairness [3]
b. Who bears migration cost? - (Migration cost attribution)Quoted 5 lines
[1] https://lore.kernel.org/all/[email protected]/
--
Best Regards,
Yan, Zi
Quoted 29 lines · Shivank Garg
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Quoted 16 lines
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilization
c. Using Non-temporal instructions for CPU-based memcpy
d. Upscaling/downscaling worker threads based on migration size, CPU
availability (system load), bandwidth saturation, etc.
2. Interface requirements with DMA hardware:
a. Standardizing APIs for DMA drivers and support for different DMA
drivers
b. Enhancing DMA drivers for bulk copying (e.g., SDXi Engine)
3. Resources Accounting:
a. CPU cgroups accounting and fairness [3]
b. Who bears migration cost? - (Migration cost attribution)
Hi all,
For reference, here is the link to the latest RFC v2:
https://lore.kernel.org/linux-mm/[email protected]
This version combines the ideas discussed in [1] and [2] and includes details on performance improvements and experimental findings to provide more context for discussion.
Quoted 4 lines
References:
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/CAHbLzkpoKP0fVZP5b10wdzAMDLWysDy7oH0qaUssiUXj80R6bw@mail.gmail.com
Looking forward to your feedback!
Thanks,
Shivank
Quoted 51 lines · Shivank Garg
Quoted 29 lines · Shivank Garg
Hi all,
Zi Yan and I would like to propose the topic: Enhancements to Page Migration with Multi-threading and Batch Offloading to DMA.
Page migration is a critical operation in NUMA systems that can incur significant overheads, affecting memory management performance across various workloads. For example, copying folios between DRAM NUMA nodes can take ~25% of the total migration cost for migrating 256MB of data.
Modern systems are equipped with powerful DMA engines for bulk data copying, GPUs, and high CPU core counts. Leveraging these hardware capabilities becomes essential for systems where frequent page promotion and demotion occur - from large-scale tiered-memory systems with CXL nodes to CPU-GPU coherent system with GPU memory exposed as NUMA nodes.
Existing page migration performs sequential page copying, underutilizing modern CPU architectures and high-bandwidth memory subsystems.
We have proposed and posted RFCs to enhance page migration through three key techniques:
1. Batching migration operations for bulk copying data [1]
2. Multi-threaded folio copying [2]
3. DMA offloading to hardware accelerators [1]By employing batching and multi-threaded folio copying, we are able to achieve significant improvements in page migration throughput for large pages.
Quoted 16 lines
Discussion points:
1. Performance:
a. Policy decision for DMA and CPU selection
b. Platform-specific scheduling of folio-copy worker threads for better
bandwidth utilization
c. Using Non-temporal instructions for CPU-based memcpy
d. Upscaling/downscaling worker threads based on migration size, CPU
availability (system load), bandwidth saturation, etc.
2. Interface requirements with DMA hardware:
a. Standardizing APIs for DMA drivers and support for different DMA
drivers
b. Enhancing DMA drivers for bulk copying (e.g., SDXi Engine)
3. Resources Accounting:
a. CPU cgroups accounting and fairness [3]
b. Who bears migration cost? - (Migration cost attribution)Hi all,
Quoted 7 lines
For reference, here is the link to the latest RFC v2:
https://lore.kernel.org/linux-mm/[email protected]
This version combines the ideas discussed in [1] and [2] and includes details on performance improvements and experimental findings to provide more context for discussion.
Sharing the slides from today’s presentation:
Main Slide Deck: https://docs.google.com/presentation/d/1mjl5-jiz-TMVRK9bQcQ_IsSXrIP82CqWS8Q6em3mJi0/edit?usp=sharing
Multi-threading Slide Deck: https://docs.google.com/presentation/d/10czypcUbRMOUn6knp340Cwv4bf83Ha2gUX8TwNXUwCs/edit#slide=id.p6
Thanks,
Shivank
Quoted 11 lines
Quoted 5 lines
References:
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/CAHbLzkpoKP0fVZP5b10wdzAMDLWysDy7oH0qaUssiUXj80R6bw@mail.gmail.comLooking forward to your feedback!
Thanks,
Shivank