mm: limit THP alignment – performance gain observed in AI inference workloads
PATCH 1/1 mm: limit THP alignment – performance gain observed in AI inference workloads
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload, which was significantly impacted by the THP alignment logic introduced in commit efa7df3e3bb5.
Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake), we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:
mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)
Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link: https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.html
Quoted 28 lines
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload,
which was significantly impacted by the THP alignment logic introduced in commitefa7df3e3bb5.Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake), we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:
mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link:
https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.html
+cc Vlata
Quoted 7 lines · On Fri, Jun 27, 2025 at 04:09:16PM +0530, [email protected] wrote:
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload,
which was significantly impacted by the THP alignment logic introduced in commitefa7df3e3bb5.
Quoted 6 lines
Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake),
we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
All congratulations are owed to Vlastimil Babka for doing this, cc'd :)
I gather he enjoys novelty beer mugs as tokens of thanks ;)
Quoted 16 lines
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link: https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.html
Quoted 15 lines · Lorenzo Stoakes
+cc Vlata
Quoted 12 lines · On Fri, Jun 27, 2025 at 04:09:16PM +0530, [email protected] wrote:
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload,
which was significantly impacted by the THP alignment logic introduced in commitefa7df3e3bb5.Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake),
we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:
Quoted 4 lines
Quoted 1 line
mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
All congratulations are owed to Vlastimil Babka for doing this, cc'd :)
I gather he enjoys novelty beer mugs as tokens of thanks ;)
I was wondering how the change can get us such a big optimization - the alignment causes us to gain at most 1 extra PMD-THP mapping. Is there something else I am missing?
I ask because when I was reading the code I was thinking whether a similar change can be done for mTHPs.
Quoted 16 lines
Quoted 16 lines
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link: https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.html
Quoted 19 lines · Dev Jain
Quoted 18 lines · Lorenzo Stoakes
+cc Vlata
Quoted 7 lines · On Fri, Jun 27, 2025 at 04:09:16PM +0530, [email protected] wrote:
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload,
which was significantly impacted by the THP alignment logic introduced in commitefa7df3e3bb5.Quoted 6 lines
Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake),
we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
All congratulations are owed to Vlastimil Babka for doing this, cc'd :)
Quoted 25 lines
Quoted 1 line
I gather he enjoys novelty beer mugs as tokens of thanks ;)
I was wondering how the change can get us such a big optimization - the alignment causes us to gain at most 1 extra PMD-THP mapping. Is there something else I am missing?
I ask because when I was reading the code I was thinking whether a similar change can be done for mTHPs.
Quoted 16 lines
Quoted 16 lines
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link: https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.html
Hi Dev Jain,
Thank you for reviewing and for your thoughtful question.
You're absolutely right that, in isolation, gaining one additional PMD-THP mapping wouldn't explain a 3200% speedup. But in our use case (Hugging Face inference workloads with dynamic input sizes and many allocations), the original PMD alignment logic caused a cascade of side effects:
The performance improvement comes from how that interacts with dynamic memory allocation patterns in AI inference workloads, especially those using frameworks like Hugging Face Transformers.
In our specific use case, the workloads were running on Intel Developer Cloud, but I no longer have access to that particular environment or the original profiling output. However, I’d like to highlight why this patch had such an outsized effect:
🔹 1. Fragmentation Avoidance
In model shard loading (e.g., large BERT or GPT2 models split into multiple memory segments), many medium-sized anonymous allocations occur in rapid succession. These workloads tend to allocate many 512 KB – 1.5 MB buffers dynamically (token buffers, intermediate tensors). Aligning each one to PMD size, even when their length wasn’t PMD-aligned, led to gaps between them — defeating natural coalescing into a single THP.
🔹 2. TLB aliasing and cache index pressure
These fragmented mappings caused frequent TLB misses and poor L1/L2 cache reuse.
The result was what looks like “memory thrashing,” with slow memory access dominating total inference time.
When every mapping is PMD-aligned (even if not PMD-sized), the gaps between them prevent Transparent Huge Pages (THPs) from activating effectively.
This breaks THP coalescence and causes fragmented page tables and higher memory overhead per shard.
🔹 3. Latency & Throughput Penalty from Memory Misalignment
This leads to higher TLB miss rates, especially under multi-threaded load, which dramatically slows down token embedding and attention calculations.
When loading model shards, memory initialization becomes cache-unfriendly, with poor reuse across cores.
This affects not only inference latency but also model cold-start time — which is critical in autoscaling deployments.
🔹 4. Qualitative Observation
Without this patch: shard loading stuttered, warm-up was slow, and we saw CPU cycles dominated by page_fault and TLB miss handlers.
With this patch: shard loading smoothed out, THPs were correctly applied (based on smaps), and throughput shot up by an order of magnitude.
🔹 5. Measured Impact
On Intel Xeon (Cooper Lake), a 6.6.0 kernel with PMD alignment on non-aligned sizes showed 11–32× worse performance.
With the patched kernel (which skips alignment unless the length is PMD-aligned), memory layout was contiguous again and THP was consistently utilized.
This isn’t about one extra THP — it’s about preventing widespread THP fragmentation and the resulting dramatic cache/TLB degradation. For AI workloads with high concurrency and dynamic shapes, this small patch has a massive effect on layout and locality.
So, it's not just “1 more huge page” — it's avoiding massive fragmentation that leads to:
1. TLB miss storms
2. Poor locality
3. Cache index thrashing
4. Improvement in latency and throughput
This applies across many adjacent, odd-length allocations typical of AI inference workloads.
The original alignment logic created a pattern of broken contiguity — defeating THP benefits altogether.
In AI workloads using Hugging Face Transformers, model shards and intermediate tensors are dynamically allocated during inference. These allocations often fall just below or above the 2MB threshold that THP relies on. Misalignment or forced alignment to PMD boundaries causes fragmentation and disrupts huge page coalescence, affecting performance.
📊 Memory Allocation Pattern Diagram
Without Patch (PMD Alignment Forced):
|<--2MB-->|<--Gap-->|<--2MB-->|<--Gap-->|<--2MB--> | Alloc A | | Alloc B | | Alloc C |
Each allocation is PMD-aligned, even if it’s not PMD-sized
Gaps prevent THP coalescence → TLB/cache fragmentation
With Patch (PMD Alignment Conditional):
|<---------6MB Contiguous Region--------->| | Alloc A | Alloc B | Alloc C | Padding |
Contiguous anonymous memory region
Coalesced into one or more THPs
Improved locality and TLB efficiency
While I regret not having the raw perf output at hand, I’d be happy to replicate a similar test locally and share reproducible results if helpful.
Best Regards,
Siddhartha Sharma
Quoted 123 lines · On 30/06/25 6:13 am, [email protected] wrote:
Quoted 19 lines · Dev Jain
Quoted 18 lines · Lorenzo Stoakes
+cc Vlata
Quoted 7 lines · On Fri, Jun 27, 2025 at 04:09:16PM +0530, [email protected] wrote:
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload,
which was significantly impacted by the THP alignment logic introduced in commitefa7df3e3bb5.Quoted 6 lines
Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake),
we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
All congratulations are owed to Vlastimil Babka for doing this, cc'd :)
Quoted 25 lines
Quoted 1 line
I gather he enjoys novelty beer mugs as tokens of thanks ;)
I was wondering how the change can get us such a big optimization - the alignment causes us to gain at most 1 extra PMD-THP mapping. Is there something else I am missing?
I ask because when I was reading the code I was thinking whether a similar change can be done for mTHPs.
Quoted 16 lines
Quoted 16 lines
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link: https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.htmlHi Dev Jain,
Thank you for reviewing and for your thoughtful question.
You're absolutely right that, in isolation, gaining one additional PMD-THP mapping wouldn't explain a 3200% speedup. But in our use case (Hugging Face inference workloads with dynamic input sizes and many allocations), the original PMD alignment logic caused a cascade of side effects:
The performance improvement comes from how that interacts with dynamic memory allocation patterns in AI inference workloads, especially those using frameworks like Hugging Face Transformers.
In our specific use case, the workloads were running on Intel Developer Cloud, but I no longer have access to that particular environment or the original profiling output. However, I’d like to highlight why this patch had such an outsized effect:
🔹 1. Fragmentation Avoidance
In model shard loading (e.g., large BERT or GPT2 models split into multiple memory segments), many medium-sized anonymous allocations occur in rapid succession. These workloads tend to allocate many 512 KB – 1.5 MB buffers dynamically (token buffers, intermediate tensors). Aligning each one to PMD size, even when their length wasn’t PMD-aligned, led to gaps between them — defeating natural coalescing into a single THP.🔹 2. TLB aliasing and cache index pressure
These fragmented mappings caused frequent TLB misses and poor L1/L2 cache reuse.
The result was what looks like “memory thrashing,” with slow memory access dominating total inference time.
When every mapping is PMD-aligned (even if not PMD-sized), the gaps between them prevent Transparent Huge Pages (THPs) from activating effectively.This breaks THP coalescence and causes fragmented page tables and higher memory overhead per shard.
🔹 3. Latency & Throughput Penalty from Memory Misalignment
This leads to higher TLB miss rates, especially under multi-threaded load, which dramatically slows down token embedding and attention calculations.When loading model shards, memory initialization becomes cache-unfriendly, with poor reuse across cores.
This affects not only inference latency but also model cold-start time — which is critical in autoscaling deployments.
🔹 4. Qualitative Observation
Without this patch: shard loading stuttered, warm-up was slow, and we saw CPU cycles dominated bypage_faultand TLB miss handlers.With this patch: shard loading smoothed out, THPs were correctly applied (based on smaps), and throughput shot up by an order of magnitude.
🔹 5. Measured Impact
On Intel Xeon (Cooper Lake), a 6.6.0 kernel with PMD alignment on non-aligned sizes showed 11–32× worse performance.With the patched kernel (which skips alignment unless the length is PMD-aligned), memory layout was contiguous again and THP was consistently utilized.
This isn’t about one extra THP — it’s about preventing widespread THP fragmentation and the resulting dramatic cache/TLB degradation. For AI workloads with high concurrency and dynamic shapes, this small patch has a massive effect on layout and locality.
So, it's not just “1 more huge page” — it's avoiding massive fragmentation that leads to:
1. TLB miss storms
2. Poor locality
3. Cache index thrashing
4. Improvement in latency and throughput
This applies across many adjacent, odd-length allocations typical of AI inference workloads.
The original alignment logic created a pattern of broken contiguity — defeating THP benefits altogether.
In AI workloads using Hugging Face Transformers, model shards and intermediate tensors are dynamically allocated during inference. These allocations often fall just below or above the 2MB threshold that THP relies on. Misalignment or forced alignment to PMD boundaries causes fragmentation and disrupts huge page coalescence, affecting performance.
📊 Memory Allocation Pattern Diagram
Without Patch (PMD Alignment Forced):
|<--2MB-->|<--Gap-->|<--2MB-->|<--Gap-->|<--2MB--> | Alloc A | | Alloc B | | Alloc C |Each allocation is PMD-aligned, even if it’s not PMD-sized
Gaps prevent THP coalescence → TLB/cache fragmentation
With Patch (PMD Alignment Conditional):
|<---------6MB Contiguous Region--------->| | Alloc A | Alloc B | Alloc C | Padding |Contiguous anonymous memory region
Coalesced into one or more THPs
Quoted 7 lines
Improved locality and TLB efficiency
While I regret not having the raw perf output at hand, I’d be happy to replicate a similar test locally and share reproducible results if helpful.
Best Regards,
Siddhartha Sharma
Thanks for your detailed explanation! I misunderstood that the optimization you were talking about
was due to efa7df3e3bb5, instead it was due to the alignment. Your explanation makes a lot of
sense!
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
the mTHP case and I'd be grateful if you can get me some results : )
Quoted 2 lines
Quoted 135 lines · Dev Jain
Quoted 123 lines · On 30/06/25 6:13 am, [email protected] wrote:
Quoted 19 lines · Dev Jain
Quoted 18 lines · Lorenzo Stoakes
+cc Vlata
Quoted 7 lines · On Fri, Jun 27, 2025 at 04:09:16PM +0530, [email protected] wrote:
Hi all,
I wanted to share validation data from a Hugging Face-based AI inferencing workload,
which was significantly impacted by the THP alignment logic introduced in commitefa7df3e3bb5.Quoted 6 lines
Using transformer models with dynamic input lengths on Intel Xeon (Cooper Lake),
we observed up to a 3200% throughput improvement after applying the patch from Oct 2024:mm: limit THP alignment of anonymous mappings to PMD-aligned sizes
All congratulations are owed to Vlastimil Babka for doing this, cc'd :)
Quoted 25 lines
Quoted 1 line
I gather he enjoys novelty beer mugs as tokens of thanks ;)
I was wondering how the change can get us such a big optimization - the alignment causes us to gain at most 1 extra PMD-THP mapping. Is there something else I am missing?
I ask because when I was reading the code I was thinking whether a similar change can be done for mTHPs.
Quoted 16 lines
Quoted 16 lines
Metrics:
- Model: BERT-base
- Inference engine: Transformers + ONNX Runtime
- Kernel: 6.6 vs patched 6.6.8
- Batch size: 8-32, input length: 64-512 tokens
- Metric: inference throughput (samples/sec)Thanks for the fix -- this change had real impact on a production-relevant workload.
Best Regards,
Siddhartha Sharma
ISV @ Kenip
Solution Link: https://www.intel.com/content/www/us/en/partner/showcase/offering/a5bHo00000045YUIAY/deadlock-clearance.htmlHi Dev Jain,
Thank you for reviewing and for your thoughtful question.
You're absolutely right that, in isolation, gaining one additional PMD-THP mapping wouldn't explain a 3200% speedup. But in our use case (Hugging Face inference workloads with dynamic input sizes and many allocations), the original PMD alignment logic caused a cascade of side effects:
The performance improvement comes from how that interacts with dynamic memory allocation patterns in AI inference workloads, especially those using frameworks like Hugging Face Transformers.
In our specific use case, the workloads were running on Intel Developer Cloud, but I no longer have access to that particular environment or the original profiling output. However, I’d like to highlight why this patch had such an outsized effect:
🔹 1. Fragmentation Avoidance
In model shard loading (e.g., large BERT or GPT2 models split into multiple memory segments), many medium-sized anonymous allocations occur in rapid succession. These workloads tend to allocate many 512 KB – 1.5 MB buffers dynamically (token buffers, intermediate tensors). Aligning each one to PMD size, even when their length wasn’t PMD-aligned, led to gaps between them — defeating natural coalescing into a single THP.🔹 2. TLB aliasing and cache index pressure
These fragmented mappings caused frequent TLB misses and poor L1/L2 cache reuse.
The result was what looks like “memory thrashing,” with slow memory access dominating total inference time.
When every mapping is PMD-aligned (even if not PMD-sized), the gaps between them prevent Transparent Huge Pages (THPs) from activating effectively.This breaks THP coalescence and causes fragmented page tables and higher memory overhead per shard.
🔹 3. Latency & Throughput Penalty from Memory Misalignment
This leads to higher TLB miss rates, especially under multi-threaded load, which dramatically slows down token embedding and attention calculations.When loading model shards, memory initialization becomes cache-unfriendly, with poor reuse across cores.
This affects not only inference latency but also model cold-start time — which is critical in autoscaling deployments.
🔹 4. Qualitative Observation
Without this patch: shard loading stuttered, warm-up was slow, and we saw CPU cycles dominated bypage_faultand TLB miss handlers.With this patch: shard loading smoothed out, THPs were correctly applied (based on smaps), and throughput shot up by an order of magnitude.
🔹 5. Measured Impact
On Intel Xeon (Cooper Lake), a 6.6.0 kernel with PMD alignment on non-aligned sizes showed 11–32× worse performance.With the patched kernel (which skips alignment unless the length is PMD-aligned), memory layout was contiguous again and THP was consistently utilized.
This isn’t about one extra THP — it’s about preventing widespread THP fragmentation and the resulting dramatic cache/TLB degradation. For AI workloads with high concurrency and dynamic shapes, this small patch has a massive effect on layout and locality.
So, it's not just “1 more huge page” — it's avoiding massive fragmentation that leads to:
1. TLB miss storms
2. Poor locality
3. Cache index thrashing
4. Improvement in latency and throughput
This applies across many adjacent, odd-length allocations typical of AI inference workloads.
The original alignment logic created a pattern of broken contiguity — defeating THP benefits altogether.
In AI workloads using Hugging Face Transformers, model shards and intermediate tensors are dynamically allocated during inference. These allocations often fall just below or above the 2MB threshold that THP relies on. Misalignment or forced alignment to PMD boundaries causes fragmentation and disrupts huge page coalescence, affecting performance.
📊 Memory Allocation Pattern Diagram
Without Patch (PMD Alignment Forced):
|<--2MB-->|<--Gap-->|<--2MB-->|<--Gap-->|<--2MB--> | Alloc A | | Alloc B | | Alloc C |Each allocation is PMD-aligned, even if it’s not PMD-sized
Gaps prevent THP coalescence → TLB/cache fragmentation
With Patch (PMD Alignment Conditional):
|<---------6MB Contiguous Region--------->| | Alloc A | Alloc B | Alloc C | Padding |Contiguous anonymous memory region
Coalesced into one or more THPs
Quoted 7 lines
Improved locality and TLB efficiency
While I regret not having the raw perf output at hand, I’d be happy to replicate a similar test locally and share reproducible results if helpful.
Best Regards,
Siddhartha Sharma
Thanks for your detailed explanation! I misunderstood that the optimization you were talking about
Quoted 8 lines
was due to efa7df3e3bb5, instead it was due to the alignment. Your explanation makes a lot of
sense!
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
the mTHP case and I'd be grateful if you can get me some results : )
Oh I see that you are using the 6.6 kernel, which probably won't have the mTHP patches.
Quoted 4 lines
Quoted 3 lines
+cc Vlastimil, please keep him cc'd on discussions here as the author of this fix in the conversation.
Quoted 6 lines · Dev Jain
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
the mTHP case and I'd be grateful if you can get me some results : )
I'd urge caution here.
The reason there was a big perf improvement is that, for certain workloads, the original patch by Rik caused issues with VMA fragmentation. So rather than getting adjacent VMAs that might later be khugepage'd, you'd get a bunch of VMAs that were auto-aligned and thus fragmented from one another.
So while you got speed ups on some workloads, you got really bad perf impact on some that were subject to this.
The observed speed up was on a very specific benchmark also. While it's a great improvement, it's important to understand the context (see the original patch for details [0]).
I do think it's worth considering changing thp_get_unmapped_area_vmflags() for mTHP, as it's currently very limited (just PMD alignment) and it'd possibly be sensible to change this to checking against allowed THP alignments, but I'd not assume this is going to get some crazy speed up as observed here.
Note that any such change would probably require some refactoring in THP first to make it not quite so awful.
I also think for Siddharta's usecase mTHP isn't really relevant is it, as intel do not support mTHP currently do they?
Regards, Lorenzo
Quoted 34 lines · Lorenzo Stoakes
+cc Vlastimil, please keep him cc'd on discussions here as the author of this
fix in the conversation.Quoted 6 lines · Dev Jain
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
the mTHP case and I'd be grateful if you can get me some results : )
I'd urge caution here.
The reason there was a big perf improvement is that, for certain workloads, the original patch by Rik caused issues with VMA fragmentation. So rather than getting adjacent VMAs that might later be khugepage'd, you'd get a bunch of VMAs that were auto-aligned and thus fragmented from one another.
So while you got speed ups on some workloads, you got really bad perf impact on some that were subject to this.
The observed speed up was on a very specific benchmark also. While it's a great improvement, it's important to understand the context (see the original patch for details [0]).
I do think it's worth considering changing
thp_get_unmapped_area_vmflags()for mTHP, as it's currently very limited (just PMD alignment) and it'd possibly be sensible to change this to checking against allowed THP alignments, but I'd not assume this is going to get some crazy speed up as observed here.Note that any such change would probably require some refactoring in THP first to make it not quite so awful.
Quoted 6 lines
I also think for Siddharta's usecase mTHP isn't really relevant is it, as intel do not support mTHP currently do they?
Regards, Lorenzo
Hi Lorenzo, Dev, All,
Thank you for the thoughtful responses and for engaging with the performance implications of the patch.
You're absolutely right that the observed speedup came from a specific class of workloads — in this case, token-length-variable AI inference pipelines based on Hugging Face Transformers and ONNX Runtime. These workloads trigger highly dynamic, anonymous memory allocation patterns, often in bursts aligned with model shard loading and attention map resizing. In such cases, VMA fragmentation due to PMD-aligned, non-PMD-sized mappings led to near-complete loss of THP utilization.
Once the alignment restriction was lifted (via Rik’s patch), we observed substantial restoration of THP behavior, which is where the performance gains came from. That said, I completely agree that:
Not all workloads benefit from this
Some may even regress if the underlying VMAs aren't THP-coalescible for other reasons
Still, for high-throughput inference workloads on modern Intel CPUs, this behavior isn’t a corner case. The shift toward multi-model concurrent serving (e.g., LLM-as-a-Service) means this dynamic allocation pattern is becoming common, especially in edge/latency-sensitive deployments.
🧠 On mTHP: Intel Does Support It
Regarding mTHP — yes, Intel platforms (especially server-grade Xeon processors from Cascade Lake onward) do support mapped transparent huge pages, including via:
tmpfs-backed files
madvise(MADV_HUGEPAGE) on file mappings
shmem usage with shmem_enabled in the kernel
So I’d say mTHP is certainly relevant for workloads where model weights or tensors are pre-cached or memory-mapped — a pattern we’re also seeing as Hugging Face, ONNX, and PyTorch ecosystems move toward zero-copy tensor sharing.
Given that, I'd absolutely be interested in testing any mTHP-targeted patch — and I’d be happy to help validate it, especially if it avoids the VMA fragmentation pitfall you rightly pointed out.
Thanks again for the detailed feedback, and I’ll try to replicate and share further traces (from my local testbed) since I currently don’t have access to the original Intel Developer Cloud logs.
Best regards,
Siddhartha Sharma
Quoted 8 lines · Lorenzo Stoakes
+cc Vlastimil, please keep him cc'd on discussions here as the author of this fix in the conversation.
Quoted 4 lines · Dev Jain
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
Quoted 7 lines
Quoted 1 line
the mTHP case and I'd be grateful if you can get me some results : )
I'd urge caution here.
The reason there was a big perf improvement is that, for certain workloads, the original patch by Rik caused issues with VMA fragmentation. So rather than getting adjacent VMAs that might later be khugepage'd, you'd get a bunch of VMAs that were auto-aligned and thus fragmented from one another.
How does getting two different adjacent VMAs allow them to be khugepage'd if both are less than PMD size? khugepaged operates per vma, I'm missing something.
Quoted 22 lines
So while you got speed ups on some workloads, you got really bad perf impact on some that were subject to this.
The observed speed up was on a very specific benchmark also. While it's a great improvement, it's important to understand the context (see the original patch for details [0]).
I do think it's worth considering changing
thp_get_unmapped_area_vmflags()for mTHP, as it's currently very limited (just PMD alignment) and it'd possibly be sensible to change this to checking against allowed THP alignments, but I'd not assume this is going to get some crazy speed up as observed here.Note that any such change would probably require some refactoring in THP first to make it not quite so awful.
I also think for Siddharta's usecase mTHP isn't really relevant is it, as intel do not support mTHP currently do they?
Regards, Lorenzo
Quoted 13 lines · Dev Jain
Quoted 11 lines · Lorenzo Stoakes
+cc Vlastimil, please keep him cc'd on discussions here as the author of this fix in the conversation.
Quoted 5 lines · Dev Jain
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
the mTHP case and I'd be grateful if you can get me some results : )
I'd urge caution here.
Quoted 7 lines
Quoted 4 lines
The reason there was a big perf improvement is that, for certain workloads, the original patch by Rik caused issues with VMA fragmentation. So rather than getting adjacent VMAs that might later be khugepage'd, you'd get a bunch of VMAs that were auto-aligned and thus fragmented from one another.
How does getting two different adjacent VMAs allow them to be khugepage'd if both are less than PMD size? khugepaged operates per vma, I'm missing something.
(future) VMA merge
Consider allocations that are >PMD but < 2*PMD for instance. Now you get fragmentation. For some workloads you would have previously eventually got PMD leaf mapping, PMD leaf mapping, PMD leaf mapping, etc. contiguouosly, with this arragenement you get PMD mapping, <bunch of PTE mappings>, PMD mapping, etc.
Quoted 16 lines · Lorenzo Stoakes
Quoted 15 lines · Dev Jain
Quoted 14 lines · Lorenzo Stoakes
+cc Vlastimil, please keep him cc'd on discussions here as the author of this fix in the conversation.
Quoted 4 lines · Dev Jain
For this workload, do you enable mTHPs on your system? My plan is to make a similar patch for
the mTHP case and I'd be grateful if you can get me some results : )
I'd urge caution here.
The reason there was a big perf improvement is that, for certain workloads, the original patch by Rik caused issues with VMA fragmentation. So rather than getting adjacent VMAs that might later be khugepage'd, you'd get a bunch of VMAs that were auto-aligned and thus fragmented from one another.
Quoted 8 lines
Quoted 2 lines
How does getting two different adjacent VMAs allow them to be khugepage'd if both are less than PMD size? khugepaged operates per vma, I'm missing something.
(future) VMA merge
Consider allocations that are >PMD but < 2*PMD for instance. Now you get fragmentation. For some workloads you would have previously eventually got PMD leaf mapping, PMD leaf mapping, PMD leaf mapping, etc. contiguouosly, with this arragenement you get PMD mapping, <bunch of PTE mappings>, PMD mapping, etc.
Sorry I am not following, don't know in detail about the VMA merge stuff. Are you saying the after the patch, the VMAs will eventually get merged? Is it possible in the kernel to get a merge in the "future"; as I understand it only happens at mmap() time?
Suppose before the patch, you have two consecutive VMAs between (PMD, 2*PMD) size. If they are able to get merged after the patch, why won't they be merged before the patch, since the VMA characteristics are the same?
Quoted 10 lines · Dev Jain
Sorry I am not following, don't know in detail about the VMA merge stuff. Are you saying the after the patch, the VMAs will eventually get merged? Is it possible in the kernel to get a merge in the "future"; as I understand it only happens at
mmap()time?Suppose before the patch, you have two consecutive VMAs between (PMD, 2*PMD) size. If they are able to get merged after the patch, why won't they be merged before the patch, since the VMA characteristics are the same?
Rik's patch aligned each to 2 MiB boundary. So you'd get gaps:
0 2MB 4MB 6MB 8MB 10MB
|-------------.------| |-------------.------| |-------------.------|
| . | | . | | . |
| . | | . | | . |
|-------------.------| |-------------.------| |-------------.------|
huge mapped 4k m'dIf you don't force alignment then subsequent mappings will be adjacent to one another and those non-huge page parts can be merged.
Vlasta's fix up means we only try to get the THP up-front if the length is already aligned at which point you won't end up with these gaps.
Quoted 14 lines · Lorenzo Stoakes
Quoted 10 lines · Dev Jain
Sorry I am not following, don't know in detail about the VMA merge stuff. Are you saying the after the patch, the VMAs will eventually get merged? Is it possible in the kernel to get a merge in the "future"; as I understand it only happens at
mmap()time?Suppose before the patch, you have two consecutive VMAs between (PMD, 2*PMD) size. If they are able to get merged after the patch, why won't they be merged before the patch, since the VMA characteristics are the same?
Rik's patch aligned each to 2 MiB boundary. So you'd get gaps:
Quoted 6 lines
0 2MB 4MB 6MB 8MB 10MB
|-------------.------| |-------------.------| |-------------.------| | . | | . | | . | | . | | . | | . | |-------------.------| |-------------.------| |-------------.------| huge mapped 4k m'd
The effort to draw this is appreciated!
I understood the alignment, what I am asking is this:
In __get_unmapped_area(), we will return a THP-aligned addr from thp_get_unmapped_area_vmflags(). Now for the diagram you have drawn, suppose that before the patch, we first mmap() the 8MB-start chunk. Then we mmap the 4MB start chunk.
We go to __mmap_region(), and we see that the 8MB-start chunk has mergeable characteristics, so we merge. So the gap goes away?
Quoted 6 lines
If you don't force alignment then subsequent mappings will be adjacent to one another and those non-huge page parts can be merged.
Vlasta's fix up means we only try to get the THP up-front if the length is already aligned at which point you won't end up with these gaps.
Quoted 25 lines · Dev Jain
Quoted 14 lines · Lorenzo Stoakes
Quoted 10 lines · Dev Jain
Sorry I am not following, don't know in detail about the VMA merge stuff. Are you saying the after the patch, the VMAs will eventually get merged? Is it possible in the kernel to get a merge in the "future"; as I understand it only happens at
mmap()time?Suppose before the patch, you have two consecutive VMAs between (PMD, 2*PMD) size. If they are able to get merged after the patch, why won't they be merged before the patch, since the VMA characteristics are the same?
Rik's patch aligned each to 2 MiB boundary. So you'd get gaps:
Quoted 6 lines
0 2MB 4MB 6MB 8MB 10MB
|-------------.------| |-------------.------| |-------------.------| | . | | . | | . | | . | | . | | . | |-------------.------| |-------------.------| |-------------.------| huge mapped 4k m'dThe effort to draw this is appreciated!
Quoted 8 lines
I understood the alignment, what I am asking is this:
In
__get_unmapped_area(), we will return a THP-aligned addr fromthp_get_unmapped_area_vmflags(). Now for the diagram you have drawn, suppose that before the patch, we firstmmap()the 8MB-start chunk. Then we mmap the 4MB start chunk.
We go to__mmap_region(), and we see that the 8MB-start chunk has mergeable characteristics, so we merge. So the gap goes away?
No because there's a gap, we only merge immedaitely adjacent VMAs. And obviously gaps mean page tables wouldn't be adjacent either...
The get_unmmaped_area() would have otherwise given adjacent mappings. Vlasta's patch means in this case we no longer bother trying to align these because their _length_ isn't PMD aligned.
Quoted 26 lines · Lorenzo Stoakes
Quoted 25 lines · Dev Jain
Quoted 14 lines · Lorenzo Stoakes
Quoted 10 lines · Dev Jain
Sorry I am not following, don't know in detail about the VMA merge stuff. Are you saying the after the patch, the VMAs will eventually get merged? Is it possible in the kernel to get a merge in the "future"; as I understand it only happens at
mmap()time?Suppose before the patch, you have two consecutive VMAs between (PMD, 2*PMD) size. If they are able to get merged after the patch, why won't they be merged before the patch, since the VMA characteristics are the same?
Rik's patch aligned each to 2 MiB boundary. So you'd get gaps:
Quoted 6 lines
0 2MB 4MB 6MB 8MB 10MB
|-------------.------| |-------------.------| |-------------.------|
| . | | . | | . |
| . | | . | | . |
|-------------.------| |-------------.------| |-------------.------|
huge mapped 4k m'dThe effort to draw this is appreciated!
I understood the alignment, what I am asking is this:
Quoted 8 lines
Quoted 6 lines
In
__get_unmapped_area(), we will return a THP-aligned addr fromthp_get_unmapped_area_vmflags(). Now for the diagram you have drawn, suppose that before the patch, we firstmmap()the 8MB-start chunk. Then we mmap the 4MB start chunk.
We go to__mmap_region(), and we see that the 8MB-start chunk has mergeable characteristics, so we merge. So the gap goes away?No because there's a gap, we only merge immedaitely adjacent VMAs. And obviously gaps mean page tables wouldn't be adjacent either...
Ah shoot. That is prev->vm_end == vmg->start in can_vma_merge_left(). Thanks.
Quoted 4 lines
The
get_unmmaped_area()would have otherwise given adjacent mappings. Vlasta's patch means in this case we no longer bother trying to align these because their _length_ isn't PMD aligned.
Quoted 38 lines · Dev Jain
Quoted 37 lines · Lorenzo Stoakes
Quoted 36 lines · Dev Jain
Quoted 19 lines · Lorenzo Stoakes
Quoted 10 lines · Dev Jain
Sorry I am not following, don't know in detail about the VMA merge stuff. Are you saying the after the patch, the VMAs will eventually get merged? Is it possible in the kernel to get a merge in the "future"; as I understand it only happens at
mmap()time?Suppose before the patch, you have two consecutive VMAs between (PMD, 2*PMD) size. If they are able to get merged after the patch, why won't they be merged before the patch, since the VMA characteristics are the same?
Rik's patch aligned each to 2 MiB boundary. So you'd get gaps:
0 2MB 4MB 6MB
8MB 10MB
|-------------.------| |-------------.------|
|-------------.------|
| . | | . |Quoted 6 lines
| . | | . | | . | | . | |-------------.------| |-------------.------| |-------------.------| huge mapped 4k m'dThe effort to draw this is appreciated!
I understood the alignment, what I am asking is this:
In
__get_unmapped_area(), we will return a THP-aligned addr fromthp_get_unmapped_area_vmflags(). Now for the diagram you have drawn, suppose that before the patch, we firstmmap()the 8MB-start chunk. Then we mmap the 4MB start chunk.
We go to__mmap_region(), and we see that the 8MB-start chunk has mergeable characteristics, so we merge. So the gap goes away?
Quoted 9 lines
Quoted 2 lines
No because there's a gap, we only merge immedaitely adjacent VMAs. And obviously gaps mean page tables wouldn't be adjacent either...
Ah shoot. That is
prev->vm_end==vmg->startincan_vma_merge_left(). Thanks.Quoted 4 lines
The
get_unmmaped_area()would have otherwise given adjacent mappings. Vlasta's patch means in this case we no longer bother trying to align these because their _length_ isn't PMD aligned.
Hi Lorenzo, Dev, all
Thank you for raising excellent points — I’ll respond to each in order to clarify the mechanics and relevance of this behavior in the context of AI inference workloads.
🧩 1. Does the patch cause VMAs to be merged eventually?
You're correct: VMA merging only happens at mmap() time (via __mmap_region()). What the patch affects is the behavior of thp_get_unmapped_area_vmflags() before the mmap is placed.
Before the patch (with Rik’s logic):
Every mmap() returned an address rounded up to the next 2MB boundary — regardless of whether the requested size was 2MB-aligned.
Result: even consecutive mmap()s (e.g., 1.5MB + 1.5MB) are now non-adjacent, so merging is impossible, even if their VMA flags match.
After this patch:
If the allocation is not PMD-aligned in size, the returned address is not forcibly aligned, increasing the likelihood that the next mmap() lands directly after the previous one → enabling merging.
So, to be clear: this patch doesn’t cause merging, but it prevents unnecessary pre-mmap gaps, which previously blocked merges from ever happening exactly like a deadlock which has been cleared now.
📐 2. Why aren’t the VMAs mergeable before the patch?
Great question. Even if the VMA flags are identical, gaps introduced by forced alignment from get_unmapped_area() break the precondition for merging:
can_vma_merge_left()
→ return prev->vm_end == vma->vm_start
With Rik’s patch in place:
Suppose you mmap() 1.5MB → gets aligned to 2MB
Next 1.5MB → gets aligned to 4MB
→ The kernel sees: prev->vm_end = 3.5MB, vma->vm_start = 4MB → No merge
With this patch, non-aligned lengths don’t get forcibly aligned, so consecutive mmap()s often fall exactly after the previous, and merging becomes possible again.
🤖 3. How does this impact AI workloads like Hugging Face Transformers? Tokenization and dynamic batching create non-deterministic memory allocation patterns:
Models like BERT and T5 dynamically allocate intermediate buffers per token-length, batch size, and attention window.
Hugging Face + ONNX Runtime uses multiple small-ish anonymous mmap()s, often 512KB–1.8MB.
These allocations come in bursts — but due to forced alignment, the kernel was placing them with artificial gaps, defeating THP eligibility entirely.
By not force-aligning non-PMD-sized mappings, we avoid injecting gaps. The result is that:
a. VMAs remain adjacent → mergeable
b. Physical memory is contiguous → eligible for khugepaged collapse
c. THP utilization increases → fewer TLB misses → lower latency → higher throughput
💡 4. Why this patch complements Rik’s rather than contradicts it:
Rik's patch made it easier to guarantee alignment for workloads that benefit from explicit huge pages — but at the cost of breaking coalescence in workloads with non-PMD-sized mappings, like ML inference.
This patch simply refines that logic:
If the length is PMD-aligned → keep alignment
If it’s not → don’t inject alignment gaps that block merging
So, for workloads that can’t benefit from THP due to misalignment, this patch removes artificial fragmentation without harming the original intent.
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.
Thanks again for the detailed discussion. Let me know if you’d like a trace or VMA map from a Hugging Face benchmarked run (happy to generate one locally).
Best Regards,
Siddhartha Sharma
+91 9015185601
Quoted 4 lines · On Tue, Jul 01, 2025 at 05:45:51PM +0530, [email protected] wrote:
🧩 1. Does the patch cause VMAs to be merged eventually? You're correct: VMA merging only happens at
mmap()time (via__mmap_region()). What the patch affects is the behavior ofthp_get_unmapped_area_vmflags()before the mmap is placed.
[...]
Quoted 5 lines
📐 2. Why aren’t the VMAs mergeable before the patch?
Great question. Even if the VMA flags are identical, gaps introduced by forced alignment fromget_unmapped_area()break the precondition for merging:
[...]
Quoted 1 line
💡 4. Why this patch complements Rik’s rather than contradicts it:
I'm really perplexed as to why you felt the need to (seemingly via LLM) reply with the explanation I've already provided here?...
There's errors in things you say here too.
With respect, please don't do this.
(I'm the co-maintainer of pretty much all the relevant code here and wrote the VMA merge logic you're referring to.)
Quoted 17 lines
🤖 3. How does this impact AI workloads like Hugging Face Transformers? Tokenization and dynamic batching create non-deterministic memory allocation patterns:
Models like BERT and T5 dynamically allocate intermediate buffers per token-length, batch size, and attention window.
Hugging Face + ONNX Runtime uses multiple small-ish anonymous mmap()s, often 512KB–1.8MB.
These allocations come in bursts — but due to forced alignment, the kernel was placing them with artificial gaps, defeating THP eligibility entirely.
By not force-aligning non-PMD-sized mappings, we avoid injecting gaps. The result is that:
Quoted 7 lines
a. VMAs remain adjacent → mergeable
b. Physical memory is contiguous → eligible for khugepaged collapse
c. THP utilization increases → fewer TLB misses → lower latency → higher throughput
This is very useful information and it's appreciated! Let's not drown this out with restatements of stuff already covered.
Quoted 6 lines
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.
Dev - could we hold off on any effort to do something like this until I've had a chance to refactor THP somewhat? This is already a mess and I'd like to avoid us piling on more complexity.
We can revisit this at a later stage.
Quoted 5 lines
Thanks again for the detailed discussion. Let me know if you’d like a trace or VMA map from a Hugging Face benchmarked run (happy to generate one locally).
Thanks! Much appreciated.
Cheers, Lorenzo
Quoted 70 lines · Lorenzo Stoakes
Quoted 4 lines · On Tue, Jul 01, 2025 at 05:45:51PM +0530, [email protected] wrote:
🧩 1. Does the patch cause VMAs to be merged eventually? You're correct: VMA merging only happens at
mmap()time (via__mmap_region()). What the patch affects is the behavior ofthp_get_unmapped_area_vmflags()before the mmap is placed.[...]
Quoted 5 lines
📐 2. Why aren’t the VMAs mergeable before the patch?
Great question. Even if the VMA flags are identical, gaps introduced by forced alignment fromget_unmapped_area()break the precondition for merging:[...]
Quoted 1 line
💡 4. Why this patch complements Rik’s rather than contradicts it:
I'm really perplexed as to why you felt the need to (seemingly via LLM) reply with the explanation I've already provided here?...
There's errors in things you say here too.
With respect, please don't do this.
(I'm the co-maintainer of pretty much all the relevant code here and wrote the VMA merge logic you're referring to.)
Quoted 17 lines
🤖 3. How does this impact AI workloads like Hugging Face Transformers? Tokenization and dynamic batching create non-deterministic memory allocation patterns:
Models like BERT and T5 dynamically allocate intermediate buffers per token-length, batch size, and attention window.
Hugging Face + ONNX Runtime uses multiple small-ish anonymous mmap()s, often 512KB–1.8MB.
These allocations come in bursts — but due to forced alignment, the kernel was placing them with artificial gaps, defeating THP eligibility entirely.
By not force-aligning non-PMD-sized mappings, we avoid injecting gaps. The result is that:
Quoted 7 lines
a. VMAs remain adjacent → mergeable
b. Physical memory is contiguous → eligible for khugepaged collapse
c. THP utilization increases → fewer TLB misses → lower latency → higher throughput
This is very useful information and it's appreciated! Let's not drown this out with restatements of stuff already covered.
Quoted 6 lines
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.Dev - could we hold off on any effort to do something like this until I've had a chance to refactor THP somewhat? This is already a mess and I'd like to avoid us piling on more complexity.
We can revisit this at a later stage.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Thanks again for the detailed discussion. Let me know if you’d like a trace or VMA map from a Hugging Face benchmarked run (happy to generate one locally).
Thanks! Much appreciated.
Cheers, Lorenzo
Hi Lorenzo,
Thanks for your clarification, and I appreciate your patience — especially given your role in maintaining and designing the VMA merge logic.
I understand now that my earlier phrasing may have repeated your explanation for VMA adjacency, and I regret unintentionally restating it.
I’ll make sure to be more careful and direct going forward.
As for the THP alignment condition now being IS_ALIGNED(len, PMD_SIZE), I agree this resolves the regressions by removing alignment for non-aligned sizes, which was exactly what broke workloads like cactusBSSN or some AI inference loads.
Thanks again for the guidance — I’m learning a lot from this thread.
Best Regards,
Siddhartha Sharma
Quoted 9 lines · On Tue, Jul 01, 2025 at 06:53:47PM +0530, [email protected] wrote:
Hi Lorenzo,
Thanks for your clarification, and I appreciate your patience — especially given your role in maintaining and designing the VMA merge logic.
I understand now that my earlier phrasing may have repeated your explanation for VMA adjacency, and I regret unintentionally restating it.
I’ll make sure to be more careful and direct going forward.
Thanks, no problem. Mostly avoids confusion.
Quoted 5 lines
As for the THP alignment condition now being
IS_ALIGNED(len, PMD_SIZE), I agree this resolves the regressions by removing alignment for non-aligned sizes, which was exactly what broke workloads like cactusBSSN or some AI inference loads.
Ack - we're really happy to hear about workloads that this has helped as this kind of input is very important as to getting insight into how THP-related stuff impacts real users so we can best optimise especially for workloads that are very important in the industry right now.
Quoted 2 lines
Thanks again for the guidance — I’m learning a lot from this thread.
Glad to have helped, thanks again for reporting!
Quoted 4 lines
Best Regards,
Siddhartha Sharma
Cheers, Lorenzo
Quoted 27 lines · Lorenzo Stoakes
Quoted 9 lines · On Tue, Jul 01, 2025 at 06:53:47PM +0530, [email protected] wrote:
Hi Lorenzo,
Thanks for your clarification, and I appreciate your patience — especially given your role in maintaining and designing the VMA merge logic.
I understand now that my earlier phrasing may have repeated your explanation for VMA adjacency, and I regret unintentionally restating it.
I’ll make sure to be more careful and direct going forward.
Thanks, no problem. Mostly avoids confusion.
Quoted 5 lines
As for the THP alignment condition now being
IS_ALIGNED(len, PMD_SIZE), I agree this resolves the regressions by removing alignment for non-aligned sizes, which was exactly what broke workloads like cactusBSSN or some AI inference loads.Ack - we're really happy to hear about workloads that this has helped as this kind of input is very important as to getting insight into how THP-related stuff impacts real users so we can best optimise especially for workloads that are very important in the industry right now.
Quoted 2 lines
Thanks again for the guidance — I’m learning a lot from this thread.
Quoted 8 lines
Glad to have helped, thanks again for reporting!
Quoted 4 lines
Best Regards,
Siddhartha SharmaCheers, Lorenzo
Hi Lorenzo,
Thanks for the acknowledgement of my work, I really appreciate it. Please let me know if there is anything I can do here now moving forwards with integrating. Furthermore, once integrated and tested, I would like to see all performance metrics that have seen improvements if possible.
Best Regards,
Siddhartha Sharma
Quoted 55 lines · Lorenzo Stoakes
Quoted 4 lines · On Tue, Jul 01, 2025 at 05:45:51PM +0530, [email protected] wrote:
🧩 1. Does the patch cause VMAs to be merged eventually? You're correct: VMA merging only happens at
mmap()time (via__mmap_region()). What the patch affects is the behavior ofthp_get_unmapped_area_vmflags()before the mmap is placed.[...]
Quoted 4 lines
📐 2. Why aren’t the VMAs mergeable before the patch?
Great question. Even if the VMA flags are identical, gaps introduced by forced alignment fromget_unmapped_area()break the precondition for merging:[...]
Quoted 1 line
💡 4. Why this patch complements Rik’s rather than contradicts it:
I'm really perplexed as to why you felt the need to (seemingly via LLM) reply with the explanation I've already provided here?...
There's errors in things you say here too.
With respect, please don't do this.
(I'm the co-maintainer of pretty much all the relevant code here and wrote the VMA merge logic you're referring to.)
Quoted 16 lines
🤖 3. How does this impact AI workloads like Hugging Face Transformers? Tokenization and dynamic batching create non-deterministic memory allocation patterns:
Models like BERT and T5 dynamically allocate intermediate buffers per token-length, batch size, and attention window.
Hugging Face + ONNX Runtime uses multiple small-ish anonymous mmap()s, often 512KB–1.8MB.
These allocations come in bursts — but due to forced alignment, the kernel was placing them with artificial gaps, defeating THP eligibility entirely.
By not force-aligning non-PMD-sized mappings, we avoid injecting gaps. The result is that:
Quoted 7 lines
a. VMAs remain adjacent → mergeable
b. Physical memory is contiguous → eligible for khugepaged collapse
c. THP utilization increases → fewer TLB misses → lower latency → higher throughput
This is very useful information and it's appreciated! Let's not drown this out with restatements of stuff already covered.
Quoted 5 lines
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.
Quoted 5 lines
Dev - could we hold off on any effort to do something like this until I've had a chance to refactor THP somewhat? This is already a mess and I'd like to avoid us piling on more complexity.
We can revisit this at a later stage.
Yes of course. I had run a small benchmark on a quick dumb patch I wrote and I don't see any measurable perf improvement, probably because the highest THP order getting chosen is always PMD size.
Out of curiosity, where do you plan to do the refactoring?
Quoted 8 lines
Quoted 5 lines
Thanks again for the detailed discussion. Let me know if you’d like a trace or VMA map from a Hugging Face benchmarked run (happy to generate one locally).
Thanks! Much appreciated.
Cheers, Lorenzo
Quoted 60 lines · Dev Jain
Quoted 59 lines · Lorenzo Stoakes
Quoted 4 lines · On Tue, Jul 01, 2025 at 05:45:51PM +0530, [email protected] wrote:
🧩 1. Does the patch cause VMAs to be merged eventually? You're correct: VMA merging only happens at
mmap()time (via__mmap_region()). What the patch affects is the behavior ofthp_get_unmapped_area_vmflags()before the mmap is placed.[...]
Quoted 4 lines
📐 2. Why aren’t the VMAs mergeable before the patch?
Great question. Even if the VMA flags are identical, gaps introduced by forced alignment fromget_unmapped_area()break the precondition for merging:[...]
Quoted 1 line
💡 4. Why this patch complements Rik’s rather than contradicts it:
I'm really perplexed as to why you felt the need to (seemingly via LLM) reply with the explanation I've already provided here?...
There's errors in things you say here too.
With respect, please don't do this.
(I'm the co-maintainer of pretty much all the relevant code here and wrote the VMA merge logic you're referring to.)
Quoted 16 lines
🤖 3. How does this impact AI workloads like Hugging Face Transformers? Tokenization and dynamic batching create non-deterministic memory allocation patterns:
Models like BERT and T5 dynamically allocate intermediate buffers per token-length, batch size, and attention window.
Hugging Face + ONNX Runtime uses multiple small-ish anonymous mmap()s, often 512KB–1.8MB.
These allocations come in bursts — but due to forced alignment, the kernel was placing them with artificial gaps, defeating THP eligibility entirely.
By not force-aligning non-PMD-sized mappings, we avoid injecting gaps. The result is that:
Quoted 7 lines
a. VMAs remain adjacent → mergeable
b. Physical memory is contiguous → eligible for khugepaged collapse
c. THP utilization increases → fewer TLB misses → lower latency → higher throughput
This is very useful information and it's appreciated! Let's not drown this out with restatements of stuff already covered.
Quoted 5 lines
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.Dev - could we hold off on any effort to do something like this until I've had a chance to refactor THP somewhat? This is already a mess and I'd like to avoid us piling on more complexity.
Quoted 5 lines
Quoted 1 line
We can revisit this at a later stage.
Yes of course. I had run a small benchmark on a quick dumb patch I wrote and I don't see any measurable perf improvement, probably because the highest THP order getting chosen is always PMD size.
I think mTHP is much more complicated, since mTHP has many sizes. Trying to adjust VMA alignments to get mTHP might not work well, since you never know what sizes new VMAs are going to have.
IMHO, it might be better to align VMA to PMD or the largest mTHP size (for example, on ARM64 with 64KB base page, PMD THP is 512MB, a 2MB mTHP sounds more reasonable there) if possible and enable VMA merging as much as possible for future huge page collapse. mTHP can be used to fill the non faulted holes in VMAs if necessary.
Quoted 2 lines
Out of curiosity, where do you plan to do the refactoring?
Best Regards,
Yan, Zi
Quoted 14 lines · Zi Yan
Quoted 14 lines
Quoted 13 lines
This is very useful information and it's appreciated! Let's not drown this out with restatements of stuff already covered.
Quoted 5 lines
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.Dev - could we hold off on any effort to do something like this until I've had a chance to refactor THP somewhat? This is already a mess and I'd like to avoid us piling on more complexity.
We can revisit this at a later stage.
Quoted 7 lines
Quoted 3 lines
Yes of course. I had run a small benchmark on a quick dumb patch I wrote and I don't see any measurable perf improvement, probably because the highest THP order getting chosen is always PMD size.
I think mTHP is much more complicated, since mTHP has many sizes. Trying to adjust VMA alignments to get mTHP might not work well, since you never know what sizes new VMAs are going to have.
Yes I agree it's more complicated. In case there would be a stream of allocations of varying small-ish sizes, aligning each of them to its smallest applicable mTHP could create gaps that wouldn't exist if we ignored the alignment and just find any free area and in the end merge it to an existing one. Basically we'd risk recreating the issue with gaps.
Sticking to one size (2MB) mitigates this to some extent. Unfortunately even after my fix the heuristics might be prone to gaps:
- all allocations not multiple of 2MB - will merge freely
- all allocations multiple of 2MB - the alignment heuristic will kick in, but as a result allocations should still merge as all boundaries are 2MB alignned
- allocations alternate between multiple of 2MB and non-multiple of 2MB - this will still create gaps
Note we already had a report about ebizzy regressing due to my commit [1] and I suspect it might be due to this kind of scenario. A proper investigation would be useful but I didn't get to it.
Maybe the solution is to first check if unaligned search gives us a range that will merge with adjacent area, and only try the alignment heuristics if it doesn't. This will still fail if mmap() is followed by e.g. mprotect() or madvise() that will change an initially un-mergeable area to a mergeable one. I have no ideas around that though. Just some thoughts to consider for anyone wanting to change things here further :)
[1] https://lore.kernel.org/all/019401db769f%24961e7e20%24c25b7a60%[email protected]/
Quoted 13 lines
IMHO, it might be better to align VMA to PMD or the largest mTHP size (for example, on ARM64 with 64KB base page, PMD THP is 512MB, a 2MB mTHP sounds more reasonable there) if possible and enable VMA merging as much as possible for future huge page collapse. mTHP can be used to fill the non faulted holes in VMAs if necessary.
Quoted 2 lines
Out of curiosity, where do you plan to do the refactoring?
Best Regards,
Yan, Zi
Quoted 47 lines · Vlastimil Babka
Quoted 14 lines · Zi Yan
Quoted 14 lines
Quoted 13 lines
This is very useful information and it's appreciated! Let's not drown this out with restatements of stuff already covered.
Quoted 5 lines
⚙️ 5. mTHP note
Although this patch doesn’t target mTHP directly, I believe a similar logic tweak could apply there too — especially with shmem-backed workloads (common in model servers using shared tensor memory). I’d be happy to help test any changes proposed there to derive the consequent results.Dev - could we hold off on any effort to do something like this until I've had a chance to refactor THP somewhat? This is already a mess and I'd like to avoid us piling on more complexity.
We can revisit this at a later stage.
Quoted 7 lines
Quoted 3 lines
Yes of course. I had run a small benchmark on a quick dumb patch I wrote and I don't see any measurable perf improvement, probably because the highest THP order getting chosen is always PMD size.
I think mTHP is much more complicated, since mTHP has many sizes. Trying to adjust VMA alignments to get mTHP might not work well, since you never know what sizes new VMAs are going to have.
Yes I agree it's more complicated. In case there would be a stream of allocations of varying small-ish sizes, aligning each of them to its smallest applicable mTHP could create gaps that wouldn't exist if we ignored the alignment and just find any free area and in the end merge it to an existing one. Basically we'd risk recreating the issue with gaps.
Sticking to one size (2MB) mitigates this to some extent. Unfortunately even after my fix the heuristics might be prone to gaps:
- all allocations not multiple of 2MB - will merge freely
- all allocations multiple of 2MB - the alignment heuristic will kick in,
but as a result allocations should still merge as all boundaries are 2MB alignned- allocations alternate between multiple of 2MB and non-multiple of 2MB -
this will still create gapsNote we already had a report about ebizzy regressing due to my commit [1] and I suspect it might be due to this kind of scenario. A proper investigation would be useful but I didn't get to it.
Quoted 22 lines
Maybe the solution is to first check if unaligned search gives us a range that will merge with adjacent area, and only try the alignment heuristics if it doesn't. This will still fail if
mmap()is followed by e.g.mprotect()ormadvise()that will change an initially un-mergeable area to a mergeable one. I have no ideas around that though. Just some thoughts to consider for anyone wanting to change things here further :)[1] https://lore.kernel.org/all/019401db769f%24961e7e20%24c25b7a60%[email protected]/
Quoted 13 lines
IMHO, it might be better to align VMA to PMD or the largest mTHP size (for example, on ARM64 with 64KB base page, PMD THP is 512MB, a 2MB mTHP sounds more reasonable there) if possible and enable VMA merging as much as possible for future huge page collapse. mTHP can be used to fill the non faulted holes in VMAs if necessary.
Quoted 2 lines
Out of curiosity, where do you plan to do the refactoring?
Best Regards,
Yan, Zi
Hi Lorenzo, Dev, Mel,
I'm following up on this patch submission from earlier this month:
"[PATCH] mm: limit THP alignment – performance gain observed in AI inference workloads."
The change limits THP alignment to PMD-sized mappings, avoiding unnecessary hugepage over-allocations in scenarios where 2MB alignment is not beneficial. We’ve observed consistent performance improvements in inference pipelines (specifically with OpenVINO) where the workload profile includes a mix of small and large allocations.
Please let me know if:
- There has been any progress or feedback from your end,
- The patch needs to align with ongoing THP refactoring efforts,
- Additional benchmarks, test traces, or system-level profiles would help.
Happy to revise or refine the patch based on further discussion. Thanks again for your time and input!
For your information, I have also posted the same at Openvino and Huggingface forums and currently waiting for review for the commit on the Openvino github repository.
Best regards,
Siddhartha Sharma
Quoted 6 lines · On 7/28/25 07:41, [email protected] wrote:
On 2025-07-07 14:26, Vlastimil Babka wrote:
Hi Lorenzo, Dev, Mel,I'm following up on this patch submission from earlier this month:
"[PATCH] mm: limit THP alignment – performance gain observed in AI inference workloads."
I'm confused. That wasn't a patch submission, but reporting performance results for my patch from late 2024? (and thanks for those!)
The patch was also already merged in late 2024:
commit d4148aeab412432bf928f311eca8a2ba52bb05df
Author: Vlastimil Babka <[email protected]> Date: Thu Oct 24 17:12:29 2024 +0200
mm, mmap: limit THP alignment of anonymous mappings to PMD-aligned sizes
So there's nothing more to do here AFAIK.
Quoted 10 lines
🤖 3. How does this impact AI workloads like Hugging Face Transformers? Tokenization and dynamic batching create non-deterministic memory allocation patterns:
Models like BERT and T5 dynamically allocate intermediate buffers per token-length, batch size, and attention window.
Hugging Face + ONNX Runtime uses multiple small-ish anonymous mmap()s, often 512KB–1.8MB.
If I remember correctly, Rik's patch should just force PMD alignment when the allocation size is greater than PMD size. Such VMA fragmentation should be caused by allocations greater than 2M but not PMD aligned, so they create 2M PMD + a bunch of 4K PTEs. Less than 2M allocations should be right next to each other and mergeable. Did I miss something?
Thanks,
Yang
Quoted 5 lines
These allocations come in bursts — but due to forced alignment, the kernel was placing them with artificial gaps, defeating THP eligibility entirely.