binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP
Subsystems (4)
v4 cover binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP
Changes since [v3]:
* Fixed compilation failure under !CONFIG_TRANSPARENT_HUGEPAGE.
* No functional changes otherwise.
Changes since [v2]:
* Renamed align_to_pmd() to should_align_to_pmd().
* Added benchmark results to the commit message.
Changes since [v1]:
* Dropped the Kconfig option CONFIG_ELF_RO_LOAD_THP_ALIGNMENT.
* Moved the alignment logic into a helper align_to_pmd() for clarity.
* Improved the comment explaining why we skip the optimization when PMD_SIZE > 32MB.
When Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.
For ELF executables loaded by the kernel ELF binary loader, PT_LOAD segments are normally aligned according to p_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.
A segment is considered eligible if:
* THP is in "always" mode,
* it is not writable,
* both p_vaddr and p_offset are PMD-aligned,
* its file size is at least PMD_SIZE, and
* its existing p_align is smaller than PMD_SIZE.
To avoid excessive address space padding on systems with very large PMD_SIZE values, this optimization is applied only when PMD_SIZE <= 32MB, since requiring larger alignments would be unreasonable, especially on 32-bit systems with a much more limited virtual address space.
This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.
This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic linker) are not affected.
Benchmark
Machine: AMD Ryzen 9 7950X (x86_64)
Binutils: 2.46
GCC: 15.2.1 (built with -z,noseparate-code + --enable-host-pie)
Workload: building Linux v7.0-rc1 vmlinux with x86_64_defconfig.
Without patch With patch instructions 8,246,133,611,932 8,246,025,137,750 cpu-cycles 8,001,028,142,928 7,565,925,107,502 itlb-misses 3,672,158,331 26,821,242 time elapsed 64.66 s 61.97 s
Instructions are basically unchanged. iTLB misses drop from ~3.67B to ~26M (~99.27% reduction), which results in about a ~5.44% reduction in cycles and ~4.18% shorter wall time for this workload.
[v3]: https://lore.kernel.org/linux-fsdevel/[email protected]
[v2]: https://lore.kernel.org/linux-fsdevel/[email protected]
[v1]: https://lore.kernel.org/linux-fsdevel/[email protected]
WANG Rui (2):
huge_mm: add stubs for THP-disabled configs
binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for
THP
fs/binfmt_elf.c | 29 +++++++++++++++++++++++++++++ include/linux/huge_mm.h | 10 ++++++++++ 2 files changed, 39 insertions(+)
Signature · 1 line
2.53.0
PATCH v4 1/2 huge_mm: add stubs for THP-disabled configs
hugepage_global_enabled() and hugepage_global_always() only exist when CONFIG_TRANSPARENT_HUGEPAGE is set. Add inline stubs that return false to let code compile when THP is disabled.
--- include/linux/huge_mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/linux/huge_mm.h +10 -0
@@ -570,6 +570,16 @@ void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,#else /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool hugepage_global_enabled (void ) +{ +return false; +} + +static inline bool hugepage_global_always (void ) +{ +return false; +} +static inline bool folio_test_pmd_mappable (struct folio *folio) {return false;
Signature · 1 line
2.53.0
David Hildenbrand (Arm)
maintainerRe: [PATCH v4 1/2] huge_mm: add stubs for THP-disabled configs
Quoted 20 lines · WANG Rui
hugepage_global_enabled()andhugepage_global_always()only exist whenCONFIG_TRANSPARENT_HUGEPAGEis set. Add inline stubs that return false to let code compile when THP is disabled.Signed-off-by WANG Rui <[email protected]>--- include/linux/huge_mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)include/linux/huge_mm.h +3
-0@@ -570,6 +570,16 @@ void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,#else /* CONFIG_TRANSPARENT_HUGEPAGE */ +static inline bool hugepage_global_enabled (void ) +{ +return false;
include/linux/huge_mm.h void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd, · line 585
Quoted 10 lines
include/linux/huge_mm.h +7
-0+} + +static inline bool hugepage_global_always (void ) +{ +return false; +} +static inline bool folio_test_pmd_mappable (struct folio *folio) {return false;
There are other ways to enable PMD THP. So I don't quite think this is the right tool for the job.
Signature · 3 lines
Cheers, David
David Hildenbrand (Arm)
maintainerRe: [PATCH v4 1/2] huge_mm: add stubs for THP-disabled configs
Quoted 18 lines · David Hildenbrand (Arm)
Quoted 17 lines · WANG Rui
hugepage_global_enabled()andhugepage_global_always()only exist whenCONFIG_TRANSPARENT_HUGEPAGEis set. Add inline stubs that return false to let code compile when THP is disabled.Signed-off-by WANG Rui <[email protected]>--- include/linux/huge_mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)include/linux/huge_mm.h +0
-0@@ -570,6 +570,16 @@ void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,#else /* CONFIG_TRANSPARENT_HUGEPAGE */
Quoted 16 lines
include/linux/huge_mm.h void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd, · line 585
Quoted 13 lines
include/linux/huge_mm.h +10
-0+static inline bool hugepage_global_enabled (void ) +{ +return false; +} + +static inline bool hugepage_global_always (void ) +{ +return false; +} +static inline bool folio_test_pmd_mappable (struct folio *folio) {return false;There are other ways to enable PMD THP. So I don't quite think this is the right tool for the job.
Ah, you care about file THPs ... gah.
Why can't we simply do the alignment without considering the current setting?
Signature · 3 lines
Cheers, David
Re: [PATCH v4 1/2] huge_mm: add stubs for THP-disabled configs
Hi David,
Quoted 37 lines · David Hildenbrand (Arm)
Quoted 18 lines · David Hildenbrand (Arm)
Quoted 17 lines · WANG Rui
hugepage_global_enabled()andhugepage_global_always()only exist whenCONFIG_TRANSPARENT_HUGEPAGEis set. Add inline stubs that return false to let code compile when THP is disabled.Signed-off-by WANG Rui <[email protected]>--- include/linux/huge_mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+)include/linux/huge_mm.h +0
-0@@ -570,6 +570,16 @@ void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd,#else /* CONFIG_TRANSPARENT_HUGEPAGE */ Quoted 16 lines
include/linux/huge_mm.h void map_anon_folio_pmd_nopf(struct folio *folio, pmd_t *pmd, · line 585
Quoted 13 lines
include/linux/huge_mm.h +10
-0+static inline bool hugepage_global_enabled (void ) +{ +return false; +} + +static inline bool hugepage_global_always (void ) +{ +return false; +} +static inline bool folio_test_pmd_mappable (struct folio *folio) {return false;There are other ways to enable PMD THP. So I don't quite think this is the right tool for the job.
Quoted 4 lines
Ah, you care about file THPs ... gah.
Why can't we simply do the alignment without considering the current setting?
The main motivation of raising the alignment here is to increase the chance of getting PMD-sized THPs for executable mappings.
If THP is not in "always" mode, the kernel will not automatically collapse file-backed mappings into THPs, so the increased alignment would not actually improve THP usage.
In that case we would only be introducing additional padding in the virtual address layout, which slightly reduces ASLR entropy without providing a practical benefit.
That's why the current code limits this to the "always" mode.
Thanks,
Rui
David Hildenbrand (Arm)
maintainerRe: [PATCH v4 1/2] huge_mm: add stubs for THP-disabled configs
Quoted 18 lines · hev
Hi David,
Quoted 10 lines · David Hildenbrand (Arm)
Quoted 3 lines · David Hildenbrand (Arm)
There are other ways to enable PMD THP. So I don't quite think this is the right tool for the job.
Ah, you care about file THPs ... gah.
Why can't we simply do the alignment without considering the current setting?
The main motivation of raising the alignment here is to increase the chance of getting PMD-sized THPs for executable mappings.
Quoted 7 lines
If THP is not in "always" mode, the kernel will not automatically collapse file-backed mappings into THPs, so the increased alignment would not actually improve THP usage.
In that case we would only be introducing additional padding in the virtual address layout, which slightly reduces ASLR entropy without providing a practical benefit.
Well, that parameter can get toggled at runtime later? Also, I think that readahead code could end up allocating a PMD THP (I might be wrong about that, the code is confusing).
Let's take a look at __get_unmapped_area(), where we don't care about ASLR entropy for anonymous memory:
} else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && !file
&& !addr /* no hint */
&& IS_ALIGNED(len, PMD_SIZE)) {Interestingly we had:
commit 34d7cf637c437d5c2a8a6ef23ea45193bad8a91c
Author: Kefeng Wang <[email protected]> Date: Fri Dec 6 15:03:45 2024 +0800
mm: don't try THP alignment for FS without get_unmapped_area
Commit ed48e87c7df3 ("thp: add thp_get_unmapped_area_vmflags()") changes thp_get_unmapped_area() to thp_get_unmapped_area_vmflags() in __get_unmapped_area(), which doesn't initialize local get_area for anonymous mappings. This leads to us always trying THP alignment even for file_operations which have a NULL ->get_unmapped_area() callback.
Since commit efa7df3e3bb5 ("mm: align larger anonymous mappings on THP boundaries") we only want to enable THP alignment for anonymous mappings, so add a !file check to avoid attempting THP alignment for file mappings.
Found issue by code inspection. THP alignment is used for easy or more pmd mappings, from vma side. This may cause unnecessary VMA fragmentation and potentially worse performance on filesystems that do not actually support THPs and thus cannot benefit from the alignment.
I'm not sure about the "VMA fragmentation" argument, really. We only consider stuff that is already multiples of PMD_SIZE.
Filesystem support for THPs is also not really something you would handle, and it's a problem that solves itself over time as more filesystems keep adding support for large folios.
So I think we should try limiting it to IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE), but not checking the runtime toggle.
Signature · 3 lines
Cheers, David
Re: [PATCH v4 1/2] huge_mm: add stubs for THP-disabled configs
Quoted 66 lines · David Hildenbrand (Arm)
Quoted 18 lines · hev
Hi David,
Quoted 10 lines · David Hildenbrand (Arm)
Quoted 3 lines · David Hildenbrand (Arm)
There are other ways to enable PMD THP. So I don't quite think this is the right tool for the job.
Ah, you care about file THPs ... gah.
Why can't we simply do the alignment without considering the current setting?
The main motivation of raising the alignment here is to increase the chance of getting PMD-sized THPs for executable mappings.
Quoted 7 lines
If THP is not in "always" mode, the kernel will not automatically collapse file-backed mappings into THPs, so the increased alignment would not actually improve THP usage.
In that case we would only be introducing additional padding in the virtual address layout, which slightly reduces ASLR entropy without providing a practical benefit.
Well, that parameter can get toggled at runtime later? Also, I think that readahead code could end up allocating a PMD THP (I might be wrong about that, the code is confusing).
Let's take a look at
__get_unmapped_area(), where we don't care about ASLR entropy for anonymous memory:} else if (
IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)&& !file&& !addr /* no hint */ && IS_ALIGNED(len, PMD_SIZE)) {Interestingly we had:
commit
34d7cf637c437d5c2a8a6ef23ea45193bad8a91cAuthor: Kefeng Wang <[email protected]> Date: Fri Dec 6 15:03:45 2024 +0800mm: don't try THP alignment for FS without
get_unmapped_areaCommit
ed48e87c7df3("thp: addthp_get_unmapped_area_vmflags()") changesthp_get_unmapped_area()tothp_get_unmapped_area_vmflags()in__get_unmapped_area(), which doesn't initialize localget_areafor anonymous mappings. This leads to us always trying THP alignment even forfile_operationswhich have a NULL ->get_unmapped_area() callback.Since commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") we only want to enable THP alignment for anonymous mappings, so add a !file check to avoid attempting THP alignment for file mappings.Found issue by code inspection. THP alignment is used for easy or more pmd mappings, from vma side. This may cause unnecessary VMA fragmentation and potentially worse performance on filesystems that do not actually support THPs and thus cannot benefit from the alignment.
I'm not sure about the "VMA fragmentation" argument, really. We only consider stuff that is already multiples of
PMD_SIZE.
Quoted 6 lines
Filesystem support for THPs is also not really something you would handle, and it's a problem that solves itself over time as more filesystems keep adding support for large folios.
So I think we should try limiting it to
IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE), but not checking the runtime toggle.
That's a fair point about the runtime toggle. Limiting it to IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) sounds reasonable.
Thanks,
Rui
Lance Yang
reviewerRe: [PATCH v4 1/2] huge_mm: add stubs for THP-disabled configs
In reply to David Hildenbrand (Arm)Quoted 23 lines · David Hildenbrand (Arm)
Quoted 22 lines · hev
Hi David,
Quoted 10 lines · David Hildenbrand (Arm)
Quoted 3 lines · David Hildenbrand (Arm)
There are other ways to enable PMD THP. So I don't quite think this is the right tool for the job.
Ah, you care about file THPs ... gah.
Why can't we simply do the alignment without considering the current setting?
The main motivation of raising the alignment here is to increase the chance of getting PMD-sized THPs for executable mappings.
If THP is not in "always" mode, the kernel will not automatically collapse file-backed mappings into THPs, so the increased alignment would not actually improve THP usage.
Quoted 7 lines
Quoted 3 lines
In that case we would only be introducing additional padding in the virtual address layout, which slightly reduces ASLR entropy without providing a practical benefit.
Well, that parameter can get toggled at runtime later? Also, I think that readahead code could end up allocating a PMD THP (I might be wrong about that, the code is confusing).
Right. In do_sync_mmap_readahead(), if the VMA has VM_HUGEPAGE, force_thp_readahead becomes true and ra->order is set to HPAGE_PMD_ORDER, IIUC.
/* Use the readahead code, even if readahead is disabled */ if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) force_thp_readahead = true;
That order is then passed down to page_cache_ra_order() and finally to filemap_alloc_folio().
if (force_thp_readahead) {
[...]
ra->async_size = HPAGE_PMD_NR;
ra->order = HPAGE_PMD_ORDER;
page_cache_ra_order(&ractl, ra);
return fpin;
}For plain VM_EXEC, the code starts from exec_folio_order(), not HPAGE_PMD_ORDER.
if (vm_flags & VM_EXEC) {
[...]
ra->order = exec_folio_order();
[...]
ra->async_size = 0;
}The default exec_folio_order() is small, and arm64 only overrides it to 64K.
/*
* Request exec memory is read into pagecache in at least 64K folios.
This size
* can be contpte-mapped when 4K base pages are in use (16 pages into 1
iTLB
* entry), and HPA can coalesce it (4 pages into 1 TLB entry) when 16K base
* pages are in use.
*/
#define exec_folio_order() ilog2(SZ_64K >> PAGE_SHIFT)
Quoted 7 lines
Let's take a look at
__get_unmapped_area(), where we don't care about ASLR entropy for anonymous memory:} else if (
IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)&& !file&& !addr /* no hint */ && IS_ALIGNED(len, PMD_SIZE)) {
Yeah. For anonymous memory, the kernel is willing to do THP-friendly alignment, but it is constrained, of course :)
Quoted 14 lines
Interestingly we had:
commit
34d7cf637c437d5c2a8a6ef23ea45193bad8a91cAuthor: Kefeng Wang <[email protected]> Date: Fri Dec 6 15:03:45 2024 +0800mm: don't try THP alignment for FS without
get_unmapped_areaCommit
ed48e87c7df3("thp: addthp_get_unmapped_area_vmflags()") changesthp_get_unmapped_area()tothp_get_unmapped_area_vmflags()in__get_unmapped_area(), which doesn't initialize localget_areafor anonymous mappings. This leads to us always trying THP alignment even forfile_operationswhich have a NULL ->get_unmapped_area() callback.
Quoted 8 lines
Since commit
efa7df3e3bb5("mm: align larger anonymous mappings on THP boundaries") we only want to enable THP alignment for anonymous mappings, so add a !file check to avoid attempting THP alignment for file mappings.Found issue by code inspection. THP alignment is used for easy or more pmd mappings, from vma side. This may cause unnecessary VMA fragmentation and potentially worse performance on filesystems that do not actually support THPs and thus cannot benefit from the alignment.
Looks like this commit does not *ban* file-backed THP-friendly alignment completely. It only prevents file mappings from getting it accidentally via the generic fallback path.
Note that some filesystems still explicitly opt in with their own
.get_unmapped_area = thp_get_unmapped_area
for example ext4, xfs, and btrfs.
So explicit filesystem opt-in is still allowed :)
Quoted 9 lines
I'm not sure about the "VMA fragmentation" argument, really. We only consider stuff that is already multiples of
PMD_SIZE.Filesystem support for THPs is also not really something you would handle, and it's a problem that solves itself over time as more filesystems keep adding support for large folios.
So I think we should try limiting it to
IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE), but not checking the runtime toggle.
Good point! ELF layout is decided once at exec time, while the runtime THP mode can change later.
PATCH v4 2/2 binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP
In reply to WANG RuiWhen Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.
For ELF executables loaded by the kernel ELF binary loader, PT_LOAD segments are normally aligned according to p_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.
A segment is considered eligible if:
* THP is in "always" mode,
* it is not writable,
* both p_vaddr and p_offset are PMD-aligned,
* its file size is at least PMD_SIZE, and
* its existing p_align is smaller than PMD_SIZE.
To avoid excessive address space padding on systems with very large PMD_SIZE values, this optimization is applied only when PMD_SIZE <= 32MB, since requiring larger alignments would be unreasonable, especially on 32-bit systems with a much more limited virtual address space.
This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.
This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic linker) are not affected.
Benchmark
Machine: AMD Ryzen 9 7950X (x86_64)
Binutils: 2.46
GCC: 15.2.1 (built with -z,noseparate-code + --enable-host-pie)
Workload: building Linux v7.0-rc1 vmlinux with x86_64_defconfig.
Without patch With patch instructions 8,246,133,611,932 8,246,025,137,750 cpu-cycles 8,001,028,142,928 7,565,925,107,502 itlb-misses 3,672,158,331 26,821,242 time elapsed 64.66 s 61.97 s
Instructions are basically unchanged. iTLB misses drop from ~3.67B to ~26M (~99.27% reduction), which results in about a ~5.44% reduction in cycles and ~4.18% shorter wall time for this workload.
--- fs/binfmt_elf.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
fs/binfmt_elf.c +29 -0
@@ -28,6 +28,7 @@#include <linux/highuid.h> #include <linux/compiler.h> #include <linux/highmem.h> +#include <linux/huge_mm.h> #include <linux/hugetlb.h> #include <linux/pagemap.h> #include <linux/vmalloc.h>
@@ -489,6 +490,30 @@ static int elf_read(struct file *file, void *buf, size_t len, loff_t pos)return 0 ; } +static inline bool should_align_to_pmd (const struct elf_phdr *cmd) +{ +/* +* Avoid excessive virtual address space padding when PMD_SIZE is very +* large (e.g. some 64K base-page configurations). +*/ +if (PMD_SIZE >SZ_32M ) +return false; + +if (!hugepage_global_always ()) +return false; + +if (!IS_ALIGNED (cmd->p_vaddr | cmd->p_offset,PMD_SIZE )) +return false; + +if (cmd->p_filesz <PMD_SIZE ) +return false; + +if (cmd->p_flags &PF_W ) +return false; + +return true; +} +static unsigned long maximum_alignment (struct elf_phdr *cmds,int nr) {unsigned long alignment =0 ;
@@ -501,6 +526,10 @@ static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr)/* skip non-power of two alignments as invalid */ if (!is_power_of_2 (p_align))continue ; + +if (should_align_to_pmd (&cmds[i]) && p_align <PMD_SIZE ) + p_align =PMD_SIZE ; + alignment =max (alignment, p_align); } }
Signature · 1 line
2.53.0
Baolin Wang
reviewerCC Usama
Quoted 36 lines · WANG Rui
Changes since [v3]:
* Fixed compilation failure under !CONFIG_TRANSPARENT_HUGEPAGE.
* No functional changes otherwise.Changes since [v2]:
* Renamedalign_to_pmd()toshould_align_to_pmd().
* Added benchmark results to the commit message.Changes since [v1]:
* Dropped theKconfigoptionCONFIG_ELF_RO_LOAD_THP_ALIGNMENT.
* Moved the alignment logic into a helperalign_to_pmd()for clarity.
* Improved the comment explaining why we skip the optimization whenPMD_SIZE> 32MB.When Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.
For ELF executables loaded by the kernel ELF binary loader,
PT_LOADsegments are normally aligned according top_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.A segment is considered eligible if:
* THP is in "always" mode,
* it is not writable,
* bothp_vaddrandp_offsetare PMD-aligned,
* its file size is at leastPMD_SIZE, and
* its existingp_alignis smaller thanPMD_SIZE.To avoid excessive address space padding on systems with very large
PMD_SIZEvalues, this optimization is applied only whenPMD_SIZE<= 32MB, since requiring larger alignments would be unreasonable, especially on 32-bit systems with a much more limited virtual address space.
Quoted 7 lines
This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.
This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic linker) are not affected.
Usama posted a similar patchset[1], and I think using exec_folio_order() for exec-segment alignment is reasonable. In your case, you can override exec_folio_order() to return a PMD‑sized order.
[1] https://lore.kernel.org/all/[email protected]/
Quoted 31 lines
Benchmark
Machine: AMD Ryzen 9 7950X (
x86_64)
Binutils: 2.46
GCC: 15.2.1 (built with -z,noseparate-code + --enable-host-pie)Workload: building Linux v7.0-rc1 vmlinux with
x86_64_defconfig.Without patch With patch
instructions 8,246,133,611,932 8,246,025,137,750 cpu-cycles 8,001,028,142,928 7,565,925,107,502 itlb-misses 3,672,158,331 26,821,242 time elapsed 64.66 s 61.97 sInstructions are basically unchanged. iTLB misses drop from ~3.67B to ~26M (~99.27% reduction), which results in about a ~5.44% reduction in cycles and ~4.18% shorter wall time for this workload.
[v3]: https://lore.kernel.org/linux-fsdevel/[email protected]
[v2]: https://lore.kernel.org/linux-fsdevel/[email protected]
[v1]: https://lore.kernel.org/linux-fsdevel/[email protected]WANG Rui (2):
huge_mm: add stubs for THP-disabled configs
binfmt_elf: Align eligible read-onlyPT_LOADsegments toPMD_SIZEfor
THPfs/binfmt_elf.c | 29 +++++++++++++++++++++++++++++ include/linux/huge_mm.h | 10 ++++++++++ 2 files changed, 39 insertions(+)
Usama Arif
reviewerQuoted 43 lines · Baolin Wang
CC Usama
Quoted 40 lines · WANG Rui
Changes since [v3]:
* Fixed compilation failure under !CONFIG_TRANSPARENT_HUGEPAGE.
* No functional changes otherwise.Changes since [v2]:
* Renamedalign_to_pmd()toshould_align_to_pmd().
* Added benchmark results to the commit message.Changes since [v1]:
* Dropped theKconfigoptionCONFIG_ELF_RO_LOAD_THP_ALIGNMENT.
* Moved the alignment logic into a helperalign_to_pmd()for clarity.
* Improved the comment explaining why we skip the optimization whenPMD_SIZE> 32MB.When Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.
For ELF executables loaded by the kernel ELF binary loader,
PT_LOADsegments are normally aligned according top_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.A segment is considered eligible if:
* THP is in "always" mode,
* it is not writable,
* bothp_vaddrandp_offsetare PMD-aligned,
* its file size is at leastPMD_SIZE, and
* its existingp_alignis smaller thanPMD_SIZE.To avoid excessive address space padding on systems with very large
PMD_SIZEvalues, this optimization is applied only whenPMD_SIZE<= 32MB, since requiring larger alignments would be unreasonable, especially on 32-bit systems with a much more limited virtual address space.This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.
Quoted 8 lines
Quoted 3 lines
This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic linker) are not affected.
Usama posted a similar patchset[1], and I think using
exec_folio_order()for exec-segment alignment is reasonable. In your case, you can overrideexec_folio_order()to return a PMD‑sized order.
Thanks for the CC Baolin! Happy to see someone else noticed the same issue!
Yeah I agree, I think piggybacking off exec_folio_order() as done in 1 should be the right appproach.
I also think there is maybe a bug in do_sync_mmap_readahead that needs to be fixed when it comes to mmap_miss counter [2].
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Quoted 32 lines
Quoted 31 lines
Benchmark
Machine: AMD Ryzen 9 7950X (
x86_64)
Binutils: 2.46
GCC: 15.2.1 (built with -z,noseparate-code + --enable-host-pie)Workload: building Linux v7.0-rc1 vmlinux with
x86_64_defconfig.Without patch With patch instructions 8,246,133,611,932 8,246,025,137,750 cpu-cycles 8,001,028,142,928 7,565,925,107,502 itlb-misses 3,672,158,331 26,821,242 time elapsed 64.66 s 61.97 s
Instructions are basically unchanged. iTLB misses drop from ~3.67B to ~26M (~99.27% reduction), which results in about a ~5.44% reduction in cycles and ~4.18% shorter wall time for this workload.
[v3]: https://lore.kernel.org/linux-fsdevel/[email protected]
[v2]: https://lore.kernel.org/linux-fsdevel/[email protected]
[v1]: https://lore.kernel.org/linux-fsdevel/[email protected]WANG Rui (2):
huge_mm: add stubs for THP-disabled configs
binfmt_elf: Align eligible read-onlyPT_LOADsegments toPMD_SIZEfor THPfs/binfmt_elf.c | 29 +++++++++++++++++++++++++++++
include/linux/huge_mm.h| 10 ++++++++++2 files changed, 39 insertions(+)
Quoted 9 lines · Baolin Wang
CC Usama
Usama posted a similar patchset[1], and I think using
exec_folio_order()for exec-segment alignment is reasonable. In your case, you can overrideexec_folio_order()to return a PMD‑sized order.
Thanks for the pointer!
Cheers,
Rui