binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP
Subsystems (3)
PATCH v2 1/1 binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP
When Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.
For ELF executables loaded by the kernel ELF binary loader, PT_LOAD segments are normally aligned according to p_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.
A segment is considered eligible if:
* THP is in "always" mode,
* it is not writable,
* both p_vaddr and p_offset are PMD-aligned,
* its file size is at least PMD_SIZE, and
* its existing p_align is smaller than PMD_SIZE.
To avoid excessive address space padding on systems with very large PMD_SIZE values, this optimization is applied only when PMD_SIZE <= 32MB, since requiring larger alignments would be unreasonable, especially on 32-bit systems with a much more limited virtual address space.
This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.
This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic linker) are not affected.
---
Changes since [v1]:
* Dropped the Kconfig option CONFIG_ELF_RO_LOAD_THP_ALIGNMENT.
* Moved the alignment logic into a helper align_to_pmd() for clarity.
* Improved the comment explaining why we skip the optimization when PMD_SIZE > 32MB.
[v1]: https://lore.kernel.org/linux-fsdevel/[email protected]
--- fs/binfmt_elf.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
fs/binfmt_elf.c +29 -0
@@ -28,6 +28,7 @@#include <linux/highuid.h> #include <linux/compiler.h> #include <linux/highmem.h> +#include <linux/huge_mm.h> #include <linux/hugetlb.h> #include <linux/pagemap.h> #include <linux/vmalloc.h>
@@ -489,6 +490,30 @@ static int elf_read(struct file *file, void *buf, size_t len, loff_t pos)return 0 ; } +static inline bool align_to_pmd (const struct elf_phdr *cmd) +{ +/* +* Avoid excessive virtual address space padding when PMD_SIZE is very +* large (e.g. some 64K base-page configurations). +*/ +if (PMD_SIZE >SZ_32M ) +return false; + +if (!hugepage_global_always ()) +return false; + +if (!IS_ALIGNED (cmd->p_vaddr | cmd->p_offset,PMD_SIZE )) +return false; + +if (cmd->p_filesz <PMD_SIZE ) +return false; + +if (cmd->p_flags &PF_W ) +return false; + +return true; +} +static unsigned long maximum_alignment (struct elf_phdr *cmds,int nr) {unsigned long alignment =0 ;
@@ -501,6 +526,10 @@ static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr)/* skip non-power of two alignments as invalid */ if (!is_power_of_2 (p_align))continue ; + +if (align_to_pmd (&cmds[i]) && p_align <PMD_SIZE ) + p_align =PMD_SIZE ; + alignment =max (alignment, p_align); } }
Signature · 1 line
2.53.0
Hi,
I ran a quick benchmark on x86_64 as well.
Machine: AMD Ryzen 9 7950X
Binutils: 2.46
GCC: 15.2.1 (built with -z,noseparate-code + --enable-host-pie)
Workload: building Linux v7.0-rc1 with x86_64_defconfig.
Without patch:
* instructions: 8,246,133,611,932 * cpu-cycles: 8,001,028,142,928 * itlb-misses: 3,672,158,331 * time elapsed: 64.66 s
With patch:
* instructions: 8,246,025,137,750 * cpu-cycles: 7,565,925,107,502 * itlb-misses: 26,821,242 * time elapsed: 61.97 s
Instructions are basically unchanged. iTLB misses drop from ~3.67B to ~26M (~99.27% reduction), which results in about a ~5.44% reduction in cycles and ~4.18% shorter wall time for this workload.
Thanks,
Rui
Quoted 87 lines · WANG Rui
When Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.
For ELF executables loaded by the kernel ELF binary loader,
PT_LOADsegments are normally aligned according top_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.A segment is considered eligible if:
* THP is in "always" mode,
* it is not writable,
* bothp_vaddrandp_offsetare PMD-aligned,
* its file size is at leastPMD_SIZE, and
* its existingp_alignis smaller thanPMD_SIZE.To avoid excessive address space padding on systems with very large
PMD_SIZEvalues, this optimization is applied only whenPMD_SIZE<= 32MB, since requiring larger alignments would be unreasonable, especially on 32-bit systems with a much more limited virtual address space.This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.
This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic linker) are not affected.
Signed-off-by WANG Rui <[email protected]>---
Changes since [v1]:
* Dropped theKconfigoptionCONFIG_ELF_RO_LOAD_THP_ALIGNMENT.
* Moved the alignment logic into a helperalign_to_pmd()for clarity.
* Improved the comment explaining why we skip the optimization whenPMD_SIZE> 32MB.[v1]: https://lore.kernel.org/linux-fsdevel/[email protected]
--- fs/binfmt_elf.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)fs/binfmt_elf.c +25
-0@@ -28,6 +28,7 @@#include <linux/highuid.h> #include <linux/compiler.h> #include <linux/highmem.h> +#include <linux/huge_mm.h> #include <linux/hugetlb.h> #include <linux/pagemap.h> #include <linux/vmalloc.h> @@ -489,6 +490,30 @@ static int elf_read(struct file *file, void *buf, size_t len, loff_t pos)return 0 ; } +static inline bool align_to_pmd (const struct elf_phdr *cmd) +{ +/* +* Avoid excessive virtual address space padding when PMD_SIZE is very +* large (e.g. some 64K base-page configurations). +*/ +if (PMD_SIZE >SZ_32M ) +return false; + +if (!hugepage_global_always ()) +return false; + +if (!IS_ALIGNED (cmd->p_vaddr | cmd->p_offset,PMD_SIZE )) +return false; + +if (cmd->p_filesz <PMD_SIZE ) +return false; + +if (cmd->p_flags &PF_W ) +return false; + +return true; +} +static unsigned long maximum_alignment (struct elf_phdr *cmds,int nr) {unsigned long alignment =0 ;
fs/binfmt_elf.c static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr) · line 531
Quoted 6 lines
fs/binfmt_elf.c +3
-0@@ -501,6 +526,10 @@ static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr)/* skip non-power of two alignments as invalid */ if (!is_power_of_2 (p_align))continue ; + +if (align_to_pmd (&cmds[i]) && p_align <PMD_SIZE ) + p_align =PMD_SIZE ;
The function name does not make any sense when reading this chunk.
"should_align_XXX_to_pmd" ?
Signature · 3 lines
Cheers, David