Skip to contentSkip to conversation

binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP

RFC PATCH Superseded WANG Rui 3 participants 6 messages Mar 2 – Mar 3

Subsystems (3)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

WANG Rui

author
#0 lore raw

RFC PATCH 1/1 binfmt_elf: Align eligible read-only PT_LOAD segments to PMD_SIZE for THP

When Transparent Huge Pages (THP) are enabled in "always" mode, file-backed read-only mappings can be backed by PMD-sized huge pages if they meet the alignment and size requirements.

For ELF executables loaded by the kernel ELF binary loader, PT_LOAD segments are normally aligned according to p_align, which is often only page-sized. As a result, large read-only segments that are otherwise eligible may fail to be mapped using PMD-sized THP.

Introduce a new Kconfig option, CONFIG_ELF_RO_LOAD_THP_ALIGNMENT, to allow bumping the maximum alignment of eligible read-only PT_LOAD segments to PMD_SIZE.

A segment is considered eligible if:

* THP is in "always" mode,
* it is not writable,
* both p_vaddr and p_offset are PMD-aligned,
* its file size is at least PMD_SIZE, and
* its existing p_align is smaller than PMD_SIZE.

To avoid excessive address space padding on systems with very large PMD_SIZE values, this optimization is applied only when PMD_SIZE <= 32MB.

This increases the likelihood that large text segments of ELF executables are backed by PMD-sized THP, reducing TLB pressure and improving performance for large binaries.

This only affects ELF executables loaded directly by the kernel binary loader. Shared libraries loaded by user space (e.g. via the dynamic loader) are not affected.

The behavior is guarded by CONFIG_ELF_RO_LOAD_THP_ALIGNMENT and depends on READ_ONLY_THP_FOR_FS, so existing systems remain unchanged unless explicitly enabled.

---
 fs/Kconfig.binfmt | 12 ++++++++++++
 fs/binfmt_elf.c   |  9 +++++++++
 2 files changed, 21 insertions(+)
fs/Kconfig.binfmt +12 -0
index 1949e25c7741..8d4271769e08 100644
@@ -74,6 +74,18 @@ config ELFCORE
 	help
 	  This option enables kernel/elfcore.o.
 
+config ELF_RO_LOAD_THP_ALIGNMENT
+	bool "Align read-only ELF load segments for THP (EXPERIMENTAL)"
+	depends on READ_ONLY_THP_FOR_FS
+
+	help
+	  Align eligible read-only ELF PT_LOAD segments to PMD_SIZE so
+	  they can be mapped using PMD-sized Transparent Huge Pages
+	  when THP is enabled in "always" mode.
+
+	  This only affects ELF executables loaded by the kernel ELF
+	  binary loader.
+
 config CORE_DUMP_DEFAULT_ELF_HEADERS
 	bool "Write ELF core dumps with partial segments"
 	default y
fs/binfmt_elf.c +9 -0
index 8e89cc5b2820..2c2ccb041938 100644
@@ -28,6 +28,7 @@
 #include <linux/highuid.h>
 #include <linux/compiler.h>
 #include <linux/highmem.h>
+#include <linux/huge_mm.h>
 #include <linux/hugetlb.h>
 #include <linux/pagemap.h>
 #include <linux/vmalloc.h>
@@ -500,6 +501,14 @@ static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr)
 			/* skip non-power of two alignments as invalid */
 			if (!is_power_of_2(p_align))
 				continue;
+
+#if defined(CONFIG_ELF_RO_LOAD_THP_ALIGNMENT) && PMD_SIZE <= SZ_32M
+			if (hugepage_global_always() && !(cmds[i].p_flags & PF_W)
+				&& IS_ALIGNED(cmds[i].p_vaddr | cmds[i].p_offset, PMD_SIZE)
+				&& cmds[i].p_filesz >= PMD_SIZE && p_align < PMD_SIZE)
+				p_align = PMD_SIZE;
+#endif
+
 			alignment = max(alignment, p_align);
 		}
 	}
Signature · 1 line
2.53.0
#1 lore raw

fs/Kconfig.binfmt config ELFCORE · line 79

Quoted 3 lines · WANG Rui
fs/Kconfig.binfmt +3 -0
+config ELF_RO_LOAD_THP_ALIGNMENT
+	bool "Align read-only ELF load segments for THP (EXPERIMENTAL)"
+	depends on READ_ONLY_THP_FOR_FS

This doesn't deserve a config option.

Quoted 1 line

+#if defined(CONFIG_ELF_RO_LOAD_THP_ALIGNMENT) && PMD_SIZE <= SZ_32M

Why 32MB? This is weird and not justified anywhere.

fs/binfmt_elf.c static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr) · line 509

Quoted 4 lines
fs/binfmt_elf.c +4 -0
+			if (hugepage_global_always() && !(cmds[i].p_flags & PF_W)
+				&& IS_ALIGNED(cmds[i].p_vaddr | cmds[i].p_offset, PMD_SIZE)
+				&& cmds[i].p_filesz >= PMD_SIZE && p_align < PMD_SIZE)
+				p_align = PMD_SIZE;

Normal style is to put the '&&' at the end of the line:

			if (!(cmds[i].p_flags & PF_W) &&
			    IS_ALIGNED(cmds[i].p_vaddr | cmds[i].p_offset, PMD_SIZE) &&
			    cmds[i].p_filesz >= PMD_SIZE && p_align < PMD_SIZE))
				p_align = PMD_SIZE;

But this conditional is too complex to be at this level of indentation. Factor it out into a helper:

			if (align_to_pmd(cmds) && p_align < PMD_SIZE)
				p_align = PMD_SIZE;
author
#2 lore raw
Quoted 7 lines · Matthew Wilcox

fs/Kconfig.binfmt config ELFCORE · line 79

Quoted 3 lines · WANG Rui
fs/Kconfig.binfmt +3 -0
+config ELF_RO_LOAD_THP_ALIGNMENT
+     bool "Align read-only ELF load segments for THP (EXPERIMENTAL)"
+     depends on READ_ONLY_THP_FOR_FS

This doesn't deserve a config option.

This optimization is not entirely free. Increasing PT_LOAD alignment can waste virtual address space, which is especially significant on 32-bit systems, and it also reduces ASLR entropy by limiting the number of possible load addresses.

In addition, coarser alignment may have secondary microarchitectural effects (eg. on indirect branch prediction), depending on the workload. Because this change affects address space layout and security-related properties, providing users with a way to opt out is reasonable, rather than making it completely unconditional. This behavior fits naturally under READ_ONLY_THP_FOR_FS.

Quoted 4 lines
Quoted 2 lines

+#if defined(CONFIG_ELF_RO_LOAD_THP_ALIGNMENT) && PMD_SIZE <= SZ_32M

Why 32MB? This is weird and not justified anywhere.

The 32MB limit is intended as a practical upper bound. With 16KB base pages, which are commonly used on some 64-bit architectures, PMD_SIZE is 32MB. Larger PMD sizes (eg. 128MB or more with 32KB pages) exist but represent relatively extreme configurations where alignment costs, virtual address space padding, and ASLR entropy loss grow disproportionately relative to the expected THP benefit.

On 32-bit systems, this issue is even more pronounced: PMD_SIZE can reach 128MB or 256MB depending on configuration, which is clearly unreasonable as an alignment requirement given the already constrained virtual address space. The cap therefore avoids these pathological cases while still covering all realistic and widely deployed PMD huge page sizes.

Thanks,
Rui

Quoted 19 lines

fs/binfmt_elf.c static unsigned long maximum_alignment(struct elf_phdr *cmds, int nr) · line 509

Quoted 5 lines
fs/binfmt_elf.c +4 -0
+                     if (hugepage_global_always() && !(cmds[i].p_flags & PF_W)
+                             && IS_ALIGNED(cmds[i].p_vaddr | cmds[i].p_offset, PMD_SIZE)
+                             && cmds[i].p_filesz >= PMD_SIZE && p_align < PMD_SIZE)
+                             p_align = PMD_SIZE;

Normal style is to put the '&&' at the end of the line:

                        if (!(cmds[i].p_flags & PF_W) &&
                            IS_ALIGNED(cmds[i].p_vaddr | cmds[i].p_offset, PMD_SIZE) &&
                            cmds[i].p_filesz >= PMD_SIZE && p_align < PMD_SIZE))
                                p_align = PMD_SIZE;

But this conditional is too complex to be at this level of indentation. Factor it out into a helper:

                        if (align_to_pmd(cmds) && p_align < PMD_SIZE)
                                p_align = PMD_SIZE;
#3 lore raw
Quoted 14 lines · hev
Quoted 7 lines · Matthew Wilcox

fs/Kconfig.binfmt config ELFCORE · line 79

Quoted 3 lines · WANG Rui
fs/Kconfig.binfmt +3 -0
+config ELF_RO_LOAD_THP_ALIGNMENT
+     bool "Align read-only ELF load segments for THP (EXPERIMENTAL)"
+     depends on READ_ONLY_THP_FOR_FS

This doesn't deserve a config option.

This optimization is not entirely free. Increasing PT_LOAD alignment can waste virtual address space, which is especially significant on 32-bit systems, and it also reduces ASLR entropy by limiting the number of possible load addresses.

Quoted 6 lines

In addition, coarser alignment may have secondary microarchitectural effects (eg. on indirect branch prediction), depending on the workload. Because this change affects address space layout and security-related properties, providing users with a way to opt out is reasonable, rather than making it completely unconditional. This behavior fits naturally under READ_ONLY_THP_FOR_FS.

This isn't reasonable at all. You're asking distro maintainers to make a decision they have insufficient information to make. Almost none of our users compile their own kernels, and frankly those that do don't have enough information to make an informed decision about which way to choose.

So if we're going to have a way to opt in/out, it needs to be something different. Maybe a heuristic based on size of text segment? Maybe an ELF flag? But then, if we're going to modify the binary, why not just set p_align and then we don't need this patch at all?

author
#4 lore raw
Quoted 28 lines · Matthew Wilcox
Quoted 14 lines · hev
Quoted 7 lines · Matthew Wilcox

fs/Kconfig.binfmt config ELFCORE · line 79

Quoted 3 lines · WANG Rui
fs/Kconfig.binfmt +3 -0
+config ELF_RO_LOAD_THP_ALIGNMENT
+     bool "Align read-only ELF load segments for THP (EXPERIMENTAL)"
+     depends on READ_ONLY_THP_FOR_FS

This doesn't deserve a config option.

This optimization is not entirely free. Increasing PT_LOAD alignment can waste virtual address space, which is especially significant on 32-bit systems, and it also reduces ASLR entropy by limiting the number of possible load addresses.

Quoted 6 lines

In addition, coarser alignment may have secondary microarchitectural effects (eg. on indirect branch prediction), depending on the workload. Because this change affects address space layout and security-related properties, providing users with a way to opt out is reasonable, rather than making it completely unconditional. This behavior fits naturally under READ_ONLY_THP_FOR_FS.

This isn't reasonable at all. You're asking distro maintainers to make a decision they have insufficient information to make. Almost none of our users compile their own kernels, and frankly those that do don't have enough information to make an informed decision about which way to choose.

Quoted 4 lines

So if we're going to have a way to opt in/out, it needs to be something different. Maybe a heuristic based on size of text segment? Maybe an ELF flag? But then, if we're going to modify the binary, why not just set p_align and then we don't need this patch at all?

I agree that a compile-time config is not a good fit here, and I’m fine with dropping it in v2.

Relying on ELF-side changes is problematic. Increasing p_align in the linker inflates file size due to extra padding, and more importantly it cannot help existing binaries. The loader is therefore the only place where this can be done without ABI changes or file size regressions.

The logic here is deliberately strict rather than heuristic: the segment must be read-only, at least PMD_SIZE in length, and PMD_SIZE is capped at 32MB to avoid pathological address space waste. If these conditions are not met, the layout is unchanged.

I don’t see a reliable way to make a smarter decision at load time without workload knowledge. With READ_ONLY_THP_FOR_FS already limiting the scope and the THP policy applied at runtime, this keeps the behavior constrained.

Thanks,
Rui

#5 lore raw
Quoted 40 lines · hev
Quoted 20 lines · Matthew Wilcox
Quoted 6 lines · hev

This optimization is not entirely free. Increasing PT_LOAD alignment can waste virtual address space, which is especially significant on 32-bit systems, and it also reduces ASLR entropy by limiting the number of possible load addresses.

Quoted 6 lines

In addition, coarser alignment may have secondary microarchitectural effects (eg. on indirect branch prediction), depending on the workload. Because this change affects address space layout and security-related properties, providing users with a way to opt out is reasonable, rather than making it completely unconditional. This behavior fits naturally under READ_ONLY_THP_FOR_FS.

This isn't reasonable at all. You're asking distro maintainers to make a decision they have insufficient information to make. Almost none of our users compile their own kernels, and frankly those that do don't have enough information to make an informed decision about which way to choose.

Quoted 4 lines

So if we're going to have a way to opt in/out, it needs to be something different. Maybe a heuristic based on size of text segment? Maybe an ELF flag? But then, if we're going to modify the binary, why not just set p_align and then we don't need this patch at all?

I agree that a compile-time config is not a good fit here, and I’m fine with dropping it in v2.

Relying on ELF-side changes is problematic. Increasing p_align in the linker inflates file size due to extra padding, and more importantly it cannot help existing binaries. The loader is therefore the only place where this can be done without ABI changes or file size regressions.

The logic here is deliberately strict rather than heuristic: the segment must be read-only, at least PMD_SIZE in length, and PMD_SIZE is capped at 32MB to avoid pathological address space waste. If these conditions are not met, the layout is unchanged.

Quoted 4 lines

I don’t see a reliable way to make a smarter decision at load time without workload knowledge. With READ_ONLY_THP_FOR_FS already limiting the scope and the THP policy applied at runtime, this keeps the behavior constrained.

A note that READ_ONLY_THP_FOR_FS will likely go away soon.

Signature · 3 lines
Cheers,

David