mm: remove VM_EXEC requirement for THP eligibility
Subsystems (2)
PATCH 1/1 mm: remove VM_EXEC requirement for THP eligibility
Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking in transparent_hugepage_enabled()") introduced the VM_EXEC requirement, which is not strictly needed.
lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD segment without the PF_X flag, which should be eligible for THP.
Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only PT_LOAD segment, which should be eligible for THP as well.
--- include/linux/huge_mm.h | 1 - 1 file changed, 1 deletion(-)
include/linux/huge_mm.h +0 -1
@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) inode = vma->vm_file->f_inode;return (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS )) && - (vma->vm_flags &VM_EXEC ) && !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); }
Signature · 1 line
2.43.0.472.g3155946c3a-goog
Quoted 5 lines · Fangrui Song
Commit
e6be37b2e7bd("mm/huge_memory.c: add missing read-only THP checking intransparent_hugepage_enabled()") introduced theVM_EXECrequirement, which is not strictly needed.
Quoted 7 lines
lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only
PT_LOADsegment without thePF_Xflag, which should be eligible for THP.Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only
PT_LOADsegment, which should be eligible for THP as well.
Yeah, it doesn't have to be VM_EXEC. The original implementation was restricted to VM_EXEC to minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?
Quoted 21 lines
Signed-off-by Fangrui Song <[email protected]>--- include/linux/huge_mm.h | 1 - 1 file changed, 1 deletion(-)include/linux/huge_mm.h +0
-2@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) inode = vma->vm_file->f_inode;return (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS )) && - (vma->vm_flags &VM_EXEC ) && !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); } --2.43.0.472.g3155946c3a-goog
Quoted 11 lines · Yang Shi
Quoted 9 lines · Fangrui Song
Commit
e6be37b2e7bd("mm/huge_memory.c: add missing read-only THP checking intransparent_hugepage_enabled()") introduced theVM_EXECrequirement, which is not strictly needed.lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only
PT_LOADsegment without thePF_Xflag, which should be eligible for THP.
Quoted 8 lines
Quoted 3 lines
Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only
PT_LOADsegment, which should be eligible for THP as well.Yeah, it doesn't have to be
VM_EXEC. The original implementation was restricted toVM_EXECto minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?
Hi Yang,
Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this VM_EXEC condition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment about CONFIG_READ_ONLY_THP_FOR_FS in
https://mazzo.li/posts/check-huge-page.html).
As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the VM_EXEC condition :)
On my Arch linux machine, the r--p page gets split if I invoke madvise(__ehdr_start, HPAGE_SIZE, MADV_HUGEPAGE) I haven't figured out why it behaves so in the presence of the VM_EXEC check.
% g++ test.cc -o ~/tmp/test -O2 -fuse-ld=lld -Wl,-z,max-page-size=2097152 && sudo ~/tmp/test __ehdr_start: 0x55f3b1c00000 55f3b1c00000-55f3b1e00000 r--p 00000000 103:03 555277119
/home/ray/tmp/test
55f3b1e00000-55f3b1e01000 r--p 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2000000-55f3b2002000 r-xp 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2201000-55f3b2202000 r--p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b2401000-55f3b2402000 rw-p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b3a9a000-55f3b3abb000 rw-p 00000000 00:00 0 [heap]
It'd be greatly appreciated if someone familiar with CONFIG_READ_ONLY_THP_FOR_FS could provide some notes on how to use this feature:)
Quoted 21 lines
Quoted 21 lines
Signed-off-by Fangrui Song <[email protected]>--- include/linux/huge_mm.h | 1 - 1 file changed, 1 deletion(-)include/linux/huge_mm.h +0
-2@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) inode = vma->vm_file->f_inode;return (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS )) && - (vma->vm_flags &VM_EXEC ) && !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); } --2.43.0.472.g3155946c3a-goog
Signature · 1 line
宋方睿
Quoted 24 lines · Fangrui Song
Quoted 11 lines · Yang Shi
Quoted 9 lines · Fangrui Song
Commit
e6be37b2e7bd("mm/huge_memory.c: add missing read-only THP checking intransparent_hugepage_enabled()") introduced theVM_EXECrequirement, which is not strictly needed.lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only
PT_LOADsegment without thePF_Xflag, which should be eligible for THP.Quoted 8 lines
Quoted 3 lines
Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only
PT_LOADsegment, which should be eligible for THP as well.Yeah, it doesn't have to be
VM_EXEC. The original implementation was restricted toVM_EXECto minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?Hi Yang,
Quoted 7 lines
Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this
VM_EXECcondition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment aboutCONFIG_READ_ONLY_THP_FOR_FSin
https://mazzo.li/posts/check-huge-page.html).
Thanks for sharing the article, learnt something about linker and loader.
Quoted 4 lines
As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the
VM_EXECcondition :)
Yeah, as long as the use case is valid, it is definitely fine to lift the restriction.
Quoted 4 lines
On my Arch linux machine, the r--p page gets split if I invoke
madvise(__ehdr_start, HPAGE_SIZE, MADV_HUGEPAGE)I haven't figured out why it behaves so in the presence of theVM_EXECcheck.
What do you mean about "split"? THP got split into small pages? It depends on the address of __ehdr_start. If it is in the middle of a VMA, the VMA is going to be split due to the different huge page attributes.
Quoted 17 lines
% g++ test.cc -o ~/tmp/test -O2 -fuse-ld=lld -Wl,-z,max-page-size=2097152 && sudo ~/tmp/test
__ehdr_start:0x55f3b1c0000055f3b1c00000-55f3b1e00000 r--p 00000000 103:03 555277119
/home/ray/tmp/test
55f3b1e00000-55f3b1e01000 r--p 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2000000-55f3b2002000 r-xp 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2201000-55f3b2202000 r--p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b2401000-55f3b2402000 rw-p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b3a9a000-55f3b3abb000 rw-p 00000000 00:00 0 [heap]
Quoted 3 lines
It'd be greatly appreciated if someone familiar with
CONFIG_READ_ONLY_THP_FOR_FScould provide some notes on how to use this feature:)
I think your blog covered all the points. If you don't mind, you could add some notes in Documentation/admin-guide/mm/transhuge.rst.
Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Signed-off-by Fangrui Song <[email protected]>--- include/linux/huge_mm.h | 1 - 1 file changed, 1 deletion(-)include/linux/huge_mm.h +0
-0@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) inode = vma->vm_file->f_inode;Quoted 8 lines
Quoted 8 lines
include/linux/huge_mm.h +0
-2return (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS )) && - (vma->vm_flags &VM_EXEC ) && !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); } --2.43.0.472.g3155946c3a-goog
--
宋方睿
Quoted 34 lines · Yang Shi
Quoted 24 lines · Fangrui Song
Quoted 11 lines · Yang Shi
Quoted 9 lines · Fangrui Song
Commit
e6be37b2e7bd("mm/huge_memory.c: add missing read-only THP checking intransparent_hugepage_enabled()") introduced theVM_EXECrequirement, which is not strictly needed.lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only
PT_LOADsegment without thePF_Xflag, which should be eligible for THP.Quoted 8 lines
Quoted 3 lines
Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only
PT_LOADsegment, which should be eligible for THP as well.Yeah, it doesn't have to be
VM_EXEC. The original implementation was restricted toVM_EXECto minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?Hi Yang,
Quoted 7 lines
Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this
VM_EXECcondition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment aboutCONFIG_READ_ONLY_THP_FOR_FSin
https://mazzo.li/posts/check-huge-page.html).
Quoted 1 line
Thanks for sharing the article, learnt something about linker and loader.
BTW, kernel should try to map the segments (the size has to be >= 2M) to 2M aligned address even though the loading address is not 2M aligned for ext4/xfs/btrfs since v5.18. See commit 1854bc6e2420 ("mm/readahead: Align file mappings for non-DAX"). Did you see this behavior?
Quoted 70 lines
Quoted 5 lines
As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the
VM_EXECcondition :)Yeah, as long as the use case is valid, it is definitely fine to lift the restriction.
Quoted 4 lines
On my Arch linux machine, the r--p page gets split if I invoke
madvise(__ehdr_start, HPAGE_SIZE, MADV_HUGEPAGE)I haven't figured out why it behaves so in the presence of theVM_EXECcheck.What do you mean about "split"? THP got split into small pages? It depends on the address of
__ehdr_start. If it is in the middle of a VMA, the VMA is going to be split due to the different huge page attributes.Quoted 17 lines
% g++ test.cc -o ~/tmp/test -O2 -fuse-ld=lld -Wl,-z,max-page-size=2097152 && sudo ~/tmp/test
__ehdr_start:0x55f3b1c0000055f3b1c00000-55f3b1e00000 r--p 00000000 103:03 555277119
/home/ray/tmp/test
55f3b1e00000-55f3b1e01000 r--p 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2000000-55f3b2002000 r-xp 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2201000-55f3b2202000 r--p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b2401000-55f3b2402000 rw-p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b3a9a000-55f3b3abb000 rw-p 00000000 00:00 0 [heap]Quoted 3 lines
It'd be greatly appreciated if someone familiar with
CONFIG_READ_ONLY_THP_FOR_FScould provide some notes on how to use this feature:)I think your blog covered all the points. If you don't mind, you could add some notes in
Documentation/admin-guide/mm/transhuge.rst.Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
Signed-off-by Fangrui Song <[email protected]>--- include/linux/huge_mm.h | 1 - 1 file changed, 1 deletion(-)include/linux/huge_mm.h +0
-0@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) inode = vma->vm_file->f_inode;Quoted 8 lines
Quoted 8 lines
include/linux/huge_mm.h +0
-2return (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS )) && - (vma->vm_flags &VM_EXEC ) && !inode_is_open_for_write (inode) &&S_ISREG (inode->i_mode); } --2.43.0.472.g3155946c3a-goog
--
宋方睿
Quoted 7 lines · Fangrui Song
Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this
VM_EXECcondition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment aboutCONFIG_READ_ONLY_THP_FOR_FSin
https://mazzo.li/posts/check-huge-page.html).
CONFIG_READ_ONLY_THP_FOR_FS is a preliminary hack which solves some problems. The real solution is using large folios, which at the moment means that you should test on XFS or AFS; filesystem authors have not been enthusiastic about adding support to their filesystems so far.
In your blog, you write:
: In -z noseparate-code layouts, the file content starts somewhere at : the first page, potentially wasting half a huge page on unrelated : content. Switching to -z separate-code allows reclaiming the benefits : of the half huge page but increases the file size. Balancing : these aspects poses a challenge. One potential solution is using : fallocate(FALLOC_FL_PUNCH_HOLE), which introduces complexity into the : linker. However, this approach feels like a workaround to address a : kernel limitation. It would be preferable if a file-backed huge page : didn't necessitate a file offset aligned to a huge page boundary.
You should distinguish between file size (ie st_size in stat(3)) and amount of space occupied on storage (st_blocks). The linker should be fine with creating a sparse file. If it doesn't, cp --sparse will do the trick.
Yes, it's a kernel limitation that folios have to be aligned within the file as well as in both virtual and physical address space. It's a huge complexity win to do that; I don't think we'd be able to tile the page cache effectively if we allowed folios to be placed at arbitrary offsets (I think it turns into a knapsack problem at that point).
Quoted 3 lines
As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the
VM_EXECcondition :)
I'm not particularly enthusiastic about making CONFIG_READ_ONLY_THP_FOR_FS better. Large folios are the future. Indeed, I'd like to see CONFIG_READ_ONLY_THP_FOR_FS go away in the next year or two once btrfs and ext4 have support for large folios.