Skip to contentSkip to conversation

mm: remove VM_EXEC requirement for THP eligibility

PATCH Applied Fangrui Song 3 participants 6 messages Dec 20, 2023 – Dec 21, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: remove VM_EXEC requirement for THP eligibility

Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking in transparent_hugepage_enabled()") introduced the VM_EXEC requirement, which is not strictly needed.

lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD segment without the PF_X flag, which should be eligible for THP.

Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only PT_LOAD segment, which should be eligible for THP as well.

---
 include/linux/huge_mm.h | 1 -
 1 file changed, 1 deletion(-)
include/linux/huge_mm.h +0 -1
index fa0350b0812a..4c9e67e9000f 100644
@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
 	inode = vma->vm_file->f_inode;
 
 	return (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS)) &&
-	       (vma->vm_flags & VM_EXEC) &&
 	       !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
 }
 
Signature · 1 line
2.43.0.472.g3155946c3a-goog
#1 lore raw
Quoted 5 lines · Fangrui Song

Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking in transparent_hugepage_enabled()") introduced the VM_EXEC requirement, which is not strictly needed.

Quoted 7 lines

lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD segment without the PF_X flag, which should be eligible for THP.

Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only PT_LOAD segment, which should be eligible for THP as well.

Yeah, it doesn't have to be VM_EXEC. The original implementation was restricted to VM_EXEC to minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?

Quoted 21 lines
---
 include/linux/huge_mm.h | 1 -
 1 file changed, 1 deletion(-)
include/linux/huge_mm.h +0 -2
index fa0350b0812a..4c9e67e9000f 100644
@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
        inode = vma->vm_file->f_inode;
 
        return (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS)) &&
-              (vma->vm_flags & VM_EXEC) &&
               !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
 }
 
--

2.43.0.472.g3155946c3a-goog

#2 lore raw
Quoted 11 lines · Yang Shi
Quoted 9 lines · Fangrui Song

Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking in transparent_hugepage_enabled()") introduced the VM_EXEC requirement, which is not strictly needed.

lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD segment without the PF_X flag, which should be eligible for THP.

Quoted 8 lines
Quoted 3 lines

Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only PT_LOAD segment, which should be eligible for THP as well.

Yeah, it doesn't have to be VM_EXEC. The original implementation was restricted to VM_EXEC to minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?

Hi Yang,

Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this VM_EXEC condition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment about CONFIG_READ_ONLY_THP_FOR_FS in
https://mazzo.li/posts/check-huge-page.html).

As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the VM_EXEC condition :)

On my Arch linux machine, the r--p page gets split if I invoke madvise(__ehdr_start, HPAGE_SIZE, MADV_HUGEPAGE) I haven't figured out why it behaves so in the presence of the VM_EXEC check.

% g++ test.cc -o ~/tmp/test -O2 -fuse-ld=lld -Wl,-z,max-page-size=2097152 && sudo ~/tmp/test __ehdr_start: 0x55f3b1c00000 55f3b1c00000-55f3b1e00000 r--p 00000000 103:03 555277119
/home/ray/tmp/test
55f3b1e00000-55f3b1e01000 r--p 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2000000-55f3b2002000 r-xp 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2201000-55f3b2202000 r--p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b2401000-55f3b2402000 rw-p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b3a9a000-55f3b3abb000 rw-p 00000000 00:00 0 [heap]

It'd be greatly appreciated if someone familiar with CONFIG_READ_ONLY_THP_FOR_FS could provide some notes on how to use this feature:)

Quoted 21 lines
Quoted 21 lines
---
 include/linux/huge_mm.h | 1 -
 1 file changed, 1 deletion(-)
include/linux/huge_mm.h +0 -2
index fa0350b0812a..4c9e67e9000f 100644
@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
        inode = vma->vm_file->f_inode;
 
        return (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS)) &&
-              (vma->vm_flags & VM_EXEC) &&
               !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
 }
 
--

2.43.0.472.g3155946c3a-goog

Signature · 1 line
宋方睿
#3 lore raw
Quoted 24 lines · Fangrui Song
Quoted 11 lines · Yang Shi
Quoted 9 lines · Fangrui Song

Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking in transparent_hugepage_enabled()") introduced the VM_EXEC requirement, which is not strictly needed.

lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD segment without the PF_X flag, which should be eligible for THP.

Quoted 8 lines
Quoted 3 lines

Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only PT_LOAD segment, which should be eligible for THP as well.

Yeah, it doesn't have to be VM_EXEC. The original implementation was restricted to VM_EXEC to minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?

Hi Yang,

Quoted 7 lines

Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this VM_EXEC condition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment about CONFIG_READ_ONLY_THP_FOR_FS in
https://mazzo.li/posts/check-huge-page.html).

Thanks for sharing the article, learnt something about linker and loader.

Quoted 4 lines

As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the VM_EXEC condition :)

Yeah, as long as the use case is valid, it is definitely fine to lift the restriction.

Quoted 4 lines

On my Arch linux machine, the r--p page gets split if I invoke madvise(__ehdr_start, HPAGE_SIZE, MADV_HUGEPAGE) I haven't figured out why it behaves so in the presence of the VM_EXEC check.

What do you mean about "split"? THP got split into small pages? It depends on the address of __ehdr_start. If it is in the middle of a VMA, the VMA is going to be split due to the different huge page attributes.

Quoted 17 lines

% g++ test.cc -o ~/tmp/test -O2 -fuse-ld=lld -Wl,-z,max-page-size=2097152 && sudo ~/tmp/test __ehdr_start: 0x55f3b1c00000 55f3b1c00000-55f3b1e00000 r--p 00000000 103:03 555277119
/home/ray/tmp/test
55f3b1e00000-55f3b1e01000 r--p 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2000000-55f3b2002000 r-xp 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2201000-55f3b2202000 r--p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b2401000-55f3b2402000 rw-p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b3a9a000-55f3b3abb000 rw-p 00000000 00:00 0 [heap]

Quoted 3 lines

It'd be greatly appreciated if someone familiar with CONFIG_READ_ONLY_THP_FOR_FS could provide some notes on how to use this feature:)

I think your blog covered all the points. If you don't mind, you could add some notes in Documentation/admin-guide/mm/transhuge.rst.

Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
---
 include/linux/huge_mm.h | 1 -
 1 file changed, 1 deletion(-)
include/linux/huge_mm.h +0 -0
index fa0350b0812a..4c9e67e9000f 100644
@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
        inode = vma->vm_file->f_inode;
 
Quoted 8 lines
Quoted 8 lines
include/linux/huge_mm.h +0 -2
        return (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS)) &&
-              (vma->vm_flags & VM_EXEC) &&
               !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
 }
 
--

2.43.0.472.g3155946c3a-goog

--
宋方睿

#4 lore raw
Quoted 34 lines · Yang Shi
Quoted 24 lines · Fangrui Song
Quoted 11 lines · Yang Shi
Quoted 9 lines · Fangrui Song

Commit e6be37b2e7bd ("mm/huge_memory.c: add missing read-only THP checking in transparent_hugepage_enabled()") introduced the VM_EXEC requirement, which is not strictly needed.

lld's default --rosegment option and GNU ld's -z separate-code option (default on Linux/x86 since binutils 2.31) create a read-only PT_LOAD segment without the PF_X flag, which should be eligible for THP.

Quoted 8 lines
Quoted 3 lines

Certain architectures support medium and large code models, where .lrodata may be placed in a separate read-only PT_LOAD segment, which should be eligible for THP as well.

Yeah, it doesn't have to be VM_EXEC. The original implementation was restricted to VM_EXEC to minimize the blast radius and the targe use case is for large text segments. Out of curiosity, did you see any noticeable improvement with this change?

Hi Yang,

Quoted 7 lines

Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this VM_EXEC condition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment about CONFIG_READ_ONLY_THP_FOR_FS in
https://mazzo.li/posts/check-huge-page.html).

Quoted 1 line

Thanks for sharing the article, learnt something about linker and loader.

BTW, kernel should try to map the segments (the size has to be >= 2M) to 2M aligned address even though the loading address is not 2M aligned for ext4/xfs/btrfs since v5.18. See commit 1854bc6e2420 ("mm/readahead: Align file mappings for non-DAX"). Did you see this behavior?

Quoted 70 lines
Quoted 5 lines

As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the VM_EXEC condition :)

Yeah, as long as the use case is valid, it is definitely fine to lift the restriction.

Quoted 4 lines

On my Arch linux machine, the r--p page gets split if I invoke madvise(__ehdr_start, HPAGE_SIZE, MADV_HUGEPAGE) I haven't figured out why it behaves so in the presence of the VM_EXEC check.

What do you mean about "split"? THP got split into small pages? It depends on the address of __ehdr_start. If it is in the middle of a VMA, the VMA is going to be split due to the different huge page attributes.

Quoted 17 lines

% g++ test.cc -o ~/tmp/test -O2 -fuse-ld=lld -Wl,-z,max-page-size=2097152 && sudo ~/tmp/test __ehdr_start: 0x55f3b1c00000 55f3b1c00000-55f3b1e00000 r--p 00000000 103:03 555277119
/home/ray/tmp/test
55f3b1e00000-55f3b1e01000 r--p 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2000000-55f3b2002000 r-xp 00200000 103:03 555277119
/home/ray/tmp/test
55f3b2201000-55f3b2202000 r--p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b2401000-55f3b2402000 rw-p 00201000 103:03 555277119
/home/ray/tmp/test
55f3b3a9a000-55f3b3abb000 rw-p 00000000 00:00 0 [heap]

Quoted 3 lines

It'd be greatly appreciated if someone familiar with CONFIG_READ_ONLY_THP_FOR_FS could provide some notes on how to use this feature:)

I think your blog covered all the points. If you don't mind, you could add some notes in Documentation/admin-guide/mm/transhuge.rst.

Quoted 27 lines
Quoted 14 lines
Quoted 14 lines
---
 include/linux/huge_mm.h | 1 -
 1 file changed, 1 deletion(-)
include/linux/huge_mm.h +0 -0
index fa0350b0812a..4c9e67e9000f 100644
@@ -126,7 +126,6 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma)
        inode = vma->vm_file->f_inode;
 
Quoted 8 lines
Quoted 8 lines
include/linux/huge_mm.h +0 -2
        return (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS)) &&
-              (vma->vm_flags & VM_EXEC) &&
               !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode);
 }
 
--

2.43.0.472.g3155946c3a-goog

--
宋方睿

#5 lore raw
In reply to Fangrui Song
Quoted 7 lines · Fangrui Song

Thanks for the comment. Frankly, I am not familiar with huge pages... I noticed this VM_EXEC condition when I was writing this hugepage-related section in
https://maskray.me/blog/2023-12-17-exploring-the-section-layout-in-linker-output#transparent-huge-pages-for-mapped-files
(Thanks to Alexander Monakov's comment about CONFIG_READ_ONLY_THP_FOR_FS in
https://mazzo.li/posts/check-huge-page.html).

CONFIG_READ_ONLY_THP_FOR_FS is a preliminary hack which solves some problems. The real solution is using large folios, which at the moment means that you should test on XFS or AFS; filesystem authors have not been enthusiastic about adding support to their filesystems so far.

In your blog, you write:

: In -z noseparate-code layouts, the file content starts somewhere at : the first page, potentially wasting half a huge page on unrelated : content. Switching to -z separate-code allows reclaiming the benefits : of the half huge page but increases the file size. Balancing : these aspects poses a challenge. One potential solution is using : fallocate(FALLOC_FL_PUNCH_HOLE), which introduces complexity into the : linker. However, this approach feels like a workaround to address a : kernel limitation. It would be preferable if a file-backed huge page : didn't necessitate a file offset aligned to a huge page boundary.

You should distinguish between file size (ie st_size in stat(3)) and amount of space occupied on storage (st_blocks). The linker should be fine with creating a sparse file. If it doesn't, cp --sparse will do the trick.

Yes, it's a kernel limitation that folios have to be aligned within the file as well as in both virtual and physical address space. It's a huge complexity win to do that; I don't think we'd be able to tile the page cache effectively if we allowed folios to be placed at arbitrary offsets (I think it turns into a knapsack problem at that point).

Quoted 3 lines

As dTLB for read-only data is also an important optimization of file-backed THP, it seems straightforward that we should drop the VM_EXEC condition :)

I'm not particularly enthusiastic about making CONFIG_READ_ONLY_THP_FOR_FS better. Large folios are the future. Indeed, I'd like to see CONFIG_READ_ONLY_THP_FOR_FS go away in the next year or two once btrfs and ext4 have support for large folios.