mm, thp: Relax the VM_DENYWRITE constraint on file-backed THPs
Subsystems (3)
PATCH 1/1 mm, thp: Relax the VM_DENYWRITE constraint on file-backed THPs
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write in do_dentry_open(). Since sys_mmap ignores MAP_DENYWRITE, this constrains the use of file THPs to vmas produced by execve().
Systems that make heavy use of shared libraries (e.g. Android) are unable to apply VM_DENYWRITE through the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.
This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.
Restricting the use of THPs to executable mappings eliminates the risk that a read-only file later opened for write would encounter significant latencies due to page cache truncation.
The ld linker flag '-z max-page-size=(hugepage size)' can be used to produce executables with the necessary layout. The dynamic linker must map these file's segments at a hugepage size aligned vma for the mapping to be backed with THPs.
--- fs/open.c | 13 +++++++++++-- mm/khugepaged.c | 16 +++++++++++++++- 2 files changed, 26 insertions(+), 3 deletions(-)
fs/open.c +11 -2
@@ -852,8 +852,17 @@ static int do_dentry_open(struct file *f,* XXX: Huge page cache doesn't support writing yet. Drop all page * cache for this file before processing writes. */ -if ((f->f_mode &FMODE_WRITE ) &&filemap_nr_thps (inode->i_mapping)) -truncate_pagecache (inode,0 ); +if (f->f_mode &FMODE_WRITE ) { +/* +* Paired with smp_mb() in collapse_file() to ensure nr_thps +* is up to date and the update to i_writecount by +* get_write_access() is visible. Ensures subsequent insertion +* of THPs into the page cache will fail. +*/ +smp_mb (); +if (filemap_nr_thps (inode->i_mapping)) +truncate_pagecache (inode,0 ); + }return 0 ;
mm/khugepaged.c +15 -1
@@ -459,7 +459,8 @@ static bool hugepage_vma_check(struct vm_area_struct *vma,/* Read-only file mappings need to be aligned for THP to work. */ if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && vma->vm_file && - (vm_flags &VM_DENYWRITE )) { + !inode_is_open_for_write (vma->vm_file->f_inode) && + (vm_flags &VM_EXEC )) {return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff,HPAGE_PMD_NR ); }
@@ -1872,6 +1873,19 @@ static void collapse_file(struct mm_struct *mm,else {__mod_lruvec_page_state (new_page,NR_FILE_THPS , nr);filemap_nr_thps_inc (mapping); +/* +* Paired with smp_mb() in do_dentry_open() to ensure +* i_writecount is up to date and the update to nr_thps is +* visible. Ensures the page cache will be truncated if the +* file is opened writable. +*/ +smp_mb (); +if (inode_is_open_for_write (mapping->host)) { + result =SCAN_FAIL ; +__mod_lruvec_page_state (new_page,NR_FILE_THPS , -nr); +filemap_nr_thps_dec (mapping); + goto xa_locked; + } }if (nr_none) {
Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
Quoted 9 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().
Quoted 8 lines
Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.
Thanks for working on this. We could also use this in many data center workloads.
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?
Thanks,
Song
Quoted 5 lines
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?
That's correct, while a file is opened for write it will refault into 4kB pages and block use of THPs. Once all writers complete (i_writecount <=0), the file can fault into THPs again and khugepaged can collapse existing page ranges provided that it can successfully allocate new huge pages.
From,
Collin
Quoted 31 lines · Song Liu
Quoted 9 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Quoted 8 lines
Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Thanks for working on this. We could also use this in many data center workloads.
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?Thanks,
Song
Quoted 10 lines · Collin Fijalkovich
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?That's correct, while a file is opened for write it will refault into 4kB pages and block use of THPs. Once all writers complete (
i_writecount<=0), the file can fault into THPs again and khugepaged can collapse existing page ranges provided that it can successfully allocate new huge pages.
Will it be a problem if a slow writer (say a slow scp) writes to the shared library while the shared library is in use?
Thanks,
Song
Quoted 35 lines
From,
CollinOn Mon, Mar 22, 2021 at 4:55 PM Song Liu <[email protected]> wrote:
Quoted 9 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Quoted 8 lines
Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Thanks for working on this. We could also use this in many data center workloads.
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?Thanks,
Song
There will be an immediate performance hit when the file is opened for write, as its associated pages are removed from the page cache. While the writer is present there will be the usual overhead of using 4kB pages instead of THPs, but there should not be an additional penalty. It is problematic if a file is repeatedly opened for write, as it will need to refault each time.
- Collin
Quoted 54 lines · Song Liu
Quoted 11 lines · Collin Fijalkovich
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?That's correct, while a file is opened for write it will refault into 4kB pages and block use of THPs. Once all writers complete (
i_writecount<=0), the file can fault into THPs again and khugepaged can collapse existing page ranges provided that it can successfully allocate new huge pages.Will it be a problem if a slow writer (say a slow scp) writes to the shared library while the shared library is in use?
Thanks,
SongQuoted 35 lines
From,
CollinOn Mon, Mar 22, 2021 at 4:55 PM Song Liu <[email protected]> wrote:
Quoted 9 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Quoted 8 lines
Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Thanks for working on this. We could also use this in many data center workloads.
Question: when we use this on shared library, the library is still writable. When the
shared library is opened for write, these pages will refault in as 4kB pages, right?Thanks,
Song
I like this, it reminds me of the changes I proposed a few years ago to try to automatically map read-only text regions of appropriate sizes and alignment with THPs.
My concern had always been whether commercial software and distro vendors would buy into supplying the appropriate linker flags when compiling; your solution is at the very least a good head start down that road.
Matthew Wilcox and I had noticed a lot of ordinary apps such as gcc, Chrome and Firefox would benefit from such mappings; have you tried building any of those with the appropriate linker flag to see how they might benefit from the change?
Thanks,
-- Bill
Quoted 96 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Restricting the use of THPs to executable mappings eliminates the risk that a read-only file later opened for write would encounter significant latencies due to page cache truncation.
The ld linker flag '-z max-page-size=(hugepage size)' can be used to produce executables with the necessary layout. The dynamic linker must map these file's segments at a hugepage size aligned vma for the mapping to be backed with THPs.
Signed-off-by Collin Fijalkovich <[email protected]>---
fs/open.c| 13 +++++++++++--mm/khugepaged.c| 16 +++++++++++++++-2 files changed, 26 insertions(+), 3 deletions(-)fs/open.c +11
-2@@ -852,8 +852,17 @@ static int do_dentry_open(struct file *f,* XXX: Huge page cache doesn't support writing yet. Drop all page * cache for this file before processing writes. */ -if ((f->f_mode &FMODE_WRITE ) &&filemap_nr_thps (inode->i_mapping)) -truncate_pagecache (inode,0 ); +if (f->f_mode &FMODE_WRITE ) { +/* +* Paired with smp_mb() in collapse_file() to ensure nr_thps +* is up to date and the update to i_writecount by +* get_write_access() is visible. Ensures subsequent insertion +* of THPs into the page cache will fail. +*/ +smp_mb (); +if (filemap_nr_thps (inode->i_mapping)) +truncate_pagecache (inode,0 ); + }return 0 ;mm/khugepaged.c +15
-1@@ -459,7 +459,8 @@ static bool hugepage_vma_check(struct vm_area_struct *vma,/* Read-only file mappings need to be aligned for THP to work. */ if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && vma->vm_file && - (vm_flags &VM_DENYWRITE )) { + !inode_is_open_for_write (vma->vm_file->f_inode) && + (vm_flags &VM_EXEC )) {return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff,HPAGE_PMD_NR ); }@@ -1872,6 +1873,19 @@ static void collapse_file(struct mm_struct *mm,else {__mod_lruvec_page_state (new_page,NR_FILE_THPS , nr);filemap_nr_thps_inc (mapping); +/* +* Paired with smp_mb() in do_dentry_open() to ensure +* i_writecount is up to date and the update to nr_thps is +* visible. Ensures the page cache will be truncated if the +* file is opened writable. +*/ +smp_mb (); +if (inode_is_open_for_write (mapping->host)) { + result =SCAN_FAIL ; +__mod_lruvec_page_state (new_page,NR_FILE_THPS , -nr); +filemap_nr_thps_dec (mapping); + goto xa_locked; + } }if (nr_none) {Signature · 1 line
2.31.0.rc2.261.g7f71774620-goog
v2 has been uploaded with performance testing results:
https://lore.kernel.org/patchwork/patch/1408266/
Quoted 94 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Restricting the use of THPs to executable mappings eliminates the risk that a read-only file later opened for write would encounter significant latencies due to page cache truncation.
The ld linker flag '-z max-page-size=(hugepage size)' can be used to produce executables with the necessary layout. The dynamic linker must map these file's segments at a hugepage size aligned vma for the mapping to be backed with THPs.
Signed-off-by Collin Fijalkovich <[email protected]>--- fs/open.c | 13 +++++++++++-- mm/khugepaged.c | 16 +++++++++++++++- 2 files changed, 26 insertions(+), 3 deletions(-)fs/open.c +11
-2@@ -852,8 +852,17 @@ static int do_dentry_open(struct file *f,* XXX: Huge page cache doesn't support writing yet. Drop all page * cache for this file before processing writes. */ -if ((f->f_mode &FMODE_WRITE ) &&filemap_nr_thps (inode->i_mapping)) -truncate_pagecache (inode,0 ); +if (f->f_mode &FMODE_WRITE ) { +/* +* Paired with smp_mb() in collapse_file() to ensure nr_thps +* is up to date and the update to i_writecount by +* get_write_access() is visible. Ensures subsequent insertion +* of THPs into the page cache will fail. +*/ +smp_mb (); +if (filemap_nr_thps (inode->i_mapping)) +truncate_pagecache (inode,0 ); + }return 0 ;mm/khugepaged.c +15
-2@@ -459,7 +459,8 @@ static bool hugepage_vma_check(struct vm_area_struct *vma,/* Read-only file mappings need to be aligned for THP to work. */ if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && vma->vm_file && - (vm_flags &VM_DENYWRITE )) { + !inode_is_open_for_write (vma->vm_file->f_inode) && + (vm_flags &VM_EXEC )) {return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff,HPAGE_PMD_NR ); }@@ -1872,6 +1873,19 @@ static void collapse_file(struct mm_struct *mm,else {__mod_lruvec_page_state (new_page,NR_FILE_THPS , nr);filemap_nr_thps_inc (mapping); +/* +* Paired with smp_mb() in do_dentry_open() to ensure +* i_writecount is up to date and the update to nr_thps is +* visible. Ensures the page cache will be truncated if the +* file is opened writable. +*/ +smp_mb (); +if (inode_is_open_for_write (mapping->host)) { + result =SCAN_FAIL ; +__mod_lruvec_page_state (new_page,NR_FILE_THPS , -nr); +filemap_nr_thps_dec (mapping); + goto xa_locked; + } }if (nr_none) { --2.31.0.rc2.261.g7f71774620-goog
I saw a similar change a few years ago with my prototype:
https://lore.kernel.org/linux-mm/[email protected]/
the key being a very nice drop in iTLB-load-misses, so it looks like your code is having the right effect.
What about the call to filemap_nr_thps_dec() in unaccount_page_cache_page() - does that need an smp_mb() as well?
-- Bill
Quoted 104 lines · Collin Fijalkovich
v2 has been uploaded with performance testing results:
https://lore.kernel.org/patchwork/patch/1408266/Quoted 94 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Restricting the use of THPs to executable mappings eliminates the risk that a read-only file later opened for write would encounter significant latencies due to page cache truncation.
The ld linker flag '-z max-page-size=(hugepage size)' can be used to produce executables with the necessary layout. The dynamic linker must map these file's segments at a hugepage size aligned vma for the mapping to be backed with THPs.
Signed-off-by Collin Fijalkovich <[email protected]>---
fs/open.c| 13 +++++++++++--mm/khugepaged.c| 16 +++++++++++++++-2 files changed, 26 insertions(+), 3 deletions(-)fs/open.c +11
-2@@ -852,8 +852,17 @@ static int do_dentry_open(struct file *f,* XXX: Huge page cache doesn't support writing yet. Drop all page * cache for this file before processing writes. */ -if ((f->f_mode &FMODE_WRITE ) &&filemap_nr_thps (inode->i_mapping)) -truncate_pagecache (inode,0 ); +if (f->f_mode &FMODE_WRITE ) { +/* +* Paired with smp_mb() in collapse_file() to ensure nr_thps +* is up to date and the update to i_writecount by +* get_write_access() is visible. Ensures subsequent insertion +* of THPs into the page cache will fail. +*/ +smp_mb (); +if (filemap_nr_thps (inode->i_mapping)) +truncate_pagecache (inode,0 ); + }return 0 ;mm/khugepaged.c +15
-2@@ -459,7 +459,8 @@ static bool hugepage_vma_check(struct vm_area_struct *vma,/* Read-only file mappings need to be aligned for THP to work. */ if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && vma->vm_file && - (vm_flags &VM_DENYWRITE )) { + !inode_is_open_for_write (vma->vm_file->f_inode) && + (vm_flags &VM_EXEC )) {return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff,HPAGE_PMD_NR ); }@@ -1872,6 +1873,19 @@ static void collapse_file(struct mm_struct *mm,else {__mod_lruvec_page_state (new_page,NR_FILE_THPS , nr);filemap_nr_thps_inc (mapping); +/* +* Paired with smp_mb() in do_dentry_open() to ensure +* i_writecount is up to date and the update to nr_thps is +* visible. Ensures the page cache will be truncated if the +* file is opened writable. +*/ +smp_mb (); +if (inode_is_open_for_write (mapping->host)) { + result =SCAN_FAIL ; +__mod_lruvec_page_state (new_page,NR_FILE_THPS , -nr); +filemap_nr_thps_dec (mapping); + goto xa_locked; + } }if (nr_none) { --2.31.0.rc2.261.g7f71774620-goog
Instrumenting filemap_nr_thps_inc() should be sufficient for ensuring writable file mappings will not be THP-backed.
If filemap_nr_thps_dec() in unaccount_page_cache_page() and filemap_nr_thps() in do_dentry_open() race, the worst case is an unnecessary truncation. We could introduce a memory barrier in unaccount_page_cache_page(), but I'm unsure it would significantly mitigate the risk of spurious truncations. Barring synchronization between do_dentry_open() and ongoing page cache operations, there could be an in-progress delete_from_page_cache_batch() that has not yet updated accounting for THPs in its targeted range.
-- Collin
Quoted 119 lines · William Kucharski
I saw a similar change a few years ago with my prototype:
https://lore.kernel.org/linux-mm/[email protected]/
the key being a very nice drop in iTLB-load-misses, so it looks like your code is having the right effect.
What about the call to
filemap_nr_thps_dec()inunaccount_page_cache_page()- does that need ansmp_mb()as well?-- Bill
Quoted 104 lines · Collin Fijalkovich
v2 has been uploaded with performance testing results:
https://lore.kernel.org/patchwork/patch/1408266/Quoted 94 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Restricting the use of THPs to executable mappings eliminates the risk that a read-only file later opened for write would encounter significant latencies due to page cache truncation.
The ld linker flag '-z max-page-size=(hugepage size)' can be used to produce executables with the necessary layout. The dynamic linker must map these file's segments at a hugepage size aligned vma for the mapping to be backed with THPs.
Signed-off-by Collin Fijalkovich <[email protected]>---
fs/open.c| 13 +++++++++++--mm/khugepaged.c| 16 +++++++++++++++-2 files changed, 26 insertions(+), 3 deletions(-)fs/open.c +11
-2@@ -852,8 +852,17 @@ static int do_dentry_open(struct file *f,* XXX: Huge page cache doesn't support writing yet. Drop all page * cache for this file before processing writes. */ -if ((f->f_mode &FMODE_WRITE ) &&filemap_nr_thps (inode->i_mapping)) -truncate_pagecache (inode,0 ); +if (f->f_mode &FMODE_WRITE ) { +/* +* Paired with smp_mb() in collapse_file() to ensure nr_thps +* is up to date and the update to i_writecount by +* get_write_access() is visible. Ensures subsequent insertion +* of THPs into the page cache will fail. +*/ +smp_mb (); +if (filemap_nr_thps (inode->i_mapping)) +truncate_pagecache (inode,0 ); + }return 0 ;mm/khugepaged.c +15
-2@@ -459,7 +459,8 @@ static bool hugepage_vma_check(struct vm_area_struct *vma,/* Read-only file mappings need to be aligned for THP to work. */ if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && vma->vm_file && - (vm_flags &VM_DENYWRITE )) { + !inode_is_open_for_write (vma->vm_file->f_inode) && + (vm_flags &VM_EXEC )) {return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff,HPAGE_PMD_NR ); }@@ -1872,6 +1873,19 @@ static void collapse_file(struct mm_struct *mm,else {__mod_lruvec_page_state (new_page,NR_FILE_THPS , nr);filemap_nr_thps_inc (mapping); +/* +* Paired with smp_mb() in do_dentry_open() to ensure +* i_writecount is up to date and the update to nr_thps is +* visible. Ensures the page cache will be truncated if the +* file is opened writable. +*/ +smp_mb (); +if (inode_is_open_for_write (mapping->host)) { + result =SCAN_FAIL ; +__mod_lruvec_page_state (new_page,NR_FILE_THPS , -nr); +filemap_nr_thps_dec (mapping); + goto xa_locked; + } }if (nr_none) { --2.31.0.rc2.261.g7f71774620-goog
William Kucharski
Reviewed-bySounds good.
Quoted 137 lines · Collin Fijalkovich
Instrumenting
filemap_nr_thps_inc()should be sufficient for ensuring writable file mappings will not be THP-backed.If
filemap_nr_thps_dec()inunaccount_page_cache_page()andfilemap_nr_thps()indo_dentry_open()race, the worst case is an unnecessary truncation. We could introduce a memory barrier inunaccount_page_cache_page(), but I'm unsure it would significantly mitigate the risk of spurious truncations. Barring synchronization betweendo_dentry_open()and ongoing page cache operations, there could be an in-progressdelete_from_page_cache_batch()that has not yet updated accounting for THPs in its targeted range.-- Collin
Quoted 119 lines · William Kucharski
I saw a similar change a few years ago with my prototype:
https://lore.kernel.org/linux-mm/[email protected]/
the key being a very nice drop in iTLB-load-misses, so it looks like your code is having the right effect.
What about the call to
filemap_nr_thps_dec()inunaccount_page_cache_page()- does that need ansmp_mb()as well?-- Bill
Quoted 104 lines · Collin Fijalkovich
v2 has been uploaded with performance testing results:
https://lore.kernel.org/patchwork/patch/1408266/Quoted 94 lines · Collin Fijalkovich
Transparent huge pages are supported for read-only non-shmem filesystems, but are only used for vmas with
VM_DENYWRITE. This condition ensures that file THPs are protected from writes while an application is running (ETXTBSY). Any existing file THPs are then dropped from the page cache when a file is opened for write indo_dentry_open(). Sincesys_mmapignoresMAP_DENYWRITE, this constrains the use of file THPs to vmas produced byexecve().Systems that make heavy use of shared libraries (e.g. Android) are unable to apply
VM_DENYWRITEthrough the dynamic linker, preventing them from benefiting from the resultant reduced contention on the TLB.This patch reduces the constraint on file THPs allowing use with any executable mapping from a file not opened for write (see
inode_is_open_for_write()). It also introduces additional conditions to ensure that files opened for write will never be backed by file THPs.Restricting the use of THPs to executable mappings eliminates the risk that a read-only file later opened for write would encounter significant latencies due to page cache truncation.
The ld linker flag '-z max-page-size=(hugepage size)' can be used to produce executables with the necessary layout. The dynamic linker must map these file's segments at a hugepage size aligned vma for the mapping to be backed with THPs.
Signed-off-by Collin Fijalkovich <[email protected]>---
fs/open.c| 13 +++++++++++--mm/khugepaged.c| 16 +++++++++++++++-2 files changed, 26 insertions(+), 3 deletions(-)fs/open.c +11
-2@@ -852,8 +852,17 @@ static int do_dentry_open(struct file *f,* XXX: Huge page cache doesn't support writing yet. Drop all page * cache for this file before processing writes. */ -if ((f->f_mode &FMODE_WRITE ) &&filemap_nr_thps (inode->i_mapping)) -truncate_pagecache (inode,0 ); +if (f->f_mode &FMODE_WRITE ) { +/* +* Paired with smp_mb() in collapse_file() to ensure nr_thps +* is up to date and the update to i_writecount by +* get_write_access() is visible. Ensures subsequent insertion +* of THPs into the page cache will fail. +*/ +smp_mb (); +if (filemap_nr_thps (inode->i_mapping)) +truncate_pagecache (inode,0 ); + }return 0 ;mm/khugepaged.c +15
-2@@ -459,7 +459,8 @@ static bool hugepage_vma_check(struct vm_area_struct *vma,/* Read-only file mappings need to be aligned for THP to work. */ if (IS_ENABLED (CONFIG_READ_ONLY_THP_FOR_FS ) && vma->vm_file && - (vm_flags &VM_DENYWRITE )) { + !inode_is_open_for_write (vma->vm_file->f_inode) && + (vm_flags &VM_EXEC )) {return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff,HPAGE_PMD_NR ); }@@ -1872,6 +1873,19 @@ static void collapse_file(struct mm_struct *mm,else {__mod_lruvec_page_state (new_page,NR_FILE_THPS , nr);filemap_nr_thps_inc (mapping); +/* +* Paired with smp_mb() in do_dentry_open() to ensure +* i_writecount is up to date and the update to nr_thps is +* visible. Ensures the page cache will be truncated if the +* file is opened writable. +*/ +smp_mb (); +if (inode_is_open_for_write (mapping->host)) { + result =SCAN_FAIL ; +__mod_lruvec_page_state (new_page,NR_FILE_THPS , -nr); +filemap_nr_thps_dec (mapping); + goto xa_locked; + } }if (nr_none) { --2.31.0.rc2.261.g7f71774620-goog