mm, thp: introduce a new sysfs interface to facilitate file THP for .text
Subsystems (6)
cover mm, thp: introduce a new sysfs interface to facilitate file THP for .text
Hi, all
Recently, our team focus on huge pages of executable binary files and share libraries, refer to these huge pages as 'hugetext' in the next description. The hugetext indeed to improve the performance of application, e.g. mysql. It has been shown in [1][2]. And with the increase of the text section, the improvement will be more obvious. Base on [1][2], we make some improvement to make file-backed THP more usability and easy for applications.
In current kernel, ref[1] introduced READ_ONLY_THP_FOR_FS, and ref[2] add the support for share libraries based on the previous one. However, Until now, hugetext is not convenient to use at present. For example, we need to explicitly madvise MADV_HUGEPAGE for .text and set "transparent_hugepage/enabled" to always or madvise . On the other hand, hugetext requires 2M alignment of vma->vm_start and vma->vm_pgoff, which is not guaranteed by kernel or loader.
Our design:
To solve the drawback mentioned above of file THP in using, we have mainly improved two points that shows below.
(1) introduce a new sysfs interface "transparent_hugepage/hugetext_enabled" in order to automatically (i.e., transparently) enable file THP for suitable .text vmas. The usage belows:
to disable hugetext:
$ echo 0 > /sys/kernel/mm/transparent_hugepage/hugetext_enabled
to enable hugetext:
$ echo 1 > /sys/kernel/mm/transparent_hugepage/hugetext_enabled
to enable or disable in boot options: hugetext=1 or hugetext=0
Q: Why not add a new option, e.g., "text_always", in addition to "always", "madvise", and "never" to "transparent_hugepage/enabled" ?
A: A new option to "transparent_hugepage/enabled" cannot handle such scenario, where THP always for .text, and madivse/never for others (e.g., anon vma).
The .text is usually small in size. In our production environment, at most 10G out of 500G total memory is used as .text. The .text is also performance critical. More important, We don't want to change the user's default behavior too much. So we think that a new independent sysfs interface for file THP is worthy.
(2) make vm_start of .text 2M align with vm_pgoff, especially for PIE/PIC binaries and shared libraries.
For binaries that are compiled with '--pie -fPIC' and with LOAD alignment smaller than 2M (typically 4K, 64K), change maximum_alignment to 2M.
For shared libraries, ld.so seems not to consider p_align well, as shown below.
$ readelf -l /usr/lib64/libc-2.17.so
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000001c2fe8 0x00000000001c2fe8 R E 200000
$ cat /proc/1/smaps
7fecc4072000-7fecc4235000 r-xp 00000000 08:03 655802 /usr/lib64/libc-2.17.soFinally, why this feasure is implemented in kernel, not in userspace, or ld.so?
Userspace methods like libhugetlbfs have various disadvantages:
* require recompiling applications;
* the anonymous mapping cannot be shared;
* debugging is not convenient.
To madvise MADV_HUGEPAGE for .text in ld.so has been suggested in the glibc mailing list[3], but there was no response.
Finally, considering that this feature requires very little code and is not too difficult to implement based on the existing file-backed THP support, it was finally chosen to be implemented in the kernel.
Thanks!
Reference:
[1] https://patchwork.kernel.org/project/linux-mm/cover/[email protected]/
[2] https://patchwork.kernel.org/project/linux-fsdevel/patch/[email protected]/
[3] https://sourceware.org/pipermail/libc-alpha/2021-February/122334.html
Rongwei Wang (3):
mm, thp: support binaries transparent use of file THP mm, thp: make mapping address of libraries THP align mm, thp: make mapping address of PIC binaries THP align
fs/binfmt_elf.c | 5 +++ include/linux/huge_mm.h | 36 +++++++++++++++++++ include/linux/khugepaged.h | 9 +++++ mm/Kconfig | 11 ++++++ mm/huge_memory.c | 72 ++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 4 +++ mm/memory.c | 12 +++++++ mm/mmap.c | 18 ++++++++++ 8 files changed, 167 insertions(+)
Signature · 1 line
2.27.0
PATCH 1/3 mm, thp: support binaries transparent use of file THP
The file THP for .text is not convenient to use at present. Applications need to explicitly madvise MADV_HUGEPAGE for .text, which is not friendly for tasks in the production environment.
This patch extends READ_ONLY_THP_FOR_FS, introduces a new sysfs
interface: hugetext_enabled, to make the read-only file-backed pages THPeligible proactively and transparently.
Compared with original design, It not depend on 'madvise()' any more. And because of 'hugetext_enabled' introduced, users are no longer limited to 'enabled' setting (e.g., always, madvise and never).
There are two methods to enable or disable this feature:
To enable hugetext:
1. echo 1 > /sys/kernel/mm/transparent_hugepage/hugetext_enabled
2. hugetext=1 in boot cmdline
To disable hugetext:
1. echo 0 > /sys/kernel/mm/transparent_hugepage/hugetext_enabled
2. hugetext=0 in boot cmdline
This feature is disabled by default.
--- include/linux/huge_mm.h | 24 ++++++++++++++++ include/linux/khugepaged.h | 9 ++++++ mm/Kconfig | 11 ++++++++ mm/huge_memory.c | 57 ++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 4 +++ mm/memory.c | 12 ++++++++ 6 files changed, 117 insertions(+)
include/linux/huge_mm.h +24 -0
@@ -87,6 +87,9 @@ enum transparent_hugepage_flag {TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG ,TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG ,TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG , +#ifdef CONFIG_HUGETEXT + TRANSPARENT_HUGEPAGE_HUGETEXT_ENABLED_FLAG, +#endif };struct kobject ;
@@ -140,6 +143,27 @@ static inline bool transhuge_vma_enabled(struct vm_area_struct *vma,return true; } +#ifdef CONFIG_HUGETEXT +#define hugetext_enabled () \ + (transparent_hugepage_flags & \ + (1<<TRANSPARENT_HUGEPAGE_HUGETEXT_ENABLED_FLAG)) +#else +#define hugetext_enabled () false +#endif /* CONFIG_HUGETEXT */ + +static inline bool vma_is_hugetext (struct vm_area_struct *vma, +unsigned long vm_flags) +{ +if (!(vm_flags &VM_EXEC )) +return false; + +if (vma->vm_file && !inode_is_open_for_write (vma->vm_file->f_inode)) +return IS_ALIGNED ((vma->vm_start >>PAGE_SHIFT ) - vma->vm_pgoff, +HPAGE_PMD_NR ); + +return false; +} +/* * to be used on vmas which are known to support THP. * Use transparent_hugepage_active otherwise
include/linux/khugepaged.h +9 -0
@@ -26,10 +26,18 @@ static inline void collapse_pte_mapped_thp(struct mm_struct *mm, }#endif +#ifdef CONFIG_HUGETEXT +#define khugepaged_enabled () \ + (transparent_hugepage_flags & \ + ((1<<TRANSPARENT_HUGEPAGE_FLAG) | \ + (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG) | \ + (1<<TRANSPARENT_HUGEPAGE_HUGETEXT_ENABLED_FLAG))) +#else #define khugepaged_enabled () \ (transparent_hugepage_flags & \ ((1<<TRANSPARENT_HUGEPAGE_FLAG) | \ (1<<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG))) +#endif #define khugepaged_always () \ (transparent_hugepage_flags & \ (1<<TRANSPARENT_HUGEPAGE_FLAG))
@@ -59,6 +67,7 @@ static inline int khugepaged_enter(struct vm_area_struct *vma,if (!test_bit (MMF_VM_HUGEPAGE , &vma->vm_mm->flags))if ((khugepaged_always () || (shmem_file (vma->vm_file) &&shmem_huge_enabled (vma)) || + (hugetext_enabled () &&vma_is_hugetext (vma, vm_flags)) || (khugepaged_req_madv () && (vm_flags &VM_HUGEPAGE ))) && !(vm_flags &VM_NOHUGEPAGE ) && !test_bit (MMF_DISABLE_THP , &vma->vm_mm->flags))
mm/Kconfig +11 -0
@@ -868,6 +868,17 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config HUGETEXT + bool "THP for text segments" + depends on READ_ONLY_THP_FOR_FS + + help + Allow khugepaged to put read-only file-backed pages, including + shared libraries, as well as the anonymous and executable pages + in THP. + + This feature builds on and extends READ_ONLY_THP_FOR_FS. + config ARCH_HAS_PTE_SPECIAL bool
mm/huge_memory.c +57 -0
75 lines not on this page · raw message
mm/khugepaged.c +4 -0
@@ -451,6 +451,10 @@ static bool hugepage_vma_check(struct vm_area_struct *vma, HPAGE_PMD_NR); } +/* Make hugetext independent of THP settings */ +if (hugetext_enabled () &&vma_is_hugetext (vma, vm_flags)) +return true; +/* THP settings require madvise. */ if (!(vm_flags &VM_HUGEPAGE ) && !khugepaged_always ())return false;
mm/memory.c +12 -0
@@ -73,6 +73,7 @@#include <linux/perf_event.h> #include <linux/ptrace.h> #include <linux/vmalloc.h> +#include <linux/khugepaged.h> #include <trace/events/kmem.h>
@@ -4157,6 +4158,17 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;vm_fault_t ret =0 ; +#ifdef CONFIG_HUGETEXT +/* Add the candidate hugetext vma into khugepaged scan list */ +if (pmd_none (*vmf->pmd) &&hugetext_enabled () + &&vma_is_hugetext (vma, vma->vm_flags)) { +unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; + +if (transhuge_vma_suitable (vma, haddr)) +khugepaged_enter (vma, vma->vm_flags); + } +#endif +/* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or
Signature · 1 line
2.27.0
PATCH 2/3 mm, thp: make mapping address of libraries THP align
In reply to Rongwei WangFor shared libraries, ld.so seems not to consider p_align well, as shown below.
$ readelf -l /usr/lib64/libc-2.17.so
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000001c2fe8 0x00000000001c2fe8 R E 200000
$ cat /proc/1/smaps
7fecc4072000-7fecc4235000 r-xp 00000000 08:03 655802 /usr/lib64/libc-2.17.soThis makes the mapping address allocated by 'get_unmapped_area' align with 2M for libraries, to facilitate file THP for .text section as far as possible.
--- include/linux/huge_mm.h | 12 ++++++++++++ mm/huge_memory.c | 15 +++++++++++++++ mm/mmap.c | 18 ++++++++++++++++++ 3 files changed, 45 insertions(+)
include/linux/huge_mm.h +12 -0
@@ -147,8 +147,20 @@ static inline bool transhuge_vma_enabled(struct vm_area_struct *vma,#define hugetext_enabled () \ (transparent_hugepage_flags & \ (1<<TRANSPARENT_HUGEPAGE_HUGETEXT_ENABLED_FLAG)) + +extern unsigned long hugetext_get_unmapped_area (struct file *filp, +unsigned long addr,unsigned long len,unsigned long pgoff, +unsigned long flags);#else #define hugetext_enabled () false + +static inline unsigned long hugetext_get_unmapped_area (struct file *filp, +unsigned long addr,unsigned long len,unsigned long pgoff, +unsigned long flags) +{ +BUILD_BUG (); +return 0 ; +}#endif /* CONFIG_HUGETEXT */ static inline bool vma_is_hugetext(struct vm_area_struct *vma,
mm/huge_memory.c +15 -0
@@ -650,6 +650,21 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, }EXPORT_SYMBOL_GPL (thp_get_unmapped_area); +#ifdef CONFIG_HUGETEXT +unsigned long hugetext_get_unmapped_area (struct file *filp,unsigned long addr, +unsigned long len,unsigned long pgoff,unsigned long flags) +{ +unsigned long ret; +loff_t off = (loff_t )pgoff <<PAGE_SHIFT ; + + ret =__thp_get_unmapped_area (filp, addr, len, off, flags,PMD_SIZE ); +if (ret) +return ret; + +return current->mm->get_unmapped_area (filp, addr, len, pgoff, flags); +} +#endif /* CONFIG_HUGETEXT */ +static vm_fault_t __do_huge_pmd_anonymous_page (struct vm_fault *vmf,struct page *page,gfp_t gfp) {
mm/mmap.c +18 -0
@@ -2242,8 +2242,26 @@ get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, get_area = current->mm->get_unmapped_area;if (file) { +#ifdef CONFIG_HUGETEXT +/* +* Prior to the file->f_op->get_unmapped_area. +* +* If hugetext is enabled, except for MAP_FIXED, it always +* make the mapping address of files that have executable +* attribute be mapped in 2MB alignment. +*/ +struct inode *inode =file_inode (file); + +if (hugetext_enabled () && (inode->i_mode &0111 ) && + (!file->f_op->get_unmapped_area || + file->f_op->get_unmapped_area == thp_get_unmapped_area)) + get_area = hugetext_get_unmapped_area; +else if (file->f_op->get_unmapped_area) + get_area = file->f_op->get_unmapped_area; +#else if (file->f_op->get_unmapped_area) get_area = file->f_op->get_unmapped_area; +#endif }else if (flags &MAP_SHARED ) {/* * mmap_region() will call shmem_zero_setup() to create a file,
Signature · 1 line
2.27.0
PATCH 3/3 mm, thp: make mapping address of PIC binaries THP align
In reply to Rongwei WangFor binaries that are compiled with '--pie -fPIC' and with LOAD alignment smaller than 2M (typically 4K, 64K), the load address is least likely to be 2M aligned.
This changes the maximum_alignment of such binaries to 2M to facilitate file THP for .text section as far as possible.
--- fs/binfmt_elf.c | 5 +++++ 1 file changed, 5 insertions(+)
fs/binfmt_elf.c +5 -0
@@ -1136,6 +1136,11 @@ static int load_elf_binary(struct linux_binprm *bprm) retval = -EINVAL ; goto out_free_dentry; } +#ifdef CONFIG_HUGETEXT +if (hugetext_enabled () && interpreter && + total_size >=HPAGE_PMD_SIZE ) + load_bias &=HPAGE_PMD_MASK ; +#endif } error = elf_map(bprm->file, load_bias + vaddr, elf_ppnt,
Signature · 1 line
2.27.0
Can we please just get proper pagecache THP (through folios) merged instead of piling hacks over hacks here? The whole readonly THP already was more than painful enough due to all the hacks involved.
Quoted 3 lines · Christoph Hellwig
Can we please just get proper pagecache THP (through folios) merged instead of piling hacks over hacks here? The whole readonly THP already was more than painful enough due to all the hacks involved.
This was my initial reaction too.
But read the patches. They're nothing to do with the implementation of THP / folios in the page cache. They're all to make sure that mappings are PMD aligned.
I think there's a lot to criticise in the patches (eg, a system-wide setting is probably a bad idea. and a lot of this stuff seems to be fixing userspace bugs in the kernel). But let's criticise what's actually in the patches, because these are problems that exist regardless of RO_THP vs folios.
Quoted 10 lines · Matthew Wilcox
Quoted 3 lines · Christoph Hellwig
Can we please just get proper pagecache THP (through folios) merged instead of piling hacks over hacks here? The whole readonly THP already was more than painful enough due to all the hacks involved.
This was my initial reaction too.
But read the patches. They're nothing to do with the implementation of THP / folios in the page cache. They're all to make sure that mappings are PMD aligned.
Hi, Matthew
In fact, we had thought about realizing this by handling page cache directly. And then, we found that we just need to align the mapping address and make khugepaged can scan these 'mm_struct' base on READ_ONLY_THP_FOR_FS.
Quoted 3 lines
I think there's a lot to criticise in the patches (eg, a system-wide setting is probably a bad idea. and a lot of this stuff seems to
At the beginning, we don't introduce the new sysfs interface, just re-use 'transparent_hugepage/enabled'. But In some production system, they
disable the THP directly, especially those applications that are sensitive to THP. So, Considering these scenarios, we had to design a new sysfs interface ('transparent_hugepage/hugetext_enabled').
And if you have other idea, we are willing to take to improve these patches.
Thanks!
Quoted 4 lines
be fixing userspace bugs in the kernel). But let's criticise what's actually in the patches, because these are problems that exist regardless of
RO_THPvs folios.