mm, thp: Add mm flag to control THP
Subsystems (9)
v4 cover mm, thp: Add mm flag to control THP
(First send had too big of a cc list to make it into all the mailing lists.)
This patch is based on some of my work combined with some suggestions/patches given by Oleg Nesterov. The main goal here is to add a prctl switch to allow us to disable to THP on a per mm_struct basis.
Changes for v4:
* Added a bit of documentation for flag changes
* Changed the prctl switch code so that get/set operations are processed in separate cases
* Added information about the new flags to the user-facing prctl docs (this is in a separate patch, everyone will be cc'd there as well)
The main motivation behind this patch is to provide a way to disable THP for jobs where the code cannot be modified, and using a malloc hook with madvise is not an option (i.e. statically allocated data). This patch allows us to do just that, without affecting other jobs running on the system.
We need to do this sort of thing for jobs where THP hurts performance, due to the possibility of increased remote memory accesses that can be created by situations such as the following:
When you touch 1 byte of an untouched, contiguous 2MB chunk, a THP will be handed out, and the THP will be stuck on whatever node the chunk was originally referenced from. If many remote nodes need to do work on that same chunk, they'll be making remote accesses.
With THP disabled, 4K pages can be handed out to separate nodes as they're needed, greatly reducing the amount of remote accesses to memory.
Here's a bit of test data with the new patch in place...
First with the flag unset:
# perf stat -a ./prctl_wrapper_mmv3 0 ./thp_pthread -C 0 -m 0 -c 512 -b 256g Setting thp_disabled for this task... thp_disable: 0 Set thp_disabled state to 0 Process pid = 18027
PF/
MAX MIN TOTCPU/ TOT_PF/ TOT_PF/ WSEC/
TYPE: CPUS WALL WALL SYS USER TOTCPU CPU WALL_SEC SYS_SEC CPU NODES 512 1.120 0.060 0.000 0.110 0.110 0.000 28571428864 -9223372036854775808 55803572 23
Performance counter stats for './prctl_wrapper_mmv3_hack 0 ./thp_pthread -C 0 -m 0 -c 512 -b 256g':
273719072.841402 task-clock # 641.026 CPUs utilized [100.00%]
1,008,986 context-switches # 0.000 M/sec [100.00%]
7,717 CPU-migrations # 0.000 M/sec [100.00%]
1,698,932 page-faults # 0.000 M/sec
355,222,544,890,379 cycles # 1.298 GHz [100.00%]
536,445,412,234,588 stalled-cycles-frontend # 151.02% frontend cycles idle [100.00%]
409,110,531,310,223 stalled-cycles-backend # 115.17% backend cycles idle [100.00%]
148,286,797,266,411 instructions # 0.42 insns per cycle
# 3.62 stalled cycles per insn [100.00%]
27,061,793,159,503 branches # 98.867 M/sec [100.00%]
1,188,655,196 branch-misses # 0.00% of all branches
427.001706337 seconds time elapsedNow with the flag set:
# perf stat -a ./prctl_wrapper_mmv3 1 ./thp_pthread -C 0 -m 0 -c 512 -b 256g Setting thp_disabled for this task... thp_disable: 1 Set thp_disabled state to 1 Process pid = 144957
PF/
MAX MIN TOTCPU/ TOT_PF/ TOT_PF/ WSEC/
TYPE: CPUS WALL WALL SYS USER TOTCPU CPU WALL_SEC SYS_SEC CPU NODES 512 0.620 0.260 0.250 0.320 0.570 0.001 51612901376 128000000000 100806448 23
Performance counter stats for './prctl_wrapper_mmv3_hack 1 ./thp_pthread -C 0 -m 0 -c 512 -b 256g':
138789390.540183 task-clock # 641.959 CPUs utilized [100.00%]
534,205 context-switches # 0.000 M/sec [100.00%]
4,595 CPU-migrations # 0.000 M/sec [100.00%]
63,133,119 page-faults # 0.000 M/sec
147,977,747,269,768 cycles # 1.066 GHz [100.00%]
200,524,196,493,108 stalled-cycles-frontend # 135.51% frontend cycles idle [100.00%]
105,175,163,716,388 stalled-cycles-backend # 71.07% backend cycles idle [100.00%]
180,916,213,503,160 instructions # 1.22 insns per cycle
# 1.11 stalled cycles per insn [100.00%]
26,999,511,005,868 branches # 194.536 M/sec [100.00%]
714,066,351 branch-misses # 0.00% of all branches
216.196778807 seconds time elapsedAs with previous versions of the patch, We're getting about a 2x performance increase here. Here's a link to the test case I used, along with the little wrapper to activate the flag:
http://oss.sgi.com/projects/memtests/thp_pthread_mmprctlv3.tar.gz
Let me know if anybody has any further suggestions here. Thanks!
(Sorry for the big cc list!)
Alex Thorlton (3):
Revert "thp: make MADV_HUGEPAGE check for mm->def_flags" Add VM_INIT_DEF_MASK and PRCTL_THP_DISABLE
exec: kill the unnecessary mm->def_flags setting in load_elf_binary()
arch/s390/mm/pgtable.c | 3 +++ fs/binfmt_elf.c | 4 ---- include/linux/mm.h | 3 +++ include/uapi/linux/prctl.h | 3 +++ kernel/fork.c | 11 ++++++++--- kernel/sys.c | 15 +++++++++++++++ mm/huge_memory.c | 4 ---- 7 files changed, 32 insertions(+), 11 deletions(-)
Signature · 1 line
1.7.12.4
PATCH v4 1/3 Revert "thp: make MADV_HUGEPAGE check for mm->def_flags"
This reverts commit 8e72033f2a489b6c98c4e3c7cc281b1afd6cb85c, and adds in code to fix up any issues caused by the revert.
The revert is necessary because hugepage_madvise would return -EINVAL when VM_NOHUGEPAGE is set, which will break subsequent chunks of this patch set.
Here's a snip of an e-mail from Gerald detailing the original purpose of this code, and providing justification for the revert:
<snip>
The intent of 8e72033f2a48 was to guard against any future programming errors that may result in an madvice(MADV_HUGEPAGE) on guest mappings, which would crash the kernel.
Martin suggested adding the bit to arch/s390/mm/pgtable.c, if 8e72033f2a48 was to be reverted, because that check will also prevent a kernel crash in the case described above, it will now send a SIGSEGV instead.
This would now also allow to do the madvise on other parts, if needed, so it is a more flexible approach. One could also say that it would have been better to do it this way right from the beginning... </snip>
--- arch/s390/mm/pgtable.c | 3 +++ mm/huge_memory.c | 4 ---- 2 files changed, 3 insertions(+), 4 deletions(-)
arch/s390/mm/pgtable.c +3 -0
@@ -504,6 +504,9 @@ static int gmap_connect_pgtable(unsigned long address, unsigned long segment,if (!pmd_present (*pmd) &&__pte_alloc (mm, vma, pmd, vmaddr))return -ENOMEM ; +/* large pmds cannot yet be handled */ +if (pmd_large (*pmd)) +return -EFAULT ;/* pmd now points to a valid segment table entry. */ rmap =kmalloc (sizeof (*rmap),GFP_KERNEL |__GFP_REPEAT);if (!rmap)
mm/huge_memory.c +0 -4
@@ -1968,8 +1968,6 @@ out:int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice) { -struct mm_struct *mm = vma->vm_mm; -switch (advice) {case MADV_HUGEPAGE :/*
@@ -1977,8 +1975,6 @@ int hugepage_madvise(struct vm_area_struct *vma,*/ if (*vm_flags & (VM_HUGEPAGE |VM_NO_THP ))return -EINVAL ; -if (mm->def_flags &VM_NOHUGEPAGE ) -return -EINVAL ; *vm_flags &= ~VM_NOHUGEPAGE ; *vm_flags |=VM_HUGEPAGE ;/*
Signature · 7 lines
1.7.12.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v4 2/3 mm, thp: Add VM_INIT_DEF_MASK and PRCTL_THP_DISABLE
In reply to Alex ThorltonThis patch adds a VM_INIT_DEF_MASK, to allow us to set the default flags for VMs. It also adds a prctl control which alllows us to set the THP disable bit in mm->def_flags so that VMs will pick up the setting as they are created.
--- include/linux/mm.h | 3 +++ include/uapi/linux/prctl.h | 3 +++ kernel/fork.c | 11 ++++++++--- kernel/sys.c | 15 +++++++++++++++ 4 files changed, 29 insertions(+), 3 deletions(-)
include/linux/mm.h +3 -0
@@ -177,6 +177,9 @@ extern unsigned int kobjsize(const void *objp);*/ #define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP) +/* This mask defines which mm->def_flags a process can inherit its parent */ +#define VM_INIT_DEF_MASK VM_NOHUGEPAGE +/* * mapping from the currently active vm_flags protection bits (the * low four bits) to a page protection mask..
include/uapi/linux/prctl.h +3 -0
@@ -149,4 +149,7 @@#define PR_GET_TID_ADDRESS 40 +#define PR_SET_THP_DISABLE 41 +#define PR_GET_THP_DISABLE 42 +#endif /* _LINUX_PRCTL_H */
kernel/fork.c +8 -3
@@ -529,8 +529,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)atomic_set (&mm->mm_count,1 );init_rwsem (&mm->mmap_sem);INIT_LIST_HEAD (&mm->mmlist); - mm->flags = (current->mm) ? - (current->mm->flags &MMF_INIT_MASK ) : default_dump_filter; mm->core_state =NULL ;atomic_long_set (&mm->nr_ptes,0 );memset (&mm->rss_stat,0 ,sizeof (mm->rss_stat));
@@ -539,8 +537,15 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)mm_init_owner (mm, p);clear_tlb_flush_pending (mm); -if (likely (!mm_alloc_pgd (mm))) { +if (current->mm) { + mm->flags = current->mm->flags &MMF_INIT_MASK ; + mm->def_flags = current->mm->def_flags &VM_INIT_DEF_MASK ; + }else { + mm->flags = default_dump_filter; mm->def_flags =0 ; + } + +if (likely (!mm_alloc_pgd (mm))) {mmu_notifier_mm_init (mm);return mm; }
kernel/sys.c +15 -0
@@ -1996,6 +1996,21 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,if (arg2 || arg3 || arg4 || arg5)return -EINVAL ;return current->no_new_privs ?1 :0 ; +case PR_GET_THP_DISABLE : +if (arg2 || arg3 || arg4 || arg5) +return -EINVAL ; + error = !!(me->mm->def_flags &VM_NOHUGEPAGE ); +break ; +case PR_SET_THP_DISABLE : +if (arg3 || arg4 || arg5) +return -EINVAL ; +down_write (&me->mm->mmap_sem); +if (arg2) + me->mm->def_flags |=VM_NOHUGEPAGE ; +else + me->mm->def_flags &= ~VM_NOHUGEPAGE ; +up_write (&me->mm->mmap_sem); +break ;default : error = -EINVAL ;break ;
Signature · 7 lines
1.7.12.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 2/3] mm, thp: Add VM_INIT_DEF_MASK and PRCTL_THP_DISABLE
Quoted 7 lines · Alex Thorlton
This patch adds a
VM_INIT_DEF_MASK, to allow us to set the default flags for VMs. It also adds a prctl control which alllows us to set the THP disable bit inmm->def_flagsso that VMs will pick up the setting as they are created.Signed-off-by Alex Thorlton <[email protected]>Suggested-by Oleg Nesterov <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v4 3/3 exec: kill the unnecessary mm->def_flags setting in load_elf_binary()
In reply to Alex Thorltonload_elf_binary() sets current->mm->def_flags = def_flags and def_flags is always zero. Not only this looks strange, this is unnecessary because mm_init() has already set ->def_flags = 0.
--- fs/binfmt_elf.c | 4 ---- 1 file changed, 4 deletions(-)
fs/binfmt_elf.c +0 -4
@@ -579,7 +579,6 @@ static int load_elf_binary(struct linux_binprm *bprm)unsigned long start_code, end_code, start_data, end_data;unsigned long reloc_func_desc__maybe_unused =0 ;int executable_stack =EXSTACK_DEFAULT ; -unsigned long def_flags =0 ;struct pt_regs *regs =current_pt_regs ();struct {struct elfhdr elf_ex;
@@ -719,9 +718,6 @@ static int load_elf_binary(struct linux_binprm *bprm)if (retval) goto out_free_dentry; -/* OK, This is the point of no return */ - current->mm->def_flags = def_flags; -/* Do this immediately, since STACK_TOP as used in setup_arg_pages may depend on the personality. */ SET_PERSONALITY (loc->elf_ex);
Signature · 7 lines
1.7.12.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 3/3] exec: kill the unnecessary mm->def_flags setting in load_elf_binary()
Quoted 6 lines · Alex Thorlton
load_elf_binary()setscurrent->mm->def_flags=def_flagsanddef_flagsis always zero. Not only this looks strange, this is unnecessary becausemm_init()has already set ->def_flags = 0.Signed-off-by Alex Thorlton <[email protected]>Suggested-by Oleg Nesterov <[email protected]>
Signature · 7 lines
All rights reversed -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Alex Thorlton
(First send had too big of a cc list to make it into all the mailing lists.)
Sorry for the double-send to a couple of lists/people. I got bumped from linux-api for having too long of a cc list, and I figured it had gotten bumped elsewhere as well.
- Alex
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>