mm: add new syscall set_mempolicy_home_node
Subsystems (20)
v6 cover mm: add new syscall set_mempolicy_home_node
Changes from v5:
* Add missing mpol_put()
* use untagged_addr
* return error for memory policy other than MPOL_BIND and MPOL_PREFERRED_MANY
* update documentation/commit message/code comments
Changes from v4:
* Add flags == 0 check
* Make sure the home node is online before updating memory policy.
Changes from v3:
* Fix build warning reported by kernel test robot
Changes from RFC v2:
* Rebase to latest kernel
* Update numa_memory_policy.rst
Aneesh Kumar K.V (3):
mm/mempolicy: use policy_node helper with MPOL_PREFERRED_MANY mm/mempolicy: add set_mempolicy_home_node syscall mm/mempolicy: wire up syscall set_mempolicy_home_node
.../admin-guide/mm/numa_memory_policy.rst | 16 +++- arch/alpha/kernel/syscalls/syscall.tbl | 1 + arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 + arch/ia64/kernel/syscalls/syscall.tbl | 1 + arch/m68k/kernel/syscalls/syscall.tbl | 1 + arch/microblaze/kernel/syscalls/syscall.tbl | 1 + arch/mips/kernel/syscalls/syscall_n32.tbl | 1 + arch/mips/kernel/syscalls/syscall_n64.tbl | 1 + arch/mips/kernel/syscalls/syscall_o32.tbl | 1 + arch/parisc/kernel/syscalls/syscall.tbl | 1 + arch/powerpc/kernel/syscalls/syscall.tbl | 1 + arch/s390/kernel/syscalls/syscall.tbl | 1 + arch/sh/kernel/syscalls/syscall.tbl | 1 + arch/sparc/kernel/syscalls/syscall.tbl | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/xtensa/kernel/syscalls/syscall.tbl | 1 + include/linux/mempolicy.h | 1 + include/linux/syscalls.h | 3 + include/uapi/asm-generic/unistd.h | 5 +- kernel/sys_ni.c | 1 + mm/mempolicy.c | 84 ++++++++++++++++++- 24 files changed, 125 insertions(+), 5 deletions(-)
Signature · 1 line
2.31.1
PATCH v6 2/3 mm/mempolicy: add set_mempolicy_home_node syscall
This syscall can be used to set a home node for the MPOL_BIND and MPOL_PREFERRED_MANY memory policy. Users should use this syscall after setting up a memory policy for the specified range as shown below.
mbind(p, nr_pages * page_size, MPOL_BIND, new_nodes->maskp,
new_nodes->size + 1, 0); sys_set_mempolicy_home_node((unsigned long)p, nr_pages * page_size, home_node, 0);
The syscall allows specifying a home node/preferred node from which kernel will fulfill memory allocation requests first.
For address range with MPOL_BIND memory policy, if nodemask specifies more than one node, page allocations will come from the node in the nodemask with sufficient free memory that is closest to the home node/preferred node.
For MPOL_PREFERRED_MANY if the nodemask specifies more than one node, page allocation will come from the node in the nodemask with sufficient free memory that is closest to the home node/preferred node. If there is not enough memory in all the nodes specified in the nodemask, the allocation will be attempted from the closest numa node to the home node in the system.
This helps applications to hint at a memory allocation preference node and fallback to _only_ a set of nodes if the memory is not available on the preferred node. Fallback allocation is attempted from the node which is nearest to the preferred node.
This helps applications to have control on memory allocation numa nodes and avoids default fallback to slow memory NUMA nodes. For example a system with NUMA nodes 1,2 and 3 with DRAM memory and 10, 11 and 12 of slow memory
new_nodes = numa_bitmask_alloc(nr_nodes);
numa_bitmask_setbit(new_nodes, 1);
numa_bitmask_setbit(new_nodes, 2);
numa_bitmask_setbit(new_nodes, 3);
p = mmap(NULL, nr_pages * page_size, protflag, mapflag, -1, 0); mbind(p, nr_pages * page_size, MPOL_BIND, new_nodes->maskp, new_nodes->size + 1, 0);
sys_set_mempolicy_home_node(p, nr_pages * page_size, 2, 0);
This will allocate from nodes closer to node 2 and will make sure the kernel will only allocate from nodes 1, 2, and 3. Memory will not be allocated from slow memory nodes 10, 11, and 12. This differs from default MPOL_BIND behavior in that with default MPOL_BIND the allocation will be attempted from node closer to the local node. One of the reasons to specify a home node is to allow allocations from cpu less NUMA node and its nearby NUMA nodes.
With MPOL_PREFERRED_MANY on the other hand will first try to allocate from the closest node to node 2 from the node list 1, 2 and 3. If those nodes don't have enough memory, kernel will allocate from slow memory node 10, 11 and 12 which ever is closer to node 2.
--- .../admin-guide/mm/numa_memory_policy.rst | 16 +++- include/linux/mempolicy.h | 1 + mm/mempolicy.c | 79 +++++++++++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/numa_memory_policy.rst +15 -1
@@ -408,7 +408,7 @@ follows: Memory Policy APIs ================== -Linux supports 3 system calls for controlling memory policy. These APIS +Linux supports 4 system calls for controlling memory policy. These APIS always affect only the calling task, the calling task's address space, or some shared object mapped into the calling task's address space.
@@ -460,6 +460,20 @@ requested via the 'flags' argument. See the mbind(2) man page for more details. +Set home node for a Range of Task's Address Spacec:: + + long sys_set_mempolicy_home_node(unsigned long start, unsigned long len, + unsigned long home_node, + unsigned long flags); + +sys_set_mempolicy_home_node set the home node for a VMA policy present in the +task's address range. The system call updates the home node only for the existing +mempolicy range. Other address ranges are ignored. A home node is the NUMA node +closest to which page allocation will come from. Specifying the home node override +the default allocation policy to allocate memory close to the local node for an +executing CPU. + + Memory Policy Command Line Interface ====================================
include/linux/mempolicy.h +1 -0
@@ -46,6 +46,7 @@ struct mempolicy {unsigned short mode;/* See MPOL_* above */ unsigned short flags;/* See set_mempolicy() MPOL_F_* above */ nodemask_t nodes;/* interleave/bind/perfer */ +int home_node;/* Home node to use for MPOL_BIND and MPOL_PREFERRED_MANY */ union {nodemask_t cpuset_mems_allowed;/* relative to these nodes */
mm/mempolicy.c +79 -0
103 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v6 1/3 mm/mempolicy: use policy_node helper with MPOL_PREFERRED_MANY
In reply to Aneesh Kumar K.VA followup patch will enable setting a home node with MPOL_PREFERRED_MANY memory policy. To facilitate that switch to using policy_node helper. There is no functional change in this patch.
--- mm/mempolicy.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/mempolicy.c +3 -2
@@ -2061,7 +2061,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); page =__alloc_pages (preferred_gfp, order, nid, &pol->nodes);if (!page) - page =__alloc_pages (gfp, order,numa_node_id (),NULL ); + page =__alloc_pages (gfp, order, nid,NULL );return page; }
@@ -2102,6 +2102,7 @@ struct page *alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, }if (pol->mode ==MPOL_PREFERRED_MANY ) { + node =policy_node (gfp, pol, node); page =alloc_pages_preferred_many (gfp, order, node, pol);mpol_cond_put (pol); goto out;
@@ -2186,7 +2187,7 @@ struct page *alloc_pages(gfp_t gfp, unsigned order) page =alloc_page_interleave (gfp, order,interleave_nodes (pol)); else if (pol->mode == MPOL_PREFERRED_MANY) page =alloc_pages_preferred_many (gfp, order, -numa_node_id (), pol); +policy_node (gfp, pol,numa_node_id ()), pol); else page = __alloc_pages(gfp, order, policy_node(gfp, pol, numa_node_id()),
Signature · 1 line
2.31.1
PATCH v6 3/3 mm/mempolicy: wire up syscall set_mempolicy_home_node
In reply to Aneesh Kumar K.VCc: Ben Widawsky <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Feng Tang <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Andrea Arcangeli <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Mike Kravetz <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Andi Kleen <[email protected]>
Cc: Dan Williams <[email protected]>
Cc: Huang Ying <[email protected]>
Cc: [email protected]
Fix build failure
--- arch/alpha/kernel/syscalls/syscall.tbl | 1 + arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 ++ arch/ia64/kernel/syscalls/syscall.tbl | 1 + arch/m68k/kernel/syscalls/syscall.tbl | 1 + arch/microblaze/kernel/syscalls/syscall.tbl | 1 + arch/mips/kernel/syscalls/syscall_n32.tbl | 1 + arch/mips/kernel/syscalls/syscall_n64.tbl | 1 + arch/mips/kernel/syscalls/syscall_o32.tbl | 1 + arch/parisc/kernel/syscalls/syscall.tbl | 1 + arch/powerpc/kernel/syscalls/syscall.tbl | 1 + arch/s390/kernel/syscalls/syscall.tbl | 1 + arch/sh/kernel/syscalls/syscall.tbl | 1 + arch/sparc/kernel/syscalls/syscall.tbl | 1 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/xtensa/kernel/syscalls/syscall.tbl | 1 + include/linux/syscalls.h | 3 +++ include/uapi/asm-generic/unistd.h | 5 ++++- kernel/sys_ni.c | 1 + 21 files changed, 27 insertions(+), 2 deletions(-)
arch/alpha/kernel/syscalls/syscall.tbl +1 -0
@@ -489,3 +489,4 @@ # 557 reserved for memfd_secret 558 common process_mrelease sys_process_mrelease 559 common futex_waitv sys_futex_waitv +560 common set_mempolicy_home_node sys_ni_syscall
arch/arm/tools/syscall.tbl +1 -0
@@ -463,3 +463,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/arm64/include/asm/unistd.h +1 -1
@@ -38,7 +38,7 @@#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5)#define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) -#define __NR_compat_syscalls 450 +#define __NR_compat_syscalls 451#endif #define __ARCH_WANT_SYS_CLONE
arch/arm64/include/asm/unistd32.h +2 -0
@@ -905,6 +905,8 @@ __SYSCALL(__NR_landlock_restrict_self, sys_landlock_restrict_self) __SYSCALL(__NR_process_mrelease, sys_process_mrelease)#define __NR_futex_waitv 449 __SYSCALL(__NR_futex_waitv, sys_futex_waitv) +#define __NR_set_mempolicy_home_node 450 +__SYSCALL (__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)/* * Please add new compat syscalls above this comment and update
arch/ia64/kernel/syscalls/syscall.tbl +1 -0
@@ -370,3 +370,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/m68k/kernel/syscalls/syscall.tbl +1 -0
@@ -449,3 +449,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/microblaze/kernel/syscalls/syscall.tbl +1 -0
@@ -455,3 +455,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/mips/kernel/syscalls/syscall_n32.tbl +1 -0
@@ -388,3 +388,4 @@ # 447 reserved for memfd_secret 448 n32 process_mrelease sys_process_mrelease 449 n32 futex_waitv sys_futex_waitv +450 n32 set_mempolicy_home_node sys_set_mempolicy_home_node
arch/mips/kernel/syscalls/syscall_n64.tbl +1 -0
@@ -364,3 +364,4 @@ # 447 reserved for memfd_secret 448 n64 process_mrelease sys_process_mrelease 449 n64 futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/mips/kernel/syscalls/syscall_o32.tbl +1 -0
@@ -437,3 +437,4 @@ # 447 reserved for memfd_secret 448 o32 process_mrelease sys_process_mrelease 449 o32 futex_waitv sys_futex_waitv +450 o32 set_mempolicy_home_node sys_set_mempolicy_home_node
arch/parisc/kernel/syscalls/syscall.tbl +1 -0
@@ -447,3 +447,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/powerpc/kernel/syscalls/syscall.tbl +1 -0
@@ -529,3 +529,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 nospu set_mempolicy_home_node sys_set_mempolicy_home_node
arch/s390/kernel/syscalls/syscall.tbl +1 -0
@@ -452,3 +452,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node sys_set_mempolicy_home_node
arch/sh/kernel/syscalls/syscall.tbl +1 -0
@@ -452,3 +452,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/sparc/kernel/syscalls/syscall.tbl +1 -0
@@ -495,3 +495,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -454,3 +454,4 @@ 447 i386 memfd_secret sys_memfd_secret 448 i386 process_mrelease sys_process_mrelease 449 i386 futex_waitv sys_futex_waitv +450 i386 set_mempolicy_home_node sys_set_mempolicy_home_node
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -371,6 +371,7 @@ 447 common memfd_secret sys_memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node # # Due to a historical design error, certain syscalls are numbered differently
arch/xtensa/kernel/syscalls/syscall.tbl +1 -0
@@ -420,3 +420,4 @@ # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv +450 common set_mempolicy_home_node sys_set_mempolicy_home_node
include/linux/syscalls.h +3 -0
@@ -1057,6 +1057,9 @@ asmlinkage long sys_landlock_add_rule(int ruleset_fd, enum landlock_rule_type ru const void __user *rule_attr, __u32 flags);asmlinkage long sys_landlock_restrict_self (int ruleset_fd,__u32 flags);asmlinkage long sys_memfd_secret (unsigned int flags); +asmlinkage long sys_set_mempolicy_home_node (unsigned long start,unsigned long len, +unsigned long home_node, +unsigned long flags);/* * Architecture-specific system calls
include/uapi/asm-generic/unistd.h +4 -1
@@ -883,8 +883,11 @@ __SYSCALL(__NR_process_mrelease, sys_process_mrelease)#define __NR_futex_waitv 449 __SYSCALL(__NR_futex_waitv, sys_futex_waitv) +#define __NR_set_mempolicy_home_node 450 +__SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node) +#undef __NR_syscalls -#define __NR_syscalls 450 +#define __NR_syscalls 451/* * 32 bit systems traditionally used different
kernel/sys_ni.c +1 -0
@@ -297,6 +297,7 @@ COND_SYSCALL(get_mempolicy);COND_SYSCALL (set_mempolicy);COND_SYSCALL (migrate_pages);COND_SYSCALL (move_pages); +COND_SYSCALL (set_mempolicy_home_node);COND_SYSCALL (perf_event_open);COND_SYSCALL (accept4);
Signature · 1 line
2.31.1
Re: [PATCH v6 2/3] mm/mempolicy: add set_mempolicy_home_node syscall
In reply to Aneesh Kumar K.VQuoted 55 lines · Aneesh Kumar K.V
This syscall can be used to set a home node for the
MPOL_BINDandMPOL_PREFERRED_MANYmemory policy. Users should use this syscall after setting up a memory policy for the specified range as shown below.mbind(p,
nr_pages*page_size,MPOL_BIND,new_nodes->maskp,new_nodes->size + 1, 0); sys_set_mempolicy_home_node((unsigned long)p, nr_pages * page_size, home_node, 0);The syscall allows specifying a home node/preferred node from which kernel will fulfill memory allocation requests first.
For address range with
MPOL_BINDmemory policy, if nodemask specifies more than one node, page allocations will come from the node in the nodemask with sufficient free memory that is closest to the home node/preferred node.For
MPOL_PREFERRED_MANYif the nodemask specifies more than one node, page allocation will come from the node in the nodemask with sufficient free memory that is closest to the home node/preferred node. If there is not enough memory in all the nodes specified in the nodemask, the allocation will be attempted from the closest numa node to the home node in the system.This helps applications to hint at a memory allocation preference node and fallback to _only_ a set of nodes if the memory is not available on the preferred node. Fallback allocation is attempted from the node which is nearest to the preferred node.
This helps applications to have control on memory allocation numa nodes and avoids default fallback to slow memory NUMA nodes. For example a system with NUMA nodes 1,2 and 3 with DRAM memory and 10, 11 and 12 of slow memory
new_nodes=numa_bitmask_alloc(nr_nodes);
numa_bitmask_setbit(new_nodes, 1);
numa_bitmask_setbit(new_nodes, 2);
numa_bitmask_setbit(new_nodes, 3);p =
mmap(NULL, nr_pages * page_size, protflag, mapflag, -1, 0);mbind(p, nr_pages * page_size, MPOL_BIND, new_nodes->maskp, new_nodes->size + 1, 0);
sys_set_mempolicy_home_node(p, nr_pages * page_size, 2, 0);This will allocate from nodes closer to node 2 and will make sure the kernel will only allocate from nodes 1, 2, and 3. Memory will not be allocated from slow memory nodes 10, 11, and 12. This differs from default
MPOL_BINDbehavior in that with defaultMPOL_BINDthe allocation will be attempted from node closer to the local node. One of the reasons to specify a home node is to allow allocations from cpu less NUMA node and its nearby NUMA nodes.With
MPOL_PREFERRED_MANYon the other hand will first try to allocate from the closest node to node 2 from the node list 1, 2 and 3. If those nodes don't have enough memory, kernel will allocate from slow memory node 10, 11 and 12 which ever is closer to node 2.
Quoted 14 lines
Cc Ben Widawsky <[email protected]>Cc Dave Hansen <[email protected]>Cc Feng Tang <[email protected]>Cc Michal Hocko <[email protected]>Cc Andrea Arcangeli <[email protected]>Cc Mel Gorman <[email protected]>Cc Mike Kravetz <[email protected]>Cc Randy Dunlap <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Andi Kleen <[email protected]>Cc Dan Williams <[email protected]>Cc Huang Ying <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>
LGTM
Thanks!
Signature · 2 lines
Michal Hocko SUSE Labs