Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0DE4C27C40 for ; Wed, 22 Nov 2023 21:35:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51DA08D0008; Wed, 22 Nov 2023 16:35:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CE548D0005; Wed, 22 Nov 2023 16:35:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 396548D0008; Wed, 22 Nov 2023 16:35:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2B73E8D0005 for ; Wed, 22 Nov 2023 16:35:52 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id EE85340CD2 for ; Wed, 22 Nov 2023 21:35:51 +0000 (UTC) X-FDA: 81486897702.23.46B712E Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf23.hostedemail.com (Postfix) with ESMTP id 3A365140009 for ; Wed, 22 Nov 2023 21:35:50 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="PlfsiR/7"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of vpetrucci@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=vpetrucci@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700688950; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=T8YHcJrVA3B+7BnFIZDKn+9VseQtxV3EDqLRBy0nrns=; b=XttKsblee6Ux8KIFNQPEpws6DEAn05cUqR9yW2uqpxQ+ox9TDKGXlIZSOfvbDWiBbclaEk QpzI2JvBGxHRXP/2fzbSfyn2K++HOaHFrQo/eT7bS7MI48f7mbzm19Av7T3qZFXzaevoZF qsWUXnzucm2vZtdyjIOIq1XqrauIesQ= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="PlfsiR/7"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of vpetrucci@gmail.com designates 209.85.167.176 as permitted sender) smtp.mailfrom=vpetrucci@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700688950; a=rsa-sha256; cv=none; b=JUHsRRhY7KA/S2ujLlNV1ptuVxOe+qbzucRIsytyMQ7zOlAXZLRri5AghmJ/5o2THPeDod rDVKuU3093BwZDTns71BXy4NAJZ2LeDNYvsWR/3Yopm+HmFs3n6GHH8Smc6RTxrZbdZQg+ NzdYgB0n80kA8IJr921D5wt8sw0vSiA= Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3b844357f7cso183029b6e.1 for ; Wed, 22 Nov 2023 13:35:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700688949; x=1701293749; darn=kvack.org; h=content-disposition:mime-version:message-id:subject:to:from:date :from:to:cc:subject:date:message-id:reply-to; bh=T8YHcJrVA3B+7BnFIZDKn+9VseQtxV3EDqLRBy0nrns=; b=PlfsiR/7ReI3Zg304mk+LuQEi2munt1s1QMdCC+jVOGVbEQVuf9frrV/HqEsIfSYpy 7hg4WPLnQHZHYk2b5menuaDUAcK9R/pujQ08KVYtWHNugUZgsdlouwCTGb+tKDxMgNhd os1dF1THlVnzbBNa/pvuMwogzu0zpoIFAv37SaFyTBo+UzXUNNpInVCRIeOWFVHutNs8 2oXXvpfZjXksm42+DbW8dzT7ssnBuEugR/MNpv/QRvZkS6Lzru5DZNSRvNunFUvxiLuN aHVGf5Bkpt2jpRTn32WpUqGvKwJcfRu30Tf1pRAjS+qNKt+DID8xOezX0WXilFGa1X8F dbNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700688949; x=1701293749; h=content-disposition:mime-version:message-id:subject:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T8YHcJrVA3B+7BnFIZDKn+9VseQtxV3EDqLRBy0nrns=; b=MNT24QOG13btmwcgAvevw9Uu96GUh+V9ltfRwHvfM52MDVhqK6EXrSvwQuz3PwRzwD om/xjOS4TryYR6BcLeKvnVyTpoiL+tYzQ+x51LBsMbr6v/jiAeIH0MXVgjCEdiITHiGX LfZSSjHlQccFQbCg6ixjTijbeejyxZcpLXaB0uT2H6GJdDytlJ1/57R1k3rvRsZVN5Is frxgiDLKMnsI7piUT5SgVkrkzUhwjapKvnc/zPXHbyvfwkEVsuRy6QO8VPjlOKzmLc0n ciIheFBfsewZGLt7J6whKIUTrueHD7070mZVfBjXcTV8qy3jplOoL/qGWOnQFVBf84lV sXEA== X-Gm-Message-State: AOJu0Yw9XmSjZ8rFydItj2/0Fe5w8h4ZyxQ96n9k6DNRg78r8lO2H7S5 E3wjAsdCjh/Y44PXGHtk7auFIz/QQlk= X-Google-Smtp-Source: AGHT+IHzI1Mah+zvRRNiDGtCfLuIYA7dIifB8nLcmkwAPVad+/5g6DxyI01y5wOqA8S6IzsnUD6mig== X-Received: by 2002:a05:6808:2788:b0:3ae:a6e:6ccc with SMTP id es8-20020a056808278800b003ae0a6e6cccmr3371125oib.49.1700688948773; Wed, 22 Nov 2023 13:35:48 -0800 (PST) Received: from r13-u19.micron.com ([165.225.243.112]) by smtp.gmail.com with ESMTPSA id 37-20020a630f65000000b005c200b11b77sm141731pgp.86.2023.11.22.13.35.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Nov 2023 13:35:48 -0800 (PST) Date: Wed, 22 Nov 2023 15:35:45 -0600 From: Vinicius Petrucci To: linux-mm@kvack.org Subject: [RFC PATCH] mm/mbind: Introduce process_mbind() syscall for external memory binding Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 3A365140009 X-Stat-Signature: tqcn67z6rfrmi1gcnwza3ygtdi9biuer X-HE-Tag: 1700688950-406058 X-HE-Meta: U2FsdGVkX1/zl2ndNmJtCdoMZuRQMWB2DcJvBYtASh+59YZTKtPjjvpIw6XC5rkzsQQvOTyTpkhpacLs5zfCHduzg/8uIsbSa4mvjrMajYuO6v5PWv7jiFQeU4zf+oMcMiAxf5OrrwgLtGdJDcZ7Iw9AAoTvIwqg17cnfTkk4qWuyAyDEmjCNZH3D0s/L1VnjBDVMLYuOUWo0ucVhNwMac7Ge+7Ku3vmFS29x2auXyUg4VdO3DOtTv88zJjqRt0pcuWtB7Ua5bRpHXJI3gWh3oB7PBnykHT2R0P1reXPh2vz6OJT90UjaEmh/vXcrJfEny9RY5H/yU5S7unKiRMv6/Y/djUM2N37krX1oUXVSS+qKu9wCDeU7YzkZwp2JSeKMGVWXc29Ny2EPWijAQAcmfFdXgBTVGnvxQrjjO4n07vW09kGAbOcyeNCr30WjRT4mUI5fIGevKrNPycvhUZHrhjSNtGuOsaznkS0SuQBlvB5TSrpdviXyNj4+Up6/52dvVno/BwpyC9jcKTm5PC97HBALVQwyi1gtMQ43ajAGhMRxNHDxC+vzTG3lr0GCoulOtYTfPcbCQ0AGHAxl+qw01FiHVMuua8rWkvHEaFNA9wajCvZ8PBCM+3zYtQz/BrvQxxNdGn8B1Ts0FauR7fsmGhtQW99O+RrqJT+I4wv0gdR8K8v1+Es1G+WkfgeS1qYYGT1hfiY8UGBhleOjZuFp2VKMG1opa7PqbIPJa27VMqoRHOqEqh0Lo0+bik/OohLY8gGKSXBzpJ+ZsruunCaoO4nSYQvQJLjeoX9MSAmhSKVoP53U0uh8e+Xo3kIL4i26Rsj0Crrz/HFpMC35bgZiuOV9j8HAso/v5EEtZdBD7QNpDEv+O6m8435ukxb3vJ/PRCAu1eVj2iS4PttwQCNLIlQkHkFbn0P9Ekb0n9MapKGqW7iZhRPLuQQ9d/99MZhoyh+abCwvrCsFtHDqPv xexje5O8 7FvN6bVpCuuvQFrvUpQCJPC3eXgSjEN28Vx6jwRrSNd11HdPX45b4z0fFiyI5FKRQYjfxvAqUQzCfB4O0JE1MoUaFNgbGGTbzIAMK2QFKB+2KeIOWsIpN6+3ljtStXrGVXd1V8XO3NbhDjJLaMr1XWKxOKhabKfeUYXqsBNSwru9xnVjScPaRtgqLA8Q3Z+d+LH8cHIpJ7NARox3Pq+0NEOcoCgLLWE/Lk4hr8ukkwH4adcUIhF+C8EinyGUYI2i9kSduteSzgauSdtdo6vOcdCU6CQpE3HqluT+SKU1ktCOcdoqo0gWTVuH2vRlw+nscZV3qiaFqo7TVP6W1MakIqDvyOkpU1zOb2FTwt4IQoJddoO0BZCxngEQdfaZYBnyRkD8RtryeBoAvkZWChLU16NvEYQ9YqWMSdagFySGxI7mCgO25koqisu+IzD3Ptrlp9mYeCPzqhtMtZVQ6ePb8/zQ0u7G5saNqgby0AqGA+nJERWe8QSmEisWHDfI6RxP2GKapYyOdBgySjIYkADksv9gWS8RId6/qXBpZi/wDnq9i3DYT5mbUjeUd+Dr2NFaMr6p0M7+U2hhu0OKwuN6Xht+UwO6Vv/slpbegE6a/ITMZiOw22fHr5v8ObtjH14kpBEQyYq+07uIDQks/8t8vr6zeX3hWFwG052LIkBsKWV56At8nOqFpYWNwYMjX4NofhK3T1kVLscZqvJUq3LZqrW/DkVQWmr727Jhkvr5mKaeeKpAAmXF3Om9GqXG8P7QmFUuO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Vinicius Tavares Petrucci This patch introduces `process_mbind()` to enable a userspace orchestrator with an understanding of another process's memory layout to alter its memory policy. As system memory configurations become more and more complex (e.g., DDR+HBM+CXL memories), such a userspace orchestrator can explore more advanced techniques to guide memory placement to individual NUMA nodes across memory tiers. This allows for a more efficient allocation of memory resources, leading to enhanced application performance. Alternatively, there are existing methods such as LD_PRELOAD (https://pmem.io/memkind/) or syscall_intercept (https://github.com/pmem/syscall_intercept), but these techniques, beyond the lack of portability/universality, can lead to system incompatibility issues, inconsistency in application behavior, potential risks due to global system-wide settings, and increased complexity in implementation. The concept of an external entity that understands the layout of another process's VM is already present with `process_madvise()`. Thus, it seems reasonable to introduce the `process_mbind` variant of `mbind`. The implementation framework of `process_mbind()` is akin to `process_madvise()`. It uses pidfd of an external process to direct the memory policy and supports a vector of memory address ranges. The general use case here is similar to the prior RFC `pidfd_set_mempolicy()` (https://lore.kernel.org/linux-mm/20221010094842.4123037-1-hezhongkun.hzk@bytedance.com/), but offers a more fine-grained external control by binding specific memory regions (say, heap data structures) to specific NUMA nodes. Another concrete use case was described by a prior work showing up to 2X runtime improvement (compared to AutoNUMA tiering) using memory object/region-based memory placement for workloads with irregular access patterns such as graph analytics: https://iiswc.org/iiswc2022/IISWC2022_42.pdf The proposed API is as follows: long process_mbind(int pidfd, const struct iovec *iovec, unsigned long vlen, unsigned long mode, const unsigned long *nmask, unsigned int flags); The `pidfd` argument is used to select the process that is identified by the PID file descriptor provided in pidfd. (See pidofd_open(2) for more information) The pointer `iovec` points to an array of iovec structures (as described in ): struct iovec { void *iov_base; /* starting address of region */ size_t iov_len; /* size of region (in bytes) */ }; The `iovec` defines memory regions that start at the address (iov_base) and have a size measured in bytes (iov_len). The `vlen` indicates the quantity of elements contained in iovec. Please note the initial `maxnode` parameter from `mbind` was omitted to ensure the API doesn't exceed 6 arguments. Instead, the constant MAX_NUMNODES was utilized. Please see the mbind(2) man page for more details about other's arguments. Additionally, it is worth noting the following: - Using a vector of address ranges as an argument in `process_mbind` provides more flexibility than the original `mbind` system call, even when invoked from a current or local process. - In contrast to `move_pages`, which requires an array of fixed-size pages, `process_mbind` (with flags MPOL_MF_MOVE*) offers a more convinient and flexible page migration capability on a per object or region basis. - Similar to `process_madvise`, manipulating the memory binding of external processes necessitates `CAP_SYS_NICE` and `PTRACE_MODE_READ_FSCREDS` checks (refer to ptrace(2)). Suggested-by: Frank van der Linden Signed-off-by: Vinicius Tavares Petrucci Signed-off-by: Hasan Al Maruf --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 4 ++ include/uapi/asm-generic/unistd.h | 4 +- kernel/sys_ni.c | 1 + mm/mempolicy.c | 86 +++++++++++++++++++++++++- 5 files changed, 92 insertions(+), 4 deletions(-) diff --git a/arch/x86/entry/syscalls/syscall_64.tbl b/arch/x86/entry/syscalls/syscall_64.tbl index 8cb8bf68721c..9d9db49a3242 100644 --- a/arch/x86/entry/syscalls/syscall_64.tbl +++ b/arch/x86/entry/syscalls/syscall_64.tbl @@ -378,6 +378,7 @@ 454 common futex_wake sys_futex_wake 455 common futex_wait sys_futex_wait 456 common futex_requeue sys_futex_requeue +457 common process_mbind sys_process_mbind # # Due to a historical design error, certain syscalls are numbered differently diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index fd9d12de7e92..def5250ed625 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -816,6 +816,10 @@ asmlinkage long sys_mbind(unsigned long start, unsigned long len, const unsigned long __user *nmask, unsigned long maxnode, unsigned flags); +asmlinkage long sys_process_mbind(int pidfd, const struct iovec __user *vec, + size_t vlen, unsigned long mode, + const unsigned long __user *nmask, + unsigned flags); asmlinkage long sys_get_mempolicy(int __user *policy, unsigned long __user *nmask, unsigned long maxnode, diff --git a/include/uapi/asm-generic/unistd.h b/include/uapi/asm-generic/unistd.h index 756b013fb832..9ed2c91940d6 100644 --- a/include/uapi/asm-generic/unistd.h +++ b/include/uapi/asm-generic/unistd.h @@ -828,9 +828,11 @@ __SYSCALL(__NR_futex_wake, sys_futex_wake) __SYSCALL(__NR_futex_wait, sys_futex_wait) #define __NR_futex_requeue 456 __SYSCALL(__NR_futex_requeue, sys_futex_requeue) +#define __NR_process_mbind 457 +__SYSCALL(__NR_process_mbind, sys_process_mbind) #undef __NR_syscalls -#define __NR_syscalls 457 +#define __NR_syscalls 458 /* * 32 bit systems traditionally used different diff --git a/kernel/sys_ni.c b/kernel/sys_ni.c index e1a6e3c675c0..cc5cb5ae3ae7 100644 --- a/kernel/sys_ni.c +++ b/kernel/sys_ni.c @@ -187,6 +187,7 @@ COND_SYSCALL(process_madvise); COND_SYSCALL(process_mrelease); COND_SYSCALL(remap_file_pages); COND_SYSCALL(mbind); +COND_SYSCALL(process_mbind); COND_SYSCALL(get_mempolicy); COND_SYSCALL(set_mempolicy); COND_SYSCALL(migrate_pages); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 10a590ee1c89..91ee300fa728 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1215,11 +1215,10 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src, } #endif -static long do_mbind(unsigned long start, unsigned long len, +static long do_mbind(struct mm_struct *mm, unsigned long start, unsigned long len, unsigned short mode, unsigned short mode_flags, nodemask_t *nmask, unsigned long flags) { - struct mm_struct *mm = current->mm; struct vm_area_struct *vma, *prev; struct vma_iterator vmi; struct migration_mpol mmpol; @@ -1465,10 +1464,84 @@ static inline int sanitize_mpol_flags(int *mode, unsigned short *flags) return 0; } +static long kernel_mbind_process(int pidfd, const struct iovec __user *vec, + size_t vlen, unsigned long mode, + const unsigned long __user *nmask, unsigned int flags) +{ + ssize_t ret; + struct iovec iovstack[UIO_FASTIOV]; + struct iovec *iov = iovstack; + struct iov_iter iter; + struct task_struct *task; + struct mm_struct *mm; + unsigned int f_flags; + unsigned short mode_flags; + int lmode = mode; + unsigned long maxnode = MAX_NUMNODES; + int err; + nodemask_t nodes; + + err = sanitize_mpol_flags(&lmode, &mode_flags); + if (err) + goto out; + + err = get_nodes(&nodes, nmask, maxnode); + if (err) + goto out; + + ret = import_iovec(ITER_DEST, vec, vlen, ARRAY_SIZE(iovstack), + &iov, &iter); + if (ret < 0) + goto out; + + task = pidfd_get_task(pidfd, &f_flags); + if (IS_ERR(task)) { + ret = PTR_ERR(task); + goto free_iov; + } + + /* From process_madvise: Require PTRACE_MODE_READ + * to avoid leaking ASLR metadata. */ + mm = mm_access(task, PTRACE_MODE_READ_FSCREDS); + if (IS_ERR_OR_NULL(mm)) { + ret = IS_ERR(mm) ? PTR_ERR(mm) : -ESRCH; + goto release_task; + } + + /* From process_madvise: Require CAP_SYS_NICE for + * influencing process performance. */ + if (!capable(CAP_SYS_NICE)) { + ret = -EPERM; + goto release_mm; + } + + while (iov_iter_count(&iter)) { + unsigned long start = untagged_addr( + (unsigned long)iter_iov_addr(&iter)); + unsigned long len = iter_iov_len(&iter); + + ret = do_mbind(mm, start, len, lmode, mode_flags, + &nodes, flags); + if (ret < 0) + break; + iov_iter_advance(&iter, iter_iov_len(&iter)); + } + +release_mm: + mmput(mm); +release_task: + put_task_struct(task); +free_iov: + kfree(iov); +out: + return ret; +} + static long kernel_mbind(unsigned long start, unsigned long len, unsigned long mode, const unsigned long __user *nmask, unsigned long maxnode, unsigned int flags) { + struct mm_struct *mm = current->mm; unsigned short mode_flags; nodemask_t nodes; int lmode = mode; @@ -1483,7 +1556,7 @@ static long kernel_mbind(unsigned long start, unsigned long len, if (err) return err; - return do_mbind(start, len, lmode, mode_flags, &nodes, flags); + return do_mbind(mm, start, len, lmode, mode_flags, &nodes, flags); } SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, len, @@ -1553,6 +1626,13 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le return err; } +SYSCALL_DEFINE6(process_mbind, int, pidfd, const struct iovec __user *, vec, + size_t, vlen, unsigned long, mode, + const unsigned long __user *, nmask, unsigned int, flags) +{ + return kernel_mbind_process(pidfd, vec, vlen, mode, nmask, flags); +} + SYSCALL_DEFINE6(mbind, unsigned long, start, unsigned long, len, unsigned long, mode, const unsigned long __user *, nmask, unsigned long, maxnode, unsigned int, flags) -- 2.41.0