mm/userfaultfd/poll: Scale userfaultfd wakeups
Subsystems (14)
Peter Xu
reviewercover mm/userfaultfd/poll: Scale userfaultfd wakeups
Userfaultfd is the type of file that doesn't need wake-all semantics: if there is a message enqueued (for either a fault address, or an event), we only need to wake up one service thread to handle it. Waking up more normally means a waste of cpu cycles. Besides that, and more importantly, that just doesn't scale.
Andrea used to have one patch that made read() to be O(1) but never hit upstream. This is my effort to try upstreaming that (which is a oneliner..), meanwhile on top of that I also made poll() O(1) on wakeup, too (more or less bring EPOLLEXCLUSIVE to poll()), with some tests showing that effect.
To verify this, I added a test called uffd-perf (leveraging the refactored uffd selftest suite) that will measure the messaging channel latencies on wakeups, and the waitqueue optimizations can be reflected by the new test:
Constants: 40 uffd threads, on N_CPUS=40, memsize=512M
Units: milliseconds (to finish the test)
|-----------------+--------+-------+------------|
| test case | before | after | diff (%) |
|-----------------+--------+-------+------------|
| workers=8,poll | 1762 | 1133 | -55.516328 |
| workers=8,read | 1437 | 585 | -145.64103 |
| workers=16,poll | 1117 | 1097 | -1.8231541 |
| workers=16,read | 1159 | 759 | -52.700922 |
| workers=32,poll | 1001 | 973 | -2.8776978 |
| workers=32,read | 866 | 713 | -21.458626 |
|-----------------+--------+-------+------------|The more threads hanging on the fd_wqh, a bigger difference will be there shown in the numbers. "8 worker threads" is the worst case here because it means there can be a worst case of 40-8=32 threads hanging idle on fd_wqh queue.
In real life, workers can be more than this, but small number of active worker threads will cause similar effect.
This is currently based on Andrew's mm-unstable branch, but assuming this is applicable to most of the not-so-old trees.
Comments welcomed, thanks.
Andrea Arcangeli (1):
mm/userfaultfd: Make uffd read() wait event exclusive
Peter Xu (6):
poll: Add a poll_flags for poll_queue_proc()
poll: POLL_ENQUEUE_EXCLUSIVE
fs/userfaultfd: Use exclusive waitqueue for poll() selftests/mm: Replace uffd_read_mutex with a semaphore selftests/mm: Create uffd_fault_thread_create|join() selftests/mm: uffd perf test
drivers/vfio/virqfd.c | 4 +- drivers/vhost/vhost.c | 2 +- drivers/virt/acrn/irqfd.c | 2 +- fs/aio.c | 2 +- fs/eventpoll.c | 2 +- fs/select.c | 9 +- fs/userfaultfd.c | 8 +- include/linux/poll.h | 25 ++- io_uring/poll.c | 4 +- mm/memcontrol.c | 4 +- net/9p/trans_fd.c | 3 +- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/uffd-common.c | 65 +++++++ tools/testing/selftests/mm/uffd-common.h | 7 + tools/testing/selftests/mm/uffd-perf.c | 207 +++++++++++++++++++++++ tools/testing/selftests/mm/uffd-stress.c | 53 +----- virt/kvm/eventfd.c | 2 +- 17 files changed, 337 insertions(+), 64 deletions(-) create mode 100644 tools/testing/selftests/mm/uffd-perf.c
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH 1/7 mm/userfaultfd: Make uffd read() wait event exclusive
From: Andrea Arcangeli <[email protected]>
When a new message is generated for an userfaultfd, instead of waking up all the readers, we can wake up only one exclusive reader to process the event. Waking up >1 readers for 1 message will be a waste of resource, where the rest readers will see nothing again and re-queue.
This should make userfaultfd read() O(1) on wakeups.
Note that queuing on head is intended (rather than tail) to make sure the readers are waked up in LIFO fashion; fairness doesn't matter much here, but caching does.
--- fs/userfaultfd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/userfaultfd.c +5 -1
@@ -1061,7 +1061,11 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,/* always take the fd_wqh lock before the fault_pending_wqh lock */ spin_lock_irq (&ctx->fd_wqh.lock); -__add_wait_queue (&ctx->fd_wqh, &wait); +/* +* Only wake up one exclusive reader each time there's an event. +* Paired with wake_up_poll() when e.g. a new page fault msg generated. +*/ +__add_wait_queue_exclusive (&ctx->fd_wqh, &wait);for (;;) {set_current_state (TASK_INTERRUPTIBLE );spin_lock (&ctx->fault_pending_wqh.lock);
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH 2/7 poll: Add a poll_flags for poll_queue_proc()
In reply to Peter XuAllows the poll enqueue function to pass over a flag into it.
--- drivers/vfio/virqfd.c | 4 ++-- drivers/vhost/vhost.c | 2 +- drivers/virt/acrn/irqfd.c | 2 +- fs/aio.c | 2 +- fs/eventpoll.c | 2 +- fs/select.c | 4 ++-- include/linux/poll.h | 7 +++++-- io_uring/poll.c | 4 ++-- mm/memcontrol.c | 4 +++- net/9p/trans_fd.c | 3 ++- virt/kvm/eventfd.c | 2 +- 11 files changed, 21 insertions(+), 15 deletions(-)
drivers/vfio/virqfd.c +2 -2
@@ -75,8 +75,8 @@ static int virqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, voidreturn 0 ; } -static void virqfd_ptable_queue_proc (struct file *file, -wait_queue_head_t *wqh,poll_table *pt) +static void virqfd_ptable_queue_proc (struct file *file,wait_queue_head_t *wqh, +poll_table *pt,poll_flags flags) { struct virqfd *virqfd = container_of(pt, struct virqfd, pt);add_wait_queue (wqh, &virqfd->wait);
drivers/vhost/vhost.c +1 -1
@@ -162,7 +162,7 @@ static void vhost_poll_func(struct file *file, wait_queue_head_t *wqh, }static int vhost_poll_wakeup (wait_queue_entry_t *wait,unsigned mode,int sync, -void *key) +void *key,poll_flags flags) { struct vhost_poll *poll = container_of(wait, struct vhost_poll, wait);struct vhost_work *work = &poll->work;
drivers/virt/acrn/irqfd.c +1 -1
@@ -94,7 +94,7 @@ static int hsm_irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode, }static void hsm_irqfd_poll_func (struct file *file,wait_queue_head_t *wqh, -poll_table *pt) +poll_table *pt,poll_flags flags) {struct hsm_irqfd *irqfd;
fs/aio.c +1 -1
@@ -1823,7 +1823,7 @@ struct aio_poll_table {static void aio_poll_queue_proc (struct file *file,struct wait_queue_head *head, -struct poll_table_struct *p) +struct poll_table_struct *p,poll_flags flags) { struct aio_poll_table *pt = container_of(p, struct aio_poll_table, pt);
fs/eventpoll.c +1 -1
@@ -1270,7 +1270,7 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v* target file wakeup lists. */ static void ep_ptable_queue_proc (struct file *file,wait_queue_head_t *whead, -poll_table *pt) +poll_table *pt,poll_flags flags) { struct ep_pqueue *epq = container_of(pt, struct ep_pqueue, pt);struct epitem *epi = epq->epi;
fs/select.c +2 -2
@@ -117,7 +117,7 @@ struct poll_table_page {* poll table. */ static void __pollwait (struct file *filp,wait_queue_head_t *wait_address, -poll_table *p); +poll_table *p,poll_flags flags);void poll_initwait (struct poll_wqueues *pwq) {
@@ -220,7 +220,7 @@ static int pollwake(wait_queue_entry_t *wait, unsigned mode, int sync, void *key/* Add a new entry */ static void __pollwait (struct file *filp,wait_queue_head_t *wait_address, -poll_table *p) +poll_table *p,poll_flags flags) { struct poll_wqueues *pwq = container_of(p, struct poll_wqueues, pt);struct poll_table_entry *entry =poll_get_entry (pwq);
include/linux/poll.h +5 -2
@@ -27,12 +27,15 @@#define DEFAULT_POLLMASK (EPOLLIN | EPOLLOUT | EPOLLRDNORM | EPOLLWRNORM) +typedef unsigned int poll_flags ; +struct poll_table_struct ;/* * structures and helpers for f_op->poll implementations */ -typedef void (*poll_queue_proc )(struct file *,wait_queue_head_t *,struct poll_table_struct *); +typedef void (*poll_queue_proc )(struct file *,wait_queue_head_t *, +struct poll_table_struct *,poll_flags );/* * Do not touch the structure directly, use the access functions
@@ -46,7 +49,7 @@ typedef struct poll_table_struct {static inline void poll_wait (struct file * filp,wait_queue_head_t * wait_address,poll_table *p) {if (p && p->_qproc && wait_address) - p->_qproc (filp, wait_address, p); + p->_qproc (filp, wait_address, p,0 ); }/*
io_uring/poll.c +2 -2
@@ -533,7 +533,7 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, }static void io_poll_queue_proc (struct file *file,struct wait_queue_head *head, -struct poll_table_struct *p) +struct poll_table_struct *p,poll_flags flags) { struct io_poll_table *pt = container_of(p, struct io_poll_table, pt); struct io_poll *poll = io_kiocb_to_cmd(pt->req, struct io_poll);
@@ -644,7 +644,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, }static void io_async_queue_proc (struct file *file,struct wait_queue_head *head, -struct poll_table_struct *p) +struct poll_table_struct *p,poll_flags flags) { struct io_poll_table *pt = container_of(p, struct io_poll_table, pt);struct async_poll *apoll = pt->req->apoll;
mm/memcontrol.c +3 -1
@@ -4877,7 +4877,9 @@ static int memcg_event_wake(wait_queue_entry_t *wait, unsigned mode, }static void memcg_event_ptable_queue_proc (struct file *file, -wait_queue_head_t *wqh,poll_table *pt) +wait_queue_head_t *wqh, +poll_table *pt, +poll_flags flags) {struct mem_cgroup_event *event = container_of(pt, struct mem_cgroup_event, pt);
net/9p/trans_fd.c +2 -1
@@ -550,7 +550,8 @@ static int p9_pollwake(wait_queue_entry_t *wait, unsigned int mode, int sync, vo*/ static void -p9_pollwait (struct file *filp,wait_queue_head_t *wait_address,poll_table *p) +p9_pollwait (struct file *filp,wait_queue_head_t *wait_address,poll_table *p, +poll_flags flags) { struct p9_conn *m = container_of(p, struct p9_conn, pt);struct p9_poll_wait *pwait =NULL ;
virt/kvm/eventfd.c +1 -1
@@ -246,7 +246,7 @@ irqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync, void *key)static void irqfd_ptable_queue_proc (struct file *file,wait_queue_head_t *wqh, -poll_table *pt) +poll_table *pt,poll_flags flags) {struct kvm_kernel_irqfd *irqfd = container_of(pt, struct kvm_kernel_irqfd, pt);
Signature · 1 line
2.41.0
kernel test robot
build botRe: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc()
kernel test robot noticed the following build warnings:
Hi Peter,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Xu/mm-userfaultfd-Make-uffd-read-wait-event-exclusive/20230906-054430 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230905214235.320571-3-peterx%40redhat.com patch subject: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc() config: m68k-allyesconfig (https://download.01.org/0day-ci/archive/20230906/[email protected]/config) compiler: m68k-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230906/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> net/9p/trans_fd.c:555: warning: Function parameter or member 'flags' not described in 'p9_pollwait'
vim +555 net/9p/trans_fd.c
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 542
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 543 /**
5503ac565998837 Eric Van Hensbergen 2008-10-13 544 * p9_pollwait - add poll task to the wait queue
5503ac565998837 Eric Van Hensbergen 2008-10-13 545 * @filp: file pointer being polled
5503ac565998837 Eric Van Hensbergen 2008-10-13 546 * @wait_address: wait_q to block on
5503ac565998837 Eric Van Hensbergen 2008-10-13 547 * @p: poll state
ee443996a35c1e0 Eric Van Hensbergen 2008-03-05 548 *
5503ac565998837 Eric Van Hensbergen 2008-10-13 549 * called by files poll operation to add v9fs-poll task to files wait queue
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 550 */
ee443996a35c1e0 Eric Van Hensbergen 2008-03-05 551
5503ac565998837 Eric Van Hensbergen 2008-10-13 552 static void
14649d7d7806aba Peter Xu 2023-09-05 553 p9_pollwait(struct file *filp, wait_queue_head_t *wait_address, poll_table *p,
14649d7d7806aba Peter Xu 2023-09-05 554 poll_flags flags)
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 @555 {
5503ac565998837 Eric Van Hensbergen 2008-10-13 556 struct p9_conn *m = container_of(p, struct p9_conn, pt);
5503ac565998837 Eric Van Hensbergen 2008-10-13 557 struct p9_poll_wait *pwait = NULL;
5503ac565998837 Eric Van Hensbergen 2008-10-13 558 int i;
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 559
5503ac565998837 Eric Van Hensbergen 2008-10-13 560 for (i = 0; i < ARRAY_SIZE(m->poll_wait); i++) {
5503ac565998837 Eric Van Hensbergen 2008-10-13 561 if (m->poll_wait[i].wait_addr == NULL) {
5503ac565998837 Eric Van Hensbergen 2008-10-13 562 pwait = &m->poll_wait[i];
5503ac565998837 Eric Van Hensbergen 2008-10-13 563 break;
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 564 }
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 565 }
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 566
5503ac565998837 Eric Van Hensbergen 2008-10-13 567 if (!pwait) {
5d3851530d6d685 Joe Perches 2011-11-28 568 p9_debug(P9_DEBUG_ERROR, "not enough wait_address slots\n");
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 569 return;
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 570 }
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 571
5503ac565998837 Eric Van Hensbergen 2008-10-13 572 pwait->conn = m;
5503ac565998837 Eric Van Hensbergen 2008-10-13 573 pwait->wait_addr = wait_address;
5503ac565998837 Eric Van Hensbergen 2008-10-13 574 init_waitqueue_func_entry(&pwait->wait, p9_pollwake);
5503ac565998837 Eric Van Hensbergen 2008-10-13 575 add_wait_queue(wait_address, &pwait->wait);
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 576 }
8a0dc95fd976a05 Eric Van Hensbergen 2008-02-06 577 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc()
In reply to Peter Xukernel test robot noticed the following build errors:
Hi Peter,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Xu/mm-userfaultfd-Make-uffd-read-wait-event-exclusive/20230906-054430 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230905214235.320571-3-peterx%40redhat.com patch subject: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc() config: x86_64-buildonly-randconfig-005-20230906 (https://download.01.org/0day-ci/archive/20230907/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230907/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
drivers/xen/privcmd.c: In function 'privcmd_irqfd_assign':
>> drivers/xen/privcmd.c:965:40: error: passing argument 2 of 'init_poll_funcptr' from incompatible pointer type [-Werror=incompatible-pointer-types]
965 | init_poll_funcptr(&kirqfd->pt, irqfd_poll_func);
| ^~~~~~~~~~~~~~~
| |
| void (*)(struct file *, wait_queue_head_t *, poll_table *) {aka void (*)(struct file *, struct wait_queue_head *, struct poll_table_struct *)} In file included from drivers/xen/privcmd.c:17:
include/linux/poll.h:76:70: note: expected 'poll_queue_proc' {aka 'void (*)(struct file *, struct wait_queue_head *, struct poll_table_struct *, unsigned int)'} but argument is of type 'void (*)(struct file *, wait_queue_head_t *, poll_table *)' {aka 'void (*)(struct file *, struct wait_queue_head *, struct poll_table_struct *)'}
76 | static inline void init_poll_funcptr(poll_table *pt, poll_queue_proc qproc)
| ~~~~~~~~~~~~~~~~^~~~~
cc1: some warnings being treated as errors
vim +/init_poll_funcptr +965 drivers/xen/privcmd.c
f8941e6c4c7129 Viresh Kumar 2023-08-22 924
f8941e6c4c7129 Viresh Kumar 2023-08-22 925 static int privcmd_irqfd_assign(struct privcmd_irqfd *irqfd)
f8941e6c4c7129 Viresh Kumar 2023-08-22 926 {
f8941e6c4c7129 Viresh Kumar 2023-08-22 927 struct privcmd_kernel_irqfd *kirqfd, *tmp;
f8941e6c4c7129 Viresh Kumar 2023-08-22 928 __poll_t events;
f8941e6c4c7129 Viresh Kumar 2023-08-22 929 struct fd f;
f8941e6c4c7129 Viresh Kumar 2023-08-22 930 void *dm_op;
f8941e6c4c7129 Viresh Kumar 2023-08-22 931 int ret;
f8941e6c4c7129 Viresh Kumar 2023-08-22 932
f8941e6c4c7129 Viresh Kumar 2023-08-22 933 kirqfd = kzalloc(sizeof(*kirqfd) + irqfd->size, GFP_KERNEL);
f8941e6c4c7129 Viresh Kumar 2023-08-22 934 if (!kirqfd)
f8941e6c4c7129 Viresh Kumar 2023-08-22 935 return -ENOMEM;
f8941e6c4c7129 Viresh Kumar 2023-08-22 936 dm_op = kirqfd + 1;
f8941e6c4c7129 Viresh Kumar 2023-08-22 937
f8941e6c4c7129 Viresh Kumar 2023-08-22 938 if (copy_from_user(dm_op, irqfd->dm_op, irqfd->size)) {
f8941e6c4c7129 Viresh Kumar 2023-08-22 939 ret = -EFAULT;
f8941e6c4c7129 Viresh Kumar 2023-08-22 940 goto error_kfree;
f8941e6c4c7129 Viresh Kumar 2023-08-22 941 }
f8941e6c4c7129 Viresh Kumar 2023-08-22 942
f8941e6c4c7129 Viresh Kumar 2023-08-22 943 kirqfd->xbufs.size = irqfd->size;
f8941e6c4c7129 Viresh Kumar 2023-08-22 944 set_xen_guest_handle(kirqfd->xbufs.h, dm_op);
f8941e6c4c7129 Viresh Kumar 2023-08-22 945 kirqfd->dom = irqfd->dom;
f8941e6c4c7129 Viresh Kumar 2023-08-22 946 INIT_WORK(&kirqfd->shutdown, irqfd_shutdown);
f8941e6c4c7129 Viresh Kumar 2023-08-22 947
f8941e6c4c7129 Viresh Kumar 2023-08-22 948 f = fdget(irqfd->fd);
f8941e6c4c7129 Viresh Kumar 2023-08-22 949 if (!f.file) {
f8941e6c4c7129 Viresh Kumar 2023-08-22 950 ret = -EBADF;
f8941e6c4c7129 Viresh Kumar 2023-08-22 951 goto error_kfree;
f8941e6c4c7129 Viresh Kumar 2023-08-22 952 }
f8941e6c4c7129 Viresh Kumar 2023-08-22 953
f8941e6c4c7129 Viresh Kumar 2023-08-22 954 kirqfd->eventfd = eventfd_ctx_fileget(f.file);
f8941e6c4c7129 Viresh Kumar 2023-08-22 955 if (IS_ERR(kirqfd->eventfd)) {
f8941e6c4c7129 Viresh Kumar 2023-08-22 956 ret = PTR_ERR(kirqfd->eventfd);
f8941e6c4c7129 Viresh Kumar 2023-08-22 957 goto error_fd_put;
f8941e6c4c7129 Viresh Kumar 2023-08-22 958 }
f8941e6c4c7129 Viresh Kumar 2023-08-22 959
f8941e6c4c7129 Viresh Kumar 2023-08-22 960 /*
f8941e6c4c7129 Viresh Kumar 2023-08-22 961 * Install our own custom wake-up handling so we are notified via a
f8941e6c4c7129 Viresh Kumar 2023-08-22 962 * callback whenever someone signals the underlying eventfd.
f8941e6c4c7129 Viresh Kumar 2023-08-22 963 */
f8941e6c4c7129 Viresh Kumar 2023-08-22 964 init_waitqueue_func_entry(&kirqfd->wait, irqfd_wakeup);
f8941e6c4c7129 Viresh Kumar 2023-08-22 @965 init_poll_funcptr(&kirqfd->pt, irqfd_poll_func);
f8941e6c4c7129 Viresh Kumar 2023-08-22 966
f8941e6c4c7129 Viresh Kumar 2023-08-22 967 mutex_lock(&irqfds_lock);
f8941e6c4c7129 Viresh Kumar 2023-08-22 968
f8941e6c4c7129 Viresh Kumar 2023-08-22 969 list_for_each_entry(tmp, &irqfds_list, list) {
f8941e6c4c7129 Viresh Kumar 2023-08-22 970 if (kirqfd->eventfd == tmp->eventfd) {
f8941e6c4c7129 Viresh Kumar 2023-08-22 971 ret = -EBUSY;
f8941e6c4c7129 Viresh Kumar 2023-08-22 972 mutex_unlock(&irqfds_lock);
f8941e6c4c7129 Viresh Kumar 2023-08-22 973 goto error_eventfd;
f8941e6c4c7129 Viresh Kumar 2023-08-22 974 }
f8941e6c4c7129 Viresh Kumar 2023-08-22 975 }
f8941e6c4c7129 Viresh Kumar 2023-08-22 976
f8941e6c4c7129 Viresh Kumar 2023-08-22 977 list_add_tail(&kirqfd->list, &irqfds_list);
f8941e6c4c7129 Viresh Kumar 2023-08-22 978 mutex_unlock(&irqfds_lock);
f8941e6c4c7129 Viresh Kumar 2023-08-22 979
f8941e6c4c7129 Viresh Kumar 2023-08-22 980 /*
f8941e6c4c7129 Viresh Kumar 2023-08-22 981 * Check if there was an event already pending on the eventfd before we
f8941e6c4c7129 Viresh Kumar 2023-08-22 982 * registered, and trigger it as if we didn't miss it.
f8941e6c4c7129 Viresh Kumar 2023-08-22 983 */
f8941e6c4c7129 Viresh Kumar 2023-08-22 984 events = vfs_poll(f.file, &kirqfd->pt);
f8941e6c4c7129 Viresh Kumar 2023-08-22 985 if (events & EPOLLIN)
f8941e6c4c7129 Viresh Kumar 2023-08-22 986 irqfd_inject(kirqfd);
f8941e6c4c7129 Viresh Kumar 2023-08-22 987
f8941e6c4c7129 Viresh Kumar 2023-08-22 988 /*
f8941e6c4c7129 Viresh Kumar 2023-08-22 989 * Do not drop the file until the kirqfd is fully initialized, otherwise
f8941e6c4c7129 Viresh Kumar 2023-08-22 990 * we might race against the EPOLLHUP.
f8941e6c4c7129 Viresh Kumar 2023-08-22 991 */
f8941e6c4c7129 Viresh Kumar 2023-08-22 992 fdput(f);
f8941e6c4c7129 Viresh Kumar 2023-08-22 993 return 0;
f8941e6c4c7129 Viresh Kumar 2023-08-22 994
f8941e6c4c7129 Viresh Kumar 2023-08-22 995 error_eventfd:
f8941e6c4c7129 Viresh Kumar 2023-08-22 996 eventfd_ctx_put(kirqfd->eventfd);
f8941e6c4c7129 Viresh Kumar 2023-08-22 997
f8941e6c4c7129 Viresh Kumar 2023-08-22 998 error_fd_put:
f8941e6c4c7129 Viresh Kumar 2023-08-22 999 fdput(f);
f8941e6c4c7129 Viresh Kumar 2023-08-22 1000
f8941e6c4c7129 Viresh Kumar 2023-08-22 1001 error_kfree:
f8941e6c4c7129 Viresh Kumar 2023-08-22 1002 kfree(kirqfd);
f8941e6c4c7129 Viresh Kumar 2023-08-22 1003 return ret;
f8941e6c4c7129 Viresh Kumar 2023-08-22 1004 }
f8941e6c4c7129 Viresh Kumar 2023-08-22 1005 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc()
In reply to Peter Xukernel test robot noticed the following build errors:
Hi Peter,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
url: https://github.com/intel-lab-lkp/linux/commits/Peter-Xu/mm-userfaultfd-Make-uffd-read-wait-event-exclusive/20230906-054430 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20230905214235.320571-3-peterx%40redhat.com patch subject: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc() config: hexagon-randconfig-002-20230906 (https://download.01.org/0day-ci/archive/20230907/[email protected]/config) compiler: clang version 14.0.6 (https://github.com/llvm/llvm-project.git f28c006a5895fc0e329fe15fead81e37457cb1d1) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20230907/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from drivers/vhost/vhost.c:14:
In file included from include/uapi/linux/vhost.h:14:
In file included from include/uapi/linux/vhost_types.h:16:
In file included from include/linux/virtio_config.h:7:
In file included from include/linux/virtio.h:7:
In file included from include/linux/scatterlist.h:9:
In file included from arch/hexagon/include/asm/io.h:337:
include/asm-generic/io.h:547:31: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __raw_readb(PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:560:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le16_to_cpu((__le16 __force)__raw_readw(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:37:51: note: expanded from macro '__le16_to_cpu'
#define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
^
In file included from drivers/vhost/vhost.c:14:
In file included from include/uapi/linux/vhost.h:14:
In file included from include/uapi/linux/vhost_types.h:16:
In file included from include/linux/virtio_config.h:7:
In file included from include/linux/virtio.h:7:
In file included from include/linux/scatterlist.h:9:
In file included from arch/hexagon/include/asm/io.h:337:
include/asm-generic/io.h:573:61: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
val = __le32_to_cpu((__le32 __force)__raw_readl(PCI_IOBASE + addr));
~~~~~~~~~~ ^
include/uapi/linux/byteorder/little_endian.h:35:51: note: expanded from macro '__le32_to_cpu'
#define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
^
In file included from drivers/vhost/vhost.c:14:
In file included from include/uapi/linux/vhost.h:14:
In file included from include/uapi/linux/vhost_types.h:16:
In file included from include/linux/virtio_config.h:7:
In file included from include/linux/virtio.h:7:
In file included from include/linux/scatterlist.h:9:
In file included from arch/hexagon/include/asm/io.h:337:
include/asm-generic/io.h:584:33: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writeb(value, PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:594:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writew((u16 __force)cpu_to_le16(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
include/asm-generic/io.h:604:59: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
__raw_writel((u32 __force)cpu_to_le32(value), PCI_IOBASE + addr);
~~~~~~~~~~ ^
>> drivers/vhost/vhost.c:193:41: error: incompatible function pointer types passing 'int (wait_queue_entry_t *, unsigned int, int, void *, poll_flags)' (aka 'int (struct wait_queue_entry *, unsigned int, int, void *, unsigned int)') to parameter of type 'wait_queue_func_t' (aka 'int (*)(struct wait_queue_entry *, unsigned int, int, void *)') [-Werror,-Wincompatible-function-pointer-types]
init_waitqueue_func_entry(&poll->wait, vhost_poll_wakeup);
^~~~~~~~~~~~~~~~~
include/linux/wait.h:90:80: note: passing argument to parameter 'func' here
init_waitqueue_func_entry(struct wait_queue_entry *wq_entry, wait_queue_func_t func)
^
>> drivers/vhost/vhost.c:194:34: error: incompatible function pointer types passing 'void (struct file *, wait_queue_head_t *, poll_table *)' (aka 'void (struct file *, struct wait_queue_head *, struct poll_table_struct *)') to parameter of type 'poll_queue_proc' (aka 'void (*)(struct file *, struct wait_queue_head *, struct poll_table_struct *, unsigned int)') [-Werror,-Wincompatible-function-pointer-types]
init_poll_funcptr(&poll->table, vhost_poll_func);
^~~~~~~~~~~~~~~
include/linux/poll.h:76:70: note: passing argument to parameter 'qproc' here
static inline void init_poll_funcptr(poll_table *pt, poll_queue_proc qproc)
^
>> drivers/vhost/vhost.c:215:57: error: too few arguments to function call, expected 5, have 4
vhost_poll_wakeup(&poll->wait, 0, 0, poll_to_key(mask));
~~~~~~~~~~~~~~~~~ ^
drivers/vhost/vhost.c:164:12: note: 'vhost_poll_wakeup' declared here
static int vhost_poll_wakeup(wait_queue_entry_t *wait, unsigned mode, int sync,
^6 warnings and 3 errors generated.
vim +193 drivers/vhost/vhost.c
87d6a412bd1ed8 Michael S. Tsirkin 2010-09-02 187
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 188 /* Init poll structure */
c23f3445e68e1d Tejun Heo 2010-06-02 189 void vhost_poll_init(struct vhost_poll *poll, vhost_work_fn_t fn,
493b94bf5ae0f6 Mike Christie 2023-06-26 190 __poll_t mask, struct vhost_dev *dev,
493b94bf5ae0f6 Mike Christie 2023-06-26 191 struct vhost_virtqueue *vq)
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 192 {
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 @193 init_waitqueue_func_entry(&poll->wait, vhost_poll_wakeup);
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 @194 init_poll_funcptr(&poll->table, vhost_poll_func);
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 195 poll->mask = mask;
c23f3445e68e1d Tejun Heo 2010-06-02 196 poll->dev = dev;
2b8b328b61c799 Jason Wang 2013-01-28 197 poll->wqh = NULL;
493b94bf5ae0f6 Mike Christie 2023-06-26 198 poll->vq = vq;
c23f3445e68e1d Tejun Heo 2010-06-02 199
87d6a412bd1ed8 Michael S. Tsirkin 2010-09-02 200 vhost_work_init(&poll->work, fn);
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 201 }
6ac1afbf6132df Asias He 2013-05-06 202 EXPORT_SYMBOL_GPL(vhost_poll_init);
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 203
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 204 /* Start polling a file. We add ourselves to file's wait queue. The caller must
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 205 * keep a reference to a file until after vhost_poll_stop is called. */
2b8b328b61c799 Jason Wang 2013-01-28 206 int vhost_poll_start(struct vhost_poll *poll, struct file *file)
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 207 {
e6c8adca20ba45 Al Viro 2017-07-03 208 __poll_t mask;
d47effe1be0c4f Krishna Kumar 2011-03-01 209
70181d51209cbc Jason Wang 2013-04-10 210 if (poll->wqh)
70181d51209cbc Jason Wang 2013-04-10 211 return 0;
70181d51209cbc Jason Wang 2013-04-10 212
9965ed174e7d38 Christoph Hellwig 2018-03-05 213 mask = vfs_poll(file, &poll->table);
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 214 if (mask)
3ad6f93e98d6df Al Viro 2017-07-03 @215 vhost_poll_wakeup(&poll->wait, 0, 0, poll_to_key(mask));
a9a08845e9acbd Linus Torvalds 2018-02-11 216 if (mask & EPOLLERR) {
dc6455a71c7fc5 Jason Wang 2018-03-27 217 vhost_poll_stop(poll);
896fc242bc1d26 Yunsheng Lin 2019-08-20 218 return -EINVAL;
2b8b328b61c799 Jason Wang 2013-01-28 219 }
2b8b328b61c799 Jason Wang 2013-01-28 220
896fc242bc1d26 Yunsheng Lin 2019-08-20 221 return 0;
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 222 }
6ac1afbf6132df Asias He 2013-05-06 223 EXPORT_SYMBOL_GPL(vhost_poll_start);
3a4d5c94e95935 Michael S. Tsirkin 2010-01-14 224 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Peter Xu
reviewerRe: [PATCH 2/7] poll: Add a poll_flags for poll_queue_proc()
In reply to Peter XuQuoted 3 lines · Peter Xu
Allows the poll enqueue function to pass over a flag into it.
Signed-off-by Peter Xu <[email protected]>
A fixup for this patch (should fix all syzbot errors):
===8<===
From ba55ee0539a7a80b98e7a5e3942c0ee8cabe5f73 Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Tue, 5 Sep 2023 20:05:33 -0400
Subject: [PATCH] fixup! poll: Add a poll_flags for poll_queue_proc()
--- drivers/vhost/vhost.c | 4 ++-- drivers/xen/privcmd.c | 3 ++- net/9p/trans_fd.c | 1 + 3 files changed, 5 insertions(+), 3 deletions(-)
drivers/vhost/vhost.c +2 -2
@@ -152,7 +152,7 @@ static void vhost_flush_work(struct vhost_work *work) }static void vhost_poll_func (struct file *file,wait_queue_head_t *wqh, -poll_table *pt) +poll_table *pt,poll_flags flags) {struct vhost_poll *poll;
@@ -162,7 +162,7 @@ static void vhost_poll_func(struct file *file, wait_queue_head_t *wqh, }static int vhost_poll_wakeup (wait_queue_entry_t *wait,unsigned mode,int sync, -void *key,poll_flags flags) +void *key) { struct vhost_poll *poll = container_of(wait, struct vhost_poll, wait);struct vhost_work *work = &poll->work;
drivers/xen/privcmd.c +2 -1
@@ -914,7 +914,8 @@ irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode, int sync, void *key) }static void -irqfd_poll_func (struct file *file,wait_queue_head_t *wqh,poll_table *pt) +irqfd_poll_func (struct file *file,wait_queue_head_t *wqh,poll_table *pt, +poll_flags flags) {struct privcmd_kernel_irqfd *kirqfd = container_of(pt, struct privcmd_kernel_irqfd, pt);
net/9p/trans_fd.c +1 -0
@@ -545,6 +545,7 @@ static int p9_pollwake(wait_queue_entry_t *wait, unsigned int mode, int sync, vo* @filp: file pointer being polled * @wait_address: wait_q to block on * @p: poll state +* @flags: poll flags * * called by files poll operation to add v9fs-poll task to files wait queue */
Signature · 7 lines
2.41.0 ===8<=== Thanks, -- Peter Xu
Peter Xu
reviewerPATCH 3/7 poll: POLL_ENQUEUE_EXCLUSIVE
In reply to Peter XuAdd a flag for poll_wait() showing that the caller wants the enqueue to be exclusive. It is similar to EPOLLEXCLUSIVE for epoll() but grants kernel poll users to opt-in with more efficient exclusive queuing where applicable.
--- fs/select.c | 5 ++++- include/linux/poll.h | 20 ++++++++++++++++++-- 2 files changed, 22 insertions(+), 3 deletions(-)
fs/select.c +4 -1
@@ -231,7 +231,10 @@ static void __pollwait(struct file *filp, wait_queue_head_t *wait_address, entry->key = p->_key;init_waitqueue_func_entry (&entry->wait, pollwake); entry->wait.private = pwq; -add_wait_queue (wait_address, &entry->wait); +if (flags &POLL_ENQUEUE_EXCLUSIVE ) +add_wait_queue_exclusive (wait_address, &entry->wait); +else +add_wait_queue (wait_address, &entry->wait); } static int poll_schedule_timeout(struct poll_wqueues *pwq, int state,
include/linux/poll.h +18 -2
@@ -29,6 +29,8 @@typedef unsigned int poll_flags ; +#define POLL_ENQUEUE_EXCLUSIVE BIT(0) +struct poll_table_struct ;/*
@@ -46,10 +48,24 @@ typedef struct poll_table_struct {__poll_t _key; } poll_table; -static inline void poll_wait (struct file * filp,wait_queue_head_t * wait_address,poll_table *p) +static inline void __poll_wait (struct file *filp,wait_queue_head_t *wait_address, +poll_table *p,poll_flags flags) {if (p && p->_qproc && wait_address) - p->_qproc (filp, wait_address, p,0 ); + p->_qproc (filp, wait_address, p, flags); +} + +static inline void poll_wait (struct file *filp,wait_queue_head_t *wait_address, +poll_table *p) +{ +__poll_wait (filp, wait_address, p,0 ); +} + +static inline void poll_wait_exclusive (struct file *filp, +wait_queue_head_t *wait_address, +poll_table *p) +{ +__poll_wait (filp, wait_address, p,POLL_ENQUEUE_EXCLUSIVE ); }/*
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH 4/7 fs/userfaultfd: Use exclusive waitqueue for poll()
In reply to Peter XuUserfaultfd is the kind of fd that does not need a wake all semantics when wake up. Enqueue using the new POLL_ENQUEUE_EXCLUSIVE flag.
--- fs/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/userfaultfd.c +1 -1
@@ -994,7 +994,7 @@ static __poll_t userfaultfd_poll(struct file *file, poll_table *wait)struct userfaultfd_ctx *ctx = file->private_data;__poll_t ret; -poll_wait (file, &ctx->fd_wqh, wait); +poll_wait_exclusive (file, &ctx->fd_wqh, wait);if (!userfaultfd_is_initialized (ctx))return EPOLLERR ;
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH 5/7 selftests/mm: Replace uffd_read_mutex with a semaphore
In reply to Peter XuEach uffd read threads unlocks the read mutex first, probably just to make sure the thread is reaching a stage where pthread_cancel() can always work before the main thread moves on.
However keeping the mutex locked always and unlock in the thread is a bit hacky. Replacing it with a semaphore which should be much clearer, where the main thread will wait() and the thread will just post(). Move it to uffd-common.* to be reused later.
--- tools/testing/selftests/mm/uffd-common.c | 1 + tools/testing/selftests/mm/uffd-common.h | 2 ++ tools/testing/selftests/mm/uffd-stress.c | 8 +++----- 3 files changed, 6 insertions(+), 5 deletions(-)
tools/testing/selftests/mm/uffd-common.c +1 -0
@@ -17,6 +17,7 @@ bool map_shared;bool test_uffdio_wp = true;unsigned long long *count_verify;uffd_test_ops_t *uffd_test_ops; +sem_t uffd_read_sem;static int uffd_mem_fd_create (off_t mem_size,bool hugetlb) {
tools/testing/selftests/mm/uffd-common.h +2 -0
@@ -32,6 +32,7 @@#include <inttypes.h> #include <stdint.h> #include <sys/random.h> +#include <semaphore.h> #include "../kselftest.h" #include "vm_util.h"
@@ -97,6 +98,7 @@ extern bool map_shared;extern bool test_uffdio_wp;extern unsigned long long *count_verify;extern volatile bool test_uffdio_copy_eexist; +extern sem_t uffd_read_sem;extern uffd_test_ops_t anon_uffd_test_ops;extern uffd_test_ops_t shmem_uffd_test_ops;
tools/testing/selftests/mm/uffd-stress.c +3 -5
@@ -125,14 +125,12 @@ static int copy_page_retry(int ufd, unsigned long offset)return __copy_page (ufd, offset, true, test_uffdio_wp); } -pthread_mutex_t uffd_read_mutex =PTHREAD_MUTEX_INITIALIZER ; -static void *uffd_read_thread (void *arg) {struct uffd_args *args = (struct uffd_args *)arg;struct uffd_msg msg; -pthread_mutex_unlock (&uffd_read_mutex); +sem_post (&uffd_read_sem);/* from here cancellation is ok */ for (;;) {
@@ -196,7 +194,7 @@ static int stress(struct uffd_args *args) uffd_read_thread, (void *)&args[cpu]))return 1 ; -pthread_mutex_lock (&uffd_read_mutex); +sem_wait (&uffd_read_sem); }if (pthread_create (&background_threads[cpu], &attr, background_thread, (void *)cpu))
@@ -258,7 +256,7 @@ static int userfaultfd_stress(void) zeropage = area;bzero (zeropage, page_size); -pthread_mutex_lock (&uffd_read_mutex); +sem_init (&uffd_read_sem,0 ,0 );pthread_attr_init (&attr);pthread_attr_setstacksize (&attr,16 *1024 *1024 );
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH 6/7 selftests/mm: Create uffd_fault_thread_create|join()
In reply to Peter XuMake them common functions to be reused.
--- tools/testing/selftests/mm/uffd-common.c | 46 ++++++++++++++++++++++ tools/testing/selftests/mm/uffd-common.h | 4 ++ tools/testing/selftests/mm/uffd-stress.c | 49 ++++-------------------- 3 files changed, 57 insertions(+), 42 deletions(-)
tools/testing/selftests/mm/uffd-common.c +46 -0
@@ -555,6 +555,52 @@ void *uffd_poll_thread(void *arg)return NULL ; } +void *uffd_read_thread (void *arg) +{ +struct uffd_args *args = (struct uffd_args *)arg; +struct uffd_msg msg; + +sem_post (&uffd_read_sem); +/* from here cancellation is ok */ + +for (;;) { +if (uffd_read_msg (uffd, &msg)) +continue ; +uffd_handle_page_fault (&msg, args); + } + +return NULL ; +} + +void uffd_fault_thread_create (pthread_t *thread,pthread_attr_t *attr, +struct uffd_args *args,bool poll) +{ +if (poll) { +if (pthread_create (thread, attr, uffd_poll_thread, args)) +err ("uffd_poll_thread create" ); + }else { +if (pthread_create (thread, attr, uffd_read_thread, args)) +err ("uffd_read_thread create" ); +sem_wait (&uffd_read_sem); + } +} + +void uffd_fault_thread_join (pthread_t thread,int cpu,bool poll) +{ +char c =1 ; + +if (poll) { +if (write (pipefd[cpu*2 +1 ], &c,1 ) !=1 ) +err ("pipefd write error" ); + }else { +if (pthread_cancel (thread)) +err ("pthread_cancel()" ); + } + +if (pthread_join (thread,NULL )) +err ("pthread_join()" ); +} +static void retry_copy_page (int ufd,struct uffdio_copy *uffdio_copy,unsigned long offset) {
tools/testing/selftests/mm/uffd-common.h +4 -0
@@ -114,6 +114,10 @@ void uffd_handle_page_fault(struct uffd_msg *msg, struct uffd_args *args);int __copy_page (int ufd,unsigned long offset,bool retry,bool wp);int copy_page (int ufd,unsigned long offset,bool wp);void *uffd_poll_thread (void *arg); +void *uffd_read_thread (void *arg); +void uffd_fault_thread_create (pthread_t *thread,pthread_attr_t *attr, +struct uffd_args *args,bool poll); +void uffd_fault_thread_join (pthread_t thread,int cpu,bool poll);int uffd_open_dev (unsigned int flags);int uffd_open_sys (unsigned int flags);
tools/testing/selftests/mm/uffd-stress.c +7 -42
67 lines not on this page · raw message
Signature · 1 line
2.41.0
Peter Xu
reviewerPATCH 7/7 selftests/mm: uffd perf test
In reply to Peter XuAdd a simple perf test for userfaultfd missing mode, on private anon only. It mostly only tests the messaging, so memory type / fault type may not that much yet.
--- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/uffd-common.c | 18 ++ tools/testing/selftests/mm/uffd-common.h | 1 + tools/testing/selftests/mm/uffd-perf.c | 207 +++++++++++++++++++++++ 4 files changed, 228 insertions(+) create mode 100644 tools/testing/selftests/mm/uffd-perf.c
tools/testing/selftests/mm/Makefile +2 -0
@@ -64,6 +64,7 @@ TEST_GEN_FILES += thuge-gen TEST_GEN_FILES += transhuge-stress TEST_GEN_FILES += uffd-stress TEST_GEN_FILES += uffd-unit-tests +TEST_GEN_FILES += uffd-perf TEST_GEN_FILES += split_huge_page_test TEST_GEN_FILES += ksm_tests TEST_GEN_FILES += ksm_functional_tests
@@ -120,6 +121,7 @@ $(TEST_GEN_FILES): vm_util.c $(OUTPUT)/uffd-stress: uffd-common.c $(OUTPUT)/uffd-unit-tests: uffd-common.c +$(OUTPUT)/uffd-perf: uffd-common.c ifeq ($(ARCH),x86_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32))
tools/testing/selftests/mm/uffd-common.c +18 -0
@@ -725,3 +725,21 @@ int uffd_get_features(uint64_t *features)return 0 ; } + +uint64_t get_usec (void ) +{ +uint64_t val =0 ; +struct timespec t; +int ret =clock_gettime (CLOCK_MONOTONIC , &t); + +if (ret ==-1 ) { +perror ("clock_gettime() failed" ); +/* should never happen */ +exit (-1 ); + } + + val = t.tv_nsec /1000 ;/* ns -> us */ + val += t.tv_sec *1000000 ;/* s -> us */ + +return val; +}
tools/testing/selftests/mm/uffd-common.h +1 -0
@@ -123,6 +123,7 @@ int uffd_open_dev(unsigned int flags);int uffd_open_sys (unsigned int flags);int uffd_open (unsigned int flags);int uffd_get_features (uint64_t *features); +uint64_t get_usec (void );#define TEST_ANON 1#define TEST_HUGETLB 2
tools/testing/selftests/mm/uffd-perf.c new file +207 -0
207 lines not on this page · raw message
Signature · 1 line
2.41.0
Quoted 6 lines · Peter Xu
Userfaultfd is the type of file that doesn't need wake-all semantics: if there is a message enqueued (for either a fault address, or an event), we only need to wake up one service thread to handle it. Waking up more normally means a waste of cpu cycles. Besides that, and more importantly, that just doesn't scale.
Hi Peter,
I took a quick look over the series and didn't see anything objectionable. I was planning to actually test the series out and then send out R-b's, but it will take some additional time (next week).
In the meantime, I was curious about the use case. A design I've seen for VM live migration is to have 1 thread reading events off the uffd, and then have many threads actually resolving the fault events that come in (e.g. fetching pages over the network, issuing UFFDIO_COPY or UFFDIO_CONTINUE, or whatever). In that design, since we only have a single reader anyway, I think this series doesn't help.
But, I'm curious if you have data indicating that > 1 reader is more performant overall? I suspect it might be the case that, with "enough" vCPUs, it makes sense to do so, but I don't have benchmark data to tell me what that tipping point is yet.
OTOH, if one reader is plenty in ~all cases, optimizing this path is less important.
Quoted 71 lines
Andrea used to have one patch that made
read()to be O(1) but never hit upstream. This is my effort to try upstreaming that (which is a oneliner..), meanwhile on top of that I also madepoll()O(1) on wakeup, too (more or less bring EPOLLEXCLUSIVE topoll()), with some tests showing that effect.To verify this, I added a test called uffd-perf (leveraging the refactored uffd selftest suite) that will measure the messaging channel latencies on wakeups, and the waitqueue optimizations can be reflected by the new test:
Constants: 40 uffd threads, on N_CPUS=40, memsize=512M Units: milliseconds (to finish the test) |-----------------+--------+-------+------------| | test case | before | after | diff (%) | |-----------------+--------+-------+------------| | workers=8,poll | 1762 | 1133 | -55.516328 | | workers=8,read | 1437 | 585 | -145.64103 | | workers=16,poll | 1117 | 1097 | -1.8231541 | | workers=16,read | 1159 | 759 | -52.700922 | | workers=32,poll | 1001 | 973 | -2.8776978 | | workers=32,read | 866 | 713 | -21.458626 | |-----------------+--------+-------+------------|The more threads hanging on the
fd_wqh, a bigger difference will be there shown in the numbers. "8 worker threads" is the worst case here because it means there can be a worst case of 40-8=32 threads hanging idle onfd_wqhqueue.In real life, workers can be more than this, but small number of active worker threads will cause similar effect.
This is currently based on Andrew's mm-unstable branch, but assuming this is applicable to most of the not-so-old trees.
Comments welcomed, thanks.
Andrea Arcangeli (1):
mm/userfaultfd: Make uffdread()wait event exclusivePeter Xu (6):
poll: Add apoll_flagsforpoll_queue_proc()
poll:POLL_ENQUEUE_EXCLUSIVE
fs/userfaultfd: Use exclusive waitqueue forpoll()selftests/mm: Replaceuffd_read_mutexwith a semaphore selftests/mm: Create uffd_fault_thread_create|join() selftests/mm: uffd perf testdrivers/vfio/virqfd.c | 4 +- drivers/vhost/vhost.c | 2 +- drivers/virt/acrn/irqfd.c | 2 +- fs/aio.c | 2 +- fs/eventpoll.c | 2 +- fs/select.c | 9 +- fs/userfaultfd.c | 8 +- include/linux/poll.h | 25 ++- io_uring/poll.c | 4 +- mm/memcontrol.c | 4 +- net/9p/trans_fd.c | 3 +- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/uffd-common.c | 65 +++++++ tools/testing/selftests/mm/uffd-common.h | 7 + tools/testing/selftests/mm/uffd-perf.c | 207 +++++++++++++++++++++++ tools/testing/selftests/mm/uffd-stress.c | 53 +----- virt/kvm/eventfd.c | 2 +- 17 files changed, 337 insertions(+), 64 deletions(-) create mode 100644 tools/testing/selftests/mm/uffd-perf.c--
2.41.0
Peter Xu
reviewerQuoted 9 lines · Axel Rasmussen
Quoted 6 lines · Peter Xu
Userfaultfd is the type of file that doesn't need wake-all semantics: if there is a message enqueued (for either a fault address, or an event), we only need to wake up one service thread to handle it. Waking up more normally means a waste of cpu cycles. Besides that, and more importantly, that just doesn't scale.
Hi Peter,
Hi, Axel,
Sorry to respond late.
Quoted 4 lines
I took a quick look over the series and didn't see anything objectionable. I was planning to actually test the series out and then send out R-b's, but it will take some additional time (next week).
Thanks. The 2nd patch definitely needs some fixup on some functions (either I overlooked without enough CONFIG_* chosen; I am surprised I have vhost even compiled out when testing..), hope that won't bring you too much trouble. I'll send a fixup soon on top of patch 2.
Quoted 7 lines
In the meantime, I was curious about the use case. A design I've seen for VM live migration is to have 1 thread reading events off the uffd, and then have many threads actually resolving the fault events that come in (e.g. fetching pages over the network, issuing
UFFDIO_COPYorUFFDIO_CONTINUE, or whatever). In that design, since we only have a single reader anyway, I think this series doesn't help.
Yes. If the test to carry out only uses 1 thread, it shouldn't bring much difference.
Quoted 8 lines
But, I'm curious if you have data indicating that > 1 reader is more performant overall? I suspect it might be the case that, with "enough" vCPUs, it makes sense to do so, but I don't have benchmark data to tell me what that tipping point is yet.
OTOH, if one reader is plenty in ~all cases, optimizing this path is less important.
For myself I don't yet have an application that can leverage this much indeed, because QEMU so far only uses 1 reader thread.
IIRC Anish was exactly proposing some kvm specific solutions to make single uffd scale, and this might be suitable for any use case like that where we may want to use single uffd and try to make it scale with threads. Using 1 reader + N worker is also a solution, but when using N readers (which also do the work) the app will hit this problem.
I am also aware that some apps use more than 1 reader threads (umap), but I don't really know more than that.
The problem is I think we shouldn't have that overhead easily just because an app invokes >1 readers, meanwhile it also doesn't make much sense to wake up all readers for a single event for userfaults. So it should always be something good to have.
Thanks,
Signature · 1 line
Peter Xu