Skip to contentSkip to conversation

io_uring epoll wait support

PATCH Superseded Jens Axboe 2 participants 18 messages Feb 4, 2025 – Feb 7, 2025

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

v2 cover io_uring epoll wait support

Hi,

One issue people consistently run into when converting legacy epoll event loops with io_uring is that parts of the event loop still needs to use epoll. And since event loops generally need to wait in one spot, they add the io_uring fd to the epoll set and continue to use epoll_wait(2) to wait on events. This is suboptimal on the io_uring front as there's now an active poller on the ring, and it's suboptimal as it doesn't give the application the batch waiting (with fine grained timeouts) that io_uring provides.

This patchset adds support for IORING_OP_EPOLL_WAIT, which does an async epoll_wait() operation. No sleeping or thread offload is involved, it relies on the wait_queue_entry callback for retries. With that, then the above event loops can continue to use epoll for certain parts, but bundle it all under waiting on the ring itself rather than add the ring fd to the epoll set.

Patches 1..4 are just prep patches, and patch 5 adds the epoll change to allow io_uring to queue a callback, if no events are available. Patches 6..7 are just prep patches on the io_uring side, and patch 8 finally adds IORING_OP_EPOLL_WAIT support. Patch 9 adds multishot support, which further gets rid of repeated write_lock and list manipulations on the struct eventpoll waitqueue head. This last bit should be a nice win, having a persistent waitqueue entry rather than needing to lock/add/unlock for each epoll_wait() equivalent operation.

Patches can also be found here:

https://git.kernel.dk/cgit/linux/log/?h=io_uring-epoll-wait

and are against 6.14-rc1 + already pending io_uring patches.

Since v1:
- Add provided buffer support and require it for multishot
- Fixup various multishot issues
- Various other fixes

 fs/eventpoll.c                 | 155 ++++++++++++++++-------
 include/linux/eventpoll.h      |   8 ++
 include/linux/io_uring_types.h |   4 +
 include/uapi/linux/io_uring.h  |   7 ++
 io_uring/Makefile              |   9 +-
 io_uring/cancel.c              |   5 +
 io_uring/epoll.c               | 222 ++++++++++++++++++++++++++++++++-
 io_uring/epoll.h               |  22 ++++
 io_uring/io_uring.c            |   5 +
 io_uring/opdef.c               |  15 +++
 io_uring/poll.c                |  30 +----
 io_uring/poll.h                |  32 +++++
 12 files changed, 434 insertions(+), 80 deletions(-)
Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH v2 1/11 eventpoll: abstract out main epoll reaper into a function

Add epoll_wait(), which takes a struct file and the number of events etc to reap. This can then be called by do_epoll_wait(), and used by io_uring as well.

No intended functional changes in this patch.

---
 fs/eventpoll.c            | 31 ++++++++++++++++++-------------
 include/linux/eventpoll.h |  4 ++++
 2 files changed, 22 insertions(+), 13 deletions(-)
fs/eventpoll.c +18 -13
index 7c0980db77b3..73b639caed3d 100644
@@ -2445,12 +2445,8 @@ SYSCALL_DEFINE4(epoll_ctl, int, epfd, int, op, int, fd,
 	return do_epoll_ctl(epfd, op, fd, &epds, false);
 }
 
-/*
- * Implement the event wait interface for the eventpoll file. It is the kernel
- * part of the user space epoll_wait(2).
- */
-static int do_epoll_wait(int epfd, struct epoll_event __user *events,
-			 int maxevents, struct timespec64 *to)
+int epoll_wait(struct file *file, struct epoll_event __user *events,
+	       int maxevents, struct timespec64 *to)
 {
 	struct eventpoll *ep;
 
@@ -2462,28 +2458,37 @@ static int do_epoll_wait(int epfd, struct epoll_event __user *events,
 	if (!access_ok(events, maxevents * sizeof(struct epoll_event)))
 		return -EFAULT;
 
-	/* Get the "struct file *" for the eventpoll file */
-	CLASS(fd, f)(epfd);
-	if (fd_empty(f))
-		return -EBADF;
-
 	/*
 	 * We have to check that the file structure underneath the fd
 	 * the user passed to us _is_ an eventpoll file.
 	 */
-	if (!is_file_epoll(fd_file(f)))
+	if (!is_file_epoll(file))
 		return -EINVAL;
 
 	/*
 	 * At this point it is safe to assume that the "private_data" contains
 	 * our own data structure.
 	 */
-	ep = fd_file(f)->private_data;
+	ep = file->private_data;
 
 	/* Time to fish for events ... */
 	return ep_poll(ep, events, maxevents, to);
 }
 
+/*
+ * Implement the event wait interface for the eventpoll file. It is the kernel
+ * part of the user space epoll_wait(2).
+ */
+static int do_epoll_wait(int epfd, struct epoll_event __user *events,
+			 int maxevents, struct timespec64 *to)
+{
+	/* Get the "struct file *" for the eventpoll file */
+	CLASS(fd, f)(epfd);
+	if (!fd_empty(f))
+		return epoll_wait(fd_file(f), events, maxevents, to);
+	return -EBADF;
+}
+
 SYSCALL_DEFINE4(epoll_wait, int, epfd, struct epoll_event __user *, events,
 		int, maxevents, int, timeout)
 {
include/linux/eventpoll.h +4 -0
index 0c0d00fcd131..f37fea931c44 100644
@@ -25,6 +25,10 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd, unsigned long t
 /* Used to release the epoll bits inside the "struct file" */
 void eventpoll_release_file(struct file *file);
 
+/* Use to reap events */
+int epoll_wait(struct file *file, struct epoll_event __user *events,
+	       int maxevents, struct timespec64 *to);
+
 /*
  * This is called from inside fs/file_table.c:__fput() to unlink files
  * from the eventpoll interface. We need to have this facility to cleanup
Signature · 1 line
2.47.2
author maintainer
#2 lore raw

PATCH v2 2/11 eventpoll: add helper to remove wait entry from wait queue head

In reply to Jens Axboe

__epoll_wait_remove() is the core helper, it kills a given wait_queue_entry from the eventpoll wait_queue_head. Use it internally, and provide an overall helper, epoll_wait_remove(), which takes a struct file and provides the same functionality.

---
 fs/eventpoll.c            | 58 +++++++++++++++++++++++++--------------
 include/linux/eventpoll.h |  3 ++
 2 files changed, 40 insertions(+), 21 deletions(-)
fs/eventpoll.c +37 -21
index 73b639caed3d..01edbee5c766 100644
@@ -1980,6 +1980,42 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry,
 	return ret;
 }
 
+static int __epoll_wait_remove(struct eventpoll *ep,
+			       struct wait_queue_entry *wait, int timed_out)
+{
+	int eavail;
+
+	/*
+	 * We were woken up, thus go and try to harvest some events. If timed
+	 * out and still on the wait queue, recheck eavail carefully under
+	 * lock, below.
+	 */
+	eavail = 1;
+
+	if (!list_empty_careful(&wait->entry)) {
+		write_lock_irq(&ep->lock);
+		/*
+		 * If the thread timed out and is not on the wait queue, it
+		 * means that the thread was woken up after its timeout expired
+		 * before it could reacquire the lock. Thus, when wait.entry is
+		 * empty, it needs to harvest events.
+		 */
+		if (timed_out)
+			eavail = list_empty(&wait->entry);
+		__remove_wait_queue(&ep->wq, wait);
+		write_unlock_irq(&ep->lock);
+	}
+
+	return eavail;
+}
+
+int epoll_wait_remove(struct file *file, struct wait_queue_entry *wait)
+{
+	if (is_file_epoll(file))
+		return __epoll_wait_remove(file->private_data, wait, false);
+	return -EINVAL;
+}
+
 /**
  * ep_poll - Retrieves ready events, and delivers them to the caller-supplied
  *           event buffer.
@@ -2100,27 +2136,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 							      HRTIMER_MODE_ABS);
 		__set_current_state(TASK_RUNNING);
 
-		/*
-		 * We were woken up, thus go and try to harvest some events.
-		 * If timed out and still on the wait queue, recheck eavail
-		 * carefully under lock, below.
-		 */
-		eavail = 1;
-
-		if (!list_empty_careful(&wait.entry)) {
-			write_lock_irq(&ep->lock);
-			/*
-			 * If the thread timed out and is not on the wait queue,
-			 * it means that the thread was woken up after its
-			 * timeout expired before it could reacquire the lock.
-			 * Thus, when wait.entry is empty, it needs to harvest
-			 * events.
-			 */
-			if (timed_out)
-				eavail = list_empty(&wait.entry);
-			__remove_wait_queue(&ep->wq, &wait);
-			write_unlock_irq(&ep->lock);
-		}
+		eavail = __epoll_wait_remove(ep, &wait, timed_out);
 	}
 }
 
include/linux/eventpoll.h +3 -0
index f37fea931c44..1301fc74aca0 100644
@@ -29,6 +29,9 @@ void eventpoll_release_file(struct file *file);
 int epoll_wait(struct file *file, struct epoll_event __user *events,
 	       int maxevents, struct timespec64 *to);
 
+/* Remove wait entry */
+int epoll_wait_remove(struct file *file, struct wait_queue_entry *wait);
+
 /*
  * This is called from inside fs/file_table.c:__fput() to unlink files
  * from the eventpoll interface. We need to have this facility to cleanup
Signature · 1 line
2.47.2
author maintainer
#3 lore raw

PATCH v2 3/11 eventpoll: abstract out ep_try_send_events() helper

In reply to Jens Axboe

In preparation for reusing this helper in another epoll setup helper, abstract it out.

---
 fs/eventpoll.c | 28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)
fs/eventpoll.c +18 -10
index 01edbee5c766..3cbd290503c7 100644
@@ -2016,6 +2016,22 @@ int epoll_wait_remove(struct file *file, struct wait_queue_entry *wait)
 	return -EINVAL;
 }
 
+static int ep_try_send_events(struct eventpoll *ep,
+			      struct epoll_event __user *events, int maxevents)
+{
+	int res;
+
+	/*
+	 * Try to transfer events to user space. In case we get 0 events and
+	 * there's still timeout left over, we go trying again in search of
+	 * more luck.
+	 */
+	res = ep_send_events(ep, events, maxevents);
+	if (res > 0)
+		ep_suspend_napi_irqs(ep);
+	return res;
+}
+
 /**
  * ep_poll - Retrieves ready events, and delivers them to the caller-supplied
  *           event buffer.
@@ -2067,17 +2083,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 
 	while (1) {
 		if (eavail) {
-			/*
-			 * Try to transfer events to user space. In case we get
-			 * 0 events and there's still timeout left over, we go
-			 * trying again in search of more luck.
-			 */
-			res = ep_send_events(ep, events, maxevents);
-			if (res) {
-				if (res > 0)
-					ep_suspend_napi_irqs(ep);
+			res = ep_try_send_events(ep, events, maxevents);
+			if (res)
 				return res;
-			}
 		}
 
 		if (timed_out)
Signature · 1 line
2.47.2
author maintainer
#4 lore raw

PATCH v2 4/11 eventpoll: add struct wait_queue_entry argument to epoll_wait()

In reply to Jens Axboe

In preparation for allowing an outside caller to add itself to the epoll waitqueue, pass in a struct wait_queue_entry. Unused in its current form, but will be utilized shortly.

No intended functional changes in this patch.

---
 fs/eventpoll.c            | 5 +++--
 include/linux/eventpoll.h | 3 ++-
 2 files changed, 5 insertions(+), 3 deletions(-)
fs/eventpoll.c +3 -2
index 3cbd290503c7..ecaa5591f4be 100644
@@ -2470,7 +2470,8 @@ SYSCALL_DEFINE4(epoll_ctl, int, epfd, int, op, int, fd,
 }
 
 int epoll_wait(struct file *file, struct epoll_event __user *events,
-	       int maxevents, struct timespec64 *to)
+	       int maxevents, struct timespec64 *to,
+	       struct wait_queue_entry *wait)
 {
 	struct eventpoll *ep;
 
@@ -2509,7 +2510,7 @@ static int do_epoll_wait(int epfd, struct epoll_event __user *events,
 	/* Get the "struct file *" for the eventpoll file */
 	CLASS(fd, f)(epfd);
 	if (!fd_empty(f))
-		return epoll_wait(fd_file(f), events, maxevents, to);
+		return epoll_wait(fd_file(f), events, maxevents, to, NULL);
 	return -EBADF;
 }
 
include/linux/eventpoll.h +2 -1
index 1301fc74aca0..24f9344df5a3 100644
@@ -27,7 +27,8 @@ void eventpoll_release_file(struct file *file);
 
 /* Use to reap events */
 int epoll_wait(struct file *file, struct epoll_event __user *events,
-	       int maxevents, struct timespec64 *to);
+	       int maxevents, struct timespec64 *to,
+	       struct wait_queue_entry *wait);
 
 /* Remove wait entry */
 int epoll_wait_remove(struct file *file, struct wait_queue_entry *wait);
Signature · 1 line
2.47.2
author maintainer
#5 lore raw

PATCH v2 5/11 eventpoll: add ep_poll_queue() loop

In reply to Jens Axboe

If a wait_queue_entry is passed in to epoll_wait(), then utilize this new helper for reaping events and/or adding to the epoll waitqueue rather than calling the potentially sleeping ep_poll(). It works like ep_poll(), except it doesn't block - it either returns the events that are already available, or it adds the specified entry to the struct eventpoll waitqueue to get a callback when events are triggered. It returns -EIOCBQUEUED for that case.

---
 fs/eventpoll.c | 37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
fs/eventpoll.c +36 -1
index ecaa5591f4be..a8be0c7110e4 100644
@@ -2032,6 +2032,39 @@ static int ep_try_send_events(struct eventpoll *ep,
 	return res;
 }
 
+static int ep_poll_queue(struct eventpoll *ep,
+			 struct epoll_event __user *events, int maxevents,
+			 struct wait_queue_entry *wait)
+{
+	int res, eavail;
+
+	/* See ep_poll() for commentary */
+	eavail = ep_events_available(ep);
+	while (1) {
+		if (eavail) {
+			res = ep_try_send_events(ep, events, maxevents);
+			if (res)
+				return res;
+		}
+
+		eavail = ep_busy_loop(ep, true);
+		if (eavail)
+			continue;
+
+		if (!list_empty_careful(&wait->entry))
+			return -EIOCBQUEUED;
+
+		write_lock_irq(&ep->lock);
+		eavail = ep_events_available(ep);
+		if (!eavail)
+			__add_wait_queue_exclusive(&ep->wq, wait);
+		write_unlock_irq(&ep->lock);
+
+		if (!eavail)
+			return -EIOCBQUEUED;
+	}
+}
+
 /**
  * ep_poll - Retrieves ready events, and delivers them to the caller-supplied
  *           event buffer.
@@ -2497,7 +2530,9 @@ int epoll_wait(struct file *file, struct epoll_event __user *events,
 	ep = file->private_data;
 
 	/* Time to fish for events ... */
-	return ep_poll(ep, events, maxevents, to);
+	if (!wait)
+		return ep_poll(ep, events, maxevents, to);
+	return ep_poll_queue(ep, events, maxevents, wait);
 }
 
 /*
Signature · 1 line
2.47.2
#6 lore raw

Re: [PATCH 05/11] eventpoll: add ep_poll_queue() loop

Quoted 27 lines · Jens Axboe

If a wait_queue_entry is passed in to epoll_wait(), then utilize this new helper for reaping events and/or adding to the epoll waitqueue rather than calling the potentially sleeping ep_poll(). It works like ep_poll(), except it doesn't block - it either returns the events that are already available, or it adds the specified entry to the struct eventpoll waitqueue to get a callback when events are triggered. It returns -EIOCBQUEUED for that case.

---
  fs/eventpoll.c | 37 ++++++++++++++++++++++++++++++++++++-
  1 file changed, 36 insertions(+), 1 deletion(-)
fs/eventpoll.c +6 -0
index ecaa5591f4be..a8be0c7110e4 100644
@@ -2032,6 +2032,39 @@ static int ep_try_send_events(struct eventpoll *ep,
  	return res;
  }
 
+static int ep_poll_queue(struct eventpoll *ep,
+			 struct epoll_event __user *events, int maxevents,
+			 struct wait_queue_entry *wait)
+{
+	int res, eavail;
+

fs/eventpoll.c static int ep_try_send_events(struct eventpoll *ep, · line 2050

Quoted 10 lines
fs/eventpoll.c +10 -0
+	/* See ep_poll() for commentary */
+	eavail = ep_events_available(ep);
+	while (1) {
+		if (eavail) {
+			res = ep_try_send_events(ep, events, maxevents);
+			if (res)
+				return res;
+		}
+
+		eavail = ep_busy_loop(ep, true);

I have doubts we want to busy loop here even if it's just one iteration / nonblockinf. And there is already napi polling support in io_uring done from the right for io_uring users spot.

Signature · 1 line
Pavel Begunkov
author maintainer
#7 lore raw

Re: [PATCH 05/11] eventpoll: add ep_poll_queue() loop

Quoted 33 lines · Pavel Begunkov
Quoted 32 lines · Jens Axboe

If a wait_queue_entry is passed in to epoll_wait(), then utilize this new helper for reaping events and/or adding to the epoll waitqueue rather than calling the potentially sleeping ep_poll(). It works like ep_poll(), except it doesn't block - it either returns the events that are already available, or it adds the specified entry to the struct eventpoll waitqueue to get a callback when events are triggered. It returns -EIOCBQUEUED for that case.

---
  fs/eventpoll.c | 37 ++++++++++++++++++++++++++++++++++++-

  1 file changed, 36 insertions(+), 1 deletion(-)
fs/eventpoll.c +11 -0
index ecaa5591f4be..a8be0c7110e4 100644
@@ -2032,6 +2032,39 @@ static int ep_try_send_events(struct eventpoll *ep,
       return res;
   }
   +static int ep_poll_queue(struct eventpoll *ep,
+             struct epoll_event __user *events, int maxevents,
+             struct wait_queue_entry *wait)
+{
+    int res, eavail;
+
+    /* See ep_poll() for commentary */
+    eavail = ep_events_available(ep);
+    while (1) {
+        if (eavail) {
+            res = ep_try_send_events(ep, events, maxevents);
+            if (res)
Quoted 8 lines

fs/eventpoll.c static int ep_try_send_events(struct eventpoll *ep, · line 2050

Quoted 4 lines
fs/eventpoll.c +4 -0
+                return res;
+        }
+
+        eavail = ep_busy_loop(ep, true);

I have doubts we want to busy loop here even if it's just one iteration / nonblockinf. And there is already napi polling support in io_uring done from the right for io_uring users spot.

Yeah I did ponder that which is why it's passing in the timed_out == true to just do a single loop. We could certainly get rid of that.

Signature · 1 line
Jens Axboe
author maintainer
#8 lore raw

PATCH v2 6/11 io_uring/epoll: remove CONFIG_EPOLL guards

In reply to Jens Axboe

Just have the Makefile add the object if epoll is enabled, then it's not necessary to guard the entire epoll.c file inside an CONFIG_EPOLL ifdef.

---
 io_uring/Makefile | 9 +++++----
 io_uring/epoll.c  | 2 --
 2 files changed, 5 insertions(+), 6 deletions(-)
io_uring/Makefile +5 -4
index d695b60dba4f..7114a6dbd439 100644
@@ -11,9 +11,10 @@ obj-$(CONFIG_IO_URING)		+= io_uring.o opdef.o kbuf.o rsrc.o notif.o \
 					eventfd.o uring_cmd.o openclose.o \
 					sqpoll.o xattr.o nop.o fs.o splice.o \
 					sync.o msg_ring.o advise.o openclose.o \
-					epoll.o statx.o timeout.o fdinfo.o \
-					cancel.o waitid.o register.o \
-					truncate.o memmap.o alloc_cache.o
+					statx.o timeout.o fdinfo.o cancel.o \
+					waitid.o register.o truncate.o \
+					memmap.o alloc_cache.o
 obj-$(CONFIG_IO_WQ)		+= io-wq.o
 obj-$(CONFIG_FUTEX)		+= futex.o
-obj-$(CONFIG_NET_RX_BUSY_POLL) += napi.o
+obj-$(CONFIG_EPOLL)		+= epoll.o
+obj-$(CONFIG_NET_RX_BUSY_POLL)	+= napi.o
io_uring/epoll.c +0 -2
index 89bff2068a19..7848d9cc073d 100644
@@ -12,7 +12,6 @@
 #include "io_uring.h"
 #include "epoll.h"
 
-#if defined(CONFIG_EPOLL)
 struct io_epoll {
 	struct file			*file;
 	int				epfd;
@@ -58,4 +57,3 @@ int io_epoll_ctl(struct io_kiocb *req, unsigned int issue_flags)
 	io_req_set_res(req, ret, 0);
 	return IOU_OK;
 }
-#endif
Signature · 1 line
2.47.2
author maintainer
#9 lore raw

PATCH v2 7/11 io_uring/poll: pull ownership handling into poll.h

In reply to Jens Axboe

In preparation for using it from somewhere else. Rather than try and duplicate the functionality, just make it generically available to io_uring opcodes.

Note: would have to be used carefully, cannot be used by opcodes that can trigger poll logic.

---
 io_uring/poll.c | 30 +-----------------------------
 io_uring/poll.h | 31 +++++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+), 29 deletions(-)
io_uring/poll.c +1 -29
index bb1c0cd4f809..5e44ac562491 100644
@@ -41,16 +41,6 @@ struct io_poll_table {
 	__poll_t result_mask;
 };
 
-#define IO_POLL_CANCEL_FLAG	BIT(31)
-#define IO_POLL_RETRY_FLAG	BIT(30)
-#define IO_POLL_REF_MASK	GENMASK(29, 0)
-
-/*
- * We usually have 1-2 refs taken, 128 is more than enough and we want to
- * maximise the margin between this amount and the moment when it overflows.
- */
-#define IO_POLL_REF_BIAS	128
-
 #define IO_WQE_F_DOUBLE		1
 
 static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync,
@@ -70,7 +60,7 @@ static inline bool wqe_is_double(struct wait_queue_entry *wqe)
 	return priv & IO_WQE_F_DOUBLE;
 }
 
-static bool io_poll_get_ownership_slowpath(struct io_kiocb *req)
+bool io_poll_get_ownership_slowpath(struct io_kiocb *req)
 {
 	int v;
 
@@ -85,24 +75,6 @@ static bool io_poll_get_ownership_slowpath(struct io_kiocb *req)
 	return !(atomic_fetch_inc(&req->poll_refs) & IO_POLL_REF_MASK);
 }
 
-/*
- * If refs part of ->poll_refs (see IO_POLL_REF_MASK) is 0, it's free. We can
- * bump it and acquire ownership. It's disallowed to modify requests while not
- * owning it, that prevents from races for enqueueing task_work's and b/w
- * arming poll and wakeups.
- */
-static inline bool io_poll_get_ownership(struct io_kiocb *req)
-{
-	if (unlikely(atomic_read(&req->poll_refs) >= IO_POLL_REF_BIAS))
-		return io_poll_get_ownership_slowpath(req);
-	return !(atomic_fetch_inc(&req->poll_refs) & IO_POLL_REF_MASK);
-}
-
-static void io_poll_mark_cancelled(struct io_kiocb *req)
-{
-	atomic_or(IO_POLL_CANCEL_FLAG, &req->poll_refs);
-}
-
 static struct io_poll *io_poll_get_double(struct io_kiocb *req)
 {
 	/* pure poll stashes this in ->async_data, poll driven retry elsewhere */
io_uring/poll.h +31 -0
index 04ede93113dc..2f416cd3be13 100644
@@ -21,6 +21,18 @@ struct async_poll {
 	struct io_poll		*double_poll;
 };
 
+#define IO_POLL_CANCEL_FLAG	BIT(31)
+#define IO_POLL_RETRY_FLAG	BIT(30)
+#define IO_POLL_REF_MASK	GENMASK(29, 0)
+
+bool io_poll_get_ownership_slowpath(struct io_kiocb *req);
+
+/*
+ * We usually have 1-2 refs taken, 128 is more than enough and we want to
+ * maximise the margin between this amount and the moment when it overflows.
+ */
+#define IO_POLL_REF_BIAS	128
+
 /*
  * Must only be called inside issue_flags & IO_URING_F_MULTISHOT, or
  * potentially other cases where we already "own" this poll request.
@@ -30,6 +42,25 @@ static inline void io_poll_multishot_retry(struct io_kiocb *req)
 	atomic_inc(&req->poll_refs);
 }
 
+/*
+ * If refs part of ->poll_refs (see IO_POLL_REF_MASK) is 0, it's free. We can
+ * bump it and acquire ownership. It's disallowed to modify requests while not
+ * owning it, that prevents from races for enqueueing task_work's and b/w
+ * arming poll and wakeups.
+ */
+static inline bool io_poll_get_ownership(struct io_kiocb *req)
+{
+	if (unlikely(atomic_read(&req->poll_refs) >= IO_POLL_REF_BIAS))
+		return io_poll_get_ownership_slowpath(req);
+	return !(atomic_fetch_inc(&req->poll_refs) & IO_POLL_REF_MASK);
+}
+
+static inline void io_poll_mark_cancelled(struct io_kiocb *req)
+{
+	atomic_or(IO_POLL_CANCEL_FLAG, &req->poll_refs);
+}
+
+
 int io_poll_add_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
 int io_poll_add(struct io_kiocb *req, unsigned int issue_flags);
 
Signature · 1 line
2.47.2
author maintainer
#10 lore raw

PATCH v2 8/11 io_uring/poll: add IO_POLL_FINISH_FLAG

In reply to Jens Axboe

Use the same value as the retry, doesn't need to be unique, just available if the poll owning mechanism user wishes to use it for something other than a retry condition.

---
 io_uring/poll.h | 1 +
 1 file changed, 1 insertion(+)
io_uring/poll.h +1 -0
index 2f416cd3be13..97d14b2b2751 100644
@@ -23,6 +23,7 @@ struct async_poll {
 
 #define IO_POLL_CANCEL_FLAG	BIT(31)
 #define IO_POLL_RETRY_FLAG	BIT(30)
+#define IO_POLL_FINISH_FLAG	IO_POLL_RETRY_FLAG
 #define IO_POLL_REF_MASK	GENMASK(29, 0)
 
 bool io_poll_get_ownership_slowpath(struct io_kiocb *req);
Signature · 1 line
2.47.2
#11 lore raw

Re: [PATCH 08/11] io_uring/poll: add IO_POLL_FINISH_FLAG

Quoted 14 lines · Jens Axboe

Use the same value as the retry, doesn't need to be unique, just available if the poll owning mechanism user wishes to use it for something other than a retry condition.

---
  io_uring/poll.h | 1 +
  1 file changed, 1 insertion(+)
io_uring/poll.h +0 -0
index 2f416cd3be13..97d14b2b2751 100644

io_uring/poll.h struct async_poll { · line 26

Quoted 4 lines
io_uring/poll.h +1 -0
@@ -23,6 +23,7 @@ struct async_poll {
 
  #define IO_POLL_CANCEL_FLAG	BIT(31)
  #define IO_POLL_RETRY_FLAG	BIT(30)
+#define IO_POLL_FINISH_FLAG	IO_POLL_RETRY_FLAG

The patches use io_poll_get_ownership(), which already might set the flag and with a different meaning put into it.

Quoted 3 lines
io_uring/poll.h +0 -0
  #define IO_POLL_REF_MASK	GENMASK(29, 0)
 
  bool io_poll_get_ownership_slowpath(struct io_kiocb *req);
Signature · 1 line
Pavel Begunkov
author maintainer
#12 lore raw

Re: [PATCH 08/11] io_uring/poll: add IO_POLL_FINISH_FLAG

Quoted 10 lines · Pavel Begunkov
Quoted 9 lines · Jens Axboe

Use the same value as the retry, doesn't need to be unique, just available if the poll owning mechanism user wishes to use it for something other than a retry condition.

---
  io_uring/poll.h | 1 +

  1 file changed, 1 insertion(+)
Quoted 11 lines

io_uring/poll.h struct async_poll { · line 25

Quoted 8 lines
io_uring/poll.h +1 -0
index 2f416cd3be13..97d14b2b2751 100644
@@ -23,6 +23,7 @@ struct async_poll {
     #define IO_POLL_CANCEL_FLAG    BIT(31)
   #define IO_POLL_RETRY_FLAG    BIT(30)
+#define IO_POLL_FINISH_FLAG    IO_POLL_RETRY_FLAG

The patches use io_poll_get_ownership(), which already might set the flag and with a different meaning put into it.

Oh true, I totally missed that. I'll just add another flag for it instead and shrink the ref mask.

Signature · 1 line
Jens Axboe
author maintainer
#13 lore raw

PATCH v2 9/11 io_uring/epoll: add support for IORING_OP_EPOLL_WAIT

In reply to Jens Axboe

For existing epoll event loops that can't fully convert to io_uring, the used approach is usually to add the io_uring fd to the epoll instance and use epoll_wait() to wait on both "legacy" and io_uring events. While this work, it isn't optimal as:

1) epoll_wait() is pretty limited in what it can do. It does not support partial reaping of events, or waiting on a batch of events.

2) When an io_uring ring is added to an epoll instance, it activates the io_uring "I'm being polled" logic which slows things down.

Rather than use this approach, with EPOLL_WAIT support added to io_uring, event loops can use the normal io_uring wait logic for everything, as long as an epoll wait request has been armed with io_uring.

Note that IORING_OP_EPOLL_WAIT does NOT take a timeout value, as this is an async request. Waiting on io_uring events in general has various timeout parameters, and those are the ones that should be used when waiting on any kind of request. If events are immediately available for reaping, then This opcode will return those immediately. If none are available, then it will post an async completion when they become available.

cqe->res will contain either an error code (< 0 value) for a malformed request, invalid epoll instance, etc. It will return a positive result indicating how many events were reaped.

IORING_OP_EPOLL_WAIT requests may be canceled using the normal io_uring cancelation infrastructure. The poll logic for managing ownership is adopted to guard the epoll side too.

---
 include/linux/io_uring_types.h |   4 +
 include/uapi/linux/io_uring.h  |   1 +
 io_uring/cancel.c              |   5 +
 io_uring/epoll.c               | 169 +++++++++++++++++++++++++++++++++
 io_uring/epoll.h               |  22 +++++
 io_uring/io_uring.c            |   5 +
 io_uring/opdef.c               |  14 +++
 7 files changed, 220 insertions(+)
include/linux/io_uring_types.h +4 -0
index 3def525a1da3..ee56992d31d5 100644
@@ -370,6 +370,10 @@ struct io_ring_ctx {
 	struct io_alloc_cache	futex_cache;
 #endif
 
+#ifdef CONFIG_EPOLL
+	struct hlist_head	epoll_list;
+#endif
+
 	const struct cred	*sq_creds;	/* cred used for __io_sq_thread() */
 	struct io_sq_data	*sq_data;	/* if using sq thread polling */
 
include/uapi/linux/io_uring.h +1 -0
index e11c82638527..a559e1e1544a 100644
@@ -278,6 +278,7 @@ enum io_uring_op {
 	IORING_OP_FTRUNCATE,
 	IORING_OP_BIND,
 	IORING_OP_LISTEN,
+	IORING_OP_EPOLL_WAIT,
 
 	/* this goes last, obviously */
 	IORING_OP_LAST,
io_uring/cancel.c +5 -0
index 484193567839..9cebd0145cb4 100644
@@ -17,6 +17,7 @@
 #include "timeout.h"
 #include "waitid.h"
 #include "futex.h"
+#include "epoll.h"
 #include "cancel.h"
 
 struct io_cancel {
@@ -128,6 +129,10 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd,
 	if (ret != -ENOENT)
 		return ret;
 
+	ret = io_epoll_wait_cancel(ctx, cd, issue_flags);
+	if (ret != -ENOENT)
+		return ret;
+
 	spin_lock(&ctx->completion_lock);
 	if (!(cd->flags & IORING_ASYNC_CANCEL_FD))
 		ret = io_timeout_cancel(ctx, cd);
io_uring/epoll.c +169 -0

184 lines not on this page · raw message

io_uring/epoll.h +22 -0
index 870cce11ba98..296940d89063 100644
@@ -1,6 +1,28 @@
 // SPDX-License-Identifier: GPL-2.0
 
+#include "cancel.h"
+
 #if defined(CONFIG_EPOLL)
+int io_epoll_wait_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd,
+			 unsigned int issue_flags);
+bool io_epoll_wait_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx,
+			      bool cancel_all);
+
 int io_epoll_ctl_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
 int io_epoll_ctl(struct io_kiocb *req, unsigned int issue_flags);
+int io_epoll_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
+int io_epoll_wait(struct io_kiocb *req, unsigned int issue_flags);
+#else
+static inline bool io_epoll_wait_remove_all(struct io_ring_ctx *ctx,
+					    struct io_uring_task *tctx,
+					    bool cancel_all)
+{
+	return false;
+}
+static inline int io_epoll_wait_cancel(struct io_ring_ctx *ctx,
+				       struct io_cancel_data *cd,
+				       unsigned int issue_flags)
+{
+	return 0;
+}
 #endif
io_uring/io_uring.c +5 -0
index e34a92c73a5d..78375981907d 100644
@@ -93,6 +93,7 @@
 #include "notif.h"
 #include "waitid.h"
 #include "futex.h"
+#include "epoll.h"
 #include "napi.h"
 #include "uring_cmd.h"
 #include "msg_ring.h"
@@ -358,6 +359,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
 	INIT_HLIST_HEAD(&ctx->waitid_list);
 #ifdef CONFIG_FUTEX
 	INIT_HLIST_HEAD(&ctx->futex_list);
+#endif
+#ifdef CONFIG_EPOLL
+	INIT_HLIST_HEAD(&ctx->epoll_list);
 #endif
 	INIT_DELAYED_WORK(&ctx->fallback_work, io_fallback_req_func);
 	INIT_WQ_LIST(&ctx->submit_state.compl_reqs);
@@ -3084,6 +3088,7 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx,
 	ret |= io_poll_remove_all(ctx, tctx, cancel_all);
 	ret |= io_waitid_remove_all(ctx, tctx, cancel_all);
 	ret |= io_futex_remove_all(ctx, tctx, cancel_all);
+	ret |= io_epoll_wait_remove_all(ctx, tctx, cancel_all);
 	ret |= io_uring_try_cancel_uring_cmd(ctx, tctx, cancel_all);
 	mutex_unlock(&ctx->uring_lock);
 	ret |= io_kill_timeouts(ctx, tctx, cancel_all);
io_uring/opdef.c +14 -0
index e8baef4e5146..44553a657476 100644
@@ -514,6 +514,17 @@ const struct io_issue_def io_issue_defs[] = {
 		.async_size		= sizeof(struct io_async_msghdr),
 #else
 		.prep			= io_eopnotsupp_prep,
+#endif
+	},
+	[IORING_OP_EPOLL_WAIT] = {
+		.needs_file		= 1,
+		.unbound_nonreg_file	= 1,
+		.audit_skip		= 1,
+#if defined(CONFIG_EPOLL)
+		.prep			= io_epoll_wait_prep,
+		.issue			= io_epoll_wait,
+#else
+		.prep			= io_eopnotsupp_prep,
 #endif
 	},
 };
@@ -745,6 +756,9 @@ const struct io_cold_def io_cold_defs[] = {
 	[IORING_OP_LISTEN] = {
 		.name			= "LISTEN",
 	},
+	[IORING_OP_EPOLL_WAIT] = {
+		.name			= "EPOLL_WAIT",
+	},
 };
 
 const char *io_uring_get_opcode(u8 opcode)
Signature · 1 line
2.47.2
#14 lore raw

Re: [PATCH 09/11] io_uring/epoll: add support for IORING_OP_EPOLL_WAIT

Quoted 30 lines · Jens Axboe

For existing epoll event loops that can't fully convert to io_uring, the used approach is usually to add the io_uring fd to the epoll instance and use epoll_wait() to wait on both "legacy" and io_uring events. While this work, it isn't optimal as:

1) epoll_wait() is pretty limited in what it can do. It does not support partial reaping of events, or waiting on a batch of events.

2) When an io_uring ring is added to an epoll instance, it activates the io_uring "I'm being polled" logic which slows things down.

Rather than use this approach, with EPOLL_WAIT support added to io_uring, event loops can use the normal io_uring wait logic for everything, as long as an epoll wait request has been armed with io_uring.

Note that IORING_OP_EPOLL_WAIT does NOT take a timeout value, as this is an async request. Waiting on io_uring events in general has various timeout parameters, and those are the ones that should be used when waiting on any kind of request. If events are immediately available for reaping, then This opcode will return those immediately. If none are available, then it will post an async completion when they become available.

cqe->res will contain either an error code (< 0 value) for a malformed request, invalid epoll instance, etc. It will return a positive result indicating how many events were reaped.

IORING_OP_EPOLL_WAIT requests may be canceled using the normal io_uring cancelation infrastructure. The poll logic for managing ownership is adopted to guard the epoll side too.

...

Quoted 4 lines
io_uring/epoll.c +0 -0
index 7848d9cc073d..5a47f0cce647 100644

...

Quoted 8 lines
io_uring/epoll.c +8 -0
+static void io_epoll_retry(struct io_kiocb *req, struct io_tw_state *ts)
+{
+	int v;
+
+	do {
+		v = atomic_read(&req->poll_refs);
+		if (unlikely(v != 1)) {
+			if (WARN_ON_ONCE(!(v & IO_POLL_REF_MASK)))

io_uring/epoll.c int io_epoll_ctl(struct io_kiocb *req, unsigned int issue_flags) · line 148

Quoted 10 lines
io_uring/epoll.c +10 -0
+				return;
+			if (v & IO_POLL_CANCEL_FLAG) {
+				__io_epoll_cancel(req);
+				return;
+			}
+			if (v & IO_POLL_FINISH_FLAG)
+				return;
+		}
+		v &= IO_POLL_REF_MASK;
+	} while (atomic_sub_return(v, &req->poll_refs) & IO_POLL_REF_MASK);

I haven't looked deep into the set, but this loop looks very suspicious. The entire purpose of the twin loop in poll.c is not to lose events while doing processing, which is why the processing happens before the decrement...

io_uring/epoll.c int io_epoll_ctl(struct io_kiocb *req, unsigned int issue_flags) · line 150

Quoted 1 line
io_uring/epoll.c +1 -0
+	io_req_task_submit(req, ts);

Maybe the issue is supposed to handle that, but this one is not allowed unless you fully unhash all the polling. Once you dropped refs the poll wait entry feels free to claim the request, and, for example, queue a task work, and io_req_task_submit() would decide to queue it as well. It's likely not the only race that can happen.

Quoted 14 lines
io_uring/epoll.c +14 -0
+}
+
+static int io_epoll_execute(struct io_kiocb *req)
+{
+	struct io_epoll_wait *iew = io_kiocb_to_cmd(req, struct io_epoll_wait);
+
+	list_del_init_careful(&iew->wait.entry);
+	if (io_poll_get_ownership(req)) {
+		req->io_task_work.func = io_epoll_retry;
+		io_req_task_work_add(req);
+	}
+
+	return 1;
+}

...

Quoted 22 lines
io_uring/epoll.c +22 -0
+int io_epoll_wait(struct io_kiocb *req, unsigned int issue_flags)
+{
+	struct io_epoll_wait *iew = io_kiocb_to_cmd(req, struct io_epoll_wait);
+	struct io_ring_ctx *ctx = req->ctx;
+	int ret;
+
+	io_ring_submit_lock(ctx, issue_flags);
+
+	ret = epoll_wait(req->file, iew->events, iew->maxevents, NULL, &iew->wait);
+	if (ret == -EIOCBQUEUED) {
+		if (hlist_unhashed(&req->hash_node))
+			hlist_add_head(&req->hash_node, &ctx->epoll_list);
+		io_ring_submit_unlock(ctx, issue_flags);
+		return IOU_ISSUE_SKIP_COMPLETE;
+	} else if (ret < 0) {
+		req_set_fail(req);
+	}
+	hlist_del_init(&req->hash_node);
+	io_ring_submit_unlock(ctx, issue_flags);
+	io_req_set_res(req, ret, 0);
+	return IOU_OK;
+}

...

Signature · 1 line
Pavel Begunkov
author maintainer
#15 lore raw

Re: [PATCH 09/11] io_uring/epoll: add support for IORING_OP_EPOLL_WAIT

Quoted 61 lines · Pavel Begunkov
Quoted 30 lines · Jens Axboe

For existing epoll event loops that can't fully convert to io_uring, the used approach is usually to add the io_uring fd to the epoll instance and use epoll_wait() to wait on both "legacy" and io_uring events. While this work, it isn't optimal as:

1) epoll_wait() is pretty limited in what it can do. It does not support partial reaping of events, or waiting on a batch of events.

2) When an io_uring ring is added to an epoll instance, it activates the io_uring "I'm being polled" logic which slows things down.

Rather than use this approach, with EPOLL_WAIT support added to io_uring, event loops can use the normal io_uring wait logic for everything, as long as an epoll wait request has been armed with io_uring.

Note that IORING_OP_EPOLL_WAIT does NOT take a timeout value, as this is an async request. Waiting on io_uring events in general has various timeout parameters, and those are the ones that should be used when waiting on any kind of request. If events are immediately available for reaping, then This opcode will return those immediately. If none are available, then it will post an async completion when they become available.

cqe->res will contain either an error code (< 0 value) for a malformed request, invalid epoll instance, etc. It will return a positive result indicating how many events were reaped.

IORING_OP_EPOLL_WAIT requests may be canceled using the normal io_uring cancelation infrastructure. The poll logic for managing ownership is adopted to guard the epoll side too.

...

Quoted 4 lines
io_uring/epoll.c +0 -0
index 7848d9cc073d..5a47f0cce647 100644

...

Quoted 8 lines
io_uring/epoll.c +8 -0
+static void io_epoll_retry(struct io_kiocb *req, struct io_tw_state *ts)
+{
+    int v;
+
+    do {
+        v = atomic_read(&req->poll_refs);
+        if (unlikely(v != 1)) {
+            if (WARN_ON_ONCE(!(v & IO_POLL_REF_MASK)))

io_uring/epoll.c int io_epoll_ctl(struct io_kiocb *req, unsigned int issue_flags) · line 148

Quoted 10 lines
io_uring/epoll.c +10 -0
+                return;
+            if (v & IO_POLL_CANCEL_FLAG) {
+                __io_epoll_cancel(req);
+                return;
+            }
+            if (v & IO_POLL_FINISH_FLAG)
+                return;
+        }
+        v &= IO_POLL_REF_MASK;
+    } while (atomic_sub_return(v, &req->poll_refs) & IO_POLL_REF_MASK);

I haven't looked deep into the set, but this loop looks very suspicious. The entire purpose of the twin loop in poll.c is not to lose events while doing processing, which is why the processing happens before the decrement...

Quoted 8 lines
Quoted 1 line

+ io_req_task_submit(req, ts);

Maybe the issue is supposed to handle that, but this one is not allowed unless you fully unhash all the polling. Once you dropped refs the poll wait entry feels free to claim the request, and, for example, queue a task work, and io_req_task_submit() would decide to queue it as well. It's likely not the only race that can happen.

I'm going to send out a new version with the multishot support dropped for now as it both a) simplifies the series, and b) I'm not super convinced it can be sanely used. In any case, it can be left for later. Once I do that, please take a look at the ownership side and let's continue the discussion there!

Signature · 1 line
Jens Axboe
author maintainer
#16 lore raw

PATCH v2 10/11 io_uring/epoll: add support for provided buffers

In reply to Jens Axboe

This will be a prerequisite for adding multishot support, but can be used with single shot support as well. Works like any other request that supports provided buffers - set addr to NULL and ensure that sqe->buf_group is set, and IOSQE_BUFFER_SELECT in sqe->flags. Then epoll wait will pick a buffer from that group and store the events there.

---
 io_uring/epoll.c | 31 +++++++++++++++++++++++++++----
 io_uring/opdef.c |  1 +
 2 files changed, 28 insertions(+), 4 deletions(-)
io_uring/epoll.c +27 -4
index 5a47f0cce647..134112e7a505 100644
@@ -10,6 +10,7 @@
 #include <uapi/linux/io_uring.h>
 
 #include "io_uring.h"
+#include "kbuf.h"
 #include "epoll.h"
 #include "poll.h"
 
@@ -189,11 +190,13 @@ int io_epoll_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
 	struct io_epoll_wait *iew = io_kiocb_to_cmd(req, struct io_epoll_wait);
 
-	if (sqe->off || sqe->rw_flags || sqe->buf_index || sqe->splice_fd_in)
+	if (sqe->off || sqe->rw_flags || sqe->splice_fd_in)
 		return -EINVAL;
 
 	iew->maxevents = READ_ONCE(sqe->len);
 	iew->events = u64_to_user_ptr(READ_ONCE(sqe->addr));
+	if (req->flags & REQ_F_BUFFER_SELECT && iew->events)
+		return -EINVAL;
 
 	iew->wait.flags = 0;
 	iew->wait.private = req;
@@ -207,22 +210,42 @@ int io_epoll_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 int io_epoll_wait(struct io_kiocb *req, unsigned int issue_flags)
 {
 	struct io_epoll_wait *iew = io_kiocb_to_cmd(req, struct io_epoll_wait);
+	struct epoll_event __user *evs = iew->events;
 	struct io_ring_ctx *ctx = req->ctx;
+	int maxevents = iew->maxevents;
+	unsigned int cflags = 0;
 	int ret;
 
 	io_ring_submit_lock(ctx, issue_flags);
 
-	ret = epoll_wait(req->file, iew->events, iew->maxevents, NULL, &iew->wait);
+	if (io_do_buffer_select(req)) {
+		size_t len = iew->maxevents * sizeof(*evs);
+
+		evs = io_buffer_select(req, &len, 0);
+		if (!evs) {
+			ret = -ENOBUFS;
+			goto err;
+		}
+		maxevents = len / sizeof(*evs);
+	}
+
+	ret = epoll_wait(req->file, evs, maxevents, NULL, &iew->wait);
 	if (ret == -EIOCBQUEUED) {
+		io_kbuf_recycle(req, 0);
 		if (hlist_unhashed(&req->hash_node))
 			hlist_add_head(&req->hash_node, &ctx->epoll_list);
 		io_ring_submit_unlock(ctx, issue_flags);
 		return IOU_ISSUE_SKIP_COMPLETE;
-	} else if (ret < 0) {
+	} else if (ret > 0) {
+		cflags = io_put_kbuf(req, ret * sizeof(*evs), 0);
+	} else if (!ret) {
+		io_kbuf_recycle(req, 0);
+	} else {
+err:
 		req_set_fail(req);
 	}
 	hlist_del_init(&req->hash_node);
 	io_ring_submit_unlock(ctx, issue_flags);
-	io_req_set_res(req, ret, 0);
+	io_req_set_res(req, ret, cflags);
 	return IOU_OK;
 }
io_uring/opdef.c +1 -0
index 44553a657476..04ff2b438531 100644
@@ -520,6 +520,7 @@ const struct io_issue_def io_issue_defs[] = {
 		.needs_file		= 1,
 		.unbound_nonreg_file	= 1,
 		.audit_skip		= 1,
+		.buffer_select		= 1,
 #if defined(CONFIG_EPOLL)
 		.prep			= io_epoll_wait_prep,
 		.issue			= io_epoll_wait,
Signature · 1 line
2.47.2
author maintainer
#17 lore raw

PATCH v2 11/11 io_uring/epoll: add multishot support for IORING_OP_EPOLL_WAIT

In reply to Jens Axboe

As with other multishot requests, submitting a multishot epoll wait request will keep it re-armed post the initial trigger. This allows multiple epoll wait completions per request submitted, every time events are available. If more completions are expected for this epoll wait request, then IORING_CQE_F_MORE will be set in the posted cqe->flags.

For multishot, the request remains on the epoll callback waitqueue head. This means that epoll doesn't need to juggle the ep->lock writelock (and disable/enable IRQs) for each invocation of the reaping loop. That should translate into nice efficiency gains.

Use by setting IORING_EPOLL_WAIT_MULTISHOT in the sqe->epoll_flags member. Must be used with provided buffers.

---
 include/uapi/linux/io_uring.h |  6 +++++
 io_uring/epoll.c              | 46 ++++++++++++++++++++++++++++-------
 2 files changed, 43 insertions(+), 9 deletions(-)
include/uapi/linux/io_uring.h +6 -0
index a559e1e1544a..93f504b6d4ec 100644
@@ -73,6 +73,7 @@ struct io_uring_sqe {
 		__u32		futex_flags;
 		__u32		install_fd_flags;
 		__u32		nop_flags;
+		__u32		epoll_flags;
 	};
 	__u64	user_data;	/* data to be passed back at completion time */
 	/* pack this to avoid bogus arm OABI complaints */
@@ -405,6 +406,11 @@ enum io_uring_op {
 #define IORING_ACCEPT_DONTWAIT	(1U << 1)
 #define IORING_ACCEPT_POLL_FIRST	(1U << 2)
 
+/*
+ * epoll_wait flags, stored in sqe->epoll_flags
+ */
+#define IORING_EPOLL_WAIT_MULTISHOT	(1U << 0)
+
 /*
  * IORING_OP_MSG_RING command types, stored in sqe->addr
  */
io_uring/epoll.c +37 -9

108 lines not on this page · raw message

Signature · 1 line
2.47.2