Add io_uring support for waitid
Jens Axboe
maintainerv4 cover Add io_uring support for waitid
Hi,
This adds support for IORING_OP_WAITID, which is an async variant of the waitid(2) syscall. Rather than have a parent need to block waiting on a child task state change, it can now simply get an async notication when the requested state change has occured.
Patches 1..4 are purely prep patches, and should not have functional changes. They split out parts of do_wait() into __do_wait(), so that the prepare-to-wait and sleep parts are contained within do_wait().
Patch 5 adds io_uring support.
I wrote a few basic tests for this, which can be found in the 'waitid' branch of liburing:
https://git.kernel.dk/cgit/liburing/log/?h=waitid
Also spun a custom kernel for someone to test it, and no issues reported so far.
The code can also be found here:
https://git.kernel.dk/cgit/linux/log/?h=io_uring-waitid
include/linux/io_uring_types.h | 2 + include/uapi/linux/io_uring.h | 2 + io_uring/Makefile | 3 +- io_uring/cancel.c | 5 + io_uring/io_uring.c | 3 + io_uring/opdef.c | 10 +- io_uring/waitid.c | 372 +++++++++++++++++++++++++++++++++ io_uring/waitid.h | 15 ++ kernel/exit.c | 131 ++++++------ kernel/exit.h | 30 +++ 10 files changed, 512 insertions(+), 61 deletions(-)
Changes since v3:
- Rebase on current tree
- Move it before the futex changes. Note that if you're testing this, this means that the opcode values have changed. The liburing repo has been rebased as a result as well, you'll want to update that too. liburing also has update test cases.
- Fix races between cancelation and wakeup trigger. This follows a scheme similar to the internal poll io_uring handling
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH v4 1/5 exit: abstract out should_wake helper for child_wait_callback()
Abstract out the helper that decides if we should wake up following a wake_up() callback on our internal waitqueue.
No functional changes intended in this patch.
--- kernel/exit.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
kernel/exit.c +14 -6
@@ -1520,6 +1520,17 @@ static int ptrace_do_wait(struct wait_opts *wo, struct task_struct *tsk)return 0 ; } +static bool pid_child_should_wake (struct wait_opts *wo,struct task_struct *p) +{ +if (!eligible_pid (wo, p)) +return false; + +if ((wo->wo_flags & __WNOTHREAD) && wo->child_wait.private != p->parent) +return false; + +return true; +} +static int child_wait_callback (wait_queue_entry_t *wait,unsigned mode,int sync,void *key) {
@@ -1527,13 +1538,10 @@ static int child_wait_callback(wait_queue_entry_t *wait, unsigned mode, child_wait);struct task_struct *p = key; -if (!eligible_pid (wo, p)) -return 0 ; +if (pid_child_should_wake (wo, p)) +return default_wake_function (wait, mode, sync, key); -if ((wo->wo_flags & __WNOTHREAD) && wait->private != p->parent) -return 0 ; - -return default_wake_function (wait, mode, sync, key); +return 0 ; }void __wake_up_parent (struct task_struct *p,struct task_struct *parent)
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v4 2/5 exit: move core of do_wait() into helper
In reply to Jens AxboeRather than have a maze of gotos, put the actual logic in __do_wait() and have do_wait() loop deal with waitqueue setup/teardown and whether to call __do_wait() again.
No functional changes intended in this patch.
--- kernel/exit.c | 51 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 20 deletions(-)
kernel/exit.c +31 -20
@@ -1590,16 +1590,10 @@ static int do_wait_pid(struct wait_opts *wo)return 0 ; } -static long do_wait (struct wait_opts *wo) +static long __do_wait (struct wait_opts *wo) { -int retval; - -trace_sched_process_wait (wo->wo_pid); +long retval; -init_waitqueue_func_entry (&wo->child_wait, child_wait_callback); - wo->child_wait.private = current; -add_wait_queue (¤t->signal->wait_chldexit, &wo->child_wait); -repeat:/* * If there is nothing that can match our criteria, just get out. * We will clear ->notask_error to zero if we see any child that
@@ -1611,24 +1605,23 @@ static long do_wait(struct wait_opts *wo) (!wo->wo_pid || !pid_has_task(wo->wo_pid, wo->wo_type))) goto notask; -set_current_state (TASK_INTERRUPTIBLE );read_lock (&tasklist_lock);if (wo->wo_type ==PIDTYPE_PID ) { retval =do_wait_pid (wo);if (retval) - goto end; +return retval; }else {struct task_struct *tsk = current; do { retval = do_wait_thread(wo, tsk); if (retval) - goto end; + return retval; retval = ptrace_do_wait(wo, tsk); if (retval) - goto end; + return retval; if (wo->wo_flags & __WNOTHREAD) break;
@@ -1638,14 +1631,32 @@ static long do_wait(struct wait_opts *wo) notask: retval = wo->notask_error; -if (!retval && !(wo->wo_flags &WNOHANG )) { - retval = -ERESTARTSYS ; -if (!signal_pending (current)) { -schedule (); - goto repeat; - } - } -end: +if (!retval && !(wo->wo_flags &WNOHANG )) +return -ERESTARTSYS ; + +return retval; +} + +static long do_wait (struct wait_opts *wo) +{ +int retval; + +trace_sched_process_wait (wo->wo_pid); + +init_waitqueue_func_entry (&wo->child_wait, child_wait_callback); + wo->child_wait.private = current; +add_wait_queue (¤t->signal->wait_chldexit, &wo->child_wait); + +do { +set_current_state (TASK_INTERRUPTIBLE ); + retval =__do_wait (wo); +if (retval != -ERESTARTSYS ) +break ; +if (signal_pending (current)) +break ; +schedule (); + }while (1 ); +__set_current_state (TASK_RUNNING );remove_wait_queue (¤t->signal->wait_chldexit, &wo->child_wait);return retval;
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v4 3/5 exit: add kernel_waitid_prepare() helper
In reply to Jens AxboeMove the setup logic out of kernel_waitid(), and into a separate helper.
No functional changes intended in this patch.
--- kernel/exit.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-)
kernel/exit.c +25 -13
@@ -1662,14 +1662,13 @@ static long do_wait(struct wait_opts *wo)return retval; } -static long kernel_waitid (int which,pid_t upid,struct waitid_info *infop, -int options,struct rusage *ru) +static int kernel_waitid_prepare (struct wait_opts *wo,int which,pid_t upid, +struct waitid_info *infop,int options, +struct rusage *ru) { -struct wait_opts wo; +unsigned int f_flags =0 ;struct pid *pid =NULL ;enum pid_type type; -long ret; -unsigned int f_flags =0 ;if (options & ~(WNOHANG |WNOWAIT |WEXITED |WSTOPPED |WCONTINUED | __WNOTHREAD|__WCLONE|__WALL))
@@ -1712,19 +1711,32 @@ static long kernel_waitid(int which, pid_t upid, struct waitid_info *infop,return -EINVAL ; } - wo.wo_type = type; - wo.wo_pid = pid; - wo.wo_flags = options; - wo.wo_info = infop; - wo.wo_rusage = ru; + wo->wo_type = type; + wo->wo_pid = pid; + wo->wo_flags = options; + wo->wo_info = infop; + wo->wo_rusage = ru;if (f_flags &O_NONBLOCK ) - wo.wo_flags |=WNOHANG ; + wo->wo_flags |=WNOHANG ; + +return 0 ; +} + +static long kernel_waitid (int which,pid_t upid,struct waitid_info *infop, +int options,struct rusage *ru) +{ +struct wait_opts wo; +long ret; + + ret =kernel_waitid_prepare (&wo, which, upid, infop, options, ru); +if (ret) +return ret; ret =do_wait (&wo); -if (!ret && !(options &WNOHANG ) && (f_flags &O_NONBLOCK )) +if (!ret && !(options &WNOHANG ) && (wo.wo_flags &WNOHANG )) ret = -EAGAIN ; -put_pid (pid); +put_pid (wo.wo_pid);return ret; }
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v4 4/5 exit: add internal include file with helpers
In reply to Jens AxboeMove struct wait_opts and waitid_info into kernel/exit.h, and include function declarations for the recently added helpers. Make them non-static as well.
This is in preparation for adding a waitid operation through io_uring. With the abtracted helpers, this is now possible.
--- kernel/exit.c | 32 +++++++------------------------- kernel/exit.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 25 deletions(-) create mode 100644 kernel/exit.h
kernel/exit.c +7 -25
@@ -74,6 +74,8 @@#include <asm/unistd.h> #include <asm/mmu_context.h> +#include "exit.h" +/* * The default value should be high enough to not crash a system that randomly * crashes its kernel from time to time, but low enough to at least not permit
@@ -1037,26 +1039,6 @@ SYSCALL_DEFINE1(exit_group, int, error_code)return 0 ; } -struct waitid_info { -pid_t pid; -uid_t uid; -int status; -int cause; -}; - -struct wait_opts { -enum pid_type wo_type; -int wo_flags; -struct pid *wo_pid; - -struct waitid_info *wo_info; -int wo_stat; -struct rusage *wo_rusage; - -wait_queue_entry_t child_wait; -int notask_error; -}; -static int eligible_pid (struct wait_opts *wo,struct task_struct *p) { return wo->wo_type == PIDTYPE_MAX ||
@@ -1520,7 +1502,7 @@ static int ptrace_do_wait(struct wait_opts *wo, struct task_struct *tsk)return 0 ; } -static bool pid_child_should_wake (struct wait_opts *wo,struct task_struct *p) +bool pid_child_should_wake (struct wait_opts *wo,struct task_struct *p) {if (!eligible_pid (wo, p))return false;
@@ -1590,7 +1572,7 @@ static int do_wait_pid(struct wait_opts *wo)return 0 ; } -static long __do_wait (struct wait_opts *wo) +long __do_wait (struct wait_opts *wo) {long retval;
@@ -1662,9 +1644,9 @@ static long do_wait(struct wait_opts *wo)return retval; } -static int kernel_waitid_prepare (struct wait_opts *wo,int which,pid_t upid, -struct waitid_info *infop,int options, -struct rusage *ru) +int kernel_waitid_prepare (struct wait_opts *wo,int which,pid_t upid, +struct waitid_info *infop,int options, +struct rusage *ru) {unsigned int f_flags =0 ;struct pid *pid =NULL ;
kernel/exit.h new file +30 -0
@@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0-only +#ifndef LINUX_WAITID_H +#define LINUX_WAITID_H + +struct waitid_info { +pid_t pid; +uid_t uid; +int status; +int cause; +}; + +struct wait_opts { +enum pid_type wo_type; +int wo_flags; +struct pid *wo_pid; + +struct waitid_info *wo_info; +int wo_stat; +struct rusage *wo_rusage; + +wait_queue_entry_t child_wait; +int notask_error; +}; + +bool pid_child_should_wake (struct wait_opts *wo,struct task_struct *p); +long __do_wait (struct wait_opts *wo); +int kernel_waitid_prepare (struct wait_opts *wo,int which,pid_t upid, +struct waitid_info *infop,int options, +struct rusage *ru); +#endif
Signature · 1 line
2.40.1
Jens Axboe
maintainerPATCH v4 5/5 io_uring: add IORING_OP_WAITID support
In reply to Jens AxboeThis adds support for an async version of waitid(2), in a fully async version. If an event isn't immediately available, wait for a callback to trigger a retry.
The format of the sqe is as follows:
sqe->len The 'which', the idtype being queried/waited for.sqe->fd The 'pid' (or id) being waited for.sqe->file_index The 'options' being set.sqe->addr2 A pointer to siginfo_t, if any, being filled in.
buf_index, add3, and waitid_flags are reserved/unused for now. waitid_flags will be used for options for this request type. One interesting use case may be to add multi-shot support, so that the request stays armed and posts a notification every time a monitored process state change occurs.
Note that this does not support rusage, on Arnd's recommendation.
See the waitid(2) man page for details on the arguments.
--- include/linux/io_uring_types.h | 2 + include/uapi/linux/io_uring.h | 2 + io_uring/Makefile | 3 +- io_uring/cancel.c | 5 + io_uring/io_uring.c | 3 + io_uring/opdef.c | 10 +- io_uring/waitid.c | 372 +++++++++++++++++++++++++++++++++ io_uring/waitid.h | 15 ++ 8 files changed, 410 insertions(+), 2 deletions(-) create mode 100644 io_uring/waitid.c create mode 100644 io_uring/waitid.h
include/linux/io_uring_types.h +2 -0
@@ -313,6 +313,8 @@ struct io_ring_ctx {struct list_head cq_overflow_list;struct io_hash_table cancel_table; +struct hlist_head waitid_list; +const struct cred *sq_creds;/* cred used for __io_sq_thread() */ struct io_sq_data *sq_data;/* if using sq thread polling */
include/uapi/linux/io_uring.h +2 -0
@@ -65,6 +65,7 @@ struct io_uring_sqe {__u32 xattr_flags;__u32 msg_ring_flags;__u32 uring_cmd_flags; +__u32 waitid_flags; };__u64 user_data;/* data to be passed back at completion time */ /* pack this to avoid bogus arm OABI complaints */
@@ -240,6 +241,7 @@ enum io_uring_op {IORING_OP_URING_CMD ,IORING_OP_SEND_ZC ,IORING_OP_SENDMSG_ZC , +IORING_OP_WAITID ,/* this goes last, obviously */ IORING_OP_LAST,
io_uring/Makefile +2 -1
@@ -7,5 +7,6 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ openclose.o uring_cmd.o epoll.o \ statx.o net.o msg_ring.o timeout.o \ sqpoll.o fdinfo.o tctx.o poll.o \ - cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o + cancel.o kbuf.o rsrc.o rw.o opdef.o \ + notif.o waitid.o obj-$(CONFIG_IO_WQ) += io-wq.o
io_uring/cancel.c +5 -0
@@ -15,6 +15,7 @@#include "tctx.h" #include "poll.h" #include "timeout.h" +#include "waitid.h" #include "cancel.h" struct io_cancel {
@@ -119,6 +120,10 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd,if (ret != -ENOENT )return ret; + ret =io_waitid_cancel (ctx, cd, issue_flags); +if (ret != -ENOENT ) +return ret; +spin_lock (&ctx->completion_lock);if (!(cd->flags &IORING_ASYNC_CANCEL_FD )) ret =io_timeout_cancel (ctx, cd);
io_uring/io_uring.c +3 -0
@@ -92,6 +92,7 @@#include "cancel.h" #include "net.h" #include "notif.h" +#include "waitid.h" #include "timeout.h" #include "poll.h"
@@ -348,6 +349,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)INIT_LIST_HEAD (&ctx->tctx_list); ctx->submit_state.free_list.next =NULL ;INIT_WQ_LIST (&ctx->locked_free_list); +INIT_HLIST_HEAD (&ctx->waitid_list);INIT_DELAYED_WORK (&ctx->fallback_work, io_fallback_req_func);INIT_WQ_LIST (&ctx->submit_state.compl_reqs);return ctx;
@@ -3303,6 +3305,7 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, ret |=io_cancel_defer_files (ctx, task, cancel_all);mutex_lock (&ctx->uring_lock); ret |=io_poll_remove_all (ctx, task, cancel_all); + ret |=io_waitid_remove_all (ctx, task, cancel_all);mutex_unlock (&ctx->uring_lock); ret |=io_kill_timeouts (ctx, task, cancel_all);if (task)
io_uring/opdef.c +9 -1
@@ -33,6 +33,7 @@#include "poll.h" #include "cancel.h" #include "rw.h" +#include "waitid.h" static int io_no_issue (struct io_kiocb *req,unsigned int issue_flags) {
@@ -428,9 +429,12 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_WAITID ] = { + .prep = io_waitid_prep, + .issue = io_waitid, + }, }; -const struct io_cold_def io_cold_defs[] = { [IORING_OP_NOP ] = { .name ="NOP" ,
@@ -648,6 +652,10 @@ const struct io_cold_def io_cold_defs[] = { .fail = io_sendrecv_fail, #endif }, + [IORING_OP_WAITID ] = { + .name ="WAITID" , + .async_size =sizeof (struct io_waitid_async ), + }, };const char *io_uring_get_opcode (u8 opcode)
io_uring/waitid.c new file +372 -0
372 lines not on this page · raw message
io_uring/waitid.h new file +15 -0
@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "../kernel/exit.h" + +struct io_waitid_async { +struct io_kiocb *req; +struct wait_opts wo; +}; + +int io_waitid_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +int io_waitid (struct io_kiocb *req,unsigned int issue_flags); +int io_waitid_cancel (struct io_ring_ctx *ctx,struct io_cancel_data *cd, +unsigned int issue_flags); +bool io_waitid_remove_all (struct io_ring_ctx *ctx,struct task_struct *task, +bool cancel_all);
Signature · 1 line
2.40.1
Jens Axboe
maintainerQuoted 13 lines · Jens Axboe
Hi,
This adds support for
IORING_OP_WAITID, which is an async variant of the waitid(2) syscall. Rather than have a parent need to block waiting on a child task state change, it can now simply get an async notication when the requested state change has occured.Patches 1..4 are purely prep patches, and should not have functional changes. They split out parts of
do_wait()into__do_wait(), so that the prepare-to-wait and sleep parts are contained withindo_wait().Patch 5 adds
io_uringsupport.
Quoted 7 lines
I wrote a few basic tests for this, which can be found in the 'waitid' branch of liburing:
https://git.kernel.dk/cgit/liburing/log/?h=waitid
Also spun a custom kernel for someone to test it, and no issues reported so far.
Forget to mention that I also ran all the ltp testcases for any wait* syscall test, and everything still passes just fine.
Signature · 1 line
Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Jens Axboe
Hi,
This adds support for
IORING_OP_WAITID, which is an async variant of the waitid(2) syscall. Rather than have a parent need to block waiting on a child task state change, it can now simply get an async notication when the requested state change has occured.Patches 1..4 are purely prep patches, and should not have functional changes. They split out parts of
do_wait()into__do_wait(), so that the prepare-to-wait and sleep parts are contained withindo_wait().Patch 5 adds
io_uringsupport.I wrote a few basic tests for this, which can be found in the 'waitid' branch of liburing:
Quoted 7 lines
Quoted 4 lines
https://git.kernel.dk/cgit/liburing/log/?h=waitid
Also spun a custom kernel for someone to test it, and no issues reported so far.
Forget to mention that I also ran all the ltp testcases for any wait* syscall test, and everything still passes just fine.
I think the struct that this ends up exposing to io_uring is pretty ugly and it would warrant a larger cleanup. I wouldn't be surprised if you get some people complain about this.
Other than that I don't have any complaints about the series.
Jens Axboe
maintainerQuoted 23 lines · Christian Brauner
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Jens Axboe
Hi,
This adds support for
IORING_OP_WAITID, which is an async variant of the waitid(2) syscall. Rather than have a parent need to block waiting on a child task state change, it can now simply get an async notication when the requested state change has occured.Patches 1..4 are purely prep patches, and should not have functional changes. They split out parts of
do_wait()into__do_wait(), so that the prepare-to-wait and sleep parts are contained withindo_wait().Patch 5 adds
io_uringsupport.I wrote a few basic tests for this, which can be found in the 'waitid' branch of liburing:
https://git.kernel.dk/cgit/liburing/log/?h=waitid
Also spun a custom kernel for someone to test it, and no issues reported so far.
Quoted 8 lines
Quoted 2 lines
Forget to mention that I also ran all the ltp testcases for any wait* syscall test, and everything still passes just fine.
I think the struct that this ends up exposing to
io_uringis pretty ugly and it would warrant a larger cleanup. I wouldn't be surprised if you get some people complain about this.Other than that I don't have any complaints about the series.
io_uring only really needs child_wait and wo_pid on the wait_opts side, for waitid_info it needs all of it. I'm assuming your worry is about the former rather than the latter.
I think we could only make this smaller if we had a separate entry point for io_uring, which would then make the code reuse a lot smaller. Right now we just have __do_wait() abstracted out, and if we added a third struct that has child_wait/wo_pid and exposed just that, we could not share this infrastructure.
So as far as I can tell, there's no way to make the sharing less than it is, at least not without adding cost of more code and less reuse.
Shrug?
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerQuoted 43 lines · Jens Axboe
Quoted 23 lines · Christian Brauner
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Jens Axboe
Hi,
This adds support for
IORING_OP_WAITID, which is an async variant of the waitid(2) syscall. Rather than have a parent need to block waiting on a child task state change, it can now simply get an async notication when the requested state change has occured.Patches 1..4 are purely prep patches, and should not have functional changes. They split out parts of
do_wait()into__do_wait(), so that the prepare-to-wait and sleep parts are contained withindo_wait().Patch 5 adds
io_uringsupport.I wrote a few basic tests for this, which can be found in the 'waitid' branch of liburing:
https://git.kernel.dk/cgit/liburing/log/?h=waitid
Also spun a custom kernel for someone to test it, and no issues reported so far.
Quoted 8 lines
Quoted 2 lines
Forget to mention that I also ran all the ltp testcases for any wait* syscall test, and everything still passes just fine.
I think the struct that this ends up exposing to
io_uringis pretty ugly and it would warrant a larger cleanup. I wouldn't be surprised if you get some people complain about this.Other than that I don't have any complaints about the series.
io_uringonly really needschild_waitandwo_pidon thewait_optsside, forwaitid_infoit needs all of it. I'm assuming your worry is about the former rather than the latter.I think we could only make this smaller if we had a separate entry point for
io_uring, which would then make the code reuse a lot smaller. Right now we just have__do_wait()abstracted out, and if we added a third struct that has child_wait/wo_pid and exposed just that, we could not share this infrastructure.
Quoted 4 lines
So as far as I can tell, there's no way to make the sharing less than it is, at least not without adding cost of more code and less reuse.
Shrug?
Took a closer look, and I don't think it's really possible to split much out of wait_opts. You may only need child_wait/wo_pid for setup, but on the wakeup side you type and flags as well. We could probably add that third struct and move wo_rusage and notask_error out, but seems very pointless at that point just to avoid those two. And if we do wire up rusage at some point, then we're left with just the one.
Signature · 1 line
Jens Axboe