Add io_uring_iowait_toggle()
PATCH 1/1 Add io_uring_iowait_toggle()
Add io_uring_iowait_toggle() helper function for the userspace liburing side of IORING_ENTER_NO_IOWAIT flag added in io_uring for 6.12.
This function toggles whether a ring sets in_iowait when waiting for completions. This is useful when waiting for multiple batched completions using e.g. io_uring_submit_and_wait_timeout() and userspace treats iowait time as CPU utilization.
It works by keeping an internal flag INT_FLAG_NO_IOWAIT, which if set will set IORING_ENTER_NO_IOWAIT on every io_uring_enter().
Manpages are added/modified, a unit test is included, and io_uring.h is synced with the kernel side.
--- man/io_uring_enter.2 | 5 + man/io_uring_iowait_toggle.3 | 45 +++++++++ src/include/liburing.h | 1 + src/include/liburing/io_uring.h | 2 + src/int_flags.h | 1 + src/liburing.map | 2 + src/queue.c | 2 + src/setup.c | 12 +++ test/Makefile | 1 + test/no-iowait.c | 162 ++++++++++++++++++++++++++++++++ 10 files changed, 233 insertions(+) create mode 100644 man/io_uring_iowait_toggle.3 create mode 100644 test/no-iowait.c
man/io_uring_enter.2 +5 -0
@@ -104,6 +104,11 @@ If the ring file descriptor has been registered through use of then setting this flag will tell the kernel that the .I ring_fd passed in is the registered ring offset rather than a normal file descriptor. +.TP +.B IORING_ENTER_NO_IOWAIT +If this flag is set, then in_iowait will not be set for the current task if +.BR io_uring_enter (2) +results in waiting. .PP .PP
man/io_uring_iowait_toggle.3 new file +45 -0
45 lines not on this page · raw message
src/include/liburing.h +1 -0
@@ -243,6 +243,7 @@ int io_uring_unregister_napi(struct io_uring *ring, struct io_uring_napi *napi);int io_uring_get_events (struct io_uring *ring);int io_uring_submit_and_get_events (struct io_uring *ring); +int io_uring_iowait_toggle (struct io_uring *ring,bool enabled);/* * io_uring syscalls.
src/include/liburing/io_uring.h +2 -0
@@ -504,6 +504,7 @@ struct io_cqring_offsets {#define IORING_ENTER_SQ_WAIT (1U << 2)#define IORING_ENTER_EXT_ARG (1U << 3)#define IORING_ENTER_REGISTERED_RING (1U << 4) +#define IORING_ENTER_NO_IOWAIT (1U << 6)/* * Passed in for io_uring_setup(2). Copied back with updated info on success
@@ -539,6 +540,7 @@ struct io_uring_params {#define IORING_FEAT_LINKED_FILE (1U << 12)#define IORING_FEAT_REG_REG_RING (1U << 13)#define IORING_FEAT_RECVSEND_BUNDLE (1U << 14) +#define IORING_FEAT_IOWAIT_TOGGLE (1U << 15)/* * io_uring_register(2) opcodes and arguments
src/int_flags.h +1 -0
@@ -6,6 +6,7 @@ enum {INT_FLAG_REG_RING =1 ,INT_FLAG_REG_REG_RING =2 ,INT_FLAG_APP_MEM =4 , + INT_FLAG_NO_IOWAIT = 8, };#endif
src/liburing.map +2 -0
@@ -97,4 +97,6 @@ LIBURING_2.7 { } LIBURING_2.6; LIBURING_2.8 { + global: + io_uring_iowait_toggle; } LIBURING_2.7;
src/queue.c +2 -0
@@ -110,6 +110,8 @@ static int _io_uring_get_cqe(struct io_uring *ring,if (ring->int_flags &INT_FLAG_REG_RING ) flags |=IORING_ENTER_REGISTERED_RING ; +if (ring->int_flags &INT_FLAG_NO_IOWAIT ) + flags |=IORING_ENTER_NO_IOWAIT ; ret =__sys_io_uring_enter2 (ring->enter_ring_fd, data->submit, data->wait_nr, flags, data->arg, data->sz);
src/setup.c +12 -0
@@ -687,3 +687,15 @@ int io_uring_free_buf_ring(struct io_uring *ring, struct io_uring_buf_ring *br,__sys_munmap (br, nentries *sizeof (struct io_uring_buf ));return 0 ; } + +int io_uring_iowait_toggle (struct io_uring *ring,bool enabled) +{ +if (!(ring->features &IORING_FEAT_IOWAIT_TOGGLE )) +return -EOPNOTSUPP ; + +if (enabled) + ring->int_flags &= ~INT_FLAG_NO_IOWAIT ; +else + ring->int_flags |=INT_FLAG_NO_IOWAIT ; +return 0 ; +}
test/Makefile +1 -0
@@ -125,6 +125,7 @@ test_srcs := \ msg-ring-flags.c \ msg-ring-overflow.c \ multicqes_drain.c \ + no-iowait.c \ no-mmap-inval.c \ nolibc.c \ nop-all-sizes.c \
test/no-iowait.c new file +162 -0
162 lines not on this page · raw message
Signature · 1 line
2.43.5
Jens Axboe
maintainerman/io_uring_enter.2 If the ring file descriptor has been registered through use of · line 111
Quoted 13 lines · David Wei
man/io_uring_enter.2 +5
-0@@ -104,6 +104,11 @@ If the ring file descriptor has been registered through use of then setting this flag will tell the kernel that the .I ring_fd passed in is the registered ring offset rather than a normal file descriptor. +.TP +.B IORING_ENTER_NO_IOWAIT +If this flag is set, then in_iowait will not be set for the current task if +.BR io_uring_enter (2) +results in waiting.
I'd probably would say something ala:
If this flag is set, then waiting on events will not be accounted as iowait for the task if
.BR io_uring_enter (2)
results in waiting.
or something like that. io_iowait is an in-kernel thing, iowait is what application writers will know about.
Quoted 6 lines
man/io_uring_enter.2 +6
-0+.B #include <liburing.h> +.PP +.BI "int io_uring_iowait_toggle(struct io_uring *" ring ", +.BI " bool " enabled ");" +.BI " +.fi
man/io_uring_enter.2 If the ring file descriptor has been registered through use of
Quoted 10 lines
man/io_uring_enter.2 +10
-0+.SH DESCRIPTION +.PP +The +.BR io_uring_iowait_toggle (3) +function toggles for a given +.I ring +whether in_iowait is set for the current task while waiting for completions. +When in_iowait is set, time spent waiting is accounted as iowait time; +otherwise, it is accounted as idle time. The default behavior is to always set +in_iowait to true.
And ditto here
man/io_uring_enter.2
Quoted 5 lines
man/io_uring_enter.2 +5
-0+Setting in_iowait achieves two things: + +1. Account time spent waiting as iowait time + +2. Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rq
This should probably be something ala:
Setting in_iowait achieves two things:
.TP
.B Account time spent waiting as iowait time
.TP
.B Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rq .PP
to make that format like a man page.
man/io_uring_enter.2
Quoted 4 lines
man/io_uring_enter.2 +4
-0+Some user tooling attributes iowait time as CPU utilization time, so high +iowait time can look like apparent high CPU utilization, even though the task +is not scheduled and the CPU is free to run other tasks. This function +provides a way to disable this behavior where it makes sense to do so.
And here. Since this is the main man page, maybe also add something about how iowait is a relic from the old days of only having one CPU, and it indicates that the task is block uninterruptibly waiting for IO and hence cannot do other things. These days it's mostly a bogus accounting value, but it does help with the cpufreq boost for certain high frequency waits. Rephrase as needed :-)
src/queue.c static int _io_uring_get_cqe(struct io_uring *ring, · line 117
Quoted 13 lines
src/queue.c +2
-0@@ -110,6 +110,8 @@ static int _io_uring_get_cqe(struct io_uring *ring,if (ring->int_flags &INT_FLAG_REG_RING ) flags |=IORING_ENTER_REGISTERED_RING ; +if (ring->int_flags &INT_FLAG_NO_IOWAIT ) + flags |=IORING_ENTER_NO_IOWAIT ; ret =__sys_io_uring_enter2 (ring->enter_ring_fd, data->submit, data->wait_nr, flags, data->arg, data->sz);
Not strictly related, and we can always do that after, but now we have two branches here. Since the INT flags are purely internal, we can renumber them so that INT_FLAG_REG_RING matches IORING_ENTER_REGISTERED_RING and INT_FLAG_NO_IOWAIT matches IORING_ENTER_NO_IOWAIT. With that, you could kill the above two branches and simply do:
flags |= (ring->int_flags & INT_TO_ENTER_MASK);
which I think would be a nice thing to do.
Rest looks good, no further comments.
Signature · 1 line
Jens Axboe
Quoted 94 lines · Jens Axboe
man/io_uring_enter.2 If the ring file descriptor has been registered through use of · line 111
Quoted 13 lines · David Wei
man/io_uring_enter.2 +5
-0@@ -104,6 +104,11 @@ If the ring file descriptor has been registered through use of then setting this flag will tell the kernel that the .I ring_fd passed in is the registered ring offset rather than a normal file descriptor. +.TP +.B IORING_ENTER_NO_IOWAIT +If this flag is set, then in_iowait will not be set for the current task if +.BR io_uring_enter (2) +results in waiting.I'd probably would say something ala:
If this flag is set, then waiting on events will not be accounted as iowait for the task if
.BRio_uring_enter(2)
results in waiting.or something like that.
io_iowaitis an in-kernel thing, iowait is what application writers will know about.Quoted 6 lines
man/io_uring_enter.2 +6
-0+.B #include <liburing.h> +.PP +.BI "int io_uring_iowait_toggle(struct io_uring *" ring ", +.BI " bool " enabled ");" +.BI " +.fiman/io_uring_enter.2 If the ring file descriptor has been registered through use of
Quoted 10 lines
man/io_uring_enter.2 +10
-0+.SH DESCRIPTION +.PP +The +.BR io_uring_iowait_toggle (3) +function toggles for a given +.I ring +whether in_iowait is set for the current task while waiting for completions. +When in_iowait is set, time spent waiting is accounted as iowait time; +otherwise, it is accounted as idle time. The default behavior is to always set +in_iowait to true.And ditto here
man/io_uring_enter.2
Quoted 5 lines
man/io_uring_enter.2 +5
-0+Setting in_iowait achieves two things: + +1. Account time spent waiting as iowait time + +2. Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rqThis should probably be something ala:
Setting
in_iowaitachieves two things:
.TP
.B Account time spent waiting as iowait time
.TP
.B Enable cpufreq optimisations, settingSCHED_CPUFREQ_IOWAITon the rq .PPto make that format like a man page.
man/io_uring_enter.2
Quoted 4 lines
man/io_uring_enter.2 +4
-0+Some user tooling attributes iowait time as CPU utilization time, so high +iowait time can look like apparent high CPU utilization, even though the task +is not scheduled and the CPU is free to run other tasks. This function +provides a way to disable this behavior where it makes sense to do so.And here. Since this is the main man page, maybe also add something about how iowait is a relic from the old days of only having one CPU, and it indicates that the task is block uninterruptibly waiting for IO and hence cannot do other things. These days it's mostly a bogus accounting value, but it does help with the cpufreq boost for certain high frequency waits. Rephrase as needed :-)
src/queue.c static int _io_uring_get_cqe(struct io_uring *ring, · line 117
Quoted 13 lines
src/queue.c +2
-0@@ -110,6 +110,8 @@ static int _io_uring_get_cqe(struct io_uring *ring,if (ring->int_flags &INT_FLAG_REG_RING ) flags |=IORING_ENTER_REGISTERED_RING ; +if (ring->int_flags &INT_FLAG_NO_IOWAIT ) + flags |=IORING_ENTER_NO_IOWAIT ; ret =__sys_io_uring_enter2 (ring->enter_ring_fd, data->submit, data->wait_nr, flags, data->arg, data->sz);Not strictly related, and we can always do that after, but now we have two branches here. Since the INT flags are purely internal, we can renumber them so that
INT_FLAG_REG_RINGmatchesIORING_ENTER_REGISTERED_RINGandINT_FLAG_NO_IOWAITmatchesIORING_ENTER_NO_IOWAIT. With that, you could kill the above two branches and simply do:
Quoted 3 lines
flags |= (ring->int_flags & INT_TO_ENTER_MASK);which I think would be a nice thing to do.
SG, I'll do this in a follow up.
Quoted 3 lines
Rest looks good, no further comments.