Skip to contentSkip to conversation

Add io_uring_iowait_toggle()

PATCH Superseded David Wei 2 participants 3 messages Aug 16, 2024 io_uring

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

David Wei

author
#0 lore raw

PATCH 1/1 Add io_uring_iowait_toggle()

Add io_uring_iowait_toggle() helper function for the userspace liburing side of IORING_ENTER_NO_IOWAIT flag added in io_uring for 6.12.

This function toggles whether a ring sets in_iowait when waiting for completions. This is useful when waiting for multiple batched completions using e.g. io_uring_submit_and_wait_timeout() and userspace treats iowait time as CPU utilization.

It works by keeping an internal flag INT_FLAG_NO_IOWAIT, which if set will set IORING_ENTER_NO_IOWAIT on every io_uring_enter().

Manpages are added/modified, a unit test is included, and io_uring.h is synced with the kernel side.

---
 man/io_uring_enter.2            |   5 +
 man/io_uring_iowait_toggle.3    |  45 +++++++++
 src/include/liburing.h          |   1 +
 src/include/liburing/io_uring.h |   2 +
 src/int_flags.h                 |   1 +
 src/liburing.map                |   2 +
 src/queue.c                     |   2 +
 src/setup.c                     |  12 +++
 test/Makefile                   |   1 +
 test/no-iowait.c                | 162 ++++++++++++++++++++++++++++++++
 10 files changed, 233 insertions(+)
 create mode 100644 man/io_uring_iowait_toggle.3
 create mode 100644 test/no-iowait.c
man/io_uring_enter.2 +5 -0
index 5e4121b..c06050a 100644
@@ -104,6 +104,11 @@ If the ring file descriptor has been registered through use of
 then setting this flag will tell the kernel that the
 .I ring_fd
 passed in is the registered ring offset rather than a normal file descriptor.
+.TP
+.B IORING_ENTER_NO_IOWAIT
+If this flag is set, then in_iowait will not be set for the current task if
+.BR io_uring_enter (2)
+results in waiting.
 
 .PP
 .PP
man/io_uring_iowait_toggle.3 new file +45 -0

45 lines not on this page · raw message

src/include/liburing.h +1 -0
index 1092f3b..ddc154c 100644
@@ -243,6 +243,7 @@ int io_uring_unregister_napi(struct io_uring *ring, struct io_uring_napi *napi);
 
 int io_uring_get_events(struct io_uring *ring);
 int io_uring_submit_and_get_events(struct io_uring *ring);
+int io_uring_iowait_toggle(struct io_uring *ring, bool enabled);
 
 /*
  * io_uring syscalls.
src/include/liburing/io_uring.h +2 -0
index 01c36a8..36cabc5 100644
@@ -504,6 +504,7 @@ struct io_cqring_offsets {
 #define IORING_ENTER_SQ_WAIT		(1U << 2)
 #define IORING_ENTER_EXT_ARG		(1U << 3)
 #define IORING_ENTER_REGISTERED_RING	(1U << 4)
+#define IORING_ENTER_NO_IOWAIT		(1U << 6)
 
 /*
  * Passed in for io_uring_setup(2). Copied back with updated info on success
@@ -539,6 +540,7 @@ struct io_uring_params {
 #define IORING_FEAT_LINKED_FILE		(1U << 12)
 #define IORING_FEAT_REG_REG_RING	(1U << 13)
 #define IORING_FEAT_RECVSEND_BUNDLE	(1U << 14)
+#define IORING_FEAT_IOWAIT_TOGGLE	(1U << 15)
 
 /*
  * io_uring_register(2) opcodes and arguments
src/int_flags.h +1 -0
index 548dd10..80ad7cb 100644
@@ -6,6 +6,7 @@ enum {
 	INT_FLAG_REG_RING	= 1,
 	INT_FLAG_REG_REG_RING	= 2,
 	INT_FLAG_APP_MEM	= 4,
+	INT_FLAG_NO_IOWAIT	= 8,
 };
 
 #endif
src/liburing.map +2 -0
index fa096bb..e64fe2d 100644
@@ -97,4 +97,6 @@ LIBURING_2.7 {
 } LIBURING_2.6;
 
 LIBURING_2.8 {
+  global:
+    io_uring_iowait_toggle;
 } LIBURING_2.7;
src/queue.c +2 -0
index c436061..bd2e6af 100644
@@ -110,6 +110,8 @@ static int _io_uring_get_cqe(struct io_uring *ring,
 
 		if (ring->int_flags & INT_FLAG_REG_RING)
 			flags |= IORING_ENTER_REGISTERED_RING;
+		if (ring->int_flags & INT_FLAG_NO_IOWAIT)
+			flags |= IORING_ENTER_NO_IOWAIT;
 		ret = __sys_io_uring_enter2(ring->enter_ring_fd, data->submit,
 					    data->wait_nr, flags, data->arg,
 					    data->sz);
src/setup.c +12 -0
index 1997d25..2647363 100644
@@ -687,3 +687,15 @@ int io_uring_free_buf_ring(struct io_uring *ring, struct io_uring_buf_ring *br,
 	__sys_munmap(br, nentries * sizeof(struct io_uring_buf));
 	return 0;
 }
+
+int io_uring_iowait_toggle(struct io_uring *ring, bool enabled)
+{
+	if (!(ring->features & IORING_FEAT_IOWAIT_TOGGLE))
+		return -EOPNOTSUPP;
+
+	if (enabled)
+		ring->int_flags &= ~INT_FLAG_NO_IOWAIT;
+	else
+		ring->int_flags |= INT_FLAG_NO_IOWAIT;
+	return 0;
+}
test/Makefile +1 -0
index 0538a75..d9a049c 100644
@@ -125,6 +125,7 @@ test_srcs := \
 	msg-ring-flags.c \
 	msg-ring-overflow.c \
 	multicqes_drain.c \
+	no-iowait.c \
 	no-mmap-inval.c \
 	nolibc.c \
 	nop-all-sizes.c \
test/no-iowait.c new file +162 -0

162 lines not on this page · raw message

Signature · 1 line
2.43.5
maintainer
#1 lore raw

man/io_uring_enter.2 If the ring file descriptor has been registered through use of · line 111

Quoted 13 lines · David Wei
man/io_uring_enter.2 +5 -0
index 5e4121b..c06050a 100644
@@ -104,6 +104,11 @@ If the ring file descriptor has been registered through use of
 then setting this flag will tell the kernel that the
 .I ring_fd
 passed in is the registered ring offset rather than a normal file descriptor.
+.TP
+.B IORING_ENTER_NO_IOWAIT
+If this flag is set, then in_iowait will not be set for the current task if
+.BR io_uring_enter (2)
+results in waiting.

I'd probably would say something ala:

If this flag is set, then waiting on events will not be accounted as iowait for the task if
.BR io_uring_enter (2)
results in waiting.

or something like that. io_iowait is an in-kernel thing, iowait is what application writers will know about.

Quoted 6 lines
man/io_uring_enter.2 +6 -0
+.B #include <liburing.h>
+.PP
+.BI "int io_uring_iowait_toggle(struct io_uring *" ring ",
+.BI "                            bool " enabled ");"
+.BI "
+.fi

man/io_uring_enter.2 If the ring file descriptor has been registered through use of

Quoted 10 lines
man/io_uring_enter.2 +10 -0
+.SH DESCRIPTION
+.PP
+The
+.BR io_uring_iowait_toggle (3)
+function toggles for a given
+.I ring
+whether in_iowait is set for the current task while waiting for completions.
+When in_iowait is set, time spent waiting is accounted as iowait time;
+otherwise, it is accounted as idle time. The default behavior is to always set
+in_iowait to true.

And ditto here

man/io_uring_enter.2

Quoted 5 lines
man/io_uring_enter.2 +5 -0
+Setting in_iowait achieves two things:
+
+1. Account time spent waiting as iowait time
+
+2. Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rq

This should probably be something ala:

Setting in_iowait achieves two things:
.TP
.B Account time spent waiting as iowait time
.TP
.B Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rq .PP

to make that format like a man page.

man/io_uring_enter.2

Quoted 4 lines
man/io_uring_enter.2 +4 -0
+Some user tooling attributes iowait time as CPU utilization time, so high
+iowait time can look like apparent high CPU utilization, even though the task
+is not scheduled and the CPU is free to run other tasks.  This function
+provides a way to disable this behavior where it makes sense to do so.

And here. Since this is the main man page, maybe also add something about how iowait is a relic from the old days of only having one CPU, and it indicates that the task is block uninterruptibly waiting for IO and hence cannot do other things. These days it's mostly a bogus accounting value, but it does help with the cpufreq boost for certain high frequency waits. Rephrase as needed :-)

src/queue.c static int _io_uring_get_cqe(struct io_uring *ring, · line 117

Quoted 13 lines
src/queue.c +2 -0
index c436061..bd2e6af 100644
@@ -110,6 +110,8 @@ static int _io_uring_get_cqe(struct io_uring *ring,
 
 		if (ring->int_flags & INT_FLAG_REG_RING)
 			flags |= IORING_ENTER_REGISTERED_RING;
+		if (ring->int_flags & INT_FLAG_NO_IOWAIT)
+			flags |= IORING_ENTER_NO_IOWAIT;
 		ret = __sys_io_uring_enter2(ring->enter_ring_fd, data->submit,
 					    data->wait_nr, flags, data->arg,
 					    data->sz);

Not strictly related, and we can always do that after, but now we have two branches here. Since the INT flags are purely internal, we can renumber them so that INT_FLAG_REG_RING matches IORING_ENTER_REGISTERED_RING and INT_FLAG_NO_IOWAIT matches IORING_ENTER_NO_IOWAIT. With that, you could kill the above two branches and simply do:

	flags |= (ring->int_flags & INT_TO_ENTER_MASK);

which I think would be a nice thing to do.

Rest looks good, no further comments.

Signature · 1 line
Jens Axboe
author
#2 lore raw
Quoted 94 lines · Jens Axboe

man/io_uring_enter.2 If the ring file descriptor has been registered through use of · line 111

Quoted 13 lines · David Wei
man/io_uring_enter.2 +5 -0
index 5e4121b..c06050a 100644
@@ -104,6 +104,11 @@ If the ring file descriptor has been registered through use of
 then setting this flag will tell the kernel that the
 .I ring_fd
 passed in is the registered ring offset rather than a normal file descriptor.
+.TP
+.B IORING_ENTER_NO_IOWAIT
+If this flag is set, then in_iowait will not be set for the current task if
+.BR io_uring_enter (2)
+results in waiting.

I'd probably would say something ala:

If this flag is set, then waiting on events will not be accounted as iowait for the task if
.BR io_uring_enter (2)
results in waiting.

or something like that. io_iowait is an in-kernel thing, iowait is what application writers will know about.

Quoted 6 lines
man/io_uring_enter.2 +6 -0
+.B #include <liburing.h>
+.PP
+.BI "int io_uring_iowait_toggle(struct io_uring *" ring ",
+.BI "                            bool " enabled ");"
+.BI "
+.fi

man/io_uring_enter.2 If the ring file descriptor has been registered through use of

Quoted 10 lines
man/io_uring_enter.2 +10 -0
+.SH DESCRIPTION
+.PP
+The
+.BR io_uring_iowait_toggle (3)
+function toggles for a given
+.I ring
+whether in_iowait is set for the current task while waiting for completions.
+When in_iowait is set, time spent waiting is accounted as iowait time;
+otherwise, it is accounted as idle time. The default behavior is to always set
+in_iowait to true.

And ditto here

man/io_uring_enter.2

Quoted 5 lines
man/io_uring_enter.2 +5 -0
+Setting in_iowait achieves two things:
+
+1. Account time spent waiting as iowait time
+
+2. Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rq

This should probably be something ala:

Setting in_iowait achieves two things:
.TP
.B Account time spent waiting as iowait time
.TP
.B Enable cpufreq optimisations, setting SCHED_CPUFREQ_IOWAIT on the rq .PP

to make that format like a man page.

man/io_uring_enter.2

Quoted 4 lines
man/io_uring_enter.2 +4 -0
+Some user tooling attributes iowait time as CPU utilization time, so high
+iowait time can look like apparent high CPU utilization, even though the task
+is not scheduled and the CPU is free to run other tasks.  This function
+provides a way to disable this behavior where it makes sense to do so.

And here. Since this is the main man page, maybe also add something about how iowait is a relic from the old days of only having one CPU, and it indicates that the task is block uninterruptibly waiting for IO and hence cannot do other things. These days it's mostly a bogus accounting value, but it does help with the cpufreq boost for certain high frequency waits. Rephrase as needed :-)

src/queue.c static int _io_uring_get_cqe(struct io_uring *ring, · line 117

Quoted 13 lines
src/queue.c +2 -0
index c436061..bd2e6af 100644
@@ -110,6 +110,8 @@ static int _io_uring_get_cqe(struct io_uring *ring,
 
 		if (ring->int_flags & INT_FLAG_REG_RING)
 			flags |= IORING_ENTER_REGISTERED_RING;
+		if (ring->int_flags & INT_FLAG_NO_IOWAIT)
+			flags |= IORING_ENTER_NO_IOWAIT;
 		ret = __sys_io_uring_enter2(ring->enter_ring_fd, data->submit,
 					    data->wait_nr, flags, data->arg,
 					    data->sz);

Not strictly related, and we can always do that after, but now we have two branches here. Since the INT flags are purely internal, we can renumber them so that INT_FLAG_REG_RING matches IORING_ENTER_REGISTERED_RING and INT_FLAG_NO_IOWAIT matches IORING_ENTER_NO_IOWAIT. With that, you could kill the above two branches and simply do:

Quoted 3 lines
	flags |= (ring->int_flags & INT_TO_ENTER_MASK);

which I think would be a nice thing to do.

SG, I'll do this in a follow up.

Quoted 3 lines

Rest looks good, no further comments.