Skip to contentSkip to conversation

io_uring: reserve word at cqring tail+4 for the user

PATCH In review Avi Kivity 2 participants 3 messages Sep 17, 2019

Subsystems (3)
lore
Share & tools
lore

Avi Kivity

author
#0 lore raw

PATCH 1/1 io_uring: reserve word at cqring tail+4 for the user

In some applications, a thread waits for I/O events generated by the kernel, and also events generated by other threads in the same application. Typically events from other threads are passed using in-memory queues that are not known to the kernel. As long as the threads is active, it polls for both kernel completions and inter-thread completions; when it is idle, it tells the other threads to use an I/O event to wait it up (e.g. an eventfd or a pipe) and then enters the kernel, waiting for such an event or an ordinary I/O completion.

When such a thread goes idle, it typically spins for a while to avoid the kernel entry/exit cost in case an event is forthcoming shortly. While it spins it polls both I/O completions and inter-thread queues.

The x86 instruction pair UMONITOR/UMWAIT allows waiting for a cache line to be written to. This can be used with io_uring to wait for a wakeup without spinning (and wasting power and slowing down the other hyperthread). Other threads can also wake up the waiter by doing a safe write to the tail word (which triggers the wakeup), but safe writes are slow as they require an atomic instruction. To speed up those wakeups, reserve a word after the tail for user writes.

A thread consuming an io_uring completion queue can then use the following sequences:

- while busy:
- pick up work from the completion queue and from other threads, and process it

- while idle:
- use UMONITOR/UMWAIT to wait on completions and notifications from other threads for a short period
- if no work is picked up, let other threads know you will need a kernel wakeup, and use io_uring_enter to wait indefinitely

---
 fs/io_uring.c                 | 5 +++--
 include/uapi/linux/io_uring.h | 4 ++++
 2 files changed, 7 insertions(+), 2 deletions(-)
fs/io_uring.c +3 -2
index cfb48bd088e1..4bd7905cee1d 100644
@@ -77,12 +77,13 @@
 
 #define IORING_MAX_ENTRIES	4096
 #define IORING_MAX_FIXED_FILES	1024
 
 struct io_uring {
-	u32 head ____cacheline_aligned_in_smp;
-	u32 tail ____cacheline_aligned_in_smp;
+	u32 head ____cacheline_aligned;
+	u32 tail ____cacheline_aligned;
+	u32 reserved_for_user; // for cq ring and UMONITOR/UMWAIT (or similar) wakeups
 };
 
 /*
  * This data is shared with the application through the mmap at offset
  * IORING_OFF_SQ_RING.
include/uapi/linux/io_uring.h +4 -0
index 1e1652f25cc1..1a6a826a66f3 100644
@@ -103,10 +103,14 @@ struct io_sqring_offsets {
  */
 #define IORING_SQ_NEED_WAKEUP	(1U << 0) /* needs io_uring_enter wakeup */
 
 struct io_cqring_offsets {
 	__u32 head;
+	// tail is guaranteed to be aligned on a cache line, and to have the
+	// following __u32 free for user use. This allows using e.g.
+	// UMONITOR/UMWAIT to wait on both writes to head and writes from
+	// other threads to the following word.
 	__u32 tail;
 	__u32 ring_mask;
 	__u32 ring_entries;
 	__u32 overflow;
 	__u32 cqes;
Signature · 1 line
2.21.0
maintainer
#1 lore raw
Quoted 30 lines · Avi Kivity

In some applications, a thread waits for I/O events generated by the kernel, and also events generated by other threads in the same application. Typically events from other threads are passed using in-memory queues that are not known to the kernel. As long as the threads is active, it polls for both kernel completions and inter-thread completions; when it is idle, it tells the other threads to use an I/O event to wait it up (e.g. an eventfd or a pipe) and then enters the kernel, waiting for such an event or an ordinary I/O completion.

When such a thread goes idle, it typically spins for a while to avoid the kernel entry/exit cost in case an event is forthcoming shortly. While it spins it polls both I/O completions and inter-thread queues.

The x86 instruction pair UMONITOR/UMWAIT allows waiting for a cache line to be written to. This can be used with io_uring to wait for a wakeup without spinning (and wasting power and slowing down the other hyperthread). Other threads can also wake up the waiter by doing a safe write to the tail word (which triggers the wakeup), but safe writes are slow as they require an atomic instruction. To speed up those wakeups, reserve a word after the tail for user writes.

A thread consuming an io_uring completion queue can then use the following sequences:

- while busy:
- pick up work from the completion queue and from other threads, and process it

Quoted 5 lines

- while idle:
- use UMONITOR/UMWAIT to wait on completions and notifications from other threads for a short period
- if no work is picked up, let other threads know you will need a kernel wakeup, and use io_uring_enter to wait indefinitely

This is cool, I like it. A few comments:

Quoted 6 lines
fs/io_uring.c +0 -0
index cfb48bd088e1..4bd7905cee1d 100644
@@ -77,12 +77,13 @@
 

fs/io_uring.c line 85

Quoted 10 lines
fs/io_uring.c +3 -2
  #define IORING_MAX_ENTRIES	4096
  #define IORING_MAX_FIXED_FILES	1024
 
  struct io_uring {
-	u32 head ____cacheline_aligned_in_smp;
-	u32 tail ____cacheline_aligned_in_smp;
+	u32 head ____cacheline_aligned;
+	u32 tail ____cacheline_aligned;
+	u32 reserved_for_user; // for cq ring and UMONITOR/UMWAIT (or similar) wakeups
  };

Since we have that full cacheline, maybe name this one a bit more appropriately as we can add others if we need it. Not a big deal. But definitely use /* */ style comments :-)

Quoted 9 lines
include/uapi/linux/io_uring.h +0 -0
index 1e1652f25cc1..1a6a826a66f3 100644
@@ -103,10 +103,14 @@ struct io_sqring_offsets {
   */
  #define IORING_SQ_NEED_WAKEUP	(1U << 0) /* needs io_uring_enter wakeup */
 
  struct io_cqring_offsets {

include/uapi/linux/io_uring.h struct io_sqring_offsets { · line 116

Quoted 10 lines
include/uapi/linux/io_uring.h +4 -0
  	__u32 head;
+	// tail is guaranteed to be aligned on a cache line, and to have the
+	// following __u32 free for user use. This allows using e.g.
+	// UMONITOR/UMWAIT to wait on both writes to head and writes from
+	// other threads to the following word.
  	__u32 tail;
  	__u32 ring_mask;
  	__u32 ring_entries;
  	__u32 overflow;
  	__u32 cqes;

Ditto on the comments here.

Would be ideal if we could pair this with an example for liburing, a basic test case would be fine. Something that shows how to use it, and verifies that it works.

Also, this patch is against master, it should be against for-5.4/io_iuring as it won't apply there right now.

Signature · 1 line
Jens Axboe
author
#2 lore raw
Quoted 47 lines · Jens Axboe
Quoted 30 lines · Avi Kivity

In some applications, a thread waits for I/O events generated by the kernel, and also events generated by other threads in the same application. Typically events from other threads are passed using in-memory queues that are not known to the kernel. As long as the threads is active, it polls for both kernel completions and inter-thread completions; when it is idle, it tells the other threads to use an I/O event to wait it up (e.g. an eventfd or a pipe) and then enters the kernel, waiting for such an event or an ordinary I/O completion.

When such a thread goes idle, it typically spins for a while to avoid the kernel entry/exit cost in case an event is forthcoming shortly. While it spins it polls both I/O completions and inter-thread queues.

The x86 instruction pair UMONITOR/UMWAIT allows waiting for a cache line to be written to. This can be used with io_uring to wait for a wakeup without spinning (and wasting power and slowing down the other hyperthread). Other threads can also wake up the waiter by doing a safe write to the tail word (which triggers the wakeup), but safe writes are slow as they require an atomic instruction. To speed up those wakeups, reserve a word after the tail for user writes.

A thread consuming an io_uring completion queue can then use the following sequences:

- while busy:
- pick up work from the completion queue and from other threads, and process it

Quoted 5 lines

- while idle:
- use UMONITOR/UMWAIT to wait on completions and notifications from other threads for a short period
- if no work is picked up, let other threads know you will need a kernel wakeup, and use io_uring_enter to wait indefinitely

This is cool, I like it. A few comments:

Quoted 9 lines
fs/io_uring.c +0 -0
index cfb48bd088e1..4bd7905cee1d 100644
@@ -77,12 +77,13 @@
 
   #define IORING_MAX_ENTRIES	4096
   #define IORING_MAX_FIXED_FILES	1024
 
Quoted 9 lines

fs/io_uring.c line 85

Quoted 7 lines
fs/io_uring.c +3 -2
   struct io_uring {
-	u32 head ____cacheline_aligned_in_smp;
-	u32 tail ____cacheline_aligned_in_smp;
+	u32 head ____cacheline_aligned;
+	u32 tail ____cacheline_aligned;
+	u32 reserved_for_user; // for cq ring and UMONITOR/UMWAIT (or similar) wakeups
   };

Since we have that full cacheline, maybe name this one a bit more appropriately as we can add others if we need it. Not a big deal.

You mean, name it for its intended purpose of serving as a write target for umonitor/umwait wakes?

Note that the user won't see the name, and that it's only accurate for an io_uring that's used for completions.

Quoted 1 line

But definitely use /* */ style comments :-)

Sorry, in C++-land for a while. You're lucky I didn't turn the whole thing into a virtual template something.

Quoted 9 lines
Quoted 9 lines
include/uapi/linux/io_uring.h +0 -0
index 1e1652f25cc1..1a6a826a66f3 100644
@@ -103,10 +103,14 @@ struct io_sqring_offsets {
    */
   #define IORING_SQ_NEED_WAKEUP	(1U << 0) /* needs io_uring_enter wakeup */
 
Quoted 12 lines

include/uapi/linux/io_uring.h struct io_sqring_offsets { · line 116

Quoted 11 lines
include/uapi/linux/io_uring.h +4 -0
   struct io_cqring_offsets {
   	__u32 head;
+	// tail is guaranteed to be aligned on a cache line, and to have the
+	// following __u32 free for user use. This allows using e.g.
+	// UMONITOR/UMWAIT to wait on both writes to head and writes from
+	// other threads to the following word.
   	__u32 tail;
   	__u32 ring_mask;
   	__u32 ring_entries;
   	__u32 overflow;
   	__u32 cqes;

Ditto on the comments here.

Sure.

Quoted 3 lines

Would be ideal if we could pair this with an example for liburing, a basic test case would be fine. Something that shows how to use it, and verifies that it works.

I'll have to look for a machine with waitpkg for that.

Quoted 2 lines

Also, this patch is against master, it should be against for-5.4/io_iuring as it won't apply there right now.

Sure, will rebase.