Skip to contentSkip to conversation

io_uring: avoid whole io_wq_work copy for requests completed inline

PATCH Superseded Xiaoguang Wang 3 participants 9 messages May 28, 2020 – May 30, 2020

Subsystems (2)

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/2 io_uring: avoid whole io_wq_work copy for requests completed inline

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

I use /dev/nullb0 to evaluate performance improvement in my physical machine:
modprobe null_blk nr_devices=1 completion_nsec=0
sudo taskset -c 60 fio -name=fiotest -filename=/dev/nullb0 -iodepth=128 -thread -rw=read -ioengine=io_uring -direct=1 -bs=4k -size=100G -numjobs=1 -time_based -runtime=120

before this patch:
Run status group 0 (all jobs):
READ: bw=724MiB/s (759MB/s), 724MiB/s-724MiB/s (759MB/s-759MB/s), io=84.8GiB (91.1GB), run=120001-120001msec

With this patch:
Run status group 0 (all jobs):
READ: bw=761MiB/s (798MB/s), 761MiB/s-761MiB/s (798MB/s-798MB/s), io=89.2GiB (95.8GB), run=120001-120001msec

About 5% improvement.

---
 fs/io-wq.h    |  5 ----
 fs/io_uring.c | 78 ++++++++++++++++++++++++++++++++++++++++-----------
 2 files changed, 62 insertions(+), 21 deletions(-)
fs/io-wq.h +0 -5
index 5ba12de7572f..3d85d365d764 100644
@@ -94,11 +94,6 @@ struct io_wq_work {
 	pid_t task_pid;
 };
 
-#define INIT_IO_WORK(work, _func)				\
-	do {							\
-		*(work) = (struct io_wq_work){ .func = _func };	\
-	} while (0)						\
-
 static inline struct io_wq_work *wq_next_work(struct io_wq_work *work)
 {
 	if (!work->list.next)
fs/io_uring.c +62 -16

184 lines not on this page · raw message

Signature · 1 line
2.17.2
#1 lore raw

PATCH v3 2/2 io_uring: avoid unnecessary io_wq_work copy for fast poll feature

Basically IORING_OP_POLL_ADD command and async armed poll handlers for regular commands don't touch io_wq_work, so only REQ_F_WORK_INITIALIZED is set, can we do io_wq_work copy and restore.

---
V3:
drop the REQ_F_WORK_NEED_RESTORE flag introduced in V2 patch, just use REQ_F_WORK_INITIALIZED to control whether to do io_wq_work copy and restore.

---
 fs/io_uring.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
fs/io_uring.c +9 -4
index 7ba8590a45a6..cade241568c6 100644
@@ -4411,7 +4411,8 @@ static void io_async_task_func(struct callback_head *cb)
 	spin_unlock_irq(&ctx->completion_lock);
 
 	/* restore ->work in case we need to retry again */
-	memcpy(&req->work, &apoll->work, sizeof(req->work));
+	if (req->flags & REQ_F_WORK_INITIALIZED)
+		memcpy(&req->work, &apoll->work, sizeof(req->work));
 	kfree(apoll);
 
 	if (!canceled) {
@@ -4508,7 +4509,8 @@ static bool io_arm_poll_handler(struct io_kiocb *req)
 		return false;
 
 	req->flags |= REQ_F_POLLED;
-	memcpy(&apoll->work, &req->work, sizeof(req->work));
+	if (req->flags & REQ_F_WORK_INITIALIZED)
+		memcpy(&apoll->work, &req->work, sizeof(req->work));
 	had_io = req->io != NULL;
 
 	get_task_struct(current);
@@ -4533,7 +4535,8 @@ static bool io_arm_poll_handler(struct io_kiocb *req)
 		if (!had_io)
 			io_poll_remove_double(req);
 		spin_unlock_irq(&ctx->completion_lock);
-		memcpy(&req->work, &apoll->work, sizeof(req->work));
+		if (req->flags & REQ_F_WORK_INITIALIZED)
+			memcpy(&req->work, &apoll->work, sizeof(req->work));
 		kfree(apoll);
 		return false;
 	}
@@ -4578,7 +4581,9 @@ static bool io_poll_remove_one(struct io_kiocb *req)
 			 * io_req_work_drop_env below when dropping the
 			 * final reference.
 			 */
-			memcpy(&req->work, &apoll->work, sizeof(req->work));
+			if (req->flags & REQ_F_WORK_INITIALIZED)
+				memcpy(&req->work, &apoll->work,
+				       sizeof(req->work));
 			kfree(apoll);
 		}
 	}
Signature · 1 line
2.17.2
maintainer
#2 lore raw
In reply to Xiaoguang Wang
Quoted 17 lines · Xiaoguang Wang

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

I use /dev/nullb0 to evaluate performance improvement in my physical machine:
modprobe null_blk nr_devices=1 completion_nsec=0
sudo taskset -c 60 fio -name=fiotest -filename=/dev/nullb0 -iodepth=128 -thread -rw=read -ioengine=io_uring -direct=1 -bs=4k -size=100G -numjobs=1 -time_based -runtime=120

before this patch:
Run status group 0 (all jobs):

   READ: bw=724MiB/s (759MB/s), 724MiB/s-724MiB/s (759MB/s-759MB/s),
   io=84.8GiB (91.1GB), run=120001-120001msec
Quoted 6 lines

With this patch:
Run status group 0 (all jobs):

   READ: bw=761MiB/s (798MB/s), 761MiB/s-761MiB/s (798MB/s-798MB/s),
   io=89.2GiB (95.8GB), run=120001-120001msec

About 5% improvement.

I think this is a big enough of a win to warrant looking closer at this. Just a quick comment from me so far:

fs/io_uring.c static int io_fsync(struct io_kiocb *req, bool force_nonblock) · line 2949

Quoted 9 lines
fs/io_uring.c +4 -1
@@ -2923,7 +2943,10 @@ static int io_fsync(struct io_kiocb *req, bool force_nonblock)
 {
 	/* fsync always requires a blocking context */
 	if (force_nonblock) {
-		req->work.func = io_fsync_finish;
+		if (!(req->flags & REQ_F_WORK_INITIALIZED))
+			init_io_work(req, io_fsync_finish);
+		else
+			req->work.func = io_fsync_finish;

This pattern is repeated enough to warrant a helper, ala:

static void io_req_init_async(req, func)

{
	if (req->flags & REQ_F_WORK_INITIALIZED)
		req->work.func = func;
	else
		init_io_work(req, func);
}

also swapped the conditions, I tend to find it easier to read without the negation.

Signature · 1 line
Jens Axboe
#3 lore raw

hi,

Quoted 23 lines · Xiaoguang Wang
Quoted 22 lines

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

I use /dev/nullb0 to evaluate performance improvement in my physical machine:
modprobe null_blk nr_devices=1 completion_nsec=0
sudo taskset -c 60 fio -name=fiotest -filename=/dev/nullb0 -iodepth=128 -thread -rw=read -ioengine=io_uring -direct=1 -bs=4k -size=100G -numjobs=1 -time_based -runtime=120

before this patch:
Run status group 0 (all jobs):

    READ: bw=724MiB/s (759MB/s), 724MiB/s-724MiB/s (759MB/s-759MB/s),
    io=84.8GiB (91.1GB), run=120001-120001msec

With this patch:
Run status group 0 (all jobs):

    READ: bw=761MiB/s (798MB/s), 761MiB/s-761MiB/s (798MB/s-798MB/s),
    io=89.2GiB (95.8GB), run=120001-120001msec
Quoted 4 lines
Quoted 1 line

About 5% improvement.

I think this is a big enough of a win to warrant looking closer at this. Just a quick comment from me so far:

Yeah, to be honest, I did't expect that we get this some obvious improvement. But I have run multiple rounds of same tests, I always get similar improvement, if you have some free time, you can have a test :)

Quoted 21 lines

fs/io_uring.c static int io_fsync(struct io_kiocb *req, bool force_nonblock) · line 2949

Quoted 10 lines
fs/io_uring.c +4 -1
@@ -2923,7 +2943,10 @@ static int io_fsync(struct io_kiocb *req, bool force_nonblock)
  {
  	/* fsync always requires a blocking context */
  	if (force_nonblock) {
-		req->work.func = io_fsync_finish;
+		if (!(req->flags & REQ_F_WORK_INITIALIZED))
+			init_io_work(req, io_fsync_finish);
+		else
+			req->work.func = io_fsync_finish;

This pattern is repeated enough to warrant a helper, ala:

static void io_req_init_async(req, func)

{
	if (req->flags & REQ_F_WORK_INITIALIZED)
		req->work.func = func;
	else
		init_io_work(req, func);
}
Quoted 2 lines

also swapped the conditions, I tend to find it easier to read without the negation.

Thanks for your suggestions. I'll prepare a V4 soon.

Regards,
Xiaoguang Wang

Quoted 1 line
#4 lore raw
In reply to Xiaoguang Wang
Quoted 4 lines · Xiaoguang Wang

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

It looks nicer. Especially if you'd add a helper as Jens supposed.

The other thing, even though I hate treating a part of the fields differently from others, I don't like ->creds tossing either.

Did you consider trying using only ->work.creds without adding req->creds? like in the untested incremental below. init_io_work() there is misleading, should be somehow played around better.

fs/io_uring.c +18 -15
index 4dd3295d74f6..4086561ce444 100644
@@ -643,7 +643,6 @@ struct io_kiocb {
 	unsigned int		flags;
 	refcount_t		refs;
 	struct task_struct	*task;
-	const struct cred	*creds;
 	unsigned long		fsize;
 	u64			user_data;
 	u32			result;
@@ -894,8 +893,16 @@ static const struct file_operations io_uring_fops;
 static inline void init_io_work(struct io_kiocb *req,
 			void (*func)(struct io_wq_work **))
 {
-	req->work = (struct io_wq_work){ .func = func };
-	req->flags |= REQ_F_WORK_INITIALIZED;
+	struct io_wq_work *work = &req->work;
+
+	/* work->creds are already initialised by a user */
+	work->list.next = NULL;
+	work->func = func;
+	work->files = NULL;
+	work->mm = NULL;
+	work->fs = NULL;
+	work->flags = REQ_F_WORK_INITIALIZED;
+	work->task_pid = 0;
 }
 struct sock *io_uring_get_socket(struct file *file)
 {
@@ -1019,15 +1026,9 @@ static inline void io_req_work_grab_env(struct io_kiocb *req,
 		mmgrab(current->mm);
 		req->work.mm = current->mm;
 	}
+	if (!req->work.creds)
+		req->work.creds = get_current_cred();
 
-	if (!req->work.creds) {
-		if (!req->creds)
-			req->work.creds = get_current_cred();
-		else {
-			req->work.creds = req->creds;
-			req->creds = NULL;
-		}
-	}
 	if (!req->work.fs && def->needs_fs) {
 		spin_lock(&current->fs->lock);
 		if (!current->fs->in_exec) {
@@ -1044,6 +1045,12 @@ static inline void io_req_work_grab_env(struct io_kiocb *req,
 
 static inline void io_req_work_drop_env(struct io_kiocb *req)
 {
+	/* always init'ed, put before REQ_F_WORK_INITIALIZED check */
+	if (req->work.creds) {
+		put_cred(req->work.creds);
+		req->work.creds = NULL;
+	}
+
 	if (!(req->flags & REQ_F_WORK_INITIALIZED))
 		return;
 
@@ -1051,10 +1058,6 @@ static inline void io_req_work_drop_env(struct io_kiocb *req)
 		mmdrop(req->work.mm);
 		req->work.mm = NULL;
 	}
-	if (req->work.creds) {
-		put_cred(req->work.creds);
-		req->work.creds = NULL;
-	}
 	if (req->work.fs) {
 		struct fs_struct *fs = req->work.fs;
 
@@ -5901,12 +5904,12 @@ static int io_init_req(struct io_ring_ctx *ctx, struct

io_kiocb *req,

id = READ_ONCE(sqe->personality);
if (id) {
- req->creds = idr_find(&ctx->personality_idr, id);
- if (unlikely(!req->creds))
+ req->work.creds = idr_find(&ctx->personality_idr, id);
+ if (unlikely(!req->work.creds))
return -EINVAL;
- get_cred(req->creds);
+ get_cred(req->work.creds);
} else
- req->creds = NULL;
+ req->work.creds = NULL;

/* same numerical values with corresponding REQ_F_*, safe to copy */ req->flags |= sqe_flags;

Signature · 1 line
Pavel Begunkov
maintainer
#5 lore raw
Quoted 6 lines · Pavel Begunkov
Quoted 4 lines · Xiaoguang Wang

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

Quoted 8 lines

It looks nicer. Especially if you'd add a helper as Jens supposed.

The other thing, even though I hate treating a part of the fields differently from others, I don't like ->creds tossing either.

Did you consider trying using only ->work.creds without adding req->creds? like in the untested incremental below. init_io_work() there is misleading, should be somehow played around better.

I had that thought too when reading the patchset, would be nice _not_ to have to add a new creds field.

Signature · 1 line
Jens Axboe
#6 lore raw
In reply to Pavel Begunkov

hi,

Quoted 7 lines · Xiaoguang Wang
Quoted 4 lines

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

It looks nicer. Especially if you'd add a helper as Jens supposed.

Sure, I'll add a helper in V4, thanks.

Quoted 7 lines

The other thing, even though I hate treating a part of the fields differently from others, I don't like ->creds tossing either.

Did you consider trying using only ->work.creds without adding req->creds? like in the untested incremental below. init_io_work() there is misleading, should be somehow played around better.

But if not adding a new req->creds, I think there will be some potential risks. In current io_uring mainline codes, look at io_kiocb's memory layout crash> struct -o io_kiocb

struct io_kiocb {
        union {
    [0]     struct file *file;
    [0]     struct io_rw rw;
    [0]     struct io_poll_iocb poll;
    [0]     struct io_accept accept;
    [0]     struct io_sync sync;
    [0]     struct io_cancel cancel;
    [0]     struct io_timeout timeout;
    [0]     struct io_connect connect;
    [0]     struct io_sr_msg sr_msg;
    [0]     struct io_open open;
    [0]     struct io_close close;
    [0]     struct io_files_update files_update;
    [0]     struct io_fadvise fadvise;
    [0]     struct io_madvise madvise;
    [0]     struct io_epoll epoll;
    [0]     struct io_splice splice;
    [0]     struct io_provide_buf pbuf;
        };
   [64] struct io_async_ctx *io;
   [72] int cflags;
   [76] u8 opcode;
   [78] u16 buf_index;
   [80] struct io_ring_ctx *ctx;
   [88] struct list_head list;
  [104] unsigned int flags;
  [108] refcount_t refs;
  [112] struct task_struct *task;
  [120] unsigned long fsize;
  [128] u64 user_data;
  [136] u32 result;
  [140] u32 sequence;
  [144] struct list_head link_list;
  [160] struct list_head inflight_entry;
  [176] struct percpu_ref *fixed_file_refs;
        union {
            struct {
  [184]         struct callback_head task_work;
  [200]         struct hlist_node hash_node;
  [216]         struct async_poll *apoll;
            };
  [184]     struct io_wq_work work;
        };
}

SIZE: 240

struct io_wq_work {
[0] struct io_wq_work_node list;
[8] void (*func)(struct io_wq_work **);
[16] struct files_struct *files;
[24] struct mm_struct *mm;
[32] const struct cred *creds;
[40] struct fs_struct *fs;
[48] unsigned int flags;
[52] pid_t task_pid;
}
SIZE: 56

The risk mainly comes from the union:

union {
	/*
	 * Only commands that never go async can use the below fields,
	 * obviously. Right now only IORING_OP_POLL_ADD uses them, and
	 * async armed poll handlers for regular commands. The latter
	 * restore the work, if needed.
	 */
	struct {
		struct callback_head	task_work;
		struct hlist_node	hash_node;
		struct async_poll	*apoll;
	};
	struct io_wq_work	work;
};

1, apoll and creds are in same memory offset, for 'async armed poll handlers' case, apoll will be used, that means creds will be overwrited. In patch "io_uring: avoid unnecessary io_wq_work copy for fast poll feature", I use REQ_F_WORK_INITIALIZED to control whether to do io_wq_work restore, then your below codes will break:

static inline void io_req_work_drop_env(struct io_kiocb *req)

{
	/* always init'ed, put before REQ_F_WORK_INITIALIZED check */
	if (req->work.creds) {

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here req->work.creds will be invalid, or I still need to use some space to record original req->work.creds, and do creds restore.

		put_cred(req->work.creds);
		req->work.creds = NULL;
	}
	if (!(req->flags & REQ_F_WORK_INITIALIZED))

return;

2, For IORING_OP_POLL_ADD case, current mainline codes will use task_work and hash_node, 32 bytes, that means io_wq_work's member list, func, files and mm would be overwrited, but will not touch creds, it's safe now. But if we will add some new member to

struct {
	struct callback_head	task_work;
	struct hlist_node	hash_node;
	struct async_poll	*apoll;
};

say callback_head adds a new member, our check will still break.

3. IMO, io_wq_work is just to describe needed running environment for reqs that will be punted to io-wq, for reqs submitted and completed inline should not touch this struct from software design view, and current io_kiocb is 240 bytes, and a new pointer will be 248 bytes, still 4 cache lines for cache line 64 bytes.

Regards,
Xiaoguang Wang

Quoted 93 lines
fs/io_uring.c +22 -19
index 4dd3295d74f6..4086561ce444 100644
@@ -643,7 +643,6 @@ struct io_kiocb {
  	unsigned int		flags;
  	refcount_t		refs;
  	struct task_struct	*task;
-	const struct cred	*creds;
  	unsigned long		fsize;
  	u64			user_data;
  	u32			result;
@@ -894,8 +893,16 @@ static const struct file_operations io_uring_fops;
  static inline void init_io_work(struct io_kiocb *req,
  			void (*func)(struct io_wq_work **))
  {
-	req->work = (struct io_wq_work){ .func = func };
-	req->flags |= REQ_F_WORK_INITIALIZED;
+	struct io_wq_work *work = &req->work;
+
+	/* work->creds are already initialised by a user */
+	work->list.next = NULL;
+	work->func = func;
+	work->files = NULL;
+	work->mm = NULL;
+	work->fs = NULL;
+	work->flags = REQ_F_WORK_INITIALIZED;
+	work->task_pid = 0;
  }
  struct sock *io_uring_get_socket(struct file *file)
  {
@@ -1019,15 +1026,9 @@ static inline void io_req_work_grab_env(struct io_kiocb *req,
  		mmgrab(current->mm);
  		req->work.mm = current->mm;
  	}
+	if (!req->work.creds)
+		req->work.creds = get_current_cred();
 
-	if (!req->work.creds) {
-		if (!req->creds)
-			req->work.creds = get_current_cred();
-		else {
-			req->work.creds = req->creds;
-			req->creds = NULL;
-		}
-	}
  	if (!req->work.fs && def->needs_fs) {
  		spin_lock(&current->fs->lock);
  		if (!current->fs->in_exec) {
@@ -1044,6 +1045,12 @@ static inline void io_req_work_grab_env(struct io_kiocb *req,
 
  static inline void io_req_work_drop_env(struct io_kiocb *req)
  {
+	/* always init'ed, put before REQ_F_WORK_INITIALIZED check */
+	if (req->work.creds) {
+		put_cred(req->work.creds);
+		req->work.creds = NULL;
+	}
+
  	if (!(req->flags & REQ_F_WORK_INITIALIZED))
  		return;
 
@@ -1051,10 +1058,6 @@ static inline void io_req_work_drop_env(struct io_kiocb *req)
  		mmdrop(req->work.mm);
  		req->work.mm = NULL;
  	}
-	if (req->work.creds) {
-		put_cred(req->work.creds);
-		req->work.creds = NULL;
-	}
  	if (req->work.fs) {
  		struct fs_struct *fs = req->work.fs;
 
@@ -5901,12 +5904,12 @@ static int io_init_req(struct io_ring_ctx *ctx, struct
 io_kiocb *req,
 
  	id = READ_ONCE(sqe->personality);
  	if (id) {
-		req->creds = idr_find(&ctx->personality_idr, id);
-		if (unlikely(!req->creds))
+		req->work.creds = idr_find(&ctx->personality_idr, id);
+		if (unlikely(!req->work.creds))
  			return -EINVAL;
-		get_cred(req->creds);
+		get_cred(req->work.creds);
  	} else
-		req->creds = NULL;
+		req->work.creds = NULL;
 
  	/* same numerical values with corresponding REQ_F_*, safe to copy */
  	req->flags |= sqe_flags;
maintainer
#7 lore raw
Quoted 123 lines · Xiaoguang Wang

hi,

Quoted 7 lines · Xiaoguang Wang
Quoted 4 lines

If requests can be submitted and completed inline, we don't need to initialize whole io_wq_work in io_init_req(), which is an expensive operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether io_wq_work is initialized.

It looks nicer. Especially if you'd add a helper as Jens supposed.

Sure, I'll add a helper in V4, thanks.

Quoted 7 lines

The other thing, even though I hate treating a part of the fields differently from others, I don't like ->creds tossing either.

Did you consider trying using only ->work.creds without adding req->creds? like in the untested incremental below. init_io_work() there is misleading, should be somehow played around better.

But if not adding a new req->creds, I think there will be some potential risks. In current io_uring mainline codes, look at io_kiocb's memory layout crash> struct -o io_kiocb

struct io_kiocb {
         union {
     [0]     struct file *file;
     [0]     struct io_rw rw;
     [0]     struct io_poll_iocb poll;
     [0]     struct io_accept accept;
     [0]     struct io_sync sync;
     [0]     struct io_cancel cancel;
     [0]     struct io_timeout timeout;
     [0]     struct io_connect connect;
     [0]     struct io_sr_msg sr_msg;
     [0]     struct io_open open;
     [0]     struct io_close close;
     [0]     struct io_files_update files_update;
     [0]     struct io_fadvise fadvise;
     [0]     struct io_madvise madvise;
     [0]     struct io_epoll epoll;
     [0]     struct io_splice splice;
     [0]     struct io_provide_buf pbuf;
         };
    [64] struct io_async_ctx *io;
    [72] int cflags;
    [76] u8 opcode;
    [78] u16 buf_index;
    [80] struct io_ring_ctx *ctx;
    [88] struct list_head list;
   [104] unsigned int flags;
   [108] refcount_t refs;
   [112] struct task_struct *task;
   [120] unsigned long fsize;
   [128] u64 user_data;
   [136] u32 result;
   [140] u32 sequence;
   [144] struct list_head link_list;
   [160] struct list_head inflight_entry;
   [176] struct percpu_ref *fixed_file_refs;
         union {
             struct {
   [184]         struct callback_head task_work;
   [200]         struct hlist_node hash_node;
   [216]         struct async_poll *apoll;
             };
   [184]     struct io_wq_work work;
         };
}

SIZE: 240

struct io_wq_work {
    [0] struct io_wq_work_node list;
    [8] void (*func)(struct io_wq_work **);
   [16] struct files_struct *files;
   [24] struct mm_struct *mm;
   [32] const struct cred *creds;
   [40] struct fs_struct *fs;
   [48] unsigned int flags;
   [52] pid_t task_pid;
}

SIZE: 56

The risk mainly comes from the union:

union {
	/*
	 * Only commands that never go async can use the below fields,
	 * obviously. Right now only IORING_OP_POLL_ADD uses them, and
	 * async armed poll handlers for regular commands. The latter
	 * restore the work, if needed.
	 */
	struct {
		struct callback_head	task_work;
		struct hlist_node	hash_node;
		struct async_poll	*apoll;
	};
	struct io_wq_work	work;
};

1, apoll and creds are in same memory offset, for 'async armed poll handlers' case, apoll will be used, that means creds will be overwrited. In patch "io_uring: avoid unnecessary io_wq_work copy for fast poll feature", I use REQ_F_WORK_INITIALIZED to control whether to do io_wq_work restore, then your below codes will break:

static inline void io_req_work_drop_env(struct io_kiocb *req)

{
	/* always init'ed, put before REQ_F_WORK_INITIALIZED check */
	if (req->work.creds) {

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here req->work.creds will be invalid, or I still need to use some space to record original req->work.creds, and do creds restore.

		put_cred(req->work.creds);
		req->work.creds = NULL;
	}
	if (!(req->flags & REQ_F_WORK_INITIALIZED))

return;

2, For IORING_OP_POLL_ADD case, current mainline codes will use task_work and hash_node, 32 bytes, that means io_wq_work's member list, func, files and mm would be overwrited, but will not touch creds, it's safe now. But if we will add some new member to

struct {
	struct callback_head	task_work;
	struct hlist_node	hash_node;
	struct async_poll	*apoll;
};
Quoted 6 lines

say callback_head adds a new member, our check will still break.

3. IMO, io_wq_work is just to describe needed running environment for reqs that will be punted to io-wq, for reqs submitted and completed inline should not touch this struct from software design view, and current io_kiocb is 240 bytes, and a new pointer will be 248 bytes, still 4 cache lines for cache line 64 bytes.

I'd say let's do the extra creds member for now. It's safer. We can always look into shrinking down the size and re-using the io_wq_work part. I'd rather get this prepped up and ready for 5.8.

Are you sending out a v4 today?

Signature · 1 line
Jens Axboe
#8 lore raw
In reply to Xiaoguang Wang
Quoted 17 lines · Xiaoguang Wang

The risk mainly comes from the union:
union {
    /*
     * Only commands that never go async can use the below fields,
     * obviously. Right now only IORING_OP_POLL_ADD uses them, and
     * async armed poll handlers for regular commands. The latter
     * restore the work, if needed.
     */
    struct {
        struct callback_head    task_work;
        struct hlist_node    hash_node;
        struct async_poll    *apoll;
    };
    struct io_wq_work    work;
};

Quoted 4 lines

1, apoll and creds are in same memory offset, for 'async armed poll handlers' case, apoll will be used, that means creds will be overwrited. In patch "io_uring: avoid unnecessary io_wq_work copy for fast poll feature", I use REQ_F_WORK_INITIALIZED to control whether to do io_wq_work restore, then your below codes will break:

Yes, that's an example, which doesn't even consider the second patch. But great that you anticipated the error. Unconditional partial copy/init probably would solve the issue, but let's keep it aside for now.

Quoted 27 lines

static inline void io_req_work_drop_env(struct io_kiocb *req) {
    /* always init'ed, put before REQ_F_WORK_INITIALIZED check */ if (req->work.creds) { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here req->work.creds will be invalid, or I still need to use some space to record original req->work.creds, and do creds restore.

        put_cred(req->work.creds);
        req->work.creds = NULL;
    }
    if (!(req->flags & REQ_F_WORK_INITIALIZED))
         return;

2, For IORING_OP_POLL_ADD case, current mainline codes will use task_work and hash_node,
32 bytes, that means io_wq_work's member list, func, files and mm would be overwrited,
but will not touch creds, it's safe now. But if we will add some new member to struct {
    struct callback_head    task_work;
    struct hlist_node    hash_node;
    struct async_poll    *apoll;
};
say callback_head adds a new member, our check will still break.

Quoted 5 lines

3. IMO, io_wq_work is just to describe needed running environment for reqs that will be
punted to io-wq, for reqs submitted and completed inline should not touch this struct
from software design view, and current io_kiocb is 240 bytes, and a new pointer

Instead, it stores an entity in 2 different places, adding yet another thing/state to keep in mind. Arguable. I'd rather say -- neither one is better. And that's why I like the simplicity of initialising it always.

Quoted 4 lines

will be
248 bytes, still 4 cache lines for cache line 64 bytes.

Signature · 1 line
Pavel Begunkov