uringlet
cover uringlet
From: Hao Xu <[email protected]>
Hi Jens and all,
This is an early RFC for a new way to do async IO. Currently io_uring works in a way like:
- issue an IO request in nowait way
here nowait means return error(EAGAIN) to io_uring layer when it would block in deeper kernel stack.
- issue an IO request in a normal(block) way
io_uring catches the EAGAIN error and create/wakeup a io-worker to redo the IO request in a block way. The original context turns to issue other requests. (some type of requests like buffered reads, leverage task work to wipe out io-workers)
This has two main disadvantages:
- we have to find every block point along the kernel code path and modify it to support nowait.
e.g. alloc_memory() ----> if (alloc_memory() fails) return -EAGAIN This hugely adds programming complexisity, especially when the code path is long and complicated. For example, buffered write, we have to handle locks, possibly journal part, meta data like extent node misses.
- By create/wakeup a new worker, we redo a IO request from the very beginning, which means we re-walk the path from beginning to the previous block point.
The original context backtracks to the io_uring layer from the block point to submit other requests. While it's better to directly start the new submission.
This RFC provides a new way to do it.
- We maintain a worker pool for each io_uring instance and each worker in it can submit requests. The original task only needs to create the first worker and return to userspace. Later it doesn't need to call io_uring_enter.[1]
- the created worker begins to submit requests. When it blocks, just let it be blocked. Create/wakeup another worker to do the submission
[1] I currently keep these workers until the io_uring context exits. In other words, a worker does submission, sleep, wake up, but won't exit. Thus the original task don't need to create/wakeup workers.
I've done some testing:
name: buffered write
fs: xfs
env: qemu box, 4 cpu, 8G mem.
tool: fio
- single file test:
fio ioengine=io_uring, size=10M, bs=1024, direct=0,
thread=1, rw=randwrite, time_based=1, runtime=180
async buffered writes:
iodepth
1 write: IOPS=428k, BW=418MiB/s (438MB/s)(73.5GiB/180000msec);
2 write: IOPS=406k, BW=396MiB/s (416MB/s)(69.7GiB/180002msec);
4 write: IOPS=382k, BW=373MiB/s (391MB/s)(65.6GiB/180000msec);
8 write: IOPS=255k, BW=249MiB/s (261MB/s)(43.7GiB/180001msec);
16 write: IOPS=399k, BW=390MiB/s (409MB/s)(68.5GiB/180000msec);
32 write: IOPS=433k, BW=423MiB/s (443MB/s)(74.3GiB/180000msec); 1 lat (nsec): min=547, max=2929.3k, avg=1074.98, stdev=6498.72
2 lat (nsec): min=607, max=84320k, avg=3619.15, stdev=109104.36
4 lat (nsec): min=891, max=195941k, avg=9062.16, stdev=213600.71
8 lat (nsec): min=684, max=204164k, avg=29308.56, stdev=542490.72
16 lat (nsec): min=1002, max=77279k, avg=38716.65, stdev=461785.55
32 lat (nsec): min=674, max=75279k, avg=72673.91, stdev=588002.49 uringlet:
iodepth
1 write: IOPS=120k, BW=117MiB/s (123MB/s)(20.6GiB/180006msec);
2 write: IOPS=273k, BW=266MiB/s (279MB/s)(46.8GiB/180010msec);
4 write: IOPS=336k, BW=328MiB/s (344MB/s)(57.7GiB/180002msec);
8 write: IOPS=373k, BW=365MiB/s (382MB/s)(64.1GiB/180000msec);
16 write: IOPS=442k, BW=432MiB/s (453MB/s)(75.9GiB/180001msec);
32 write: IOPS=444k, BW=434MiB/s (455MB/s)(76.2GiB/180010msec); 1 lat (nsec): min=684, max=10790k, avg=6781.23, stdev=10000.69
2 lat (nsec): min=650, max=91712k, avg=5690.52, stdev=136818.11
4 lat (nsec): min=785, max=79038k, avg=10297.04, stdev=227375.52
8 lat (nsec): min=862, max=97493k, avg=19804.67, stdev=350809.60
16 lat (nsec): min=823, max=81279k, avg=34681.33, stdev=478427.17
32 lat (usec): min=6, max=105935, avg=70.55, stdev=696.08uringlet behaves worse on IOPS and lantency in small iodepth. I think the reason is there are more sleep and wakeup.(not sure about it, I'll look into it later)
The downside of uringlet:
- it costs more cpu resource, the reason is similar with the sqpoll case: a uringlet worker keeps checking sqring to reduce latency.[2]
- task->plug is disabled for now since uringlet is buggy with it.
[2] For now, I allow a uringlet worker spin on the empty sqring for some times.
Any comments are welcome, This early RFC only supports buffered write for now and if the idea under it is proved to be the right way, I'll change it to a formal patchset and resolve the detail technical issues and try to support more io_uring features.
Regards,
Hao
Hao Xu (19):
io_uring: change return value of create_io_worker() and
io_wqe_create_worker()
io_uring: add IORING_SETUP_URINGLET
io_uring: make worker pool per ctx for uringlet mode
io-wq: split io_wqe_worker() to io_wqe_worker_normal() and
io_wqe_worker_let()
io_uring: add io_uringler_offload() for uringlet mode
io-wq: change the io-worker scheduling logic
io-wq: move worker state flags to io-wq.h
io-wq: add IO_WORKER_F_SUBMIT and its friends
io-wq: add IO_WORKER_F_SCHEDULED and its friends
io_uring: add io_submit_sqes_let()
io_uring: don't allocate io-wq for a worker in uringlet mode
io_uring: add uringlet worker cancellation function
io-wq: add wq->owner for uringlet mode
io_uring: modify issue_flags for uringlet mode
io_uring: don't use inline completion cache if scheduled
io_uring: release ctx->let when a ring exits
io_uring: disable task plug for now
io-wq: only do io_uringlet_end() at the first schedule time
io_uring: wire up uringlet
include/linux/io_uring_types.h | 1 + include/uapi/linux/io_uring.h | 4 + io_uring/io-wq.c | 242 +++++++++++++++++++++++++-------- io_uring/io-wq.h | 65 ++++++++- io_uring/io_uring.c | 122 +++++++++++++++-- io_uring/io_uring.h | 5 +- io_uring/tctx.c | 31 +++-- 7 files changed, 393 insertions(+), 77 deletions(-)
base-commit: 3f743e9bbb8fe20f4c477e4bf6341c4187a4a264
Signature · 1 line
2.25.1
PATCH 1/19 io_uring: change return value of create_io_worker() and io_wqe_create_worker()
From: Hao Xu <[email protected]>
Change return value of create_io_worker() and io_wqe_create_worker() so that it tells the detail error code.
--- io_uring/io-wq.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
io_uring/io-wq.c +7 -7
@@ -140,7 +140,7 @@ struct io_cb_cancel_data {bool cancel_all; }; -static bool create_io_worker (struct io_wq *wq,struct io_wqe *wqe,int index); +static int create_io_worker (struct io_wq *wq,struct io_wqe *wqe,int index);static void io_wqe_dec_running (struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wqe *wqe, struct io_wqe_acct *acct,
@@ -289,7 +289,7 @@ static bool io_wqe_activate_free_worker(struct io_wqe *wqe,* We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wqe_create_worker (struct io_wqe *wqe,struct io_wqe_acct *acct) +static int io_wqe_create_worker (struct io_wqe *wqe,struct io_wqe_acct *acct) {/* * Most likely an attempt to queue unbounded work on an io_wq that
@@ -301,7 +301,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)raw_spin_lock (&wqe->lock);if (acct->nr_workers >= acct->max_workers) {raw_spin_unlock (&wqe->lock); -return true; +return 0 ; } acct->nr_workers++;raw_spin_unlock (&wqe->lock);
@@ -790,7 +790,7 @@ static void io_workqueue_create(struct work_struct *work)kfree (worker); } -static bool create_io_worker (struct io_wq *wq,struct io_wqe *wqe,int index) +static int create_io_worker (struct io_wq *wq,struct io_wqe *wqe,int index) {struct io_wqe_acct *acct = &wqe->acct[index];struct io_worker *worker;
@@ -806,7 +806,7 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index) acct->nr_workers--;raw_spin_unlock (&wqe->lock);io_worker_ref_put (wq); -return false; +return -ENOMEM ; }refcount_set (&worker->ref,1 );
@@ -828,7 +828,7 @@ static bool create_io_worker(struct io_wq *wq, struct io_wqe *wqe, int index)schedule_work (&worker->work); } -return true; +return 0 ; }/*
@@ -933,7 +933,7 @@ static void io_wqe_enqueue(struct io_wqe *wqe, struct io_wq_work *work) !atomic_read(&acct->nr_running))) {bool did_create; - did_create =io_wqe_create_worker (wqe, acct); + did_create = !io_wqe_create_worker (wqe, acct);if (likely (did_create))return ;
Signature · 1 line
2.25.1
PATCH 2/19 io_uring: add IORING_SETUP_URINGLET
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new setup flag to turn on/off uringlet mode.
--- include/uapi/linux/io_uring.h | 4 ++++ io_uring/io_uring.c | 4 +++- 2 files changed, 7 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +4 -0
@@ -152,6 +152,10 @@ enum {* Only one task is allowed to submit requests */ #define IORING_SETUP_SINGLE_ISSUER (1U << 12) +/* +* uringlet mode +*/ +#define IORING_SETUP_URINGLET (1U << 13)enum io_uring_op {IORING_OP_NOP ,
io_uring/io_uring.c +3 -1
@@ -3226,6 +3226,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,struct file *file;int ret; +if (p->flags &IORING_SETUP_URINGLET ) +return -EINVAL ;if (!entries)return -EINVAL ;if (entries >IORING_MAX_ENTRIES ) {
@@ -3400,7 +3402,7 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params)IORING_SETUP_R_DISABLED |IORING_SETUP_SUBMIT_ALL |IORING_SETUP_COOP_TASKRUN |IORING_SETUP_TASKRUN_FLAG |IORING_SETUP_SQE128 |IORING_SETUP_CQE32 | - IORING_SETUP_SINGLE_ISSUER)) + IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_URINGLET)) return -EINVAL;return io_uring_create (entries, &p, params);
Signature · 1 line
2.25.1
PATCH 3/19 io_uring: make worker pool per ctx for uringlet mode
In reply to Hao XuFrom: Hao Xu <[email protected]>
For uringlet mode, make worker pool per ctx. This is much easier for implementation. We can make it better later if it's necessary. In uringlet mode, we need to find the specific ctx in a worker. Add a member private for this. We set wq->task to NULL for uringlet as a mark that this is a uringler io-wq.
--- include/linux/io_uring_types.h | 1 + io_uring/io-wq.c | 11 ++++++++++- io_uring/io-wq.h | 4 ++++ io_uring/io_uring.c | 9 +++++++++ io_uring/tctx.c | 8 +++++++- 5 files changed, 31 insertions(+), 2 deletions(-)
include/linux/io_uring_types.h +1 -0
@@ -356,6 +356,7 @@ struct io_ring_ctx {unsigned sq_thread_idle;/* protected by ->completion_lock */ unsigned evfd_last_cq_tail; +struct io_wq *let; };enum {
io_uring/io-wq.c +10 -1
@@ -127,6 +127,8 @@ struct io_wq {struct task_struct *task; +void *private; +struct io_wqe *wqes[]; };
@@ -392,6 +394,11 @@ static bool io_queue_worker_create(struct io_worker *worker,return false; } +static inline bool io_wq_is_uringlet (struct io_wq *wq) +{ +return wq->private; +} +static void io_wqe_dec_running (struct io_worker *worker) {struct io_wqe_acct *acct =io_wqe_get_acct (worker);
@@ -1153,6 +1160,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) wq->hash = data->hash; wq->free_work = data->free_work; wq->do_work = data->do_work; + wq->private = data->private; ret = -ENOMEM ;for_each_node (node) {
@@ -1188,7 +1196,8 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)INIT_LIST_HEAD (&wqe->all_list); } - wq->task =get_task_struct (data->task); +if (data->task) + wq->task =get_task_struct (data->task);atomic_set (&wq->worker_refs,1 );init_completion (&wq->worker_done);return wq;
io_uring/io-wq.h +4 -0
@@ -41,6 +41,7 @@ struct io_wq_data {struct task_struct *task;io_wq_work_fn *do_work;free_work_fn *free_work; +void *private; };struct io_wq *io_wq_create (unsigned bounded,struct io_wq_data *data);
@@ -80,4 +81,7 @@ static inline bool io_wq_current_is_worker(void)return in_task () && (current->flags &PF_IO_WORKER ) && current->worker_private; } + +extern struct io_wq *io_init_wq_offload (struct io_ring_ctx *ctx, +struct task_struct *task);#endif
io_uring/io_uring.c +9 -0
@@ -3318,6 +3318,15 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, ret =io_sq_offload_create (ctx, p);if (ret) goto err; + +if (ctx->flags &IORING_SETUP_URINGLET ) { + ctx->let =io_init_wq_offload (ctx, current); +if (IS_ERR (ctx->let)) { + ret =PTR_ERR (ctx->let); + goto err; + } + } +/* always set a rsrc node */ ret =io_rsrc_node_switch_start (ctx);if (ret)
io_uring/tctx.c +7 -1
@@ -12,7 +12,7 @@#include "io_uring.h" #include "tctx.h" -static struct io_wq *io_init_wq_offload (struct io_ring_ctx *ctx, +struct io_wq *io_init_wq_offload (struct io_ring_ctx *ctx,struct task_struct *task) {struct io_wq_hash *hash;
@@ -34,9 +34,15 @@ static struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx,mutex_unlock (&ctx->uring_lock); data.hash = hash; +/* for uringlet, wq->task is the iouring instance creator */ data.task = task; data.free_work = io_wq_free_work; data.do_work = io_wq_submit_work; +/* distinguish normal iowq and uringlet by wq->private for now */ +if (ctx->flags &IORING_SETUP_URINGLET ) + data.private = ctx; +else + data.private =NULL ;/* Do QD, or 4 * CPUS, whatever is smallest */ concurrency =min (ctx->sq_entries,4 *num_online_cpus ());
Signature · 1 line
2.25.1
PATCH 4/19 io-wq: split io_wqe_worker() to io_wqe_worker_normal() and io_wqe_worker_let()
In reply to Hao XuFrom: Hao Xu <[email protected]>
io_wqe_worker_normal() is the normal io worker, and io_wqe_worker_let() is the handler for uringlet mode.
--- io_uring/io-wq.c | 82 ++++++++++++++++++++++++++++++++++++++++----- io_uring/io-wq.h | 8 ++++- io_uring/io_uring.c | 8 +++-- io_uring/io_uring.h | 2 +- 4 files changed, 87 insertions(+), 13 deletions(-)
io_uring/io-wq.c +74 -8
@@ -20,6 +20,7 @@#include "io-wq.h" #include "slist.h" #include "io_uring.h" +#include "tctx.h" #define WORKER_IDLE_TIMEOUT (5 * HZ)
@@ -617,19 +618,12 @@ static void io_worker_handle_work(struct io_worker *worker) }while (1 ); } -static int io_wqe_worker (void *data) +static void io_wqe_worker_normal (struct io_worker *worker) { -struct io_worker *worker = data;struct io_wqe_acct *acct =io_wqe_get_acct (worker);struct io_wqe *wqe = worker->wqe;struct io_wq *wq = wqe->wq;bool last_timeout = false; -char buf[TASK_COMM_LEN ]; - - worker->flags |= (IO_WORKER_F_UP |IO_WORKER_F_RUNNING ); - -snprintf (buf,sizeof (buf),"iou-wrk-%d" , wq->task->pid); -set_task_comm (current, buf);while (!test_bit (IO_WQ_BIT_EXIT , &wq->state)) {long ret;
@@ -664,6 +658,78 @@ static int io_wqe_worker(void *data)if (test_bit (IO_WQ_BIT_EXIT , &wq->state))io_worker_handle_work (worker); +} + +#define IO_URINGLET_EMPTY_LIMIT 100000 +#define URINGLET_WORKER_IDLE_TIMEOUT 1 + +static void io_wqe_worker_let (struct io_worker *worker) +{ +struct io_wqe *wqe = worker->wqe; +struct io_wq *wq = wqe->wq; + +/* TODO this one breaks encapsulation */ +if (unlikely (io_uring_add_tctx_node (wq->private))) + goto out; + +while (!test_bit (IO_WQ_BIT_EXIT , &wq->state)) { +unsigned int empty_count =0 ; + +__io_worker_busy (wqe, worker); +set_current_state (TASK_INTERRUPTIBLE ); + +do { +enum io_uringlet_state submit_state; + + submit_state = wq->do_work (wq->private); +if (submit_state ==IO_URINGLET_SCHEDULED ) { + empty_count =0 ; +break ; + }else if (submit_state ==IO_URINGLET_EMPTY ) { +if (++empty_count >IO_URINGLET_EMPTY_LIMIT ) +break ; + }else { + empty_count =0 ; + } +cond_resched (); + }while (1 ); + +raw_spin_lock (&wqe->lock); +__io_worker_idle (wqe, worker); +raw_spin_unlock (&wqe->lock); +schedule_timeout (URINGLET_WORKER_IDLE_TIMEOUT ); +if (signal_pending (current)) { +struct ksignal ksig; + +if (!get_signal (&ksig)) +continue ; +break ; + } + } + +__set_current_state (TASK_RUNNING ); +out: + wq->free_work (NULL ); +} + +static int io_wqe_worker (void *data) +{ +struct io_worker *worker = data; +struct io_wqe *wqe = worker->wqe; +struct io_wq *wq = wqe->wq; +bool uringlet =io_wq_is_uringlet (wq); +char buf[TASK_COMM_LEN ]; + + worker->flags |= (IO_WORKER_F_UP |IO_WORKER_F_RUNNING ); + +snprintf (buf,sizeof (buf), uringlet ?"iou-let-%d" :"iou-wrk-%d" , + wq->task->pid); +set_task_comm (current, buf); + +if (uringlet) +io_wqe_worker_let (worker); +else +io_wqe_worker_normal (worker);io_worker_exit (worker);return 0 ;
io_uring/io-wq.h +7 -1
@@ -21,8 +21,14 @@ enum io_wq_cancel { IO_WQ_CANCEL_NOTFOUND, /* work not found */ }; +enum io_uringlet_state { +IO_URINGLET_INLINE , +IO_URINGLET_EMPTY , +IO_URINGLET_SCHEDULED , +}; +typedef struct io_wq_work *(free_work_fn )(struct io_wq_work *); -typedef void (io_wq_work_fn )(struct io_wq_work *); +typedef int (io_wq_work_fn )(struct io_wq_work *);struct io_wq_hash {refcount_t refs;
io_uring/io_uring.c +5 -3
@@ -1612,7 +1612,7 @@ struct io_wq_work *io_wq_free_work(struct io_wq_work *work)return req ? &req->work :NULL ; } -void io_wq_submit_work (struct io_wq_work *work) +int io_wq_submit_work (struct io_wq_work *work) { struct io_kiocb *req = container_of(work, struct io_kiocb, work);const struct io_op_def *def = &io_op_defs[req->opcode];
@@ -1632,7 +1632,7 @@ void io_wq_submit_work(struct io_wq_work *work)if (work->flags &IO_WQ_WORK_CANCEL ) { fail:io_req_task_queue_fail (req, err); -return ; +return 0 ; }if (!io_assign_file (req, issue_flags)) { err = -EBADF ;
@@ -1666,7 +1666,7 @@ void io_wq_submit_work(struct io_wq_work *work) }if (io_arm_poll_handler (req, issue_flags) ==IO_APOLL_OK ) -return ; +return 0 ;/* aborted or ready, in either case retry blocking */ needs_poll = false; issue_flags &= ~IO_URING_F_NONBLOCK ;
@@ -1675,6 +1675,8 @@ void io_wq_submit_work(struct io_wq_work *work)/* avoid locking problems by failing it from a clean context */ if (ret <0 )io_req_task_queue_fail (req, ret); + +return 0 ; } inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd,
io_uring/io_uring.h +1 -1
@@ -69,7 +69,7 @@ void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node);int io_req_prep_async (struct io_kiocb *req);struct io_wq_work *io_wq_free_work (struct io_wq_work *work); -void io_wq_submit_work (struct io_wq_work *work); +int io_wq_submit_work (struct io_wq_work *work);void io_free_req (struct io_kiocb *req);void io_queue_next (struct io_kiocb *req);
Signature · 1 line
2.25.1
PATCH 5/19 io_uring: add io_uringler_offload() for uringlet mode
In reply to Hao XuFrom: Hao Xu <[email protected]>
In uringlet mode, a io_uring_enter call shouldn't do the sqe submission work, but just offload it to io-workers.
--- io_uring/io-wq.c | 18 ++++++++++++++++++ io_uring/io-wq.h | 1 + io_uring/io_uring.c | 21 ++++++++++++++------- 3 files changed, 33 insertions(+), 7 deletions(-)
io_uring/io-wq.c +18 -0
@@ -771,6 +771,24 @@ void io_wq_worker_sleeping(struct task_struct *tsk)io_wqe_dec_running (worker); } +int io_uringlet_offload (struct io_wq *wq) +{ +struct io_wqe *wqe = wq->wqes[numa_node_id ()]; +struct io_wqe_acct *acct =io_get_acct (wqe, true); +bool waken; + +raw_spin_lock (&wqe->lock); +rcu_read_lock (); + waken =io_wqe_activate_free_worker (wqe, acct); +rcu_read_unlock (); +raw_spin_unlock (&wqe->lock); + +if (waken) +return 0 ; + +return io_wqe_create_worker (wqe, acct); +} +static void io_init_new_worker (struct io_wqe *wqe,struct io_worker *worker,struct task_struct *tsk) {
io_uring/io-wq.h +1 -0
@@ -90,4 +90,5 @@ static inline bool io_wq_current_is_worker(void)extern struct io_wq *io_init_wq_offload (struct io_ring_ctx *ctx,struct task_struct *task); +extern int io_uringlet_offload (struct io_wq *wq);#endif
io_uring/io_uring.c +14 -7
@@ -3051,15 +3051,22 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,if (unlikely (ret)) goto out; -mutex_lock (&ctx->uring_lock); - ret =io_submit_sqes (ctx, to_submit); -if (ret != to_submit) { +if (!(ctx->flags &IORING_SETUP_URINGLET )) { +mutex_lock (&ctx->uring_lock); + ret =io_submit_sqes (ctx, to_submit); +if (ret != to_submit) { +mutex_unlock (&ctx->uring_lock); + goto out; + } +if ((flags &IORING_ENTER_GETEVENTS ) && ctx->syscall_iopoll) + goto iopoll_locked;mutex_unlock (&ctx->uring_lock); - goto out; + }else { + ret =io_uringlet_offload (ctx->let); +if (ret) + goto out; + ret = to_submit; } -if ((flags &IORING_ENTER_GETEVENTS ) && ctx->syscall_iopoll) - goto iopoll_locked; -mutex_unlock (&ctx->uring_lock); }if (flags &IORING_ENTER_GETEVENTS ) {int ret2;
Signature · 1 line
2.25.1
PATCH 6/19 io-wq: change the io-worker scheduling logic
In reply to Hao XuFrom: Hao Xu <[email protected]>
We do io-worker creation when a io-worker gets sleeping and some condition is met. For uringlet mode, we need to do the scheduling too. A uringlet worker gets sleeping because of blocking in some place below io_uring layer in the kernel stack. So we should wake up or create a new uringlet worker in this situation. Meanwhile, setting up a flag to let the sqe submitter know it had been scheduled out.
--- io_uring/io-wq.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
io_uring/io-wq.c +18 -4
@@ -404,14 +404,28 @@ static void io_wqe_dec_running(struct io_worker *worker) {struct io_wqe_acct *acct =io_wqe_get_acct (worker);struct io_wqe *wqe = worker->wqe; +struct io_wq *wq = wqe->wq; +bool zero_refs;if (!(worker->flags &IO_WORKER_F_UP ))return ; -if (!atomic_dec_and_test (&acct->nr_running)) -return ; -if (!io_acct_run_queue (acct)) -return ; + zero_refs =atomic_dec_and_test (&acct->nr_running); + +if (io_wq_is_uringlet (wq)) { +bool activated; + +raw_spin_lock (&wqe->lock); +rcu_read_lock (); + activated =io_wqe_activate_free_worker (wqe, acct); +rcu_read_unlock (); +raw_spin_unlock (&wqe->lock); +if (activated) +return ; + }else { +if (!zero_refs || !io_acct_run_queue (acct)) +return ; + }atomic_inc (&acct->nr_running);atomic_inc (&wqe->wq->worker_refs);
Signature · 1 line
2.25.1
PATCH 7/19 io-wq: move worker state flags to io-wq.h
In reply to Hao XuFrom: Hao Xu <[email protected]>
Move worker state flags to io-wq.h so that we can levarage them later.
--- io_uring/io-wq.c | 7 ------- io_uring/io-wq.h | 8 ++++++++ 2 files changed, 8 insertions(+), 7 deletions(-)
io_uring/io-wq.c +0 -7
@@ -24,13 +24,6 @@#define WORKER_IDLE_TIMEOUT (5 * HZ) -enum { -IO_WORKER_F_UP =1 ,/* up and active */ -IO_WORKER_F_RUNNING =2 ,/* account as running */ -IO_WORKER_F_FREE =4 ,/* worker on free list */ -IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ -}; -enum {IO_WQ_BIT_EXIT =0 ,/* wq exiting */ };
io_uring/io-wq.h +8 -0
@@ -27,6 +27,14 @@ enum io_uringlet_state { IO_URINGLET_SCHEDULED, }; +enum { +IO_WORKER_F_UP =1 ,/* up and active */ +IO_WORKER_F_RUNNING =2 ,/* account as running */ +IO_WORKER_F_FREE =4 ,/* worker on free list */ +IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ +IO_WORKER_F_SCHEDULED =16 ,/* worker had been scheduled out before */ +}; +typedef struct io_wq_work *(free_work_fn )(struct io_wq_work *);typedef int (io_wq_work_fn )(struct io_wq_work *);
Signature · 1 line
2.25.1
PATCH 8/19 io-wq: add IO_WORKER_F_SUBMIT and its friends
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add IO_WORKER_F_SUBMIT to indicate that a uringlet worker is submitting sqes and thus we should do some scheduling when it blocks.
--- io_uring/io-wq.c | 20 ++++++++++++++++++++ io_uring/io-wq.h | 1 + 2 files changed, 21 insertions(+)
io_uring/io-wq.c +20 -0
@@ -393,6 +393,21 @@ static inline bool io_wq_is_uringlet(struct io_wq *wq)return wq->private; } +static inline void io_worker_set_submit (struct io_worker *worker) +{ + worker->flags |=IO_WORKER_F_SUBMIT ; +} + +static inline void io_worker_clean_submit (struct io_worker *worker) +{ + worker->flags &= ~IO_WORKER_F_SUBMIT ; +} + +static inline bool io_worker_test_submit (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_SUBMIT ; +} +static void io_wqe_dec_running (struct io_worker *worker) {struct io_wqe_acct *acct =io_wqe_get_acct (worker);
@@ -408,6 +423,9 @@ static void io_wqe_dec_running(struct io_worker *worker)if (io_wq_is_uringlet (wq)) {bool activated; +if (!io_worker_test_submit (worker)) +return ; +raw_spin_lock (&wqe->lock);rcu_read_lock (); activated =io_wqe_activate_free_worker (wqe, acct);
@@ -688,7 +706,9 @@ static void io_wqe_worker_let(struct io_worker *worker) do { enum io_uringlet_state submit_state; + io_worker_set_submit(worker); submit_state = wq->do_work(wq->private); + io_worker_clean_submit(worker); if (submit_state == IO_URINGLET_SCHEDULED) { empty_count = 0; break;
io_uring/io-wq.h +1 -0
@@ -33,6 +33,7 @@ enum {IO_WORKER_F_FREE =4 ,/* worker on free list */ IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ IO_WORKER_F_SCHEDULED =16 ,/* worker had been scheduled out before */ + IO_WORKER_F_SUBMIT = 32, /* uringlet worker is submitting sqes */ };typedef struct io_wq_work *(free_work_fn )(struct io_wq_work *);
Signature · 1 line
2.25.1
PATCH 9/19 io-wq: add IO_WORKER_F_SCHEDULED and its friends
In reply to Hao XuFrom: Hao Xu <[email protected]>
--- io_uring/io-wq.c | 29 ++--------------------------- io_uring/io-wq.h | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 27 deletions(-)
io_uring/io-wq.c +2 -27
@@ -32,33 +32,6 @@ enum { IO_ACCT_STALLED_BIT = 0, /* stalled on hash */ }; -/* -* One for each thread in a wqe pool -*/ -struct io_worker { -refcount_t ref; -unsigned flags; -struct hlist_nulls_node nulls_node; -struct list_head all_list; -struct task_struct *task; -struct io_wqe *wqe; - -struct io_wq_work *cur_work; -struct io_wq_work *next_work; -raw_spinlock_t lock; - -struct completion ref_done; - -unsigned long create_state; -struct callback_head create_work; -int create_index; - -union { -struct rcu_head rcu; -struct work_struct work; - }; -}; -#if BITS_PER_LONG ==64 #define IO_WQ_HASH_ORDER 6#else
@@ -426,6 +399,7 @@ static void io_wqe_dec_running(struct io_worker *worker)if (!io_worker_test_submit (worker))return ; +io_worker_set_scheduled (worker);raw_spin_lock (&wqe->lock);rcu_read_lock (); activated =io_wqe_activate_free_worker (wqe, acct);
@@ -706,6 +680,7 @@ static void io_wqe_worker_let(struct io_worker *worker) do { enum io_uringlet_state submit_state; + io_worker_clean_scheduled(worker); io_worker_set_submit(worker); submit_state = wq->do_work(wq->private); io_worker_clean_submit(worker);
io_uring/io-wq.h +43 -0
58 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH 10/19 io_uring: add io_submit_sqes_let()
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add io_submit_sqes_let() for submitting sqes in uringlet mode, and update logic in schedule time and the io-wq init time.
--- io_uring/io-wq.c | 1 + io_uring/io_uring.c | 55 +++++++++++++++++++++++++++++++++++++++++++++ io_uring/io_uring.h | 2 ++ io_uring/tctx.c | 8 ++++--- 4 files changed, 63 insertions(+), 3 deletions(-)
io_uring/io-wq.c +1 -0
@@ -399,6 +399,7 @@ static void io_wqe_dec_running(struct io_worker *worker)if (!io_worker_test_submit (worker))return ; +io_uringlet_end (wq->private);io_worker_set_scheduled (worker);raw_spin_lock (&wqe->lock);rcu_read_lock ();
io_uring/io_uring.c +55 -0
67 lines not on this page · raw message
io_uring/io_uring.h +2 -0
@@ -64,9 +64,11 @@ int io_uring_alloc_task_context(struct task_struct *task,int io_poll_issue (struct io_kiocb *req,bool *locked);int io_submit_sqes (struct io_ring_ctx *ctx,unsigned int nr); +int io_submit_sqes_let (struct io_wq_work *work);int io_do_iopoll (struct io_ring_ctx *ctx,bool force_nonspin);void io_free_batch_list (struct io_ring_ctx *ctx,struct io_wq_work_node *node);int io_req_prep_async (struct io_kiocb *req); +void io_uringlet_end (struct io_ring_ctx *ctx);struct io_wq_work *io_wq_free_work (struct io_wq_work *work);int io_wq_submit_work (struct io_wq_work *work);
io_uring/tctx.c +5 -3
@@ -37,12 +37,14 @@ struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx,/* for uringlet, wq->task is the iouring instance creator */ data.task = task; data.free_work = io_wq_free_work; - data.do_work = io_wq_submit_work;/* distinguish normal iowq and uringlet by wq->private for now */ -if (ctx->flags &IORING_SETUP_URINGLET ) +if (ctx->flags &IORING_SETUP_URINGLET ) { data.private = ctx; -else + data.do_work = io_submit_sqes_let; + }else { data.private =NULL ; + data.do_work = io_wq_submit_work; + }/* Do QD, or 4 * CPUS, whatever is smallest */ concurrency =min (ctx->sq_entries,4 *num_online_cpus ());
Signature · 1 line
2.25.1
PATCH 11/19 io_uring: don't allocate io-wq for a worker in uringlet mode
In reply to Hao XuFrom: Hao Xu <[email protected]>
A uringlet worker doesn't need any io-wq pool.
--- io_uring/tctx.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
io_uring/tctx.c +11 -6
@@ -81,12 +81,17 @@ __cold int io_uring_alloc_task_context(struct task_struct *task,return ret; } - tctx->io_wq =io_init_wq_offload (ctx, task); -if (IS_ERR (tctx->io_wq)) { - ret =PTR_ERR (tctx->io_wq); -percpu_counter_destroy (&tctx->inflight); -kfree (tctx); -return ret; +/* +* don't allocate io-wq in uringlet mode +*/ +if (!(ctx->flags &IORING_SETUP_URINGLET )) { + tctx->io_wq =io_init_wq_offload (ctx, task); +if (IS_ERR (tctx->io_wq)) { + ret =PTR_ERR (tctx->io_wq); +percpu_counter_destroy (&tctx->inflight); +kfree (tctx); +return ret; + } }xa_init (&tctx->xa);
Signature · 1 line
2.25.1
PATCH 12/19 io_uring: add uringlet worker cancellation function
In reply to Hao XuFrom: Hao Xu <[email protected]>
uringlet worker submits sqes, so we need to do some cancellation work before it exits.
--- io_uring/io_uring.c | 6 ++++++ io_uring/io_uring.h | 1 + io_uring/tctx.c | 2 ++ 3 files changed, 9 insertions(+)
io_uring/io_uring.c +6 -0
@@ -2922,6 +2922,12 @@ void __io_uring_cancel(bool cancel_all)io_uring_cancel_generic (cancel_all,NULL ); } +struct io_wq_work *io_uringlet_cancel (struct io_wq_work *work) +{ +__io_uring_cancel (true); +return NULL ; +} +static void *io_uring_validate_mmap_request (struct file *file,loff_t pgoff,size_t sz) {
io_uring/io_uring.h +1 -0
@@ -71,6 +71,7 @@ int io_req_prep_async(struct io_kiocb *req);void io_uringlet_end (struct io_ring_ctx *ctx);struct io_wq_work *io_wq_free_work (struct io_wq_work *work); +struct io_wq_work *io_uringlet_cancel (struct io_wq_work *work);int io_wq_submit_work (struct io_wq_work *work);void io_free_req (struct io_kiocb *req);
io_uring/tctx.c +2 -0
@@ -41,9 +41,11 @@ struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx,if (ctx->flags &IORING_SETUP_URINGLET ) { data.private = ctx; data.do_work = io_submit_sqes_let; + data.free_work = io_uringlet_cancel; }else { data.private =NULL ; data.do_work = io_wq_submit_work; + data.free_work = io_wq_free_work; }/* Do QD, or 4 * CPUS, whatever is smallest */
Signature · 1 line
2.25.1
PATCH 13/19 io-wq: add wq->owner for uringlet mode
In reply to Hao XuFrom: Hao Xu <[email protected]>
In uringlet mode, we allow exact one worker to submit sqes at the same time. nr_running is not a good choice to aim that. Add an member wq->owner and its lock to achieve that, this avoids race condition between workers.
--- io_uring/io-wq.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
io_uring/io-wq.c +30 -0
@@ -96,6 +96,9 @@ struct io_wq {void *private; +raw_spinlock_t lock; +struct io_worker *owner; +struct io_wqe *wqes[]; };
@@ -381,6 +384,8 @@ static inline bool io_worker_test_submit(struct io_worker *worker)return worker->flags &IO_WORKER_F_SUBMIT ; } +#define IO_WQ_OWNER_TRANSMIT ((struct io_worker *)-1) +static void io_wqe_dec_running (struct io_worker *worker) {struct io_wqe_acct *acct =io_wqe_get_acct (worker);
@@ -401,6 +406,10 @@ static void io_wqe_dec_running(struct io_worker *worker)io_uringlet_end (wq->private);io_worker_set_scheduled (worker); +raw_spin_lock (&wq->lock); + wq->owner =IO_WQ_OWNER_TRANSMIT ; +raw_spin_unlock (&wq->lock); +raw_spin_lock (&wqe->lock);rcu_read_lock (); activated =io_wqe_activate_free_worker (wqe, acct);
@@ -674,6 +683,17 @@ static void io_wqe_worker_let(struct io_worker *worker)while (!test_bit (IO_WQ_BIT_EXIT , &wq->state)) {unsigned int empty_count =0 ; +struct io_worker *owner; + +raw_spin_lock (&wq->lock); + owner = wq->owner; +if (owner && owner !=IO_WQ_OWNER_TRANSMIT && owner != worker) { +raw_spin_unlock (&wq->lock); +set_current_state (TASK_INTERRUPTIBLE ); + goto sleep; + } + wq->owner = worker; +raw_spin_unlock (&wq->lock);__io_worker_busy (wqe, worker);set_current_state (TASK_INTERRUPTIBLE );
@@ -697,6 +717,7 @@ static void io_wqe_worker_let(struct io_worker *worker)cond_resched (); }while (1 ); +sleep:raw_spin_lock (&wqe->lock);__io_worker_idle (wqe, worker);raw_spin_unlock (&wqe->lock);
@@ -780,6 +801,14 @@ int io_uringlet_offload(struct io_wq *wq)struct io_wqe_acct *acct =io_get_acct (wqe, true);bool waken; +raw_spin_lock (&wq->lock); +if (wq->owner) { +raw_spin_unlock (&wq->lock); +return 0 ; + } + wq->owner =IO_WQ_OWNER_TRANSMIT ; +raw_spin_unlock (&wq->lock); +raw_spin_lock (&wqe->lock);rcu_read_lock (); waken =io_wqe_activate_free_worker (wqe, acct);
@@ -1248,6 +1277,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) wq->free_work = data->free_work; wq->do_work = data->do_work; wq->private = data->private; +raw_spin_lock_init (&wq->lock); ret = -ENOMEM ;for_each_node (node) {
Signature · 1 line
2.25.1
PATCH 14/19 io_uring: modify issue_flags for uringlet mode
In reply to Hao XuFrom: Hao Xu <[email protected]>
We don't need IO_URING_F_NONBLOCK in uringlet mode.
--- io_uring/io_uring.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
io_uring/io_uring.c +5 -1
@@ -1751,8 +1751,12 @@ static inline void io_queue_sqe(struct io_kiocb *req) __must_hold(&req->ctx->uring_lock) {int ret; +unsigned int issue_flags =IO_URING_F_COMPLETE_DEFER ; - ret =io_issue_sqe (req,IO_URING_F_NONBLOCK |IO_URING_F_COMPLETE_DEFER ); +if (!(req->ctx->flags &IORING_SETUP_URINGLET )) + issue_flags |=IO_URING_F_NONBLOCK ; + + ret =io_issue_sqe (req, issue_flags);/* * We async punt it if the file wasn't marked NOWAIT, or if the file
Signature · 1 line
2.25.1
PATCH 15/19 io_uring: don't use inline completion cache if scheduled
In reply to Hao XuFrom: Hao Xu <[email protected]>
In uringlet mode, if a worker has been scheduled out during sqe submission, we cannot use inline completion for that sqe.
--- io_uring/io_uring.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
io_uring/io_uring.c +8 -1
@@ -1582,7 +1582,14 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)revert_creds (creds);if (ret ==IOU_OK ) { -if (issue_flags &IO_URING_F_COMPLETE_DEFER ) +bool uringlet = req->ctx->flags &IORING_SETUP_URINGLET ; +bool scheduled = false; + +if (uringlet) + scheduled = +io_worker_test_scheduled (current->worker_private); + +if ((issue_flags &IO_URING_F_COMPLETE_DEFER ) && !scheduled)io_req_complete_defer (req);else io_req_complete_post (req);
Signature · 1 line
2.25.1
PATCH 16/19 io_uring: release ctx->let when a ring exits
In reply to Hao XuFrom: Hao Xu <[email protected]>
Release the uringlet worker pool when a ring exits, and reclaim the resource.
--- io_uring/io_uring.c | 5 +++++ 1 file changed, 5 insertions(+)
io_uring/io_uring.c +5 -0
@@ -2699,6 +2699,11 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)unsigned long index;struct creds *creds; +if (ctx->flags &IORING_SETUP_URINGLET ) { +io_wq_exit_start (ctx->let); +io_wq_put_and_exit (ctx->let); + } +mutex_lock (&ctx->uring_lock);percpu_ref_kill (&ctx->refs);if (ctx->rings)
Signature · 1 line
2.25.1
PATCH 17/19 io_uring: disable task plug for now
In reply to Hao XuFrom: Hao Xu <[email protected]>
This is a temporary commit, the task plug causes hung and the reason is unclear for now. So disable it in uringlet mode for now.
--- io_uring/io_uring.c | 1 + 1 file changed, 1 insertion(+)
io_uring/io_uring.c +1 -0
@@ -2171,6 +2171,7 @@ int io_submit_sqes_let(struct io_wq_work *work)io_get_task_refs (entries);io_submit_state_start (&ctx->submit_state, entries); + ctx->submit_state->need_plug = false; do { const struct io_uring_sqe *sqe; struct io_kiocb *req;
Signature · 1 line
2.25.1
PATCH 18/19 io-wq: only do io_uringlet_end() at the first schedule time
In reply to Hao XuFrom: Hao Xu <[email protected]>
A request may block multiple times during its life cycle. We should only do io_uringlet_end() at the first time since this function may modify ctx->submit_state info and for the non-first time, the task already lost the control of submitting sqes. Allowing it to do so will damage the submission state.
--- io_uring/io-wq.c | 14 +++++++++++--- io_uring/io_uring.c | 2 +- 2 files changed, 12 insertions(+), 4 deletions(-)
io_uring/io-wq.c +11 -3
@@ -400,16 +400,24 @@ static void io_wqe_dec_running(struct io_worker *worker)if (io_wq_is_uringlet (wq)) {bool activated; +bool first_block;if (!io_worker_test_submit (worker))return ; -io_uringlet_end (wq->private); -io_worker_set_scheduled (worker);raw_spin_lock (&wq->lock); - wq->owner =IO_WQ_OWNER_TRANSMIT ; + first_block = (wq->owner == worker ? true : false);raw_spin_unlock (&wq->lock); +io_worker_set_scheduled (worker); + +if (first_block) { +io_uringlet_end (wq->private); +raw_spin_lock (&wq->lock); + wq->owner =IO_WQ_OWNER_TRANSMIT ; +raw_spin_unlock (&wq->lock); + } +raw_spin_lock (&wqe->lock);rcu_read_lock (); activated =io_wqe_activate_free_worker (wqe, acct);
io_uring/io_uring.c +1 -1
@@ -2171,7 +2171,7 @@ int io_submit_sqes_let(struct io_wq_work *work)io_get_task_refs (entries);io_submit_state_start (&ctx->submit_state, entries); - ctx->submit_state->need_plug = false; + ctx->submit_state.need_plug = false; do { const struct io_uring_sqe *sqe; struct io_kiocb *req;
Signature · 1 line
2.25.1
PATCH 19/19 io_uring: wire up uringlet
In reply to Hao XuFrom: Hao Xu <[email protected]>
Enable the uringlet mode.
--- io_uring/io_uring.c | 2 -- 1 file changed, 2 deletions(-)
io_uring/io_uring.c +0 -2
@@ -3313,8 +3313,6 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,struct file *file;int ret; -if (p->flags &IORING_SETUP_URINGLET ) -return -EINVAL ;if (!entries)return -EINVAL ;if (entries >IORING_MAX_ENTRIES ) {
Signature · 1 line
2.25.1
Quoted 101 lines · Hao Xu
From: Hao Xu <[email protected]>
Hi Jens and all,
This is an early RFC for a new way to do async IO. Currently
io_uringworks in a way like:
- issue an IO request in nowait way
here nowait means return error(EAGAIN) toio_uringlayer when it would block in deeper kernel stack.- issue an IO request in a normal(block) way
io_uringcatches the EAGAIN error and create/wakeup a io-worker to redo the IO request in a block way. The original context turns to issue other requests. (some type of requests like buffered reads, leverage task work to wipe out io-workers)This has two main disadvantages:
- we have to find every block point along the kernel code path and modify it to support nowait.
e.g.alloc_memory()----> if (alloc_memory()fails) return-EAGAINThis hugely adds programming complexisity, especially when the code path is long and complicated. For example, buffered write, we have to handle locks, possibly journal part, meta data like extent node misses.- By create/wakeup a new worker, we redo a IO request from the very beginning, which means we re-walk the path from beginning to the previous block point.
The original context backtracks to theio_uringlayer from the block point to submit other requests. While it's better to directly start the new submission.This RFC provides a new way to do it.
- We maintain a worker pool for eachio_uringinstance and each worker in it can submit requests. The original task only needs to create the first worker and return to userspace. Later it doesn't need to call io_uring_enter.[1]- the created worker begins to submit requests. When it blocks, just let it be blocked. Create/wakeup another worker to do the submission
[1] I currently keep these workers until the
io_uringcontext exits. In other words, a worker does submission, sleep, wake up, but won't exit. Thus the original task don't need to create/wakeup workers.I've done some testing:
name: buffered write
fs: xfs
env: qemu box, 4 cpu, 8G mem.
tool: fio- single file test:
fio ioengine=io_uring, size=10M, bs=1024, direct=0, thread=1, rw=randwrite, time_based=1, runtime=180 async buffered writes: iodepth 1 write: IOPS=428k, BW=418MiB/s (438MB/s)(73.5GiB/180000msec); 2 write: IOPS=406k, BW=396MiB/s (416MB/s)(69.7GiB/180002msec); 4 write: IOPS=382k, BW=373MiB/s (391MB/s)(65.6GiB/180000msec); 8 write: IOPS=255k, BW=249MiB/s (261MB/s)(43.7GiB/180001msec); 16 write: IOPS=399k, BW=390MiB/s (409MB/s)(68.5GiB/180000msec); 32 write: IOPS=433k, BW=423MiB/s (443MB/s)(74.3GiB/180000msec); 1 lat (nsec): min=547, max=2929.3k, avg=1074.98, stdev=6498.72 2 lat (nsec): min=607, max=84320k, avg=3619.15, stdev=109104.36 4 lat (nsec): min=891, max=195941k, avg=9062.16, stdev=213600.71 8 lat (nsec): min=684, max=204164k, avg=29308.56, stdev=542490.72 16 lat (nsec): min=1002, max=77279k, avg=38716.65, stdev=461785.55 32 lat (nsec): min=674, max=75279k, avg=72673.91, stdev=588002.49 uringlet: iodepth 1 write: IOPS=120k, BW=117MiB/s (123MB/s)(20.6GiB/180006msec); 2 write: IOPS=273k, BW=266MiB/s (279MB/s)(46.8GiB/180010msec); 4 write: IOPS=336k, BW=328MiB/s (344MB/s)(57.7GiB/180002msec); 8 write: IOPS=373k, BW=365MiB/s (382MB/s)(64.1GiB/180000msec); 16 write: IOPS=442k, BW=432MiB/s (453MB/s)(75.9GiB/180001msec); 32 write: IOPS=444k, BW=434MiB/s (455MB/s)(76.2GiB/180010msec); 1 lat (nsec): min=684, max=10790k, avg=6781.23, stdev=10000.69 2 lat (nsec): min=650, max=91712k, avg=5690.52, stdev=136818.11 4 lat (nsec): min=785, max=79038k, avg=10297.04, stdev=227375.52 8 lat (nsec): min=862, max=97493k, avg=19804.67, stdev=350809.60 16 lat (nsec): min=823, max=81279k, avg=34681.33, stdev=478427.17 32 lat (usec): min=6, max=105935, avg=70.55, stdev=696.08uringlet behaves worse on IOPS and lantency in small iodepth. I think the reason is there are more sleep and wakeup.(not sure about it, I'll look into it later)
The downside of uringlet:
- it costs more cpu resource, the reason is similar with the sqpoll case: a uringlet worker keeps checking sqring to reduce latency.[2]
-task->plugis disabled for now since uringlet is buggy with it.[2] For now, I allow a uringlet worker spin on the empty sqring for some times.
Quoted 8 lines
Any comments are welcome, This early RFC only supports buffered write for now and if the idea under it is proved to be the right way, I'll change it to a formal patchset and resolve the detail technical issues and try to support more
io_uringfeatures.Regards,
Hao
Friendly ping...
Jens, any thoughts on this one?