fixed worker
cover fixed worker
From: Hao Xu <[email protected]>
The initial feature request by users is here:
https://github.com/axboe/liburing/issues/296
Fixed worker provide a way for users to control the io-wq threads. A fixed worker is worker thread which exists no matter there are works to do or not. We provide a new register api to register fixed workers, and a register api to unregister them as well. The parameter of the register api is the number of fixed workers users want.
For example:
```c
io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5 })
do I/O works
io_uring_unregister_iowq_fixed_workers(&ring)
```After registration, there will be 5 fixed workers. User can setup their affinity, priority etc. freely, without adding any new register api to set up attributions. These workers won't be destroyed until users call unregister api.
Note, registering some fixed workers doesn't mean no creating normal workers. When there is no free workers, new normal workers can be created when works come. So a work may be picked up by fixed workers or normal workers.
If users want to offload works only to fixed workers, they can specify a flag FIXED_ONLY when registering fixed workers.
```c
io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5, .flags |=
FIXED_ONLY })
```In above case, no normal workers will be created before calling io_uring_register_iowq_fixed_workers().
Note:
- When registering fixed workers, those fixed workers are per io-wq. So if an io_uring instance is shared by multiple tasks, and you want all tasks to use fixed workers, all tasks have to call the regitser api.
- if specifying FIXED_ONLY when registering fixed workers, that is per io_uring instance. all works in this instance are handled by fixed workers.
Therefore, if an io_uring instance is shared by two tasks, and you want all requests in this instance to be handled only by fixed workers, you have to call the register api in these two tasks and specify FIXED_ONLY at least once when calling register api.
Hao Xu (11):
io-wq: fix worker counting after worker received exit signal
io-wq: add a new worker flag to indicate worker exit
io-wq: add a new type io-wq worker
io-wq: add fixed worker members in io_wq_acct
io-wq: add a new parameter for creating a new fixed worker
io-wq: return io_worker after successful inline worker creation
io_uring: add new api to register fixed workers
io_uring: add function to unregister fixed workers
io-wq: add strutures to allow to wait fixed workers exit
io-wq: distinguish fixed worker by its name
io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend
include/uapi/linux/io_uring.h | 20 +++ io_uring/io-wq.c | 275 ++++++++++++++++++++++++++++++---- io_uring/io-wq.h | 3 + io_uring/io_uring.c | 132 +++++++++++++++- 4 files changed, 397 insertions(+), 33 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/11 io-wq: fix worker counting after worker received exit signal
From: Hao Xu <[email protected]>
acct->nr_workers should be decremented when we break the loop in io_wq_worker().
--- io_uring/io-wq.c | 4 ++++ 1 file changed, 4 insertions(+)
io_uring/io-wq.c +4 -0
@@ -634,6 +634,10 @@ static int io_wq_worker(void *data)if (!get_signal (&ksig))continue ; + +raw_spin_lock (&wq->lock); + acct->nr_workers--; +raw_spin_unlock (&wq->lock);break ; }if (!ret) {
Signature · 1 line
2.25.1
Re: [PATCH 01/11] io-wq: fix worker counting after worker received exit signal
Quoted 16 lines · Hao Xu
From: Hao Xu <[email protected]>
acct->nr_workersshould be decremented when we break the loop inio_wq_worker().Fixes 78f8876c2d9f ("io-wq: exclusively gate signal based exit on get_signal() return")Signed-off-by Hao Xu <[email protected]>--- io_uring/io-wq.c | 4 ++++ 1 file changed, 4 insertions(+)io_uring/io-wq.c +0
-0
io_uring/io-wq.c static int io_wq_worker(void *data) · line 640
Quoted 7 lines
io_uring/io-wq.c +4
-0@@ -634,6 +634,10 @@ static int io_wq_worker(void *data)if (!get_signal (&ksig))continue ; + +raw_spin_lock (&wq->lock); + acct->nr_workers--; +raw_spin_unlock (&wq->lock);
Wouldn't it suffer the same race you fixed with the following?
commit 767a65e9f31789d80e41edd03a802314905e8fbf
Author: Hao Xu <[email protected]> Date: Sun Sep 12 03:40:52 2021 +0800
io-wq: fix potential race of acct->nr_workers
Even more, seems we fail to decrement nr_workers when the loop condition fails, i.e.
while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) {I.e. the patch looks legit, but what we currently have is a mess and we have more work to do.
Signature · 1 line
Pavel Begunkov
PATCH 2/11 io-wq: add a new worker flag to indicate worker exit
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new worker flag IO_WORKER_F_EXIT to indicate a worker is going to exit. This is important for fixed workers.
--- io_uring/io-wq.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
io_uring/io-wq.c +13 -1
@@ -29,6 +29,7 @@ enum {IO_WORKER_F_RUNNING =2 ,/* account as running */ IO_WORKER_F_FREE =4 ,/* worker on free list */ IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ + IO_WORKER_F_EXIT = 16, /* worker is exiting */ }; enum {
@@ -592,6 +593,11 @@ static void io_worker_handle_work(struct io_worker *worker) }while (1 ); } +static bool is_worker_exiting (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_EXIT ; +} +static int io_wq_worker (void *data) {struct io_worker *worker = data;
@@ -609,7 +615,7 @@ static int io_wq_worker(void *data)long ret;set_current_state (TASK_INTERRUPTIBLE ); -while (io_acct_run_queue (acct)) +while (!is_worker_exiting (worker) &&io_acct_run_queue (acct))io_worker_handle_work (worker);raw_spin_lock (&wq->lock);
@@ -628,6 +634,12 @@ static int io_wq_worker(void *data)raw_spin_unlock (&wq->lock);if (io_run_task_work ())continue ; +if (is_worker_exiting (worker)) { +raw_spin_lock (&wq->lock); + acct->nr_workers--; +raw_spin_unlock (&wq->lock); +break ; + } ret =schedule_timeout (WORKER_IDLE_TIMEOUT );if (signal_pending (current)) {struct ksignal ksig;
Signature · 1 line
2.25.1
Re: [PATCH 02/11] io-wq: add a new worker flag to indicate worker exit
Quoted 4 lines · Hao Xu
From: Hao Xu <[email protected]>
Add a new worker flag
IO_WORKER_F_EXITto indicate a worker is going to exit. This is important for fixed workers.
nit: would be nice to add a small sentence _how_ it's important for fixed workers
Quoted 30 lines
Signed-off-by Hao Xu <[email protected]>--- io_uring/io-wq.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)io_uring/io-wq.c +6
-0@@ -29,6 +29,7 @@ enum {IO_WORKER_F_RUNNING =2 ,/* account as running */ IO_WORKER_F_FREE =4 ,/* worker on free list */ IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ + IO_WORKER_F_EXIT = 16, /* worker is exiting */ }; enum {@@ -592,6 +593,11 @@ static void io_worker_handle_work(struct io_worker *worker) }while (1 ); } +static bool is_worker_exiting (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_EXIT ; +} +static int io_wq_worker (void *data) {struct io_worker *worker = data;
io_uring/io-wq.c static int io_wq_worker(void *data) · line 619
Quoted 6 lines
io_uring/io-wq.c +1
-1@@ -609,7 +615,7 @@ static int io_wq_worker(void *data)long ret;set_current_state (TASK_INTERRUPTIBLE ); -while (io_acct_run_queue (acct)) +while (!is_worker_exiting (worker) &&io_acct_run_queue (acct))io_worker_handle_work (worker);
Why it differs from the condition in io_wq_dec_running()? Would sth like this work?
bool io_worker_run_queue(worker) {
return !is_worker_exiting(worker) &&
io_acct_run_queue(worker_get_acct(worker));
}Quoted 15 lines
io_uring/io-wq.c +6
-0raw_spin_lock (&wq->lock);@@ -628,6 +634,12 @@ static int io_wq_worker(void *data)raw_spin_unlock (&wq->lock);if (io_run_task_work ())continue ; +if (is_worker_exiting (worker)) { +raw_spin_lock (&wq->lock); + acct->nr_workers--; +raw_spin_unlock (&wq->lock); +break ; + } ret =schedule_timeout (WORKER_IDLE_TIMEOUT );if (signal_pending (current)) {struct ksignal ksig;
Signature · 1 line
Pavel Begunkov
PATCH 3/11 io-wq: add a new type io-wq worker
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new type io-wq worker IO_WORKER_F_FIXED, this type of worker exists during the whole io-wq lifecycle.
--- io_uring/io-wq.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
io_uring/io-wq.c +12 -1
@@ -30,6 +30,7 @@ enum {IO_WORKER_F_FREE =4 ,/* worker on free list */ IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ IO_WORKER_F_EXIT =16 ,/* worker is exiting */ + IO_WORKER_F_FIXED = 32, /* is a fixed worker */ }; enum {
@@ -598,6 +599,11 @@ static bool is_worker_exiting(struct io_worker *worker)return worker->flags &IO_WORKER_F_EXIT ; } +static bool is_fixed_worker (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_FIXED ; +} +static int io_wq_worker (void *data) {struct io_worker *worker = data;
@@ -622,8 +628,13 @@ static int io_wq_worker(void *data)/* * Last sleep timed out. Exit if we're not the last worker, * or if someone modified our affinity. +* Note: fixed worker always have same lifecycle as io-wq +* itself, and cpu affinity setting doesn't work well for +* fixed worker, they can be manually reset to cpu other than +* the cpuset indicated by io_wq_worker_affinity() */ -if (last_timeout && (exit_mask || acct->nr_workers >1 )) { +if (!is_fixed_worker (worker) && last_timeout && + (exit_mask || acct->nr_workers >1 )) { acct->nr_workers--;raw_spin_unlock (&wq->lock);__set_current_state (TASK_RUNNING );
Signature · 1 line
2.25.1
Re: [PATCH 03/11] io-wq: add a new type io-wq worker
Quoted 23 lines · Hao Xu
From: Hao Xu <[email protected]>
Add a new type io-wq worker
IO_WORKER_F_FIXED, this type of worker exists during the whole io-wq lifecycle.Signed-off-by Hao Xu <[email protected]>--- io_uring/io-wq.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)io_uring/io-wq.c +1
-0@@ -30,6 +30,7 @@ enum {IO_WORKER_F_FREE =4 ,/* worker on free list */ IO_WORKER_F_BOUND =8 ,/* is doing bounded work */ IO_WORKER_F_EXIT =16 ,/* worker is exiting */ + IO_WORKER_F_FIXED = 32, /* is a fixed worker */ }; enum {
io_uring/io-wq.c static bool is_worker_exiting(struct io_worker *worker) · line 605
Quoted 7 lines
io_uring/io-wq.c +4
-0@@ -598,6 +599,11 @@ static bool is_worker_exiting(struct io_worker *worker)return worker->flags &IO_WORKER_F_EXIT ; } +static bool is_fixed_worker (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_FIXED ; +}
You move it up in Patch 5/11, I suggest to move it to the top of the file here.
Quoted 8 lines
io_uring/io-wq.c +1
-0static int io_wq_worker (void *data) {struct io_worker *worker = data;@@ -622,8 +628,13 @@ static int io_wq_worker(void *data)/* * Last sleep timed out. Exit if we're not the last worker, * or if someone modified our affinity. +* Note: fixed worker always have same lifecycle as io-wq
io_uring/io-wq.c static int io_wq_worker(void *data) · line 640
Quoted 10 lines
io_uring/io-wq.c +5
-1+* itself, and cpu affinity setting doesn't work well for +* fixed worker, they can be manually reset to cpu other than +* the cpuset indicated by io_wq_worker_affinity() */ -if (last_timeout && (exit_mask || acct->nr_workers >1 )) { +if (!is_fixed_worker (worker) && last_timeout && + (exit_mask || acct->nr_workers >1 )) { acct->nr_workers--;raw_spin_unlock (&wq->lock);__set_current_state (TASK_RUNNING );
If there is no work it'll continue to loop every WORKER_IDLE_TIMEOUT (5 * HZ), which sounds troublesome with many workers in the system.
tm = is_fixed_worker(worker) ? MAX_SCHEDULE_TIMEOUT : WORKER_IDLE_TIMEOUT; schedule_timeout(tm);
Maybe?
Signature · 1 line
Pavel Begunkov
PATCH 4/11 io-wq: add fixed worker members in io_wq_acct
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add fixed worker related members in io_wq_acct.
--- io_uring/io-wq.c | 2 ++ 1 file changed, 2 insertions(+)
io_uring/io-wq.c +2 -0
@@ -84,6 +84,8 @@ struct io_wq_acct {raw_spinlock_t lock;struct io_wq_work_list work_list;unsigned long flags; +struct io_worker **fixed_workers; +unsigned int fixed_nr; };enum {
Signature · 1 line
2.25.1
PATCH 5/11 io-wq: add a new parameter for creating a new fixed worker
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new parameter when creating new workers to indicate if users want a normal or fixed worker.
--- io_uring/io-wq.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
io_uring/io-wq.c +20 -13
@@ -137,7 +137,7 @@ struct io_cb_cancel_data {bool cancel_all; }; -static bool create_io_worker (struct io_wq *wq,int index); +static bool create_io_worker (struct io_wq *wq,int index,bool fixed);static void io_wq_dec_running (struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wq *wq, struct io_wq_acct *acct,
@@ -284,7 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,* We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wq_create_worker (struct io_wq *wq,struct io_wq_acct *acct) +static bool io_wq_create_worker (struct io_wq *wq,struct io_wq_acct *acct, +bool fixed) {/* * Most likely an attempt to queue unbounded work on an io_wq that
@@ -302,7 +303,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct)raw_spin_unlock (&wq->lock);atomic_inc (&acct->nr_running);atomic_inc (&wq->worker_refs); -return create_io_worker (wq, acct->index); +return create_io_worker (wq, acct->index, fixed); }static void io_wq_inc_running (struct io_worker *worker)
@@ -312,6 +313,11 @@ static void io_wq_inc_running(struct io_worker *worker)atomic_inc (&acct->nr_running); } +static bool is_fixed_worker (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_FIXED ; +} +static void create_worker_cb (struct callback_head *cb) {struct io_worker *worker;
@@ -331,7 +337,7 @@ static void create_worker_cb(struct callback_head *cb) }raw_spin_unlock (&wq->lock);if (do_create) { -create_io_worker (wq, worker->create_index); +create_io_worker (wq, worker->create_index,is_fixed_worker (worker)); }else {atomic_dec (&acct->nr_running);io_worker_ref_put (wq);
@@ -398,6 +404,8 @@ static void io_wq_dec_running(struct io_worker *worker)return ;if (!io_acct_run_queue (acct))return ; +if (is_fixed_worker (worker)) +return ;atomic_inc (&acct->nr_running);atomic_inc (&wq->worker_refs);
@@ -601,11 +609,6 @@ static bool is_worker_exiting(struct io_worker *worker)return worker->flags &IO_WORKER_F_EXIT ; } -static bool is_fixed_worker (struct io_worker *worker) -{ -return worker->flags &IO_WORKER_F_FIXED ; -} -static int io_wq_worker (void *data) {struct io_worker *worker = data;
@@ -806,7 +809,7 @@ static void io_workqueue_create(struct work_struct *work)kfree (worker); } -static bool create_io_worker (struct io_wq *wq,int index) +static bool create_io_worker (struct io_wq *wq,int index,bool fixed) {struct io_wq_acct *acct = &wq->acct[index];struct io_worker *worker;
@@ -833,10 +836,14 @@ static bool create_io_worker(struct io_wq *wq, int index)if (index ==IO_WQ_ACCT_BOUND ) worker->flags |=IO_WORKER_F_BOUND ; +if (fixed) + worker->flags |=IO_WORKER_F_FIXED ; + tsk =create_io_thread (io_wq_worker, worker,NUMA_NO_NODE );if (!IS_ERR (tsk)) { -io_init_new_worker (wq, worker, tsk); - }else if (!io_should_retry_thread (PTR_ERR (tsk))) { +if (!fixed) +io_init_new_worker (wq, worker, tsk); + }else if (fixed || !io_should_retry_thread (PTR_ERR (tsk))) {kfree (worker); goto fail; }else {
@@ -947,7 +954,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work) !atomic_read(&acct->nr_running))) {bool did_create; - did_create =io_wq_create_worker (wq, acct); + did_create =io_wq_create_worker (wq, acct, false);if (likely (did_create))return ;
Signature · 1 line
2.25.1
Re: [PATCH 05/11] io-wq: add a new parameter for creating a new fixed worker
Quoted 14 lines · Hao Xu
From: Hao Xu <[email protected]>
Add a new parameter when creating new workers to indicate if users want a normal or fixed worker.
Signed-off-by Hao Xu <[email protected]>--- io_uring/io-wq.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)io_uring/io-wq.c +0
-0
[...]
io_uring/io-wq.c static void io_wq_inc_running(struct io_worker *worker) · line 319
Quoted 5 lines
io_uring/io-wq.c +4
-0+static bool is_fixed_worker (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_FIXED ; +}
That's what I mentioned in the other comment.
Quoted 14 lines
io_uring/io-wq.c +2
-1+static void create_worker_cb (struct callback_head *cb) {struct io_worker *worker;@@ -331,7 +337,7 @@ static void create_worker_cb(struct callback_head *cb) }raw_spin_unlock (&wq->lock);if (do_create) { -create_io_worker (wq, worker->create_index); +create_io_worker (wq, worker->create_index,is_fixed_worker (worker)); }else {atomic_dec (&acct->nr_running);io_worker_ref_put (wq);
io_uring/io-wq.c static void io_wq_dec_running(struct io_worker *worker) · line 408
Quoted 5 lines
io_uring/io-wq.c +2
-0@@ -398,6 +404,8 @@ static void io_wq_dec_running(struct io_worker *worker)return ;if (!io_acct_run_queue (acct))return ; +if (is_fixed_worker (worker)) +return ;
Aha, it's here. I was thinking about it a little bit more. Is it even correct? If you have a mixed fixed/non-fixed setup you presumably want non-fixed workers to kick in such situations. I don't remember this creation voodoo well, maybe Jens does have an idea.
Quoted 6 lines
io_uring/io-wq.c +0
-0atomic_inc (&acct->nr_running);atomic_inc (&wq->worker_refs);@@ -601,11 +609,6 @@ static bool is_worker_exiting(struct io_worker *worker)return worker->flags &IO_WORKER_F_EXIT ; }
[...]
Quoted 8 lines
io_uring/io-wq.c +1
-1-static bool create_io_worker (struct io_wq *wq,int index) +static bool create_io_worker (struct io_wq *wq,int index,bool fixed) {struct io_wq_acct *acct = &wq->acct[index];struct io_worker *worker;@@ -833,10 +836,14 @@ static bool create_io_worker(struct io_wq *wq, int index)if (index ==IO_WQ_ACCT_BOUND ) worker->flags |=IO_WORKER_F_BOUND ;
io_uring/io-wq.c static bool create_io_worker(struct io_wq *wq, int index) · line 845
Quoted 10 lines
io_uring/io-wq.c +5
-2+if (fixed) + worker->flags |=IO_WORKER_F_FIXED ; + tsk =create_io_thread (io_wq_worker, worker,NUMA_NO_NODE );if (!IS_ERR (tsk)) { -io_init_new_worker (wq, worker, tsk); - }else if (!io_should_retry_thread (PTR_ERR (tsk))) { +if (!fixed) +io_init_new_worker (wq, worker, tsk);
Why do we skip io_init_new_worker()? I assume you putting it into lists, but what about the rest? I.e.
tsk->worker_private = worker; worker->task = tsk; set_cpus_allowed_ptr(tsk, wq->cpu_mask);
Quoted 12 lines
io_uring/io-wq.c +2
-1+ }else if (fixed || !io_should_retry_thread (PTR_ERR (tsk))) {kfree (worker); goto fail; }else {@@ -947,7 +954,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work) !atomic_read(&acct->nr_running))) {bool did_create; - did_create =io_wq_create_worker (wq, acct); + did_create =io_wq_create_worker (wq, acct, false);if (likely (did_create))return ;
Signature · 1 line
Pavel Begunkov
PATCH 6/11 io-wq: return io_worker after successful inline worker creation
In reply to Hao XuFrom: Hao Xu <[email protected]>
After creating a io worker inline successfully, return the io_worker structure. This is used by fixed worker.
--- io_uring/io-wq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
io_uring/io-wq.c +10 -10
@@ -137,7 +137,7 @@ struct io_cb_cancel_data {bool cancel_all; }; -static bool create_io_worker (struct io_wq *wq,int index,bool fixed); +static struct io_worker *create_io_worker (struct io_wq *wq,int index,bool fixed);static void io_wq_dec_running (struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wq *wq, struct io_wq_acct *acct,
@@ -284,8 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,* We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wq_create_worker (struct io_wq *wq,struct io_wq_acct *acct, -bool fixed) +static struct io_worker *io_wq_create_worker (struct io_wq *wq, +struct io_wq_acct *acct,bool fixed) {/* * Most likely an attempt to queue unbounded work on an io_wq that
@@ -297,7 +297,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,raw_spin_lock (&wq->lock);if (acct->nr_workers >= acct->max_workers) {raw_spin_unlock (&wq->lock); -return true; +return NULL ; } acct->nr_workers++;raw_spin_unlock (&wq->lock);
@@ -809,11 +809,11 @@ static void io_workqueue_create(struct work_struct *work)kfree (worker); } -static bool create_io_worker (struct io_wq *wq,int index,bool fixed) +static struct io_worker *create_io_worker (struct io_wq *wq,int index,bool fixed) {struct io_wq_acct *acct = &wq->acct[index];struct io_worker *worker; -struct task_struct *tsk; +struct task_struct *tsk =NULL ;__set_current_state (TASK_RUNNING );
@@ -825,7 +825,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) acct->nr_workers--;raw_spin_unlock (&wq->lock);io_worker_ref_put (wq); -return false; +return tsk ? (struct io_worker *)tsk :ERR_PTR (-ENOMEM ); }refcount_set (&worker->ref,1 );
@@ -841,8 +841,8 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) tsk =create_io_thread (io_wq_worker, worker,NUMA_NO_NODE );if (!IS_ERR (tsk)) { -if (!fixed) -io_init_new_worker (wq, worker, tsk); +io_init_new_worker (wq, worker, tsk); +return worker; }else if (fixed || !io_should_retry_thread (PTR_ERR (tsk))) {kfree (worker); goto fail;
@@ -851,7 +851,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)schedule_work (&worker->work); } -return true; +return (struct io_worker *)tsk; }/*
Signature · 1 line
2.25.1
Re: [PATCH 06/11] io-wq: return io_worker after successful inline worker creation
Quoted 35 lines · Hao Xu
From: Hao Xu <[email protected]>
After creating a io worker inline successfully, return the
io_workerstructure. This is used by fixed worker.Signed-off-by Hao Xu <[email protected]>--- io_uring/io-wq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)io_uring/io-wq.c +3
-3@@ -137,7 +137,7 @@ struct io_cb_cancel_data {bool cancel_all; }; -static bool create_io_worker (struct io_wq *wq,int index,bool fixed); +static struct io_worker *create_io_worker (struct io_wq *wq,int index,bool fixed);static void io_wq_dec_running (struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wq *wq, struct io_wq_acct *acct,@@ -284,8 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,* We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wq_create_worker (struct io_wq *wq,struct io_wq_acct *acct, -bool fixed) +static struct io_worker *io_wq_create_worker (struct io_wq *wq, +struct io_wq_acct *acct,bool fixed) {/* * Most likely an attempt to queue unbounded work on an io_wq that
io_uring/io-wq.c static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct, · line 300
Quoted 5 lines
io_uring/io-wq.c +1
-1@@ -297,7 +297,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,raw_spin_lock (&wq->lock);if (acct->nr_workers >= acct->max_workers) {raw_spin_unlock (&wq->lock); -return true; +return NULL ;
Something is not right here. The function could succeed even if it didn't create a new worker. Now it's a failure.
Quoted 18 lines
io_uring/io-wq.c +2
-2} acct->nr_workers++;raw_spin_unlock (&wq->lock);@@ -809,11 +809,11 @@ static void io_workqueue_create(struct work_struct *work)kfree (worker); } -static bool create_io_worker (struct io_wq *wq,int index,bool fixed) +static struct io_worker *create_io_worker (struct io_wq *wq,int index,bool fixed) {struct io_wq_acct *acct = &wq->acct[index];struct io_worker *worker; -struct task_struct *tsk; +struct task_struct *tsk =NULL ;__set_current_state (TASK_RUNNING );
io_uring/io-wq.c static bool create_io_worker(struct io_wq *wq, int index, bool fixed) · line 828
Quoted 5 lines
io_uring/io-wq.c +1
-1@@ -825,7 +825,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) acct->nr_workers--;raw_spin_unlock (&wq->lock);io_worker_ref_put (wq); -return false; +return tsk ? (struct io_worker *)tsk :ERR_PTR (-ENOMEM );
How it this conversion valid? I don't remember us overlaying struct io_worker onto task_struct
Quoted 23 lines
io_uring/io-wq.c +3
-3}refcount_set (&worker->ref,1 );@@ -841,8 +841,8 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) tsk =create_io_thread (io_wq_worker, worker,NUMA_NO_NODE );if (!IS_ERR (tsk)) { -if (!fixed) -io_init_new_worker (wq, worker, tsk); +io_init_new_worker (wq, worker, tsk); +return worker; }else if (fixed || !io_should_retry_thread (PTR_ERR (tsk))) {kfree (worker); goto fail;@@ -851,7 +851,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)schedule_work (&worker->work); } -return true; +return (struct io_worker *)tsk; }/*
Signature · 1 line
Pavel Begunkov
PATCH 7/11 io_uring: add new api to register fixed workers
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new api to register fixed workers. The api is designed to register fixed workers for the current task. For simplicity, it doesn't allow worker number update. We have a separate unregister api to uninstall all the fixed workers. And then we can register different number of fixed workers again.
--- include/uapi/linux/io_uring.h | 9 ++++ io_uring/io-wq.c | 85 +++++++++++++++++++++++++++++++++++ io_uring/io-wq.h | 1 + io_uring/io_uring.c | 71 +++++++++++++++++++++++++++++ 4 files changed, 166 insertions(+)
include/uapi/linux/io_uring.h +9 -0
@@ -535,6 +535,9 @@ enum {/* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE =25 , +/* set/get number of fixed workers */ +IORING_REGISTER_IOWQ_FIXED_WORKERS =26 , +/* this goes last */ IORING_REGISTER_LAST,
@@ -715,6 +718,12 @@ struct io_uring_recvmsg_out {__u32 flags; }; +struct io_uring_fixed_worker_arg { +__u32 nr_workers; +__u32 resv; +__u64 resv2[3 ]; +}; + #ifdef __cplusplus }#endif
io_uring/io-wq.c +85 -0
91 lines not on this page · raw message
io_uring/io-wq.h +1 -0
@@ -52,6 +52,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val);int io_wq_cpu_affinity (struct io_wq *wq,cpumask_var_t mask);int io_wq_max_workers (struct io_wq *wq,int *new_count); +int io_wq_fixed_workers (struct io_wq *wq,struct io_uring_fixed_worker_arg *count);static inline bool io_wq_is_hashed (struct io_wq_work *work) {
io_uring/io_uring.c +71 -0
83 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH 07/11] io_uring: add new api to register fixed workers
Quoted 17 lines · Hao Xu
io_uring/io-wq.c +17
-0+rcu_read_lock (); + +for (i =0 ; i <IO_WQ_ACCT_NR ; i++) { +unsigned int nr = count[i].nr_workers; + + acct = &wq->acct[i]; + acct->fixed_nr = nr; + acct->fixed_workers =kcalloc (nr,sizeof (struct io_worker *), +GFP_KERNEL ); +if (!acct->fixed_workers) { + ret = -ENOMEM ; +break ; + } + +for (j =0 ; j < nr; j++) { +struct io_worker *worker = +io_wq_create_worker (wq, acct, true);
io_uring/io-wq.c int io_wq_max_workers(struct io_wq *wq, int *new_count) · line 1448
Quoted 10 lines
io_uring/io-wq.c +10
-0+if (IS_ERR (worker)) { + ret =PTR_ERR (worker); +break ; + } + acct->fixed_workers[j] = worker; + } +if (j < nr) +break ; + } +rcu_read_unlock ();
This looks wrong. kcalloc() with GFP_KERNEL may sleep. Note that you're not allowed to sleep inside the RCU read lock critical section.
Using GFP_KERNEL implies GFP_RECLAIM, which means that direct reclaim may be triggered under memory pressure; the calling context must be allowed to sleep.
Signature · 1 line
Ammar Faizi
Re: [PATCH 07/11] io_uring: add new api to register fixed workers
Hi Ammar,
Quoted 32 lines · Ammar Faizi
Quoted 17 lines · Hao Xu
io_uring/io-wq.c +17
-0+rcu_read_lock (); + +for (i =0 ; i <IO_WQ_ACCT_NR ; i++) { +unsigned int nr = count[i].nr_workers; + + acct = &wq->acct[i]; + acct->fixed_nr = nr; + acct->fixed_workers =kcalloc (nr,sizeof (struct io_worker *), +GFP_KERNEL ); +if (!acct->fixed_workers) { + ret = -ENOMEM ; +break ; + } + +for (j =0 ; j < nr; j++) { +struct io_worker *worker = +io_wq_create_worker (wq, acct, true);io_uring/io-wq.c int io_wq_max_workers(struct io_wq *wq, int *new_count) · line 1448
Quoted 10 lines
io_uring/io-wq.c +10
-0+if (IS_ERR (worker)) { + ret =PTR_ERR (worker); +break ; + } + acct->fixed_workers[j] = worker; + } +if (j < nr) +break ; + } +rcu_read_unlock ();This looks wrong.
kcalloc()withGFP_KERNELmay sleep. Note that you're not allowed to sleep inside the RCU read lock critical section.
Quoted 4 lines
Using
GFP_KERNELimpliesGFP_RECLAIM, which means that direct reclaim may be triggered under memory pressure; the calling context must be allowed to sleep.
I think you are right, I'll fix it in v2.
Hi Jens, ask a question about this: I saw same rcu_read_lock() in io_wq_max_workers(), but what is it really protect?
Regards,
Hao
Re: [PATCH 07/11] io_uring: add new api to register fixed workers
In reply to Hao XuQuoted 18 lines · Hao Xu
io_uring/io_uring.c +18
-0+static __cold int io_register_iowq_fixed_workers (struct io_ring_ctx *ctx, +void __user *arg,int nr_args) + __must_hold(&ctx->uring_lock) +{ +struct io_uring_task *tctx =NULL ; +struct io_sq_data *sqd =NULL ; +struct io_uring_fixed_worker_arg *res; +size_t size; +int i, ret; +bool zero = true; + + size =array_size (nr_args,sizeof (*res)); +if (size ==SIZE_MAX ) +return -EOVERFLOW ; + + res =memdup_user (arg, size); +if (IS_ERR (res)) +return PTR_ERR (res);
io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4385
Quoted 10 lines
io_uring/io_uring.c +10
-0+ +for (i =0 ; i < nr_args; i++) { +if (res[i].nr_workers) { + zero = false; +break ; + } + } + +if (zero) +return 0 ;
You have a memory leak bug here. The memdup_user() needs clean up. kfree(res);
Signature · 1 line
Ammar Faizi
Re: [PATCH 07/11] io_uring: add new api to register fixed workers
Quoted 24 lines · Ammar Faizi
Quoted 23 lines · Hao Xu
io_uring/io_uring.c +23
-0+static __cold int io_register_iowq_fixed_workers (struct io_ring_ctx *ctx, +void __user *arg,int nr_args) + __must_hold(&ctx->uring_lock) +{ +struct io_uring_task *tctx =NULL ; +struct io_sq_data *sqd =NULL ; +struct io_uring_fixed_worker_arg *res; +size_t size; +int i, ret; +bool zero = true; + + size =array_size (nr_args,sizeof (*res)); +if (size ==SIZE_MAX ) +return -EOVERFLOW ; + + res =memdup_user (arg, size); +if (IS_ERR (res)) +return PTR_ERR (res); + +for (i =0 ; i < nr_args; i++) { +if (res[i].nr_workers) { + zero = false; +break ;
Quoted 9 lines
io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4385
Quoted 5 lines
io_uring/io_uring.c +5
-0+ } + } + +if (zero) +return 0 ;You have a memory leak bug here. The
memdup_user()needs clean up. kfree(res);
True, I'll fix it in v2, thanks.
Re: [PATCH 07/11] io_uring: add new api to register fixed workers
In reply to Hao XuQuoted 20 lines · Hao Xu
From: Hao Xu <[email protected]>
Add a new api to register fixed workers. The api is designed to register fixed workers for the current task. For simplicity, it doesn't allow worker number update. We have a separate unregister api to uninstall all the fixed workers. And then we can register different number of fixed workers again.
Signed-off-by Hao Xu <[email protected]>--- include/uapi/linux/io_uring.h | 9 ++++ io_uring/io-wq.c | 85 +++++++++++++++++++++++++++++++++++ io_uring/io-wq.h | 1 + io_uring/io_uring.c | 71 +++++++++++++++++++++++++++++ 4 files changed, 166 insertions(+)include/uapi/linux/io_uring.h +0
-0
[...]
Quoted 16 lines
io_uring/io_uring.c +8
-0@@ -4351,6 +4351,71 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx,return ret; } +/* +* note: this function sets fixed workers for a single task, so every +* task which wants to set the fixed workers has to call this function +*/ +static __cold int io_register_iowq_fixed_workers (struct io_ring_ctx *ctx, +void __user *arg,int nr_args) + __must_hold(&ctx->uring_lock) +{
io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4371
Quoted 10 lines
io_uring/io_uring.c +10
-0+struct io_uring_task *tctx =NULL ; +struct io_sq_data *sqd =NULL ; +struct io_uring_fixed_worker_arg *res; +size_t size; +int i, ret; +bool zero = true; + + size =array_size (nr_args,sizeof (*res)); +if (size ==SIZE_MAX ) +return -EOVERFLOW ;
I don't think the number of accounting classes is going to change, just move nr_args check from below here and have on-stack array of size 2.
struct io_uring_fixed_worker_arg res[IO_WQ_ACCT_NR];
if (nr_args != IO_WQ_ACCT_NR)
return -EINVAL;
...
Quoted 12 lines
io_uring/io_uring.c +12
-0+ + res =memdup_user (arg, size); +if (IS_ERR (res)) +return PTR_ERR (res); + +for (i =0 ; i < nr_args; i++) { +if (res[i].nr_workers) { + zero = false; +break ; + } + } +
Signature · 1 line
Pavel Begunkov
PATCH 8/11 io_uring: add function to unregister fixed workers
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new register api to unregister fixed workers.
--- include/uapi/linux/io_uring.h | 3 +++ io_uring/io-wq.c | 50 ++++++++++++++++++++++++++++++++++- io_uring/io-wq.h | 1 + io_uring/io_uring.c | 45 +++++++++++++++++++++++++++++++ 4 files changed, 98 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +3 -0
@@ -538,6 +538,9 @@ enum {/* set/get number of fixed workers */ IORING_REGISTER_IOWQ_FIXED_WORKERS =26 , +/* destroy fixed workers */ +IORING_UNREGISTER_IOWQ_FIXED_WORKERS =27 , +/* this goes last */ IORING_REGISTER_LAST,
io_uring/io-wq.c +49 -1
62 lines not on this page · raw message
io_uring/io-wq.h +1 -0
@@ -53,6 +53,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val);int io_wq_cpu_affinity (struct io_wq *wq,cpumask_var_t mask);int io_wq_max_workers (struct io_wq *wq,int *new_count);int io_wq_fixed_workers (struct io_wq *wq,struct io_uring_fixed_worker_arg *count); +int io_wq_destroy_fixed_workers (struct io_wq *wq);static inline bool io_wq_is_hashed (struct io_wq_work *work) {
io_uring/io_uring.c +45 -0
57 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH 08/11] io_uring: add function to unregister fixed workers
Quoted 17 lines · Hao Xu
From: Hao Xu <[email protected]>
Add a new register api to unregister fixed workers.
Signed-off-by Hao Xu <[email protected]>--- include/uapi/linux/io_uring.h | 3 +++ io_uring/io-wq.c | 50 ++++++++++++++++++++++++++++++++++- io_uring/io-wq.h | 1 + io_uring/io_uring.c | 45 +++++++++++++++++++++++++++++++ 4 files changed, 98 insertions(+), 1 deletion(-)include/uapi/linux/io_uring.h +0
-0
include/uapi/linux/io_uring.h enum { · line 542
Quoted 5 lines
include/uapi/linux/io_uring.h +2
-0@@ -538,6 +538,9 @@ enum {/* set/get number of fixed workers */ IORING_REGISTER_IOWQ_FIXED_WORKERS =26 , +/* destroy fixed workers */ + IORING_UNREGISTER_IOWQ_FIXED_WORKERS = 27,
Do we need a new code? I think it's cleaner if we use IORING_REGISTER_IOWQ_FIXED_WORKERS and do sth like
struct io_uring_fixed_worker_arg arg;
if (arg.nr_workers)
do_unregister_fixed_workers();
...
Signature · 1 line
Pavel Begunkov
PATCH 9/11 io-wq: add strutures to allow to wait fixed workers exit
In reply to Hao XuFrom: Hao Xu <[email protected]>
When unregister fixed workers, there should be a way to allow us to wait all the fixed workers exit.
--- io_uring/io-wq.c | 72 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 20 deletions(-)
io_uring/io-wq.c +52 -20
@@ -108,6 +108,10 @@ struct io_wq {atomic_t worker_refs;struct completion worker_done; +atomic_t fixed_worker_refs; +struct completion fixed_worker_done; +bool fixed_comp_init; +struct hlist_node cpuhp_node;struct task_struct *task;
@@ -172,10 +176,25 @@ static inline struct io_wq_acct *io_wq_get_acct(struct io_worker *worker)return io_get_acct (worker->wq, worker->flags &IO_WORKER_F_BOUND ); } -static void io_worker_ref_put (struct io_wq *wq) +static void io_worker_ref_get (struct io_wq *wq,bool fixed) +{ +atomic_inc (&wq->worker_refs); +if (fixed) +atomic_inc (&wq->fixed_worker_refs); +} + +static void io_worker_ref_put (struct io_wq *wq,bool fixed) {if (atomic_dec_and_test (&wq->worker_refs))complete (&wq->worker_done); + +if (fixed &&atomic_dec_and_test (&wq->fixed_worker_refs)) +complete (&wq->fixed_worker_done); +} + +static bool is_fixed_worker (struct io_worker *worker) +{ +return worker->flags &IO_WORKER_F_FIXED ; }static void io_worker_cancel_cb (struct io_worker *worker)
@@ -187,7 +206,7 @@ static void io_worker_cancel_cb(struct io_worker *worker)raw_spin_lock (&wq->lock); acct->nr_workers--;raw_spin_unlock (&wq->lock); -io_worker_ref_put (wq); +io_worker_ref_put (wq,is_fixed_worker (worker));clear_bit_unlock (0 , &worker->create_state);io_worker_release (worker); }
@@ -205,6 +224,7 @@ static bool io_task_worker_match(struct callback_head *cb, void *data)static void io_worker_exit (struct io_worker *worker) {struct io_wq *wq = worker->wq; +bool fixed =is_fixed_worker (worker);while (1 ) { struct callback_head *cb = task_work_cancel_match(wq->task,
@@ -230,7 +250,7 @@ static void io_worker_exit(struct io_worker *worker)preempt_enable ();kfree_rcu (worker, rcu); -io_worker_ref_put (wq); +io_worker_ref_put (wq, fixed);do_exit (0 ); }
@@ -302,7 +322,7 @@ static struct io_worker *io_wq_create_worker(struct io_wq *wq, acct->nr_workers++;raw_spin_unlock (&wq->lock);atomic_inc (&acct->nr_running); -atomic_inc (&wq->worker_refs); +io_worker_ref_get (wq, fixed);return create_io_worker (wq, acct->index, fixed); }
@@ -313,11 +333,6 @@ static void io_wq_inc_running(struct io_worker *worker)atomic_inc (&acct->nr_running); } -static bool is_fixed_worker (struct io_worker *worker) -{ -return worker->flags &IO_WORKER_F_FIXED ; -} -static void create_worker_cb (struct callback_head *cb) {struct io_worker *worker;
@@ -325,8 +340,10 @@ static void create_worker_cb(struct callback_head *cb)struct io_wq_acct *acct;bool do_create = false; +bool fixed; worker = container_of(cb, struct io_worker, create_work); + fixed =is_fixed_worker (worker); wq = worker->wq; acct = &wq->acct[worker->create_index];raw_spin_lock (&wq->lock);
@@ -337,10 +354,10 @@ static void create_worker_cb(struct callback_head *cb) }raw_spin_unlock (&wq->lock);if (do_create) { -create_io_worker (wq, worker->create_index,is_fixed_worker (worker)); +create_io_worker (wq, worker->create_index, fixed); }else {atomic_dec (&acct->nr_running); -io_worker_ref_put (wq); +io_worker_ref_put (wq, fixed); }clear_bit_unlock (0 , &worker->create_state);io_worker_release (worker);
@@ -351,6 +368,7 @@ static bool io_queue_worker_create(struct io_worker *worker, task_work_func_t func) {struct io_wq *wq = worker->wq; +bool fixed =is_fixed_worker (worker);/* raced with exit, just ignore create call */ if (test_bit (IO_WQ_BIT_EXIT , &wq->state))
@@ -367,7 +385,7 @@ static bool io_queue_worker_create(struct io_worker *worker, test_and_set_bit_lock(0, &worker->create_state)) goto fail_release; -atomic_inc (&wq->worker_refs); +io_worker_ref_get (wq, fixed);init_task_work (&worker->create_work, func); worker->create_index = acct->index;if (!task_work_add (wq->task, &worker->create_work,TWA_SIGNAL )) {
@@ -379,16 +397,16 @@ static bool io_queue_worker_create(struct io_worker *worker,*/ if (test_bit (IO_WQ_BIT_EXIT , &wq->state))io_wq_cancel_tw_create (wq); -io_worker_ref_put (wq); +io_worker_ref_put (wq, fixed);return true; } -io_worker_ref_put (wq); +io_worker_ref_put (wq, fixed);clear_bit_unlock (0 , &worker->create_state); fail_release:io_worker_release (worker); fail:atomic_dec (&acct->nr_running); -io_worker_ref_put (wq); +io_worker_ref_put (wq, fixed);return false; }
@@ -408,7 +426,7 @@ static void io_wq_dec_running(struct io_worker *worker)return ;atomic_inc (&acct->nr_running); -atomic_inc (&wq->worker_refs); +io_worker_ref_get (wq, false);io_queue_worker_create (worker, acct, create_worker_cb); }
@@ -790,7 +808,7 @@ static void create_worker_cont(struct callback_head *cb) }else {raw_spin_unlock (&wq->lock); } -io_worker_ref_put (wq); +io_worker_ref_put (wq,is_fixed_worker (worker));kfree (worker);return ; }
@@ -824,7 +842,7 @@ static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixeraw_spin_lock (&wq->lock); acct->nr_workers--;raw_spin_unlock (&wq->lock); -io_worker_ref_put (wq); +io_worker_ref_put (wq, fixed);return tsk ? (struct io_worker *)tsk :ERR_PTR (-ENOMEM ); }
@@ -1243,7 +1261,7 @@ static void io_wq_exit_workers(struct io_wq *wq)rcu_read_lock ();io_wq_for_each_worker (wq, io_wq_worker_wake,NULL );rcu_read_unlock (); -io_worker_ref_put (wq); +io_worker_ref_put (wq, false);wait_for_completion (&wq->worker_done);spin_lock_irq (&wq->hash->wait.lock);
@@ -1390,6 +1408,7 @@ static void io_wq_clean_fixed_workers(struct io_wq *wq) }kfree (workers); } +wait_for_completion (&wq->fixed_worker_done); }/*
@@ -1421,6 +1440,13 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *counrcu_read_lock (); +atomic_set (&wq->fixed_worker_refs,1 ); +if (wq->fixed_comp_init) { +reinit_completion (&wq->fixed_worker_done); + }else { +init_completion (&wq->fixed_worker_done); + wq->fixed_comp_init = true; + }for (i =0 ; i <IO_WQ_ACCT_NR ; i++) {unsigned int nr = count[i].nr_workers;
@@ -1495,12 +1521,18 @@ int io_wq_destroy_fixed_workers(struct io_wq *wq) workers[j]->flags |=IO_WORKER_F_EXIT ;wake_up_process (worker->task); } -// wait for all workers exit kfree (workers); }raw_spin_unlock (&wq->lock);rcu_read_unlock (); +// decrement the init reference +if (atomic_dec_and_test (&wq->fixed_worker_refs)) +complete (&wq->fixed_worker_done); + +wait_for_completion (&wq->fixed_worker_done); + wq->fixed_comp_init = false; +return 0 ; }
Signature · 1 line
2.25.1
PATCH 10/11 io-wq: distinguish fixed worker by its name
In reply to Hao XuFrom: Hao Xu <[email protected]>
Distinguish fixed workers and normal workers by their names.
--- io_uring/io-wq.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
io_uring/io-wq.c +4 -2
@@ -634,10 +634,12 @@ static int io_wq_worker(void *data)struct io_wq *wq = worker->wq;bool exit_mask = false, last_timeout = false;char buf[TASK_COMM_LEN ]; +bool fixed =is_fixed_worker (worker); worker->flags |= (IO_WORKER_F_UP |IO_WORKER_F_RUNNING ); -snprintf (buf,sizeof (buf),"iou-wrk-%d" , wq->task->pid); +snprintf (buf,sizeof (buf), fixed ?"iou-fixed-%d" :"iou-wrk-%d" , + wq->task->pid);set_task_comm (current, buf);while (!test_bit (IO_WQ_BIT_EXIT , &wq->state)) {
@@ -656,7 +658,7 @@ static int io_wq_worker(void *data)* fixed worker, they can be manually reset to cpu other than * the cpuset indicated by io_wq_worker_affinity() */ -if (!is_fixed_worker (worker) && last_timeout && +if (!fixed && last_timeout && (exit_mask || acct->nr_workers >1 )) { acct->nr_workers--;raw_spin_unlock (&wq->lock);
Signature · 1 line
2.25.1
Re: [PATCH 10/11] io-wq: distinguish fixed worker by its name
Quoted 14 lines · Hao Xu
From: Hao Xu <[email protected]>
Distinguish fixed workers and normal workers by their names.
Signed-off-by Hao Xu <[email protected]>--- io_uring/io-wq.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)io_uring/io-wq.c +0
-0
io_uring/io-wq.c static int io_wq_worker(void *data) · line 642
Quoted 10 lines
io_uring/io-wq.c +3
-1@@ -634,10 +634,12 @@ static int io_wq_worker(void *data)struct io_wq *wq = worker->wq;bool exit_mask = false, last_timeout = false;char buf[TASK_COMM_LEN ]; +bool fixed =is_fixed_worker (worker); worker->flags |= (IO_WORKER_F_UP |IO_WORKER_F_RUNNING ); -snprintf (buf,sizeof (buf),"iou-wrk-%d" , wq->task->pid); +snprintf (buf,sizeof (buf), fixed ?"iou-fixed-%d" :"iou-wrk-%d" , + wq->task->pid);
Minor nit: iou-wrk-fixed should be better, it still tells that it's a worker and I think it should be more familiar to users.
Quoted 12 lines
io_uring/io-wq.c +1
-1set_task_comm (current, buf);while (!test_bit (IO_WQ_BIT_EXIT , &wq->state)) {@@ -656,7 +658,7 @@ static int io_wq_worker(void *data)* fixed worker, they can be manually reset to cpu other than * the cpuset indicated by io_wq_worker_affinity() */ -if (!is_fixed_worker (worker) && last_timeout && +if (!fixed && last_timeout && (exit_mask || acct->nr_workers >1 )) { acct->nr_workers--;raw_spin_unlock (&wq->lock);
Signature · 1 line
Pavel Begunkov
PATCH 11/11 io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend
In reply to Hao XuFrom: Hao Xu <[email protected]>
Add a new setup flag to indicate that the uring instance only use fixed workers as async offload threads. Add a work flag and its code logic as well.
--- include/uapi/linux/io_uring.h | 10 +++++++++- io_uring/io-wq.c | 18 +++++++++++++----- io_uring/io-wq.h | 1 + io_uring/io_uring.c | 24 +++++++++++++++++++----- 4 files changed, 42 insertions(+), 11 deletions(-)
include/uapi/linux/io_uring.h +9 -1
@@ -185,6 +185,11 @@ enum {*/ #define IORING_SETUP_REGISTERED_FD_ONLY (1U << 15) +/* +* this ring instance only use fixed worker for async offload. +*/ +#define IORING_SETUP_FIXED_WORKER_ONLY (1U << 16) +enum io_uring_op {IORING_OP_NOP ,IORING_OP_READV ,
@@ -721,9 +726,12 @@ struct io_uring_recvmsg_out {__u32 flags; }; +#define IORING_FIXED_WORKER_F_ONLY (1U << 0) +#define IORING_FIXED_WORKER_F_VALID (IORING_FIXED_WORKER_F_ONLY) +struct io_uring_fixed_worker_arg {__u32 nr_workers; -__u32 resv; +__u32 flags;__u64 resv2[3 ]; };
io_uring/io-wq.c +13 -5
@@ -272,7 +272,7 @@ static inline bool io_acct_run_queue(struct io_wq_acct *acct)* caller must create one. */ static bool io_wq_activate_free_worker (struct io_wq *wq, -struct io_wq_acct *acct) +struct io_wq_acct *acct,bool fixed)__must_hold (RCU ) {struct hlist_nulls_node *n;
@@ -286,7 +286,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,hlist_nulls_for_each_entry_rcu (worker, n, &wq->free_list, nulls_node) {if (!io_worker_get (worker))continue ; -if (io_wq_get_acct (worker) != acct) { +if (io_wq_get_acct (worker) != acct || + (fixed && !is_fixed_worker (worker))) {io_worker_release (worker);continue ; }
@@ -492,6 +493,9 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct, work = container_of(node, struct io_wq_work, list); +if ((work->flags &IO_WQ_WORK_FIXED ) && !is_fixed_worker (worker)) +continue ; +/* not hashed, can run anytime */ if (!io_wq_is_hashed (work)) {wq_list_del (&acct->work_list, node, prev);
@@ -946,7 +950,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work)struct io_wq_acct *acct =io_work_get_acct (wq, work);struct io_cb_cancel_data match;unsigned work_flags = work->flags; -bool do_create; +bool do_create, fixed = work_flags &IO_WQ_WORK_FIXED ;/* * If io-wq is exiting for this task, or if the request has explicitly
@@ -965,11 +969,14 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work)raw_spin_lock (&wq->lock);rcu_read_lock (); - do_create = !io_wq_activate_free_worker (wq, acct); + do_create = !io_wq_activate_free_worker (wq, acct, fixed);rcu_read_unlock ();raw_spin_unlock (&wq->lock); +if (fixed) +return ; +if (do_create && ((work_flags &IO_WQ_WORK_CONCURRENT ) || !atomic_read (&acct->nr_running))) {bool did_create;
@@ -1155,7 +1162,7 @@ static int io_wq_hash_wake(struct wait_queue_entry *wait, unsigned mode,struct io_wq_acct *acct = &wq->acct[i];if (test_and_clear_bit (IO_ACCT_STALLED_BIT , &acct->flags)) -io_wq_activate_free_worker (wq, acct); +io_wq_activate_free_worker (wq, acct, false); }rcu_read_unlock ();return 1 ;
@@ -1477,6 +1484,7 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *counif (ret) goto err; +return 0 ; err:
io_uring/io-wq.h +1 -0
@@ -11,6 +11,7 @@ enum {IO_WQ_WORK_HASHED =2 ,IO_WQ_WORK_UNBOUND =4 ,IO_WQ_WORK_CONCURRENT =16 , +IO_WQ_WORK_FIXED =32 , IO_WQ_HASH_SHIFT = 24, /* upper 8 bits are used for hash key */ };
io_uring/io_uring.c +19 -5
@@ -479,6 +479,9 @@ void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use)if (WARN_ON_ONCE (!same_thread_group (req->task, current))) req->work.flags |=IO_WQ_WORK_CANCEL ; +if (req->ctx->flags &IORING_SETUP_FIXED_WORKER_ONLY ) + req->work.flags |=IO_WQ_WORK_FIXED ; +trace_io_uring_queue_async_work (req,io_wq_is_hashed (&req->work));io_wq_enqueue (tctx->io_wq, &req->work);if (link)
@@ -1971,7 +1974,12 @@ struct io_wq_work *io_wq_free_work(struct io_wq_work *work) struct io_kiocb *req = container_of(work, struct io_kiocb, work); req =io_put_req_find_next (req); -return req ? &req->work :NULL ; +if (req) { + req->work.flags |=IO_WQ_WORK_FIXED ; +return &req->work; + } + +return NULL ; }void io_wq_submit_work (struct io_wq_work *work)
@@ -4364,7 +4372,7 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,struct io_uring_fixed_worker_arg *res;size_t size;int i, ret; -bool zero = true; +bool zero = true, fixed_only = false; size =array_size (nr_args,sizeof (*res));if (size ==SIZE_MAX )
@@ -4375,15 +4383,20 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,return PTR_ERR (res);for (i =0 ; i < nr_args; i++) { -if (res[i].nr_workers) { +if (res[i].flags & ~IORING_FIXED_WORKER_F_VALID ) +return -EINVAL ; +if (res[i].flags &IORING_FIXED_WORKER_F_ONLY ) + fixed_only = true; +if (res[i].nr_workers) zero = false; -break ; - } }if (zero)return 0 ; +if (fixed_only) + ctx->flags |=IORING_SETUP_FIXED_WORKER_ONLY ; +if (ctx->flags &IORING_SETUP_SQPOLL ) { sqd = ctx->sq_data;if (sqd) {
@@ -4423,6 +4436,7 @@ static __cold int io_unregister_iowq_fixed_workers(struct io_ring_ctx *ctx)struct io_sq_data *sqd =NULL ;int ret; + ctx->flags &= ~IORING_SETUP_FIXED_WORKER_ONLY ;if (ctx->flags &IORING_SETUP_SQPOLL ) { sqd = ctx->sq_data;if (sqd) {
Signature · 1 line
2.25.1
Re: [PATCH 11/11] io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend
Quoted 5 lines · Hao Xu
From: Hao Xu <[email protected]>
Add a new setup flag to indicate that the uring instance only use fixed workers as async offload threads. Add a work flag and its code logic as well.
Do we need it? What is the effective difference from limiting workers to 0 via IORING_REGISTER_IOWQ_MAX_WORKERS? It sounds like it should rather be a liburing helper instead.
Signature · 1 line
Pavel Begunkov
Hi Jens and all,
Quoted 11 lines · Hao Xu
From: Hao Xu <[email protected]>
The initial feature request by users is here:
https://github.com/axboe/liburing/issues/296Fixed worker provide a way for users to control the io-wq threads. A fixed worker is worker thread which exists no matter there are works to do or not. We provide a new register api to register fixed workers, and a register api to unregister them as well. The parameter of the register api is the number of fixed workers users want.
Here is a liburing test case to show how it works:
https://github.com/axboe/liburing/commit/bc9e862d1317f2466381adb6243be8cc86c3bd27
Regards,
Hao
Gently ping this series
No idea why the mail threading is a mess, it looks fine in fsdevel list.
Quoted 76 lines · Hao Xu
From: Hao Xu <[email protected]>
The initial feature request by users is here:
https://github.com/axboe/liburing/issues/296Fixed worker provide a way for users to control the io-wq threads. A fixed worker is worker thread which exists no matter there are works to do or not. We provide a new register api to register fixed workers, and a register api to unregister them as well. The parameter of the register api is the number of fixed workers users want.
For example:
```c io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5 }) do I/O works io_uring_unregister_iowq_fixed_workers(&ring) ```After registration, there will be 5 fixed workers. User can setup their affinity, priority etc. freely, without adding any new register api to set up attributions. These workers won't be destroyed until users call unregister api.
Note, registering some fixed workers doesn't mean no creating normal workers. When there is no free workers, new normal workers can be created when works come. So a work may be picked up by fixed workers or normal workers.
If users want to offload works only to fixed workers, they can specify a flag
FIXED_ONLYwhen registering fixed workers.```c io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5, .flags |= FIXED_ONLY }) ```In above case, no normal workers will be created before calling
io_uring_register_iowq_fixed_workers().Note:
- When registering fixed workers, those fixed workers are per io-wq. So if anio_uringinstance is shared by multiple tasks, and you want all tasks to use fixed workers, all tasks have to call the regitser api.
- if specifyingFIXED_ONLYwhen registering fixed workers, that is perio_uringinstance. all works in this instance are handled by fixed workers.Therefore, if an
io_uringinstance is shared by two tasks, and you want all requests in this instance to be handled only by fixed workers, you have to call the register api in these two tasks and specifyFIXED_ONLYat least once when calling register api.Hao Xu (11):
io-wq: fix worker counting after worker received exit signal
io-wq: add a new worker flag to indicate worker exit
io-wq: add a new type io-wq worker
io-wq: add fixed worker members inio_wq_acct
io-wq: add a new parameter for creating a new fixed worker
io-wq: returnio_workerafter successful inline worker creation
io_uring: add new api to register fixed workers
io_uring: add function to unregister fixed workers
io-wq: add strutures to allow to wait fixed workers exit
io-wq: distinguish fixed worker by its name
io_uring: addIORING_SETUP_FIXED_WORKER_ONLYand its friendinclude/uapi/linux/io_uring.h | 20 +++ io_uring/io-wq.c | 275 ++++++++++++++++++++++++++++++---- io_uring/io-wq.h | 3 + io_uring/io_uring.c | 132 +++++++++++++++- 4 files changed, 397 insertions(+), 33 deletions(-)