Skip to contentSkip to conversation

fixed worker

RFC PATCH In review Hao Xu 3 participants 27 messages Jun 9, 2023 – Jul 5, 2023

Subsystems (2)
lore
Share & tools
lore

Hao Xu

author
#0 lore raw

cover fixed worker

From: Hao Xu <[email protected]>

The initial feature request by users is here:
https://github.com/axboe/liburing/issues/296

Fixed worker provide a way for users to control the io-wq threads. A fixed worker is worker thread which exists no matter there are works to do or not. We provide a new register api to register fixed workers, and a register api to unregister them as well. The parameter of the register api is the number of fixed workers users want.

For example:

```c
io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5 })
do I/O works
io_uring_unregister_iowq_fixed_workers(&ring)

```

After registration, there will be 5 fixed workers. User can setup their affinity, priority etc. freely, without adding any new register api to set up attributions. These workers won't be destroyed until users call unregister api.

Note, registering some fixed workers doesn't mean no creating normal workers. When there is no free workers, new normal workers can be created when works come. So a work may be picked up by fixed workers or normal workers.

If users want to offload works only to fixed workers, they can specify a flag FIXED_ONLY when registering fixed workers.

```c
io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5, .flags |=
FIXED_ONLY })

```

In above case, no normal workers will be created before calling io_uring_register_iowq_fixed_workers().

Note:
- When registering fixed workers, those fixed workers are per io-wq. So if an io_uring instance is shared by multiple tasks, and you want all tasks to use fixed workers, all tasks have to call the regitser api.
- if specifying FIXED_ONLY when registering fixed workers, that is per io_uring instance. all works in this instance are handled by fixed workers.

Therefore, if an io_uring instance is shared by two tasks, and you want all requests in this instance to be handled only by fixed workers, you have to call the register api in these two tasks and specify FIXED_ONLY at least once when calling register api.

Hao Xu (11):
io-wq: fix worker counting after worker received exit signal
io-wq: add a new worker flag to indicate worker exit
io-wq: add a new type io-wq worker
io-wq: add fixed worker members in io_wq_acct
io-wq: add a new parameter for creating a new fixed worker
io-wq: return io_worker after successful inline worker creation
io_uring: add new api to register fixed workers
io_uring: add function to unregister fixed workers
io-wq: add strutures to allow to wait fixed workers exit
io-wq: distinguish fixed worker by its name
io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend

 include/uapi/linux/io_uring.h |  20 +++
 io_uring/io-wq.c              | 275 ++++++++++++++++++++++++++++++----
 io_uring/io-wq.h              |   3 +
 io_uring/io_uring.c           | 132 +++++++++++++++-
 4 files changed, 397 insertions(+), 33 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/11 io-wq: fix worker counting after worker received exit signal

From: Hao Xu <[email protected]>

acct->nr_workers should be decremented when we break the loop in io_wq_worker().

Fixes 78f8876c2d9f ("io-wq: exclusively gate signal based exit on get_signal() return")
---
 io_uring/io-wq.c | 4 ++++
 1 file changed, 4 insertions(+)
io_uring/io-wq.c +4 -0
index b2715988791e..b70eebec2845 100644
@@ -634,6 +634,10 @@ static int io_wq_worker(void *data)
 
 			if (!get_signal(&ksig))
 				continue;
+
+			raw_spin_lock(&wq->lock);
+			acct->nr_workers--;
+			raw_spin_unlock(&wq->lock);
 			break;
 		}
 		if (!ret) {
Signature · 1 line
2.25.1
#2 lore raw

Re: [PATCH 01/11] io-wq: fix worker counting after worker received exit signal

Quoted 16 lines · Hao Xu

From: Hao Xu <[email protected]>

acct->nr_workers should be decremented when we break the loop in io_wq_worker().

Fixes 78f8876c2d9f ("io-wq: exclusively gate signal based exit on get_signal() return")
---
  io_uring/io-wq.c | 4 ++++
  1 file changed, 4 insertions(+)
io_uring/io-wq.c +0 -0
index b2715988791e..b70eebec2845 100644

io_uring/io-wq.c static int io_wq_worker(void *data) · line 640

Quoted 7 lines
io_uring/io-wq.c +4 -0
@@ -634,6 +634,10 @@ static int io_wq_worker(void *data)
 
  			if (!get_signal(&ksig))
  				continue;
+
+			raw_spin_lock(&wq->lock);
+			acct->nr_workers--;
+			raw_spin_unlock(&wq->lock);

Wouldn't it suffer the same race you fixed with the following?

commit 767a65e9f31789d80e41edd03a802314905e8fbf

Author: Hao Xu <[email protected]>
Date:   Sun Sep 12 03:40:52 2021 +0800
    io-wq: fix potential race of acct->nr_workers

Even more, seems we fail to decrement nr_workers when the loop condition fails, i.e.

	while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) {

I.e. the patch looks legit, but what we currently have is a mess and we have more work to do.

Signature · 1 line
Pavel Begunkov
author
#3 lore raw

PATCH 2/11 io-wq: add a new worker flag to indicate worker exit

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add a new worker flag IO_WORKER_F_EXIT to indicate a worker is going to exit. This is important for fixed workers.

---
 io_uring/io-wq.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
io_uring/io-wq.c +13 -1
index b70eebec2845..1717f1465613 100644
@@ -29,6 +29,7 @@ enum {
 	IO_WORKER_F_RUNNING	= 2,	/* account as running */
 	IO_WORKER_F_FREE	= 4,	/* worker on free list */
 	IO_WORKER_F_BOUND	= 8,	/* is doing bounded work */
+	IO_WORKER_F_EXIT	= 16,	/* worker is exiting */
 };
 
 enum {
@@ -592,6 +593,11 @@ static void io_worker_handle_work(struct io_worker *worker)
 	} while (1);
 }
 
+static bool is_worker_exiting(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_EXIT;
+}
+
 static int io_wq_worker(void *data)
 {
 	struct io_worker *worker = data;
@@ -609,7 +615,7 @@ static int io_wq_worker(void *data)
 		long ret;
 
 		set_current_state(TASK_INTERRUPTIBLE);
-		while (io_acct_run_queue(acct))
+		while (!is_worker_exiting(worker) && io_acct_run_queue(acct))
 			io_worker_handle_work(worker);
 
 		raw_spin_lock(&wq->lock);
@@ -628,6 +634,12 @@ static int io_wq_worker(void *data)
 		raw_spin_unlock(&wq->lock);
 		if (io_run_task_work())
 			continue;
+		if (is_worker_exiting(worker)) {
+			raw_spin_lock(&wq->lock);
+			acct->nr_workers--;
+			raw_spin_unlock(&wq->lock);
+			break;
+		}
 		ret = schedule_timeout(WORKER_IDLE_TIMEOUT);
 		if (signal_pending(current)) {
 			struct ksignal ksig;
Signature · 1 line
2.25.1
#4 lore raw

Re: [PATCH 02/11] io-wq: add a new worker flag to indicate worker exit

Quoted 4 lines · Hao Xu

From: Hao Xu <[email protected]>

Add a new worker flag IO_WORKER_F_EXIT to indicate a worker is going to exit. This is important for fixed workers.

nit: would be nice to add a small sentence _how_ it's important for fixed workers

Quoted 30 lines
---
  io_uring/io-wq.c | 14 +++++++++++++-
  1 file changed, 13 insertions(+), 1 deletion(-)
io_uring/io-wq.c +6 -0
index b70eebec2845..1717f1465613 100644
@@ -29,6 +29,7 @@ enum {
  	IO_WORKER_F_RUNNING	= 2,	/* account as running */
  	IO_WORKER_F_FREE	= 4,	/* worker on free list */
  	IO_WORKER_F_BOUND	= 8,	/* is doing bounded work */
+	IO_WORKER_F_EXIT	= 16,	/* worker is exiting */
  };
 
  enum {
@@ -592,6 +593,11 @@ static void io_worker_handle_work(struct io_worker *worker)
  	} while (1);
  }
 
+static bool is_worker_exiting(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_EXIT;
+}
+
  static int io_wq_worker(void *data)
  {
  	struct io_worker *worker = data;

io_uring/io-wq.c static int io_wq_worker(void *data) · line 619

Quoted 6 lines
io_uring/io-wq.c +1 -1
@@ -609,7 +615,7 @@ static int io_wq_worker(void *data)
  		long ret;
 
  		set_current_state(TASK_INTERRUPTIBLE);
-		while (io_acct_run_queue(acct))
+		while (!is_worker_exiting(worker) && io_acct_run_queue(acct))
  			io_worker_handle_work(worker);

Why it differs from the condition in io_wq_dec_running()? Would sth like this work?

bool io_worker_run_queue(worker) {
        return !is_worker_exiting(worker) &&
                io_acct_run_queue(worker_get_acct(worker));	
}
Quoted 15 lines
io_uring/io-wq.c +6 -0
  		raw_spin_lock(&wq->lock);
@@ -628,6 +634,12 @@ static int io_wq_worker(void *data)
  		raw_spin_unlock(&wq->lock);
  		if (io_run_task_work())
  			continue;
+		if (is_worker_exiting(worker)) {
+			raw_spin_lock(&wq->lock);
+			acct->nr_workers--;
+			raw_spin_unlock(&wq->lock);
+			break;
+		}
  		ret = schedule_timeout(WORKER_IDLE_TIMEOUT);
  		if (signal_pending(current)) {
  			struct ksignal ksig;
Signature · 1 line
Pavel Begunkov
author
#5 lore raw

PATCH 3/11 io-wq: add a new type io-wq worker

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add a new type io-wq worker IO_WORKER_F_FIXED, this type of worker exists during the whole io-wq lifecycle.

---
 io_uring/io-wq.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
io_uring/io-wq.c +12 -1
index 1717f1465613..7326fef58ca7 100644
@@ -30,6 +30,7 @@ enum {
 	IO_WORKER_F_FREE	= 4,	/* worker on free list */
 	IO_WORKER_F_BOUND	= 8,	/* is doing bounded work */
 	IO_WORKER_F_EXIT	= 16,	/* worker is exiting */
+	IO_WORKER_F_FIXED	= 32,	/* is a fixed worker */
 };
 
 enum {
@@ -598,6 +599,11 @@ static bool is_worker_exiting(struct io_worker *worker)
 	return worker->flags & IO_WORKER_F_EXIT;
 }
 
+static bool is_fixed_worker(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_FIXED;
+}
+
 static int io_wq_worker(void *data)
 {
 	struct io_worker *worker = data;
@@ -622,8 +628,13 @@ static int io_wq_worker(void *data)
 		/*
 		 * Last sleep timed out. Exit if we're not the last worker,
 		 * or if someone modified our affinity.
+		 * Note: fixed worker always have same lifecycle as io-wq
+		 * itself, and cpu affinity setting doesn't work well for
+		 * fixed worker, they can be manually reset to cpu other than
+		 * the cpuset indicated by io_wq_worker_affinity()
 		 */
-		if (last_timeout && (exit_mask || acct->nr_workers > 1)) {
+		if (!is_fixed_worker(worker) && last_timeout &&
+		    (exit_mask || acct->nr_workers > 1)) {
 			acct->nr_workers--;
 			raw_spin_unlock(&wq->lock);
 			__set_current_state(TASK_RUNNING);
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH 03/11] io-wq: add a new type io-wq worker

Quoted 23 lines · Hao Xu

From: Hao Xu <[email protected]>

Add a new type io-wq worker IO_WORKER_F_FIXED, this type of worker exists during the whole io-wq lifecycle.

---
  io_uring/io-wq.c | 13 ++++++++++++-
  1 file changed, 12 insertions(+), 1 deletion(-)
io_uring/io-wq.c +1 -0
index 1717f1465613..7326fef58ca7 100644
@@ -30,6 +30,7 @@ enum {
  	IO_WORKER_F_FREE	= 4,	/* worker on free list */
  	IO_WORKER_F_BOUND	= 8,	/* is doing bounded work */
  	IO_WORKER_F_EXIT	= 16,	/* worker is exiting */
+	IO_WORKER_F_FIXED	= 32,	/* is a fixed worker */
  };
 
  enum {

io_uring/io-wq.c static bool is_worker_exiting(struct io_worker *worker) · line 605

Quoted 7 lines
io_uring/io-wq.c +4 -0
@@ -598,6 +599,11 @@ static bool is_worker_exiting(struct io_worker *worker)
  	return worker->flags & IO_WORKER_F_EXIT;
  }
 
+static bool is_fixed_worker(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_FIXED;
+}

You move it up in Patch 5/11, I suggest to move it to the top of the file here.

Quoted 8 lines
io_uring/io-wq.c +1 -0
  static int io_wq_worker(void *data)
  {
  	struct io_worker *worker = data;
@@ -622,8 +628,13 @@ static int io_wq_worker(void *data)
  		/*
  		 * Last sleep timed out. Exit if we're not the last worker,
  		 * or if someone modified our affinity.
+		 * Note: fixed worker always have same lifecycle as io-wq

io_uring/io-wq.c static int io_wq_worker(void *data) · line 640

Quoted 10 lines
io_uring/io-wq.c +5 -1
+		 * itself, and cpu affinity setting doesn't work well for
+		 * fixed worker, they can be manually reset to cpu other than
+		 * the cpuset indicated by io_wq_worker_affinity()
  		 */
-		if (last_timeout && (exit_mask || acct->nr_workers > 1)) {
+		if (!is_fixed_worker(worker) && last_timeout &&
+		    (exit_mask || acct->nr_workers > 1)) {
  			acct->nr_workers--;
  			raw_spin_unlock(&wq->lock);
  			__set_current_state(TASK_RUNNING);

If there is no work it'll continue to loop every WORKER_IDLE_TIMEOUT (5 * HZ), which sounds troublesome with many workers in the system.

tm = is_fixed_worker(worker) ? MAX_SCHEDULE_TIMEOUT : WORKER_IDLE_TIMEOUT; schedule_timeout(tm);

Maybe?

Signature · 1 line
Pavel Begunkov
author
#7 lore raw

PATCH 4/11 io-wq: add fixed worker members in io_wq_acct

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add fixed worker related members in io_wq_acct.

---
 io_uring/io-wq.c | 2 ++
 1 file changed, 2 insertions(+)
io_uring/io-wq.c +2 -0
index 7326fef58ca7..bf9e9af8d9ca 100644
@@ -84,6 +84,8 @@ struct io_wq_acct {
 	raw_spinlock_t lock;
 	struct io_wq_work_list work_list;
 	unsigned long flags;
+	struct io_worker **fixed_workers;
+	unsigned int fixed_nr;
 };
 
 enum {
Signature · 1 line
2.25.1
author
#8 lore raw

PATCH 5/11 io-wq: add a new parameter for creating a new fixed worker

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add a new parameter when creating new workers to indicate if users want a normal or fixed worker.

---
 io_uring/io-wq.c | 33 ++++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 13 deletions(-)
io_uring/io-wq.c +20 -13
index bf9e9af8d9ca..048856eef4d4 100644
@@ -137,7 +137,7 @@ struct io_cb_cancel_data {
 	bool cancel_all;
 };
 
-static bool create_io_worker(struct io_wq *wq, int index);
+static bool create_io_worker(struct io_wq *wq, int index, bool fixed);
 static void io_wq_dec_running(struct io_worker *worker);
 static bool io_acct_cancel_pending_work(struct io_wq *wq,
 					struct io_wq_acct *acct,
@@ -284,7 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,
  * We need a worker. If we find a free one, we're good. If not, and we're
  * below the max number of workers, create one.
  */
-static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct)
+static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,
+				bool fixed)
 {
 	/*
 	 * Most likely an attempt to queue unbounded work on an io_wq that
@@ -302,7 +303,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct)
 	raw_spin_unlock(&wq->lock);
 	atomic_inc(&acct->nr_running);
 	atomic_inc(&wq->worker_refs);
-	return create_io_worker(wq, acct->index);
+	return create_io_worker(wq, acct->index, fixed);
 }
 
 static void io_wq_inc_running(struct io_worker *worker)
@@ -312,6 +313,11 @@ static void io_wq_inc_running(struct io_worker *worker)
 	atomic_inc(&acct->nr_running);
 }
 
+static bool is_fixed_worker(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_FIXED;
+}
+
 static void create_worker_cb(struct callback_head *cb)
 {
 	struct io_worker *worker;
@@ -331,7 +337,7 @@ static void create_worker_cb(struct callback_head *cb)
 	}
 	raw_spin_unlock(&wq->lock);
 	if (do_create) {
-		create_io_worker(wq, worker->create_index);
+		create_io_worker(wq, worker->create_index, is_fixed_worker(worker));
 	} else {
 		atomic_dec(&acct->nr_running);
 		io_worker_ref_put(wq);
@@ -398,6 +404,8 @@ static void io_wq_dec_running(struct io_worker *worker)
 		return;
 	if (!io_acct_run_queue(acct))
 		return;
+	if (is_fixed_worker(worker))
+		return;
 
 	atomic_inc(&acct->nr_running);
 	atomic_inc(&wq->worker_refs);
@@ -601,11 +609,6 @@ static bool is_worker_exiting(struct io_worker *worker)
 	return worker->flags & IO_WORKER_F_EXIT;
 }
 
-static bool is_fixed_worker(struct io_worker *worker)
-{
-	return worker->flags & IO_WORKER_F_FIXED;
-}
-
 static int io_wq_worker(void *data)
 {
 	struct io_worker *worker = data;
@@ -806,7 +809,7 @@ static void io_workqueue_create(struct work_struct *work)
 		kfree(worker);
 }
 
-static bool create_io_worker(struct io_wq *wq, int index)
+static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
 {
 	struct io_wq_acct *acct = &wq->acct[index];
 	struct io_worker *worker;
@@ -833,10 +836,14 @@ static bool create_io_worker(struct io_wq *wq, int index)
 	if (index == IO_WQ_ACCT_BOUND)
 		worker->flags |= IO_WORKER_F_BOUND;
 
+	if (fixed)
+		worker->flags |= IO_WORKER_F_FIXED;
+
 	tsk = create_io_thread(io_wq_worker, worker, NUMA_NO_NODE);
 	if (!IS_ERR(tsk)) {
-		io_init_new_worker(wq, worker, tsk);
-	} else if (!io_should_retry_thread(PTR_ERR(tsk))) {
+		if (!fixed)
+			io_init_new_worker(wq, worker, tsk);
+	} else if (fixed || !io_should_retry_thread(PTR_ERR(tsk))) {
 		kfree(worker);
 		goto fail;
 	} else {
@@ -947,7 +954,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work)
 	    !atomic_read(&acct->nr_running))) {
 		bool did_create;
 
-		did_create = io_wq_create_worker(wq, acct);
+		did_create = io_wq_create_worker(wq, acct, false);
 		if (likely(did_create))
 			return;
 
Signature · 1 line
2.25.1
#9 lore raw

Re: [PATCH 05/11] io-wq: add a new parameter for creating a new fixed worker

Quoted 14 lines · Hao Xu

From: Hao Xu <[email protected]>

Add a new parameter when creating new workers to indicate if users want a normal or fixed worker.

---
  io_uring/io-wq.c | 33 ++++++++++++++++++++-------------
  1 file changed, 20 insertions(+), 13 deletions(-)
io_uring/io-wq.c +0 -0
index bf9e9af8d9ca..048856eef4d4 100644

[...]

io_uring/io-wq.c static void io_wq_inc_running(struct io_worker *worker) · line 319

Quoted 5 lines
io_uring/io-wq.c +4 -0
+static bool is_fixed_worker(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_FIXED;
+} 

That's what I mentioned in the other comment.

Quoted 14 lines
io_uring/io-wq.c +2 -1
+
  static void create_worker_cb(struct callback_head *cb)
  {
  	struct io_worker *worker;
@@ -331,7 +337,7 @@ static void create_worker_cb(struct callback_head *cb)
  	}
  	raw_spin_unlock(&wq->lock);
  	if (do_create) {
-		create_io_worker(wq, worker->create_index);
+		create_io_worker(wq, worker->create_index, is_fixed_worker(worker));
  	} else {
  		atomic_dec(&acct->nr_running);
  		io_worker_ref_put(wq);

io_uring/io-wq.c static void io_wq_dec_running(struct io_worker *worker) · line 408

Quoted 5 lines
io_uring/io-wq.c +2 -0
@@ -398,6 +404,8 @@ static void io_wq_dec_running(struct io_worker *worker)
  		return;
  	if (!io_acct_run_queue(acct))
  		return;
+	if (is_fixed_worker(worker))
+		return;

Aha, it's here. I was thinking about it a little bit more. Is it even correct? If you have a mixed fixed/non-fixed setup you presumably want non-fixed workers to kick in such situations. I don't remember this creation voodoo well, maybe Jens does have an idea.

Quoted 6 lines
io_uring/io-wq.c +0 -0
  	atomic_inc(&acct->nr_running);
  	atomic_inc(&wq->worker_refs);
@@ -601,11 +609,6 @@ static bool is_worker_exiting(struct io_worker *worker)
  	return worker->flags & IO_WORKER_F_EXIT;
  }

[...]

Quoted 8 lines
io_uring/io-wq.c +1 -1
-static bool create_io_worker(struct io_wq *wq, int index)
+static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
  {
  	struct io_wq_acct *acct = &wq->acct[index];
  	struct io_worker *worker;
@@ -833,10 +836,14 @@ static bool create_io_worker(struct io_wq *wq, int index)
  	if (index == IO_WQ_ACCT_BOUND)
  		worker->flags |= IO_WORKER_F_BOUND;

io_uring/io-wq.c static bool create_io_worker(struct io_wq *wq, int index) · line 845

Quoted 10 lines
io_uring/io-wq.c +5 -2
 
+	if (fixed)
+		worker->flags |= IO_WORKER_F_FIXED;
+
  	tsk = create_io_thread(io_wq_worker, worker, NUMA_NO_NODE);
  	if (!IS_ERR(tsk)) {
-		io_init_new_worker(wq, worker, tsk);
-	} else if (!io_should_retry_thread(PTR_ERR(tsk))) {
+		if (!fixed)
+			io_init_new_worker(wq, worker, tsk);

Why do we skip io_init_new_worker()? I assume you putting it into lists, but what about the rest? I.e.

	tsk->worker_private = worker;
	worker->task = tsk;
	set_cpus_allowed_ptr(tsk, wq->cpu_mask);
Quoted 12 lines
io_uring/io-wq.c +2 -1
+	} else if (fixed || !io_should_retry_thread(PTR_ERR(tsk))) {
  		kfree(worker);
  		goto fail;
  	} else {
@@ -947,7 +954,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work)
  	    !atomic_read(&acct->nr_running))) {
  		bool did_create;
 
-		did_create = io_wq_create_worker(wq, acct);
+		did_create = io_wq_create_worker(wq, acct, false);
  		if (likely(did_create))
  			return;
Signature · 1 line
Pavel Begunkov
author
#10 lore raw

PATCH 6/11 io-wq: return io_worker after successful inline worker creation

In reply to Hao Xu

From: Hao Xu <[email protected]>

After creating a io worker inline successfully, return the io_worker structure. This is used by fixed worker.

---
 io_uring/io-wq.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
io_uring/io-wq.c +10 -10
index 048856eef4d4..4338e5b23b07 100644
@@ -137,7 +137,7 @@ struct io_cb_cancel_data {
 	bool cancel_all;
 };
 
-static bool create_io_worker(struct io_wq *wq, int index, bool fixed);
+static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixed);
 static void io_wq_dec_running(struct io_worker *worker);
 static bool io_acct_cancel_pending_work(struct io_wq *wq,
 					struct io_wq_acct *acct,
@@ -284,8 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,
  * We need a worker. If we find a free one, we're good. If not, and we're
  * below the max number of workers, create one.
  */
-static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,
-				bool fixed)
+static struct io_worker *io_wq_create_worker(struct io_wq *wq,
+					     struct io_wq_acct *acct, bool fixed)
 {
 	/*
 	 * Most likely an attempt to queue unbounded work on an io_wq that
@@ -297,7 +297,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,
 	raw_spin_lock(&wq->lock);
 	if (acct->nr_workers >= acct->max_workers) {
 		raw_spin_unlock(&wq->lock);
-		return true;
+		return NULL;
 	}
 	acct->nr_workers++;
 	raw_spin_unlock(&wq->lock);
@@ -809,11 +809,11 @@ static void io_workqueue_create(struct work_struct *work)
 		kfree(worker);
 }
 
-static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
+static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixed)
 {
 	struct io_wq_acct *acct = &wq->acct[index];
 	struct io_worker *worker;
-	struct task_struct *tsk;
+	struct task_struct *tsk = NULL;
 
 	__set_current_state(TASK_RUNNING);
 
@@ -825,7 +825,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
 		acct->nr_workers--;
 		raw_spin_unlock(&wq->lock);
 		io_worker_ref_put(wq);
-		return false;
+		return tsk ? (struct io_worker *)tsk : ERR_PTR(-ENOMEM);
 	}
 
 	refcount_set(&worker->ref, 1);
@@ -841,8 +841,8 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
 
 	tsk = create_io_thread(io_wq_worker, worker, NUMA_NO_NODE);
 	if (!IS_ERR(tsk)) {
-		if (!fixed)
-			io_init_new_worker(wq, worker, tsk);
+		io_init_new_worker(wq, worker, tsk);
+		return worker;
 	} else if (fixed || !io_should_retry_thread(PTR_ERR(tsk))) {
 		kfree(worker);
 		goto fail;
@@ -851,7 +851,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
 		schedule_work(&worker->work);
 	}
 
-	return true;
+	return (struct io_worker *)tsk;
 }
 
 /*
Signature · 1 line
2.25.1
#11 lore raw

Re: [PATCH 06/11] io-wq: return io_worker after successful inline worker creation

Quoted 35 lines · Hao Xu

From: Hao Xu <[email protected]>

After creating a io worker inline successfully, return the io_worker structure. This is used by fixed worker.

---
  io_uring/io-wq.c | 20 ++++++++++----------
  1 file changed, 10 insertions(+), 10 deletions(-)
io_uring/io-wq.c +3 -3
index 048856eef4d4..4338e5b23b07 100644
@@ -137,7 +137,7 @@ struct io_cb_cancel_data {
  	bool cancel_all;
  };
 
-static bool create_io_worker(struct io_wq *wq, int index, bool fixed);
+static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixed);
  static void io_wq_dec_running(struct io_worker *worker);
  static bool io_acct_cancel_pending_work(struct io_wq *wq,
  					struct io_wq_acct *acct,
@@ -284,8 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,
   * We need a worker. If we find a free one, we're good. If not, and we're
   * below the max number of workers, create one.
   */
-static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,
-				bool fixed)
+static struct io_worker *io_wq_create_worker(struct io_wq *wq,
+					     struct io_wq_acct *acct, bool fixed)
  {
  	/*
  	 * Most likely an attempt to queue unbounded work on an io_wq that

io_uring/io-wq.c static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct, · line 300

Quoted 5 lines
io_uring/io-wq.c +1 -1
@@ -297,7 +297,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct,
  	raw_spin_lock(&wq->lock);
  	if (acct->nr_workers >= acct->max_workers) {
  		raw_spin_unlock(&wq->lock);
-		return true;
+		return NULL;

Something is not right here. The function could succeed even if it didn't create a new worker. Now it's a failure.

Quoted 18 lines
io_uring/io-wq.c +2 -2
  	}
  	acct->nr_workers++;
  	raw_spin_unlock(&wq->lock);
@@ -809,11 +809,11 @@ static void io_workqueue_create(struct work_struct *work)
  		kfree(worker);
  }
 
-static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
+static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixed)
  {
  	struct io_wq_acct *acct = &wq->acct[index];
  	struct io_worker *worker;
-	struct task_struct *tsk;
+	struct task_struct *tsk = NULL;
 
  	__set_current_state(TASK_RUNNING);
 

io_uring/io-wq.c static bool create_io_worker(struct io_wq *wq, int index, bool fixed) · line 828

Quoted 5 lines
io_uring/io-wq.c +1 -1
@@ -825,7 +825,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
  		acct->nr_workers--;
  		raw_spin_unlock(&wq->lock);
  		io_worker_ref_put(wq);
-		return false;
+		return tsk ? (struct io_worker *)tsk : ERR_PTR(-ENOMEM);

How it this conversion valid? I don't remember us overlaying struct io_worker onto task_struct

Quoted 23 lines
io_uring/io-wq.c +3 -3
  	}
 
  	refcount_set(&worker->ref, 1);
@@ -841,8 +841,8 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
 
  	tsk = create_io_thread(io_wq_worker, worker, NUMA_NO_NODE);
  	if (!IS_ERR(tsk)) {
-		if (!fixed)
-			io_init_new_worker(wq, worker, tsk);
+		io_init_new_worker(wq, worker, tsk);
+		return worker;
  	} else if (fixed || !io_should_retry_thread(PTR_ERR(tsk))) {
  		kfree(worker);
  		goto fail;
@@ -851,7 +851,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed)
  		schedule_work(&worker->work);
  	}
 
-	return true;
+	return (struct io_worker *)tsk;
  }
 
  /*
Signature · 1 line
Pavel Begunkov
author
#12 lore raw

PATCH 7/11 io_uring: add new api to register fixed workers

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add a new api to register fixed workers. The api is designed to register fixed workers for the current task. For simplicity, it doesn't allow worker number update. We have a separate unregister api to uninstall all the fixed workers. And then we can register different number of fixed workers again.

---
 include/uapi/linux/io_uring.h |  9 ++++
 io_uring/io-wq.c              | 85 +++++++++++++++++++++++++++++++++++
 io_uring/io-wq.h              |  1 +
 io_uring/io_uring.c           | 71 +++++++++++++++++++++++++++++
 4 files changed, 166 insertions(+)
include/uapi/linux/io_uring.h +9 -0
index f222d263bc55..6dc43be5009d 100644
@@ -535,6 +535,9 @@ enum {
 	/* register a range of fixed file slots for automatic slot allocation */
 	IORING_REGISTER_FILE_ALLOC_RANGE	= 25,
 
+	/* set/get number of fixed workers */
+	IORING_REGISTER_IOWQ_FIXED_WORKERS      = 26,
+
 	/* this goes last */
 	IORING_REGISTER_LAST,
 
@@ -715,6 +718,12 @@ struct io_uring_recvmsg_out {
 	__u32 flags;
 };
 
+struct io_uring_fixed_worker_arg {
+	__u32	nr_workers;
+	__u32	resv;
+	__u64	resv2[3];
+};
+
 #ifdef __cplusplus
 }
 #endif
io_uring/io-wq.c +85 -0

91 lines not on this page · raw message

io_uring/io-wq.h +1 -0
index 31228426d192..88a1ee9fde24 100644
@@ -52,6 +52,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val);
 
 int io_wq_cpu_affinity(struct io_wq *wq, cpumask_var_t mask);
 int io_wq_max_workers(struct io_wq *wq, int *new_count);
+int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *count);
 
 static inline bool io_wq_is_hashed(struct io_wq_work *work)
 {
io_uring/io_uring.c +71 -0

83 lines not on this page · raw message

Signature · 1 line
2.25.1
#13 lore raw

Re: [PATCH 07/11] io_uring: add new api to register fixed workers

Quoted 17 lines · Hao Xu
io_uring/io-wq.c +17 -0
+	rcu_read_lock();
+
+	for (i = 0; i < IO_WQ_ACCT_NR; i++) {
+		unsigned int nr = count[i].nr_workers;
+
+		acct = &wq->acct[i];
+		acct->fixed_nr = nr;
+		acct->fixed_workers = kcalloc(nr, sizeof(struct io_worker *),
+					      GFP_KERNEL);
+		if (!acct->fixed_workers) {
+			ret = -ENOMEM;
+			break;
+		}
+
+		for (j = 0; j < nr; j++) {
+			struct io_worker *worker =
+				io_wq_create_worker(wq, acct, true);

io_uring/io-wq.c int io_wq_max_workers(struct io_wq *wq, int *new_count) · line 1448

Quoted 10 lines
io_uring/io-wq.c +10 -0
+			if (IS_ERR(worker)) {
+				ret = PTR_ERR(worker);
+				break;
+			}
+			acct->fixed_workers[j] = worker;
+		}
+		if (j < nr)
+			break;
+	}
+	rcu_read_unlock();

This looks wrong. kcalloc() with GFP_KERNEL may sleep. Note that you're not allowed to sleep inside the RCU read lock critical section.

Using GFP_KERNEL implies GFP_RECLAIM, which means that direct reclaim may be triggered under memory pressure; the calling context must be allowed to sleep.

Signature · 1 line
Ammar Faizi
author
#14 lore raw

Re: [PATCH 07/11] io_uring: add new api to register fixed workers

Hi Ammar,

Quoted 32 lines · Ammar Faizi
Quoted 17 lines · Hao Xu
io_uring/io-wq.c +17 -0
+	rcu_read_lock();
+
+	for (i = 0; i < IO_WQ_ACCT_NR; i++) {
+		unsigned int nr = count[i].nr_workers;
+
+		acct = &wq->acct[i];
+		acct->fixed_nr = nr;
+		acct->fixed_workers = kcalloc(nr, sizeof(struct io_worker *),
+					      GFP_KERNEL);
+		if (!acct->fixed_workers) {
+			ret = -ENOMEM;
+			break;
+		}
+
+		for (j = 0; j < nr; j++) {
+			struct io_worker *worker =
+				io_wq_create_worker(wq, acct, true);

io_uring/io-wq.c int io_wq_max_workers(struct io_wq *wq, int *new_count) · line 1448

Quoted 10 lines
io_uring/io-wq.c +10 -0
+			if (IS_ERR(worker)) {
+				ret = PTR_ERR(worker);
+				break;
+			}
+			acct->fixed_workers[j] = worker;
+		}
+		if (j < nr)
+			break;
+	}
+	rcu_read_unlock();

This looks wrong. kcalloc() with GFP_KERNEL may sleep. Note that you're not allowed to sleep inside the RCU read lock critical section.

Quoted 4 lines

Using GFP_KERNEL implies GFP_RECLAIM, which means that direct reclaim may be triggered under memory pressure; the calling context must be allowed to sleep.

I think you are right, I'll fix it in v2.

Hi Jens, ask a question about this: I saw same rcu_read_lock() in io_wq_max_workers(), but what is it really protect?

Regards,
Hao

#15 lore raw

Re: [PATCH 07/11] io_uring: add new api to register fixed workers

In reply to Hao Xu
Quoted 18 lines · Hao Xu
io_uring/io_uring.c +18 -0
+static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,
+					       void __user *arg, int nr_args)
+	__must_hold(&ctx->uring_lock)
+{
+	struct io_uring_task *tctx = NULL;
+	struct io_sq_data *sqd = NULL;
+	struct io_uring_fixed_worker_arg *res;
+	size_t size;
+	int i, ret;
+	bool zero = true;
+
+	size = array_size(nr_args, sizeof(*res));
+	if (size == SIZE_MAX)
+		return -EOVERFLOW;
+
+	res = memdup_user(arg, size);
+	if (IS_ERR(res))
+		return PTR_ERR(res);

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4385

Quoted 10 lines
io_uring/io_uring.c +10 -0
+
+	for (i = 0; i < nr_args; i++) {
+		if (res[i].nr_workers) {
+			zero = false;
+			break;
+		}
+	}
+
+	if (zero)
+		return 0;

You have a memory leak bug here. The memdup_user() needs clean up. kfree(res);

Signature · 1 line
Ammar Faizi
author
#16 lore raw

Re: [PATCH 07/11] io_uring: add new api to register fixed workers

Quoted 24 lines · Ammar Faizi
Quoted 23 lines · Hao Xu
io_uring/io_uring.c +23 -0
+static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,
+					       void __user *arg, int nr_args)
+	__must_hold(&ctx->uring_lock)
+{
+	struct io_uring_task *tctx = NULL;
+	struct io_sq_data *sqd = NULL;
+	struct io_uring_fixed_worker_arg *res;
+	size_t size;
+	int i, ret;
+	bool zero = true;
+
+	size = array_size(nr_args, sizeof(*res));
+	if (size == SIZE_MAX)
+		return -EOVERFLOW;
+
+	res = memdup_user(arg, size);
+	if (IS_ERR(res))
+		return PTR_ERR(res);
+
+	for (i = 0; i < nr_args; i++) {
+		if (res[i].nr_workers) {
+			zero = false;
+			break;
Quoted 9 lines

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4385

Quoted 5 lines
io_uring/io_uring.c +5 -0
+		}
+	}
+
+	if (zero)
+		return 0;

You have a memory leak bug here. The memdup_user() needs clean up. kfree(res);

True, I'll fix it in v2, thanks.

#17 lore raw

Re: [PATCH 07/11] io_uring: add new api to register fixed workers

In reply to Hao Xu
Quoted 20 lines · Hao Xu

From: Hao Xu <[email protected]>

Add a new api to register fixed workers. The api is designed to register fixed workers for the current task. For simplicity, it doesn't allow worker number update. We have a separate unregister api to uninstall all the fixed workers. And then we can register different number of fixed workers again.

---
  include/uapi/linux/io_uring.h |  9 ++++
  io_uring/io-wq.c              | 85 +++++++++++++++++++++++++++++++++++
  io_uring/io-wq.h              |  1 +
  io_uring/io_uring.c           | 71 +++++++++++++++++++++++++++++
  4 files changed, 166 insertions(+)
include/uapi/linux/io_uring.h +0 -0
index f222d263bc55..6dc43be5009d 100644

[...]

Quoted 16 lines
io_uring/io_uring.c +8 -0
index c99a7a0c3f21..bb8342b4a2c6 100644
@@ -4351,6 +4351,71 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx,
  	return ret;
  }
 
+/*
+ * note: this function sets fixed workers for a single task, so every
+ * task which wants to set the fixed workers has to call this function
+ */
+static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,
+					       void __user *arg, int nr_args)
+	__must_hold(&ctx->uring_lock)
+{

io_uring/io_uring.c static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, · line 4371

Quoted 10 lines
io_uring/io_uring.c +10 -0
+	struct io_uring_task *tctx = NULL;
+	struct io_sq_data *sqd = NULL;
+	struct io_uring_fixed_worker_arg *res;
+	size_t size;
+	int i, ret;
+	bool zero = true;
+
+	size = array_size(nr_args, sizeof(*res));
+	if (size == SIZE_MAX)
+		return -EOVERFLOW;

I don't think the number of accounting classes is going to change, just move nr_args check from below here and have on-stack array of size 2.

struct io_uring_fixed_worker_arg res[IO_WQ_ACCT_NR];

if (nr_args != IO_WQ_ACCT_NR)

	return -EINVAL;

...

Quoted 12 lines
io_uring/io_uring.c +12 -0
+
+	res = memdup_user(arg, size);
+	if (IS_ERR(res))
+		return PTR_ERR(res);
+
+	for (i = 0; i < nr_args; i++) {
+		if (res[i].nr_workers) {
+			zero = false;
+			break;
+		}
+	}
+
Signature · 1 line
Pavel Begunkov
author
#18 lore raw

PATCH 8/11 io_uring: add function to unregister fixed workers

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add a new register api to unregister fixed workers.

---
 include/uapi/linux/io_uring.h |  3 +++
 io_uring/io-wq.c              | 50 ++++++++++++++++++++++++++++++++++-
 io_uring/io-wq.h              |  1 +
 io_uring/io_uring.c           | 45 +++++++++++++++++++++++++++++++
 4 files changed, 98 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +3 -0
index 6dc43be5009d..b0a6e3106b42 100644
@@ -538,6 +538,9 @@ enum {
 	/* set/get number of fixed workers */
 	IORING_REGISTER_IOWQ_FIXED_WORKERS      = 26,
 
+	/* destroy fixed workers */
+	IORING_UNREGISTER_IOWQ_FIXED_WORKERS      = 27,
+
 	/* this goes last */
 	IORING_REGISTER_LAST,
 
io_uring/io-wq.c +49 -1

62 lines not on this page · raw message

io_uring/io-wq.h +1 -0
index 88a1ee9fde24..15e93af36511 100644
@@ -53,6 +53,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val);
 int io_wq_cpu_affinity(struct io_wq *wq, cpumask_var_t mask);
 int io_wq_max_workers(struct io_wq *wq, int *new_count);
 int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *count);
+int io_wq_destroy_fixed_workers(struct io_wq *wq);
 
 static inline bool io_wq_is_hashed(struct io_wq_work *work)
 {
io_uring/io_uring.c +45 -0

57 lines not on this page · raw message

Signature · 1 line
2.25.1
#19 lore raw

Re: [PATCH 08/11] io_uring: add function to unregister fixed workers

Quoted 17 lines · Hao Xu

From: Hao Xu <[email protected]>

Add a new register api to unregister fixed workers.

---
  include/uapi/linux/io_uring.h |  3 +++
  io_uring/io-wq.c              | 50 ++++++++++++++++++++++++++++++++++-
  io_uring/io-wq.h              |  1 +
  io_uring/io_uring.c           | 45 +++++++++++++++++++++++++++++++
  4 files changed, 98 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +0 -0
index 6dc43be5009d..b0a6e3106b42 100644

include/uapi/linux/io_uring.h enum { · line 542

Quoted 5 lines
include/uapi/linux/io_uring.h +2 -0
@@ -538,6 +538,9 @@ enum {
  	/* set/get number of fixed workers */
  	IORING_REGISTER_IOWQ_FIXED_WORKERS      = 26,
 
+	/* destroy fixed workers */
+	IORING_UNREGISTER_IOWQ_FIXED_WORKERS      = 27,

Do we need a new code? I think it's cleaner if we use IORING_REGISTER_IOWQ_FIXED_WORKERS and do sth like

struct io_uring_fixed_worker_arg arg;

if (arg.nr_workers)

	do_unregister_fixed_workers();

...

Signature · 1 line
Pavel Begunkov
author
#20 lore raw

PATCH 9/11 io-wq: add strutures to allow to wait fixed workers exit

In reply to Hao Xu

From: Hao Xu <[email protected]>

When unregister fixed workers, there should be a way to allow us to wait all the fixed workers exit.

---
 io_uring/io-wq.c | 72 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 52 insertions(+), 20 deletions(-)
io_uring/io-wq.c +52 -20
index f39e6b931d17..61cf6da2c72f 100644
@@ -108,6 +108,10 @@ struct io_wq {
 	atomic_t worker_refs;
 	struct completion worker_done;
 
+	atomic_t fixed_worker_refs;
+	struct completion fixed_worker_done;
+	bool fixed_comp_init;
+
 	struct hlist_node cpuhp_node;
 
 	struct task_struct *task;
@@ -172,10 +176,25 @@ static inline struct io_wq_acct *io_wq_get_acct(struct io_worker *worker)
 	return io_get_acct(worker->wq, worker->flags & IO_WORKER_F_BOUND);
 }
 
-static void io_worker_ref_put(struct io_wq *wq)
+static void io_worker_ref_get(struct io_wq *wq, bool fixed)
+{
+	atomic_inc(&wq->worker_refs);
+	if (fixed)
+		atomic_inc(&wq->fixed_worker_refs);
+}
+
+static void io_worker_ref_put(struct io_wq *wq, bool fixed)
 {
 	if (atomic_dec_and_test(&wq->worker_refs))
 		complete(&wq->worker_done);
+
+	if (fixed && atomic_dec_and_test(&wq->fixed_worker_refs))
+		complete(&wq->fixed_worker_done);
+}
+
+static bool is_fixed_worker(struct io_worker *worker)
+{
+	return worker->flags & IO_WORKER_F_FIXED;
 }
 
 static void io_worker_cancel_cb(struct io_worker *worker)
@@ -187,7 +206,7 @@ static void io_worker_cancel_cb(struct io_worker *worker)
 	raw_spin_lock(&wq->lock);
 	acct->nr_workers--;
 	raw_spin_unlock(&wq->lock);
-	io_worker_ref_put(wq);
+	io_worker_ref_put(wq, is_fixed_worker(worker));
 	clear_bit_unlock(0, &worker->create_state);
 	io_worker_release(worker);
 }
@@ -205,6 +224,7 @@ static bool io_task_worker_match(struct callback_head *cb, void *data)
 static void io_worker_exit(struct io_worker *worker)
 {
 	struct io_wq *wq = worker->wq;
+	bool fixed = is_fixed_worker(worker);
 
 	while (1) {
 		struct callback_head *cb = task_work_cancel_match(wq->task,
@@ -230,7 +250,7 @@ static void io_worker_exit(struct io_worker *worker)
 	preempt_enable();
 
 	kfree_rcu(worker, rcu);
-	io_worker_ref_put(wq);
+	io_worker_ref_put(wq, fixed);
 	do_exit(0);
 }
 
@@ -302,7 +322,7 @@ static struct io_worker *io_wq_create_worker(struct io_wq *wq,
 	acct->nr_workers++;
 	raw_spin_unlock(&wq->lock);
 	atomic_inc(&acct->nr_running);
-	atomic_inc(&wq->worker_refs);
+	io_worker_ref_get(wq, fixed);
 	return create_io_worker(wq, acct->index, fixed);
 }
 
@@ -313,11 +333,6 @@ static void io_wq_inc_running(struct io_worker *worker)
 	atomic_inc(&acct->nr_running);
 }
 
-static bool is_fixed_worker(struct io_worker *worker)
-{
-	return worker->flags & IO_WORKER_F_FIXED;
-}
-
 static void create_worker_cb(struct callback_head *cb)
 {
 	struct io_worker *worker;
@@ -325,8 +340,10 @@ static void create_worker_cb(struct callback_head *cb)
 
 	struct io_wq_acct *acct;
 	bool do_create = false;
+	bool fixed;
 
 	worker = container_of(cb, struct io_worker, create_work);
+	fixed = is_fixed_worker(worker);
 	wq = worker->wq;
 	acct = &wq->acct[worker->create_index];
 	raw_spin_lock(&wq->lock);
@@ -337,10 +354,10 @@ static void create_worker_cb(struct callback_head *cb)
 	}
 	raw_spin_unlock(&wq->lock);
 	if (do_create) {
-		create_io_worker(wq, worker->create_index, is_fixed_worker(worker));
+		create_io_worker(wq, worker->create_index, fixed);
 	} else {
 		atomic_dec(&acct->nr_running);
-		io_worker_ref_put(wq);
+		io_worker_ref_put(wq, fixed);
 	}
 	clear_bit_unlock(0, &worker->create_state);
 	io_worker_release(worker);
@@ -351,6 +368,7 @@ static bool io_queue_worker_create(struct io_worker *worker,
 				   task_work_func_t func)
 {
 	struct io_wq *wq = worker->wq;
+	bool fixed = is_fixed_worker(worker);
 
 	/* raced with exit, just ignore create call */
 	if (test_bit(IO_WQ_BIT_EXIT, &wq->state))
@@ -367,7 +385,7 @@ static bool io_queue_worker_create(struct io_worker *worker,
 	    test_and_set_bit_lock(0, &worker->create_state))
 		goto fail_release;
 
-	atomic_inc(&wq->worker_refs);
+	io_worker_ref_get(wq, fixed);
 	init_task_work(&worker->create_work, func);
 	worker->create_index = acct->index;
 	if (!task_work_add(wq->task, &worker->create_work, TWA_SIGNAL)) {
@@ -379,16 +397,16 @@ static bool io_queue_worker_create(struct io_worker *worker,
 		 */
 		if (test_bit(IO_WQ_BIT_EXIT, &wq->state))
 			io_wq_cancel_tw_create(wq);
-		io_worker_ref_put(wq);
+		io_worker_ref_put(wq, fixed);
 		return true;
 	}
-	io_worker_ref_put(wq);
+	io_worker_ref_put(wq, fixed);
 	clear_bit_unlock(0, &worker->create_state);
 fail_release:
 	io_worker_release(worker);
 fail:
 	atomic_dec(&acct->nr_running);
-	io_worker_ref_put(wq);
+	io_worker_ref_put(wq, fixed);
 	return false;
 }
 
@@ -408,7 +426,7 @@ static void io_wq_dec_running(struct io_worker *worker)
 		return;
 
 	atomic_inc(&acct->nr_running);
-	atomic_inc(&wq->worker_refs);
+	io_worker_ref_get(wq, false);
 	io_queue_worker_create(worker, acct, create_worker_cb);
 }
 
@@ -790,7 +808,7 @@ static void create_worker_cont(struct callback_head *cb)
 		} else {
 			raw_spin_unlock(&wq->lock);
 		}
-		io_worker_ref_put(wq);
+		io_worker_ref_put(wq, is_fixed_worker(worker));
 		kfree(worker);
 		return;
 	}
@@ -824,7 +842,7 @@ static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixe
 		raw_spin_lock(&wq->lock);
 		acct->nr_workers--;
 		raw_spin_unlock(&wq->lock);
-		io_worker_ref_put(wq);
+		io_worker_ref_put(wq, fixed);
 		return tsk ? (struct io_worker *)tsk : ERR_PTR(-ENOMEM);
 	}
 
@@ -1243,7 +1261,7 @@ static void io_wq_exit_workers(struct io_wq *wq)
 	rcu_read_lock();
 	io_wq_for_each_worker(wq, io_wq_worker_wake, NULL);
 	rcu_read_unlock();
-	io_worker_ref_put(wq);
+	io_worker_ref_put(wq, false);
 	wait_for_completion(&wq->worker_done);
 
 	spin_lock_irq(&wq->hash->wait.lock);
@@ -1390,6 +1408,7 @@ static void io_wq_clean_fixed_workers(struct io_wq *wq)
 		}
 		kfree(workers);
 	}
+	wait_for_completion(&wq->fixed_worker_done);
 }
 
 /*
@@ -1421,6 +1440,13 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *coun
 
 	rcu_read_lock();
 
+	atomic_set(&wq->fixed_worker_refs, 1);
+	if (wq->fixed_comp_init) {
+		reinit_completion(&wq->fixed_worker_done);
+	} else {
+		init_completion(&wq->fixed_worker_done);
+		wq->fixed_comp_init = true;
+	}
 	for (i = 0; i < IO_WQ_ACCT_NR; i++) {
 		unsigned int nr = count[i].nr_workers;
 
@@ -1495,12 +1521,18 @@ int io_wq_destroy_fixed_workers(struct io_wq *wq)
 			workers[j]->flags |= IO_WORKER_F_EXIT;
 			wake_up_process(worker->task);
 		}
-		// wait for all workers exit
 		kfree(workers);
 	}
 	raw_spin_unlock(&wq->lock);
 	rcu_read_unlock();
 
+	// decrement the init reference
+	if (atomic_dec_and_test(&wq->fixed_worker_refs))
+		complete(&wq->fixed_worker_done);
+
+	wait_for_completion(&wq->fixed_worker_done);
+	wq->fixed_comp_init = false;
+
 	return 0;
 }
 
Signature · 1 line
2.25.1
author
#21 lore raw

PATCH 10/11 io-wq: distinguish fixed worker by its name

In reply to Hao Xu

From: Hao Xu <[email protected]>

Distinguish fixed workers and normal workers by their names.

---
 io_uring/io-wq.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
io_uring/io-wq.c +4 -2
index 61cf6da2c72f..7a9e5fa19b81 100644
@@ -634,10 +634,12 @@ static int io_wq_worker(void *data)
 	struct io_wq *wq = worker->wq;
 	bool exit_mask = false, last_timeout = false;
 	char buf[TASK_COMM_LEN];
+	bool fixed = is_fixed_worker(worker);
 
 	worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING);
 
-	snprintf(buf, sizeof(buf), "iou-wrk-%d", wq->task->pid);
+	snprintf(buf, sizeof(buf), fixed ? "iou-fixed-%d" : "iou-wrk-%d",
+		 wq->task->pid);
 	set_task_comm(current, buf);
 
 	while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) {
@@ -656,7 +658,7 @@ static int io_wq_worker(void *data)
 		 * fixed worker, they can be manually reset to cpu other than
 		 * the cpuset indicated by io_wq_worker_affinity()
 		 */
-		if (!is_fixed_worker(worker) && last_timeout &&
+		if (!fixed && last_timeout &&
 		    (exit_mask || acct->nr_workers > 1)) {
 			acct->nr_workers--;
 			raw_spin_unlock(&wq->lock);
Signature · 1 line
2.25.1
#22 lore raw

Re: [PATCH 10/11] io-wq: distinguish fixed worker by its name

Quoted 14 lines · Hao Xu

From: Hao Xu <[email protected]>

Distinguish fixed workers and normal workers by their names.

---
  io_uring/io-wq.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
io_uring/io-wq.c +0 -0
index 61cf6da2c72f..7a9e5fa19b81 100644

io_uring/io-wq.c static int io_wq_worker(void *data) · line 642

Quoted 10 lines
io_uring/io-wq.c +3 -1
@@ -634,10 +634,12 @@ static int io_wq_worker(void *data)
  	struct io_wq *wq = worker->wq;
  	bool exit_mask = false, last_timeout = false;
  	char buf[TASK_COMM_LEN];
+	bool fixed = is_fixed_worker(worker);
 
  	worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING);
 
-	snprintf(buf, sizeof(buf), "iou-wrk-%d", wq->task->pid);
+	snprintf(buf, sizeof(buf), fixed ? "iou-fixed-%d" : "iou-wrk-%d",
+		 wq->task->pid);

Minor nit: iou-wrk-fixed should be better, it still tells that it's a worker and I think it should be more familiar to users.

Quoted 12 lines
io_uring/io-wq.c +1 -1
  	set_task_comm(current, buf);
 
  	while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) {
@@ -656,7 +658,7 @@ static int io_wq_worker(void *data)
  		 * fixed worker, they can be manually reset to cpu other than
  		 * the cpuset indicated by io_wq_worker_affinity()
  		 */
-		if (!is_fixed_worker(worker) && last_timeout &&
+		if (!fixed && last_timeout &&
  		    (exit_mask || acct->nr_workers > 1)) {
  			acct->nr_workers--;
  			raw_spin_unlock(&wq->lock);
Signature · 1 line
Pavel Begunkov
author
#23 lore raw

PATCH 11/11 io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend

In reply to Hao Xu

From: Hao Xu <[email protected]>

Add a new setup flag to indicate that the uring instance only use fixed workers as async offload threads. Add a work flag and its code logic as well.

---
 include/uapi/linux/io_uring.h | 10 +++++++++-
 io_uring/io-wq.c              | 18 +++++++++++++-----
 io_uring/io-wq.h              |  1 +
 io_uring/io_uring.c           | 24 +++++++++++++++++++-----
 4 files changed, 42 insertions(+), 11 deletions(-)
include/uapi/linux/io_uring.h +9 -1
index b0a6e3106b42..900fedaa5692 100644
@@ -185,6 +185,11 @@ enum {
  */
 #define IORING_SETUP_REGISTERED_FD_ONLY	(1U << 15)
 
+/*
+ * this ring instance only use fixed worker for async offload.
+ */
+#define IORING_SETUP_FIXED_WORKER_ONLY	(1U << 16)
+
 enum io_uring_op {
 	IORING_OP_NOP,
 	IORING_OP_READV,
@@ -721,9 +726,12 @@ struct io_uring_recvmsg_out {
 	__u32 flags;
 };
 
+#define IORING_FIXED_WORKER_F_ONLY (1U << 0)
+#define IORING_FIXED_WORKER_F_VALID (IORING_FIXED_WORKER_F_ONLY)
+
 struct io_uring_fixed_worker_arg {
 	__u32	nr_workers;
-	__u32	resv;
+	__u32	flags;
 	__u64	resv2[3];
 };
 
io_uring/io-wq.c +13 -5
index 7a9e5fa19b81..98a16abb2944 100644
@@ -272,7 +272,7 @@ static inline bool io_acct_run_queue(struct io_wq_acct *acct)
  * caller must create one.
  */
 static bool io_wq_activate_free_worker(struct io_wq *wq,
-					struct io_wq_acct *acct)
+					struct io_wq_acct *acct, bool fixed)
 	__must_hold(RCU)
 {
 	struct hlist_nulls_node *n;
@@ -286,7 +286,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq,
 	hlist_nulls_for_each_entry_rcu(worker, n, &wq->free_list, nulls_node) {
 		if (!io_worker_get(worker))
 			continue;
-		if (io_wq_get_acct(worker) != acct) {
+		if (io_wq_get_acct(worker) != acct ||
+		    (fixed && !is_fixed_worker(worker))) {
 			io_worker_release(worker);
 			continue;
 		}
@@ -492,6 +493,9 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct,
 
 		work = container_of(node, struct io_wq_work, list);
 
+		if ((work->flags & IO_WQ_WORK_FIXED) && !is_fixed_worker(worker))
+			continue;
+
 		/* not hashed, can run anytime */
 		if (!io_wq_is_hashed(work)) {
 			wq_list_del(&acct->work_list, node, prev);
@@ -946,7 +950,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work)
 	struct io_wq_acct *acct = io_work_get_acct(wq, work);
 	struct io_cb_cancel_data match;
 	unsigned work_flags = work->flags;
-	bool do_create;
+	bool do_create, fixed = work_flags & IO_WQ_WORK_FIXED;
 
 	/*
 	 * If io-wq is exiting for this task, or if the request has explicitly
@@ -965,11 +969,14 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work)
 
 	raw_spin_lock(&wq->lock);
 	rcu_read_lock();
-	do_create = !io_wq_activate_free_worker(wq, acct);
+	do_create = !io_wq_activate_free_worker(wq, acct, fixed);
 	rcu_read_unlock();
 
 	raw_spin_unlock(&wq->lock);
 
+	if (fixed)
+		return;
+
 	if (do_create && ((work_flags & IO_WQ_WORK_CONCURRENT) ||
 	    !atomic_read(&acct->nr_running))) {
 		bool did_create;
@@ -1155,7 +1162,7 @@ static int io_wq_hash_wake(struct wait_queue_entry *wait, unsigned mode,
 		struct io_wq_acct *acct = &wq->acct[i];
 
 		if (test_and_clear_bit(IO_ACCT_STALLED_BIT, &acct->flags))
-			io_wq_activate_free_worker(wq, acct);
+			io_wq_activate_free_worker(wq, acct, false);
 	}
 	rcu_read_unlock();
 	return 1;
@@ -1477,6 +1484,7 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *coun
 
 	if (ret)
 		goto err;
+
 	return 0;
 
 err:
io_uring/io-wq.h +1 -0
index 15e93af36511..d81d5f9aa602 100644
@@ -11,6 +11,7 @@ enum {
 	IO_WQ_WORK_HASHED	= 2,
 	IO_WQ_WORK_UNBOUND	= 4,
 	IO_WQ_WORK_CONCURRENT	= 16,
+	IO_WQ_WORK_FIXED	= 32,
 
 	IO_WQ_HASH_SHIFT	= 24,	/* upper 8 bits are used for hash key */
 };
io_uring/io_uring.c +19 -5
index b37224cc1d05..bf8232906605 100644
@@ -479,6 +479,9 @@ void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use)
 	if (WARN_ON_ONCE(!same_thread_group(req->task, current)))
 		req->work.flags |= IO_WQ_WORK_CANCEL;
 
+	if (req->ctx->flags & IORING_SETUP_FIXED_WORKER_ONLY)
+		req->work.flags |= IO_WQ_WORK_FIXED;
+
 	trace_io_uring_queue_async_work(req, io_wq_is_hashed(&req->work));
 	io_wq_enqueue(tctx->io_wq, &req->work);
 	if (link)
@@ -1971,7 +1974,12 @@ struct io_wq_work *io_wq_free_work(struct io_wq_work *work)
 	struct io_kiocb *req = container_of(work, struct io_kiocb, work);
 
 	req = io_put_req_find_next(req);
-	return req ? &req->work : NULL;
+	if (req) {
+		req->work.flags |= IO_WQ_WORK_FIXED;
+		return &req->work;
+	}
+
+	return NULL;
 }
 
 void io_wq_submit_work(struct io_wq_work *work)
@@ -4364,7 +4372,7 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,
 	struct io_uring_fixed_worker_arg *res;
 	size_t size;
 	int i, ret;
-	bool zero = true;
+	bool zero = true, fixed_only = false;
 
 	size = array_size(nr_args, sizeof(*res));
 	if (size == SIZE_MAX)
@@ -4375,15 +4383,20 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx,
 		return PTR_ERR(res);
 
 	for (i = 0; i < nr_args; i++) {
-		if (res[i].nr_workers) {
+		if (res[i].flags & ~IORING_FIXED_WORKER_F_VALID)
+			return -EINVAL;
+		if (res[i].flags & IORING_FIXED_WORKER_F_ONLY)
+			fixed_only = true;
+		if (res[i].nr_workers)
 			zero = false;
-			break;
-		}
 	}
 
 	if (zero)
 		return 0;
 
+	if (fixed_only)
+		ctx->flags |= IORING_SETUP_FIXED_WORKER_ONLY;
+
 	if (ctx->flags & IORING_SETUP_SQPOLL) {
 		sqd = ctx->sq_data;
 		if (sqd) {
@@ -4423,6 +4436,7 @@ static __cold int io_unregister_iowq_fixed_workers(struct io_ring_ctx *ctx)
 	struct io_sq_data *sqd = NULL;
 	int ret;
 
+	ctx->flags &= ~IORING_SETUP_FIXED_WORKER_ONLY;
 	if (ctx->flags & IORING_SETUP_SQPOLL) {
 		sqd = ctx->sq_data;
 		if (sqd) {
Signature · 1 line
2.25.1
#24 lore raw

Re: [PATCH 11/11] io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend

Quoted 5 lines · Hao Xu

From: Hao Xu <[email protected]>

Add a new setup flag to indicate that the uring instance only use fixed workers as async offload threads. Add a work flag and its code logic as well.

Do we need it? What is the effective difference from limiting workers to 0 via IORING_REGISTER_IOWQ_MAX_WORKERS? It sounds like it should rather be a liburing helper instead.

Signature · 1 line
Pavel Begunkov
author
#25 lore raw
In reply to Hao Xu

Hi Jens and all,

Quoted 11 lines · Hao Xu

From: Hao Xu <[email protected]>

The initial feature request by users is here:
https://github.com/axboe/liburing/issues/296

Fixed worker provide a way for users to control the io-wq threads. A fixed worker is worker thread which exists no matter there are works to do or not. We provide a new register api to register fixed workers, and a register api to unregister them as well. The parameter of the register api is the number of fixed workers users want.

Here is a liburing test case to show how it works:
https://github.com/axboe/liburing/commit/bc9e862d1317f2466381adb6243be8cc86c3bd27

Regards,
Hao

author
#26 lore raw
In reply to Hao Xu

Gently ping this series
No idea why the mail threading is a mess, it looks fine in fsdevel list.

Quoted 76 lines · Hao Xu

From: Hao Xu <[email protected]>

The initial feature request by users is here:
https://github.com/axboe/liburing/issues/296

Fixed worker provide a way for users to control the io-wq threads. A fixed worker is worker thread which exists no matter there are works to do or not. We provide a new register api to register fixed workers, and a register api to unregister them as well. The parameter of the register api is the number of fixed workers users want.

For example:

```c
io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5 })
do I/O works
io_uring_unregister_iowq_fixed_workers(&ring)

```

After registration, there will be 5 fixed workers. User can setup their affinity, priority etc. freely, without adding any new register api to set up attributions. These workers won't be destroyed until users call unregister api.

Note, registering some fixed workers doesn't mean no creating normal workers. When there is no free workers, new normal workers can be created when works come. So a work may be picked up by fixed workers or normal workers.

If users want to offload works only to fixed workers, they can specify a flag FIXED_ONLY when registering fixed workers.

```c
io_uring_register_iowq_fixed_workers(&ring, { .nr_workers = 5, .flags |=
FIXED_ONLY })

```

In above case, no normal workers will be created before calling io_uring_register_iowq_fixed_workers().

Note:
- When registering fixed workers, those fixed workers are per io-wq. So if an io_uring instance is shared by multiple tasks, and you want all tasks to use fixed workers, all tasks have to call the regitser api.
- if specifying FIXED_ONLY when registering fixed workers, that is per io_uring instance. all works in this instance are handled by fixed workers.

Therefore, if an io_uring instance is shared by two tasks, and you want all requests in this instance to be handled only by fixed workers, you have to call the register api in these two tasks and specify FIXED_ONLY at least once when calling register api.

Hao Xu (11):
io-wq: fix worker counting after worker received exit signal
io-wq: add a new worker flag to indicate worker exit
io-wq: add a new type io-wq worker
io-wq: add fixed worker members in io_wq_acct
io-wq: add a new parameter for creating a new fixed worker
io-wq: return io_worker after successful inline worker creation
io_uring: add new api to register fixed workers
io_uring: add function to unregister fixed workers
io-wq: add strutures to allow to wait fixed workers exit
io-wq: distinguish fixed worker by its name
io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend

  include/uapi/linux/io_uring.h |  20 +++
  io_uring/io-wq.c              | 275 ++++++++++++++++++++++++++++++----
  io_uring/io-wq.h              |   3 +
  io_uring/io_uring.c           | 132 +++++++++++++++-
  4 files changed, 397 insertions(+), 33 deletions(-)