Skip to contentSkip to conversation

iowq fix

PATCH In review Hao Xu 2 participants 15 messages Sep 11, 2021 – Sep 13, 2021

Subsystems (2)
lore
Share & tools
lore

Hao Xu

author
#0 lore raw

cover iowq fix

several iowq fixes, all in theory, haven't been manually triggered. Hao Xu (4):
io-wq: tweak return value of io_wqe_create_worker()
io-wq: code clean of io_wqe_create_worker()
io-wq: fix worker->refcount when creating worker in work exit
io-wq: fix potential race of acct->nr_workers

 fs/io-wq.c | 24 ++++++++++--------------
 1 file changed, 10 insertions(+), 14 deletions(-)
Signature · 1 line
2.24.4
author
#1 lore raw

PATCH 1/4 io-wq: tweak return value of io_wqe_create_worker()

The return value of io_wqe_create_worker() should be false if we cannot create a new worker according to the name of this function.

---
 fs/io-wq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io-wq.c +1 -1
index 382efca4812b..1b102494e970 100644
@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
 		return create_io_worker(wqe->wq, wqe, acct->index);
 	}
 
-	return true;
+	return false;
 }
 
 static void io_wqe_inc_running(struct io_worker *worker)
Signature · 1 line
2.24.4
maintainer
#2 lore raw

Re: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()

Quoted 13 lines · Hao Xu

The return value of io_wqe_create_worker() should be false if we cannot create a new worker according to the name of this function.

---
 fs/io-wq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io-wq.c +0 -0
index 382efca4812b..1b102494e970 100644

fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271

Quoted 6 lines
fs/io-wq.c +1 -1
@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
 		return create_io_worker(wqe->wq, wqe, acct->index);
 	}
 
-	return true;
+	return false;
 }

I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.

Signature · 1 line
Jens Axboe
author
#3 lore raw

Re: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()

Quoted 17 lines · Jens Axboe
Quoted 16 lines · Hao Xu

The return value of io_wqe_create_worker() should be false if we cannot create a new worker according to the name of this function.

---
  fs/io-wq.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
fs/io-wq.c +0 -0
index 382efca4812b..1b102494e970 100644
@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
  		return create_io_worker(wqe->wq, wqe, acct->index);
  	}
 
Quoted 7 lines

fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271

Quoted 3 lines
fs/io-wq.c +1 -1
-	return true;
+	return false;
  }

I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.

hmm, I think it is an error case----'we failed to create a new worker since nr_worker == max_worker'. nr_worker == max_worker doesn't mean 'no need', we may meet situation describled in 4/4: max_worker is 1, currently 1 worker is running, and we return true here:

          did_create = io_wqe_create_worker(wqe, acct);

             //*******nr_workers changes******//

          if (unlikely(!did_create)) {
                  raw_spin_lock(&wqe->lock);
                  /* fatal condition, failed to create the first worker */
                  if (!acct->nr_workers) {
                          raw_spin_unlock(&wqe->lock);
                          goto run_cancel;
                  }
                  raw_spin_unlock(&wqe->lock);
          }

we will miss the next check, but we have to do the check, since number of workers may decrease to 0 in //******// place.

or maybe we can see the return value as 'do we create a new worker or not', and the next code do safe check if it is false.

Quoted 1 line
maintainer
#4 lore raw

Re: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()

Quoted 25 lines · Hao Xu
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Hao Xu

The return value of io_wqe_create_worker() should be false if we cannot create a new worker according to the name of this function.

---
  fs/io-wq.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
fs/io-wq.c +0 -0
index 382efca4812b..1b102494e970 100644
@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
  		return create_io_worker(wqe->wq, wqe, acct->index);
  	}
 
Quoted 7 lines

fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271

Quoted 3 lines
fs/io-wq.c +1 -1
-	return true;
+	return false;
  }

I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.

Quoted 3 lines

hmm, I think it is an error case----'we failed to create a new worker since nr_worker == max_worker'. nr_worker == max_worker doesn't mean 'no need', we may meet situation describled in 4/4: max_worker is 1,

But that's not an error case in the sense of "uh oh, we need to handle this as an error". If we're at the max worker count, the work simply has to wait for another work to be done and process it.

Quoted 16 lines

currently 1 worker is running, and we return true here:

           did_create = io_wqe_create_worker(wqe, acct);

              //*******nr_workers changes******//

           if (unlikely(!did_create)) {
                   raw_spin_lock(&wqe->lock);
                   /* fatal condition, failed to create the first worker */
                   if (!acct->nr_workers) {
                           raw_spin_unlock(&wqe->lock);
                           goto run_cancel;
                   }
                   raw_spin_unlock(&wqe->lock);
           }
Quoted 2 lines

we will miss the next check, but we have to do the check, since number of workers may decrease to 0 in //******// place.

If that happens, then the work that we have inserted has already been run. Otherwise how else could we have dropped to zero workers?

Signature · 1 line
Jens Axboe
author
#5 lore raw

Re: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()

Quoted 50 lines · Jens Axboe
Quoted 25 lines · Hao Xu
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Hao Xu

The return value of io_wqe_create_worker() should be false if we cannot create a new worker according to the name of this function.

---
   fs/io-wq.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
fs/io-wq.c +0 -0
index 382efca4812b..1b102494e970 100644
@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
   		return create_io_worker(wqe->wq, wqe, acct->index);
   	}
 
Quoted 7 lines

fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271

Quoted 3 lines
fs/io-wq.c +1 -1
-	return true;
+	return false;
   }

I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.

Quoted 3 lines

hmm, I think it is an error case----'we failed to create a new worker since nr_worker == max_worker'. nr_worker == max_worker doesn't mean 'no need', we may meet situation describled in 4/4: max_worker is 1,

But that's not an error case in the sense of "uh oh, we need to handle this as an error". If we're at the max worker count, the work simply has to wait for another work to be done and process it.

Quoted 16 lines

currently 1 worker is running, and we return true here:

            did_create = io_wqe_create_worker(wqe, acct);

               //*******nr_workers changes******//

            if (unlikely(!did_create)) {
                    raw_spin_lock(&wqe->lock);
                    /* fatal condition, failed to create the first worker */
                    if (!acct->nr_workers) {
                            raw_spin_unlock(&wqe->lock);
                            goto run_cancel;
                    }
                    raw_spin_unlock(&wqe->lock);
            }
Quoted 6 lines
Quoted 2 lines

we will miss the next check, but we have to do the check, since number of workers may decrease to 0 in //******// place.

If that happens, then the work that we have inserted has already been run. Otherwise how else could we have dropped to zero workers?

Sorry, I see. I forgot the fix moved the place of nr_workers... There is no problems now. Thanks for explanation, Jens.

io_wqe_enqueue worker1
no work there and timeout nr_workers--(after fix) unlock(wqe->lock)
->insert work

->io_wqe_create_worker

                               ->io_worker_exit
                                 ->nr_workers--(before fix)
author
#6 lore raw

PATCH 2/4 io-wq: code clean of io_wqe_create_worker()

In reply to Hao Xu

Remove do_create to save a local variable.

---
 fs/io-wq.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
fs/io-wq.c +7 -12
index 1b102494e970..0e1288a549eb 100644
@@ -246,8 +246,6 @@ static bool io_wqe_activate_free_worker(struct io_wqe *wqe,
  */
 static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
 {
-	bool do_create = false;
-
 	/*
 	 * Most likely an attempt to queue unbounded work on an io_wq that
 	 * wasn't setup with any unbounded workers.
@@ -256,18 +254,15 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
 		pr_warn_once("io-wq is not configured for unbound workers");
 
 	raw_spin_lock(&wqe->lock);
-	if (acct->nr_workers < acct->max_workers) {
-		acct->nr_workers++;
-		do_create = true;
+	if (acct->nr_workers == acct->max_workers) {
+		raw_spin_unlock(&wqe->lock);
+		return false;
 	}
+	acct->nr_workers++;
 	raw_spin_unlock(&wqe->lock);
-	if (do_create) {
-		atomic_inc(&acct->nr_running);
-		atomic_inc(&wqe->wq->worker_refs);
-		return create_io_worker(wqe->wq, wqe, acct->index);
-	}
-
-	return false;
+	atomic_inc(&acct->nr_running);
+	atomic_inc(&wqe->wq->worker_refs);
+	return create_io_worker(wqe->wq, wqe, acct->index);
 }
 
 static void io_wqe_inc_running(struct io_worker *worker)
Signature · 1 line
2.24.4
maintainer
#7 lore raw

Re: [PATCH 2/4] io-wq: code clean of io_wqe_create_worker()

Quoted 1 line · Hao Xu

Remove do_create to save a local variable.

This one looks good, it's easier to follow as well. Applied.

Signature · 1 line
Jens Axboe
author
#8 lore raw

Re: [PATCH 2/4] io-wq: code clean of io_wqe_create_worker()

In reply to Hao Xu
Quoted 21 lines · Hao Xu

Remove do_create to save a local variable.

---
  fs/io-wq.c | 19 +++++++------------
  1 file changed, 7 insertions(+), 12 deletions(-)
fs/io-wq.c +0 -2
index 1b102494e970..0e1288a549eb 100644
@@ -246,8 +246,6 @@ static bool io_wqe_activate_free_worker(struct io_wqe *wqe,
   */
  static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
  {
-	bool do_create = false;
-
  	/*
  	 * Most likely an attempt to queue unbounded work on an io_wq that
  	 * wasn't setup with any unbounded workers.

fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 259

Quoted 9 lines
fs/io-wq.c +3 -3
@@ -256,18 +254,15 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)
  		pr_warn_once("io-wq is not configured for unbound workers");
 
  	raw_spin_lock(&wqe->lock);
-	if (acct->nr_workers < acct->max_workers) {
-		acct->nr_workers++;
-		do_create = true;
+	if (acct->nr_workers == acct->max_workers) {
+		raw_spin_unlock(&wqe->lock);
+		return false;

Hi Jens, would you like to tweak it to return true or would like me to send a fix.

Quoted 17 lines
fs/io-wq.c +4 -7
  	}
+	acct->nr_workers++;
  	raw_spin_unlock(&wqe->lock);
-	if (do_create) {
-		atomic_inc(&acct->nr_running);
-		atomic_inc(&wqe->wq->worker_refs);
-		return create_io_worker(wqe->wq, wqe, acct->index);
-	}
-
-	return false;
+	atomic_inc(&acct->nr_running);
+	atomic_inc(&wqe->wq->worker_refs);
+	return create_io_worker(wqe->wq, wqe, acct->index);
  }
 
  static void io_wqe_inc_running(struct io_worker *worker)
author
#9 lore raw

PATCH 3/4 io-wq: fix worker->refcount when creating worker in work exit

In reply to Hao Xu

We may enter the worker creation path from io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.
io_worker_exit

                              ref = 0
->io_wqe_dec_running
  ->io_queue_worker_create
    ->io_worker_get           failure since ref is 0
---
 fs/io-wq.c | 2 ++
 1 file changed, 2 insertions(+)
fs/io-wq.c +2 -0
index 0e1288a549eb..75e79571bdfd 100644
@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)
 	list_del_rcu(&worker->all_list);
 	acct->nr_workers--;
 	preempt_disable();
+	refcount_set(&worker->ref, 1);
 	io_wqe_dec_running(worker);
+	refcount_set(&worker->ref, 0);
 	worker->flags = 0;
 	current->flags &= ~PF_IO_WORKER;
 	preempt_enable();
Signature · 1 line
2.24.4
maintainer
#10 lore raw

Re: [PATCH 3/4] io-wq: fix worker->refcount when creating worker in work exit

Quoted 19 lines · Hao Xu

We may enter the worker creation path from io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.
io_worker_exit

                              ref = 0
->io_wqe_dec_running
  ->io_queue_worker_create
    ->io_worker_get           failure since ref is 0
---
 fs/io-wq.c | 2 ++
 1 file changed, 2 insertions(+)
fs/io-wq.c +0 -0
index 0e1288a549eb..75e79571bdfd 100644

fs/io-wq.c static void io_worker_exit(struct io_worker *worker) · line 193

Quoted 6 lines
fs/io-wq.c +2 -0
@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)
 	list_del_rcu(&worker->all_list);
 	acct->nr_workers--;
 	preempt_disable();
+	refcount_set(&worker->ref, 1);
 	io_wqe_dec_running(worker);
+	refcount_set(&worker->ref, 0);

That kind of refcount manipulation is highly suspect. And in fact it should not be needed, io_worker_exit() clears ->flags before going on with worker teardown. Hence you can't hit worker creation from io_wqe_dec_running().

Signature · 1 line
Jens Axboe
author
#11 lore raw

Re: [PATCH 3/4] io-wq: fix worker->refcount when creating worker in work exit

Quoted 15 lines · Jens Axboe
Quoted 14 lines · Hao Xu

We may enter the worker creation path from io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.
io_worker_exit

                               ref = 0
->io_wqe_dec_running
   ->io_queue_worker_create
     ->io_worker_get           failure since ref is 0
---
  fs/io-wq.c | 2 ++
  1 file changed, 2 insertions(+)
Quoted 16 lines

fs/io-wq.c static void io_worker_exit(struct io_worker *worker) · line 193

Quoted 11 lines
fs/io-wq.c +2 -0
index 0e1288a549eb..75e79571bdfd 100644
@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)
  	list_del_rcu(&worker->all_list);
  	acct->nr_workers--;
  	preempt_disable();
+	refcount_set(&worker->ref, 1);
  	io_wqe_dec_running(worker);
+	refcount_set(&worker->ref, 0);

That kind of refcount manipulation is highly suspect. And in fact it should not be needed, io_worker_exit() clears ->flags before going on with worker teardown. Hence you can't hit worker creation from io_wqe_dec_running().

Doesn't see the relationship between worker->flags and the creation. But yes, the creation path does io_worker_get() which causes failure if it's from io_worker_exit(), Now I understand it is more like a feature, isn't it? Anyway, the issue in 4/4 seems still there.

Regards,
Hao

Quoted 1 line
maintainer
#12 lore raw

Re: [PATCH 3/4] io-wq: fix worker->refcount when creating worker in work exit

Quoted 32 lines · Hao Xu
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Hao Xu

We may enter the worker creation path from io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.
io_worker_exit

                               ref = 0
->io_wqe_dec_running
   ->io_queue_worker_create
     ->io_worker_get           failure since ref is 0
---
  fs/io-wq.c | 2 ++
  1 file changed, 2 insertions(+)
Quoted 16 lines

fs/io-wq.c static void io_worker_exit(struct io_worker *worker) · line 193

Quoted 11 lines
fs/io-wq.c +2 -0
index 0e1288a549eb..75e79571bdfd 100644
@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)
  	list_del_rcu(&worker->all_list);
  	acct->nr_workers--;
  	preempt_disable();
+	refcount_set(&worker->ref, 1);
  	io_wqe_dec_running(worker);
+	refcount_set(&worker->ref, 0);

That kind of refcount manipulation is highly suspect. And in fact it should not be needed, io_worker_exit() clears ->flags before going on with worker teardown. Hence you can't hit worker creation from io_wqe_dec_running().

Quoted 4 lines

Doesn't see the relationship between worker->flags and the creation. But yes, the creation path does io_worker_get() which causes failure if it's from io_worker_exit(), Now I understand it is more like a feature, isn't it? Anyway, the issue in 4/4 seems still there.

Right, that's on purpose. In any case, the above would fail miserably if it raced with someone trying to get a reference on the worker:

A B
refcount_set(ref, 1)
io_worker_get(), succeeds now 2
refcount_set(ref, 0)
oops...

Signature · 1 line
Jens Axboe
author
#13 lore raw

PATCH 4/4 io-wq: fix potential race of acct->nr_workers

In reply to Hao Xu

Given max_worker is 1, and we currently have 1 running and it is exiting. There may be race like:
io_wqe_enqueue worker1
no work there and timeout unlock(wqe->lock)
->insert work

                               -->io_worker_exit
 lock(wqe->lock)
 ->if(!nr_workers) //it's still 1
 unlock(wqe->lock)
    goto run_cancel
                                  lock(wqe->lock)
                                  nr_workers--
                                  ->dec_running
                                    ->worker creation fails
                                  unlock(wqe->lock)

We enqueued one work but there is no workers, causes hung.

---
 fs/io-wq.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
fs/io-wq.c +1 -2
index 75e79571bdfd..b84dc8df6c68 100644
@@ -176,7 +176,6 @@ static void io_worker_ref_put(struct io_wq *wq)
 static void io_worker_exit(struct io_worker *worker)
 {
 	struct io_wqe *wqe = worker->wqe;
-	struct io_wqe_acct *acct = io_wqe_get_acct(worker);
 
 	if (refcount_dec_and_test(&worker->ref))
 		complete(&worker->ref_done);
@@ -186,7 +185,6 @@ static void io_worker_exit(struct io_worker *worker)
 	if (worker->flags & IO_WORKER_F_FREE)
 		hlist_nulls_del_rcu(&worker->nulls_node);
 	list_del_rcu(&worker->all_list);
-	acct->nr_workers--;
 	preempt_disable();
 	refcount_set(&worker->ref, 1);
 	io_wqe_dec_running(worker);
@@ -571,6 +569,7 @@ static int io_wqe_worker(void *data)
 		}
 		/* timed out, exit unless we're the last worker */
 		if (last_timeout && acct->nr_workers > 1) {
+			acct->nr_workers--;
 			raw_spin_unlock(&wqe->lock);
 			__set_current_state(TASK_RUNNING);
 			break;
Signature · 1 line
2.24.4
maintainer
#14 lore raw

Re: [PATCH 4/4] io-wq: fix potential race of acct->nr_workers

Quoted 17 lines · Hao Xu

Given max_worker is 1, and we currently have 1 running and it is exiting. There may be race like:
io_wqe_enqueue worker1
no work there and timeout unlock(wqe->lock)
->insert work

                               -->io_worker_exit
 lock(wqe->lock)
 ->if(!nr_workers) //it's still 1
 unlock(wqe->lock)
    goto run_cancel
                                  lock(wqe->lock)
                                  nr_workers--
                                  ->dec_running
                                    ->worker creation fails
                                  unlock(wqe->lock)
Quoted 1 line

We enqueued one work but there is no workers, causes hung.

This one also looks good, applied.

Signature · 1 line
Jens Axboe