iowq fix
cover iowq fix
several iowq fixes, all in theory, haven't been manually triggered. Hao Xu (4):
io-wq: tweak return value of io_wqe_create_worker()
io-wq: code clean of io_wqe_create_worker()
io-wq: fix worker->refcount when creating worker in work exit
io-wq: fix potential race of acct->nr_workers
fs/io-wq.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-)
Signature · 1 line
2.24.4
PATCH 1/4 io-wq: tweak return value of io_wqe_create_worker()
The return value of io_wqe_create_worker() should be false if we cannot create a new worker according to the name of this function.
--- fs/io-wq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/io-wq.c +1 -1
@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)return create_io_worker (wqe->wq, wqe, acct->index); } -return true; +return false; }static void io_wqe_inc_running (struct io_worker *worker)
Signature · 1 line
2.24.4
Jens Axboe
maintainerRe: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()
Quoted 13 lines · Hao Xu
The return value of
io_wqe_create_worker()should be false if we cannot create a new worker according to the name of this function.Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/io-wq.c +0
-0
fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271
Quoted 6 lines
fs/io-wq.c +1
-1@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)return create_io_worker (wqe->wq, wqe, acct->index); } -return true; +return false; }
I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.
Signature · 1 line
Jens Axboe
Re: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Hao Xu
The return value of
io_wqe_create_worker()should be false if we cannot create a new worker according to the name of this function.Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/io-wq.c +0
-0@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)return create_io_worker (wqe->wq, wqe, acct->index); }
Quoted 7 lines
fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271
Quoted 3 lines
fs/io-wq.c +1
-1-return true; +return false; }I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.
hmm, I think it is an error case----'we failed to create a new worker since nr_worker == max_worker'. nr_worker == max_worker doesn't mean 'no need', we may meet situation describled in 4/4: max_worker is 1, currently 1 worker is running, and we return true here:
did_create = io_wqe_create_worker(wqe, acct);
//*******nr_workers changes******//
if (unlikely(!did_create)) {
raw_spin_lock(&wqe->lock);
/* fatal condition, failed to create the first worker */
if (!acct->nr_workers) {
raw_spin_unlock(&wqe->lock);
goto run_cancel;
}
raw_spin_unlock(&wqe->lock);
}we will miss the next check, but we have to do the check, since number of workers may decrease to 0 in //******// place.
or maybe we can see the return value as 'do we create a new worker or not', and the next code do safe check if it is false.
Quoted 1 line
Jens Axboe
maintainerRe: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()
Quoted 25 lines · Hao Xu
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Hao Xu
The return value of
io_wqe_create_worker()should be false if we cannot create a new worker according to the name of this function.Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/io-wq.c +0
-0@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)return create_io_worker (wqe->wq, wqe, acct->index); }Quoted 7 lines
fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271
Quoted 3 lines
fs/io-wq.c +1
-1-return true; +return false; }I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.
Quoted 3 lines
hmm, I think it is an error case----'we failed to create a new worker since
nr_worker==max_worker'.nr_worker==max_workerdoesn't mean 'no need', we may meet situation describled in 4/4:max_workeris 1,
But that's not an error case in the sense of "uh oh, we need to handle this as an error". If we're at the max worker count, the work simply has to wait for another work to be done and process it.
Quoted 16 lines
currently 1 worker is running, and we return true here:
did_create = io_wqe_create_worker(wqe, acct); //*******nr_workers changes******// if (unlikely(!did_create)) { raw_spin_lock(&wqe->lock); /* fatal condition, failed to create the first worker */ if (!acct->nr_workers) { raw_spin_unlock(&wqe->lock); goto run_cancel; } raw_spin_unlock(&wqe->lock); }
Quoted 2 lines
we will miss the next check, but we have to do the check, since number of workers may decrease to 0 in //******// place.
If that happens, then the work that we have inserted has already been run. Otherwise how else could we have dropped to zero workers?
Signature · 1 line
Jens Axboe
Re: [PATCH 1/4] io-wq: tweak return value of io_wqe_create_worker()
Quoted 50 lines · Jens Axboe
Quoted 25 lines · Hao Xu
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Hao Xu
The return value of
io_wqe_create_worker()should be false if we cannot create a new worker according to the name of this function.Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/io-wq.c +0
-0@@ -267,7 +267,7 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)return create_io_worker (wqe->wq, wqe, acct->index); }Quoted 7 lines
fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 271
Quoted 3 lines
fs/io-wq.c +1
-1-return true; +return false; }I think this is just a bit confusing. It's not an error case, we just didn't need to create a worker. So don't return failure, or the caller will think that we failed while we did not.
Quoted 3 lines
hmm, I think it is an error case----'we failed to create a new worker since
nr_worker==max_worker'.nr_worker==max_workerdoesn't mean 'no need', we may meet situation describled in 4/4:max_workeris 1,But that's not an error case in the sense of "uh oh, we need to handle this as an error". If we're at the max worker count, the work simply has to wait for another work to be done and process it.
Quoted 16 lines
currently 1 worker is running, and we return true here:
did_create = io_wqe_create_worker(wqe, acct); //*******nr_workers changes******// if (unlikely(!did_create)) { raw_spin_lock(&wqe->lock); /* fatal condition, failed to create the first worker */ if (!acct->nr_workers) { raw_spin_unlock(&wqe->lock); goto run_cancel; } raw_spin_unlock(&wqe->lock); }
Quoted 6 lines
Quoted 2 lines
we will miss the next check, but we have to do the check, since number of workers may decrease to 0 in //******// place.
If that happens, then the work that we have inserted has already been run. Otherwise how else could we have dropped to zero workers?
Sorry, I see. I forgot the fix moved the place of nr_workers... There is no problems now. Thanks for explanation, Jens.
io_wqe_enqueue worker1
no work there and timeout nr_workers--(after fix) unlock(wqe->lock)
->insert work
->io_wqe_create_worker
->io_worker_exit
->nr_workers--(before fix)PATCH 2/4 io-wq: code clean of io_wqe_create_worker()
In reply to Hao XuRemove do_create to save a local variable.
--- fs/io-wq.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
fs/io-wq.c +7 -12
@@ -246,8 +246,6 @@ static bool io_wqe_activate_free_worker(struct io_wqe *wqe,*/ static bool io_wqe_create_worker (struct io_wqe *wqe,struct io_wqe_acct *acct) { -bool do_create = false; -/* * Most likely an attempt to queue unbounded work on an io_wq that * wasn't setup with any unbounded workers.
@@ -256,18 +254,15 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)pr_warn_once ("io-wq is not configured for unbound workers" );raw_spin_lock (&wqe->lock); -if (acct->nr_workers < acct->max_workers) { - acct->nr_workers++; - do_create = true; +if (acct->nr_workers == acct->max_workers) { +raw_spin_unlock (&wqe->lock); +return false; } + acct->nr_workers++;raw_spin_unlock (&wqe->lock); -if (do_create) { -atomic_inc (&acct->nr_running); -atomic_inc (&wqe->wq->worker_refs); -return create_io_worker (wqe->wq, wqe, acct->index); - } - -return false; +atomic_inc (&acct->nr_running); +atomic_inc (&wqe->wq->worker_refs); +return create_io_worker (wqe->wq, wqe, acct->index); }static void io_wqe_inc_running (struct io_worker *worker)
Signature · 1 line
2.24.4
Jens Axboe
maintainerRe: [PATCH 2/4] io-wq: code clean of io_wqe_create_worker()
Quoted 1 line · Hao Xu
Remove
do_createto save a local variable.
This one looks good, it's easier to follow as well. Applied.
Signature · 1 line
Jens Axboe
Re: [PATCH 2/4] io-wq: code clean of io_wqe_create_worker()
In reply to Hao XuQuoted 21 lines · Hao Xu
Remove
do_createto save a local variable.Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)fs/io-wq.c +0
-2@@ -246,8 +246,6 @@ static bool io_wqe_activate_free_worker(struct io_wqe *wqe,*/ static bool io_wqe_create_worker (struct io_wqe *wqe,struct io_wqe_acct *acct) { -bool do_create = false; -/* * Most likely an attempt to queue unbounded work on an io_wq that * wasn't setup with any unbounded workers.
fs/io-wq.c static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct) · line 259
Quoted 9 lines
fs/io-wq.c +3
-3@@ -256,18 +254,15 @@ static bool io_wqe_create_worker(struct io_wqe *wqe, struct io_wqe_acct *acct)pr_warn_once ("io-wq is not configured for unbound workers" );raw_spin_lock (&wqe->lock); -if (acct->nr_workers < acct->max_workers) { - acct->nr_workers++; - do_create = true; +if (acct->nr_workers == acct->max_workers) { +raw_spin_unlock (&wqe->lock); +return false;
Hi Jens, would you like to tweak it to return true or would like me to send a fix.
Quoted 17 lines
fs/io-wq.c +4
-7} + acct->nr_workers++;raw_spin_unlock (&wqe->lock); -if (do_create) { -atomic_inc (&acct->nr_running); -atomic_inc (&wqe->wq->worker_refs); -return create_io_worker (wqe->wq, wqe, acct->index); - } - -return false; +atomic_inc (&acct->nr_running); +atomic_inc (&wqe->wq->worker_refs); +return create_io_worker (wqe->wq, wqe, acct->index); }static void io_wqe_inc_running (struct io_worker *worker)
PATCH 3/4 io-wq: fix worker->refcount when creating worker in work exit
In reply to Hao XuWe may enter the worker creation path from io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.io_worker_exit
ref = 0
->io_wqe_dec_running
->io_queue_worker_create
->io_worker_get failure since ref is 0--- fs/io-wq.c | 2 ++ 1 file changed, 2 insertions(+)
fs/io-wq.c +2 -0
@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)list_del_rcu (&worker->all_list); acct->nr_workers--;preempt_disable (); +refcount_set (&worker->ref,1 );io_wqe_dec_running (worker); +refcount_set (&worker->ref,0 ); worker->flags =0 ; current->flags &= ~PF_IO_WORKER ;preempt_enable ();
Signature · 1 line
2.24.4
Jens Axboe
maintainerRe: [PATCH 3/4] io-wq: fix worker->refcount when creating worker in work exit
Quoted 19 lines · Hao Xu
We may enter the worker creation path from
io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.io_worker_exitref = 0 ->io_wqe_dec_running ->io_queue_worker_create ->io_worker_get failure since ref is 0Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 ++ 1 file changed, 2 insertions(+)fs/io-wq.c +0
-0
fs/io-wq.c static void io_worker_exit(struct io_worker *worker) · line 193
Quoted 6 lines
fs/io-wq.c +2
-0@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)list_del_rcu (&worker->all_list); acct->nr_workers--;preempt_disable (); +refcount_set (&worker->ref,1 );io_wqe_dec_running (worker); +refcount_set (&worker->ref,0 );
That kind of refcount manipulation is highly suspect. And in fact it should not be needed, io_worker_exit() clears ->flags before going on with worker teardown. Hence you can't hit worker creation from io_wqe_dec_running().
Signature · 1 line
Jens Axboe
Re: [PATCH 3/4] io-wq: fix worker->refcount when creating worker in work exit
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Hao Xu
We may enter the worker creation path from
io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.io_worker_exitref = 0 ->io_wqe_dec_running ->io_queue_worker_create ->io_worker_get failure since ref is 0Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 16 lines
fs/io-wq.c static void io_worker_exit(struct io_worker *worker) · line 193
Quoted 11 lines
fs/io-wq.c +2
-0@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)list_del_rcu (&worker->all_list); acct->nr_workers--;preempt_disable (); +refcount_set (&worker->ref,1 );io_wqe_dec_running (worker); +refcount_set (&worker->ref,0 );That kind of refcount manipulation is highly suspect. And in fact it should not be needed,
io_worker_exit()clears ->flags before going on with worker teardown. Hence you can't hit worker creation fromio_wqe_dec_running().
Doesn't see the relationship between worker->flags and the creation. But yes, the creation path does io_worker_get() which causes failure if it's from io_worker_exit(), Now I understand it is more like a feature, isn't it? Anyway, the issue in 4/4 seems still there.
Regards,
Hao
Quoted 1 line
Jens Axboe
maintainerRe: [PATCH 3/4] io-wq: fix worker->refcount when creating worker in work exit
Quoted 32 lines · Hao Xu
Quoted 15 lines · Jens Axboe
Quoted 14 lines · Hao Xu
We may enter the worker creation path from
io_worker_exit(), and refcount is already zero, which causes definite failure of worker creation.io_worker_exitref = 0 ->io_wqe_dec_running ->io_queue_worker_create ->io_worker_get failure since ref is 0Signed-off-by Hao Xu <[email protected]>--- fs/io-wq.c | 2 ++ 1 file changed, 2 insertions(+)Quoted 16 lines
fs/io-wq.c static void io_worker_exit(struct io_worker *worker) · line 193
Quoted 11 lines
fs/io-wq.c +2
-0@@ -188,7 +188,9 @@ static void io_worker_exit(struct io_worker *worker)list_del_rcu (&worker->all_list); acct->nr_workers--;preempt_disable (); +refcount_set (&worker->ref,1 );io_wqe_dec_running (worker); +refcount_set (&worker->ref,0 );That kind of refcount manipulation is highly suspect. And in fact it should not be needed,
io_worker_exit()clears ->flags before going on with worker teardown. Hence you can't hit worker creation fromio_wqe_dec_running().
Quoted 4 lines
Doesn't see the relationship between
worker->flagsand the creation. But yes, the creation path doesio_worker_get()which causes failure if it's fromio_worker_exit(), Now I understand it is more like a feature, isn't it? Anyway, the issue in 4/4 seems still there.
Right, that's on purpose. In any case, the above would fail miserably if it raced with someone trying to get a reference on the worker:
A Brefcount_set(ref, 1)
io_worker_get(), succeeds now 2refcount_set(ref, 0)
oops...
Signature · 1 line
Jens Axboe
PATCH 4/4 io-wq: fix potential race of acct->nr_workers
In reply to Hao XuGiven max_worker is 1, and we currently have 1 running and it is exiting. There may be race like:
io_wqe_enqueue worker1
no work there and timeout unlock(wqe->lock)
->insert work
-->io_worker_exit
lock(wqe->lock)
->if(!nr_workers) //it's still 1
unlock(wqe->lock)
goto run_cancel
lock(wqe->lock)
nr_workers--
->dec_running
->worker creation fails
unlock(wqe->lock)We enqueued one work but there is no workers, causes hung.
--- fs/io-wq.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/io-wq.c +1 -2
@@ -176,7 +176,6 @@ static void io_worker_ref_put(struct io_wq *wq)static void io_worker_exit (struct io_worker *worker) {struct io_wqe *wqe = worker->wqe; -struct io_wqe_acct *acct =io_wqe_get_acct (worker);if (refcount_dec_and_test (&worker->ref))complete (&worker->ref_done);
@@ -186,7 +185,6 @@ static void io_worker_exit(struct io_worker *worker)if (worker->flags &IO_WORKER_F_FREE )hlist_nulls_del_rcu (&worker->nulls_node);list_del_rcu (&worker->all_list); - acct->nr_workers--;preempt_disable ();refcount_set (&worker->ref,1 );io_wqe_dec_running (worker);
@@ -571,6 +569,7 @@ static int io_wqe_worker(void *data) }/* timed out, exit unless we're the last worker */ if (last_timeout && acct->nr_workers >1 ) { + acct->nr_workers--;raw_spin_unlock (&wqe->lock);__set_current_state (TASK_RUNNING );break ;
Signature · 1 line
2.24.4
Jens Axboe
maintainerRe: [PATCH 4/4] io-wq: fix potential race of acct->nr_workers
Quoted 17 lines · Hao Xu
Given
max_workeris 1, and we currently have 1 running and it is exiting. There may be race like:
io_wqe_enqueueworker1
no work there and timeoutunlock(wqe->lock)
->insert work-->io_worker_exit lock(wqe->lock) ->if(!nr_workers) //it's still 1 unlock(wqe->lock) goto run_cancel lock(wqe->lock) nr_workers-- ->dec_running ->worker creation fails unlock(wqe->lock)
Quoted 1 line
We enqueued one work but there is no workers, causes hung.
This one also looks good, applied.
Signature · 1 line
Jens Axboe