util/async: use atomic_mb_set in qemu_bh_cancel
PATCH 1/1 util/async: use atomic_mb_set in qemu_bh_cancel
Commit b7a745d added a qemu_bh_cancel call to the completion function as an optimization to prevent it from unnecessarily rescheduling itself.
This completion function is scheduled from worker_thread, after setting the state of a ThreadPoolElement to THREAD_DONE.
This was considered to be safe, as the completion function restarts the loop just after the call to qemu_bh_cancel. But, under certain access patterns and scheduling conditions, the loop may wrongly use a pre-fetched elem->state value, reading it as THREAD_QUEUED, and ending the completion function without having processed a pending TPE linked at pool->head.
In some situations, if there are no other independent requests in the same aio context that could eventually trigger the scheduling of the completion function, the omitted TPE and all operations pending on it will get stuck forever.
--- util/async.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
util/async.c +1 -1
@@ -174,7 +174,7 @@ void qemu_bh_schedule(QEMUBH *bh)*/ void qemu_bh_cancel (QEMUBH *bh) { - bh->scheduled =0 ; +atomic_mb_set (&bh->scheduled,0 ); }/* This func is async.The bottom half will do the delete action at the finial
Signature · 1 line
2.13.6
Quoted 6 lines · Sergio Lopez
Commit b7a745d added a
qemu_bh_cancelcall to the completion function as an optimization to prevent it from unnecessarily rescheduling itself.This completion function is scheduled from
worker_thread, after setting the state of a ThreadPoolElement toTHREAD_DONE.
Quoted 6 lines
This was considered to be safe, as the completion function restarts the loop just after the call to
qemu_bh_cancel. But, under certain access patterns and scheduling conditions, the loop may wrongly use a pre-fetchedelem->statevalue, reading it asTHREAD_QUEUED, and ending the completion function without having processed a pending TPE linked atpool->head.
The commit message probably should include an example of the scenario:
I/O thread worker thread
--------------------------------------------------------------------------------
speculatively read req->state
req->state = THREAD_DONE;
qemu_bh_schedule(pool->completion_bh)
bh->scheduled = 1;
qemu_bh_cancel(pool->completion_bh)
bh->scheduled = 0;
if (req->state == THREAD_DONE)
// sees THREAD_QUEUEDThe source of the misunderstanding was that qemu_bh_cancel is now being used by the _consumer_ rather than the producer, and therefore now needs to have acquire semantics just like e.g. aio_bh_poll.
Paolo
Quoted 24 lines
In some situations, if there are no other independent requests in the same aio context that could eventually trigger the scheduling of the completion function, the omitted TPE and all operations pending on it will get stuck forever.
Signed-off-by Sergio Lopez <[email protected]>--- util/async.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)util/async.c +1
-1@@ -174,7 +174,7 @@ void qemu_bh_schedule(QEMUBH *bh)*/ void qemu_bh_cancel (QEMUBH *bh) { - bh->scheduled =0 ; +atomic_mb_set (&bh->scheduled,0 ); }/* This func is async.The bottom half will do the delete action at the finial
Quoted 18 lines · Sergio Lopez
Commit b7a745d added a
qemu_bh_cancelcall to the completion function as an optimization to prevent it from unnecessarily rescheduling itself.This completion function is scheduled from
worker_thread, after setting the state of a ThreadPoolElement toTHREAD_DONE.This was considered to be safe, as the completion function restarts the loop just after the call to
qemu_bh_cancel. But, under certain access patterns and scheduling conditions, the loop may wrongly use a pre-fetchedelem->statevalue, reading it asTHREAD_QUEUED, and ending the completion function without having processed a pending TPE linked atpool->head.In some situations, if there are no other independent requests in the same aio context that could eventually trigger the scheduling of the completion function, the omitted TPE and all operations pending on it will get stuck forever.
Quoted 4 lines
Signed-off-by Sergio Lopez <[email protected]>--- util/async.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)