blk-mq: fix is_flush_rq
PATCH 1/1 blk-mq: fix is_flush_rq
is_flush_rq() is called from bt_iter()/bt_tags_iter(), and runs the following check:
hctx->fq->flush_rq == req
but the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:
1) memory re-order in blk_mq_rq_ctx_init():
rq->mq_hctx = data->hctx; ... refcount_set(&rq->ref, 1);
OR
2) tag re-use and ->rqs[] isn't updated with new request.
Fix the issue by re-writing is_flush_rq() as:
return rq->end_io == flush_end_io;
which turns out simpler to follow and immune to data race since we have ordered WRITE rq->end_io and refcount_set(&rq->ref, 1).
blk_mq_tagset_busy_iter")
--- block/blk-flush.c | 5 +++++ block/blk-mq.c | 2 +- block/blk.h | 6 +----- 3 files changed, 7 insertions(+), 6 deletions(-)
block/blk-flush.c +5 -0
@@ -262,6 +262,11 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error)spin_unlock_irqrestore (&fq->mq_flush_lock, flags); } +bool is_flush_rq (struct request *rq) +{ +return rq->end_io == flush_end_io; +} +/** * blk_kick_flush - consider issuing flush request * @q: request_queue being kicked
block/blk-mq.c +1 -1
@@ -911,7 +911,7 @@ static bool blk_mq_req_expired(struct request *rq, unsigned long *next)void blk_mq_put_rq_ref (struct request *rq) { -if (is_flush_rq (rq, rq->mq_hctx)) +if (is_flush_rq (rq)) rq->end_io (rq,0 );else if (refcount_dec_and_test (&rq->ref))__blk_mq_free_request (rq);
block/blk.h +1 -5
@@ -44,11 +44,7 @@ static inline void __blk_get_queue(struct request_queue *q)kobject_get (&q->kobj); } -static inline bool -is_flush_rq (struct request *req,struct blk_mq_hw_ctx *hctx) -{ -return hctx->fq->flush_rq == req; -} +bool is_flush_rq (struct request *req);struct blk_flush_queue *blk_alloc_flush_queue (int node,int cmd_size,gfp_t flags);
Signature · 1 line
2.31.1
Jens Axboe
maintainerQuoted 15 lines · Ming Lei
is_flush_rq()is called from bt_iter()/bt_tags_iter(), and runs the following check:hctx->fq->flush_rq == reqbut the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:
1) memory re-order in
blk_mq_rq_ctx_init():rq->mq_hctx = data->hctx; ... refcount_set(&rq->ref, 1);OR
Quoted 8 lines
2) tag re-use and ->rqs[] isn't updated with new request.
Fix the issue by re-writing
is_flush_rq()as:return rq->end_io == flush_end_io;which turns out simpler to follow and immune to data race since we have ordered WRITE
rq->end_ioandrefcount_set(&rq->ref, 1).
That is way better, applied thanks.
Quoted 2 lines
Fixes 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in
blk_mq_tagset_busy_iter")
I think your mailer was a bit too eager to split lines here. I fixed it up.
Signature · 1 line
Jens Axboe
block/blk-flush.c static void flush_end_io(struct request *flush_rq, blk_status_t error) · line 268
Quoted 4 lines · Ming Lei
block/blk-flush.c +4
-0+bool is_flush_rq (struct request *rq) +{ +return rq->end_io == flush_end_io; +}
My understanding is that calling is_flush_rq() is only safe if the caller guarantees that the request refcount >= 1. How about adding a comment that explains this?
Thanks,
Bart.
Quoted 9 lines · Bart Van Assche
Review
Quoted 4 lines · Ming Lei
(quoted diff) +4
-0+bool is_flush_rq(struct request *rq) +{ + return rq->end_io == flush_end_io; +}My understanding is that calling
is_flush_rq()is only safe if the caller guarantees that the request refcount >= 1. How about adding a comment that explains this?
Yeah, we can add the following words, but it isn't something urgent since is_flush_rq() is one block layer internal helper.
block/blk-flush.c +5 -0
@@ -262,6 +262,11 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error)spin_unlock_irqrestore (&fq->mq_flush_lock, flags); } +/* +* Caller has to grab refcount of this request, otherwise the flush request +* may be re-cycled, then rq->end_io is cleared and kernel panic is caused, +* see blk_mq_put_rq_ref(). +*/ bool is_flush_rq (struct request *rq) {return rq->end_io == flush_end_io;
Thanks,
Ming
Hi Ming,
Quoted 15 lines · Ming Lei
is_flush_rq()is called from bt_iter()/bt_tags_iter(), and runs the following check:hctx->fq->flush_rq == reqbut the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:
1) memory re-order in
blk_mq_rq_ctx_init():rq->mq_hctx = data->hctx; ... refcount_set(&rq->ref, 1);OR
Quoted 9 lines
2) tag re-use and ->rqs[] isn't updated with new request.
Fix the issue by re-writing
is_flush_rq()as:return rq->end_io == flush_end_io;which turns out simpler to follow and immune to data race since we have ordered WRITE
rq->end_ioandrefcount_set(&rq->ref, 1).
Recently we've run into a similar crash due to NULL rq->mq_hctx in blk_mq_put_rq_ref() on ARM, and it is a normal write request. Since memory reorder truly exists, we may also risk other uninitialized member accessing after this commit, at least we have to be more careful in busy_iter_fn...
So here you don't use memory barrier before refcount_set() is for performance consideration?
Thanks,
Joseph
Quoted 27 lines · Joseph Qi
Hi Ming,
Quoted 15 lines · Ming Lei
is_flush_rq()is called from bt_iter()/bt_tags_iter(), and runs the following check:hctx->fq->flush_rq == reqbut the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:
1) memory re-order in
blk_mq_rq_ctx_init():rq->mq_hctx = data->hctx; ... refcount_set(&rq->ref, 1);OR
Quoted 9 lines
2) tag re-use and ->rqs[] isn't updated with new request.
Fix the issue by re-writing
is_flush_rq()as:return rq->end_io == flush_end_io;which turns out simpler to follow and immune to data race since we have ordered WRITE
rq->end_ioandrefcount_set(&rq->ref, 1).
Quoted 7 lines
Recently we've run into a similar crash due to NULL
rq->mq_hctxinblk_mq_put_rq_ref()on ARM, and it is a normal write request. Since memory reorder truly exists, we may also risk other uninitialized member accessing after this commit, at least we have to be more careful inbusy_iter_fn...
So here you don't use memory barrier beforerefcount_set()is for performance consideration?
Yes, also it is much simpler to check ->end_io in concept.
Thanks,
Ming