Skip to contentSkip to conversation

blk-mq: fix is_flush_rq

PATCH In review Ming Lei 4 participants 6 messages Aug 18, 2021 – Aug 23, 2021 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 blk-mq: fix is_flush_rq

is_flush_rq() is called from bt_iter()/bt_tags_iter(), and runs the following check:

	hctx->fq->flush_rq == req

but the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:

1) memory re-order in blk_mq_rq_ctx_init():

	rq->mq_hctx = data->hctx;
	...
	refcount_set(&rq->ref, 1);

OR

2) tag re-use and ->rqs[] isn't updated with new request.

Fix the issue by re-writing is_flush_rq() as:

	return rq->end_io == flush_end_io;

which turns out simpler to follow and immune to data race since we have ordered WRITE rq->end_io and refcount_set(&rq->ref, 1).

Fixes 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in

blk_mq_tagset_busy_iter")

Cc "Blank-Burian, Markus, Dr." <[email protected]>
Cc Yufen Yu <[email protected]>
---
 block/blk-flush.c | 5 +++++
 block/blk-mq.c    | 2 +-
 block/blk.h       | 6 +-----
 3 files changed, 7 insertions(+), 6 deletions(-)
block/blk-flush.c +5 -0
index 4912c8dbb1d8..4201728bf3a5 100644
@@ -262,6 +262,11 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error)
 	spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
 }
 
+bool is_flush_rq(struct request *rq)
+{
+	return rq->end_io == flush_end_io;
+}
+
 /**
  * blk_kick_flush - consider issuing flush request
  * @q: request_queue being kicked
block/blk-mq.c +1 -1
index b5237211ccb7..6d7395ad6d94 100644
@@ -911,7 +911,7 @@ static bool blk_mq_req_expired(struct request *rq, unsigned long *next)
 
 void blk_mq_put_rq_ref(struct request *rq)
 {
-	if (is_flush_rq(rq, rq->mq_hctx))
+	if (is_flush_rq(rq))
 		rq->end_io(rq, 0);
 	else if (refcount_dec_and_test(&rq->ref))
 		__blk_mq_free_request(rq);
block/blk.h +1 -5
index 56f33fbcde59..d4fb5b72d12b 100644
@@ -44,11 +44,7 @@ static inline void __blk_get_queue(struct request_queue *q)
 	kobject_get(&q->kobj);
 }
 
-static inline bool
-is_flush_rq(struct request *req, struct blk_mq_hw_ctx *hctx)
-{
-	return hctx->fq->flush_rq == req;
-}
+bool is_flush_rq(struct request *req);
 
 struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,
 					      gfp_t flags);
Signature · 1 line
2.31.1
maintainer
#1 lore raw
Quoted 15 lines · Ming Lei

is_flush_rq() is called from bt_iter()/bt_tags_iter(), and runs the following check:

	hctx->fq->flush_rq == req

but the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:

1) memory re-order in blk_mq_rq_ctx_init():

	rq->mq_hctx = data->hctx;
	...
	refcount_set(&rq->ref, 1);

OR

Quoted 8 lines

2) tag re-use and ->rqs[] isn't updated with new request.

Fix the issue by re-writing is_flush_rq() as:

	return rq->end_io == flush_end_io;

which turns out simpler to follow and immune to data race since we have ordered WRITE rq->end_io and refcount_set(&rq->ref, 1).

That is way better, applied thanks.

Quoted 2 lines
Fixes 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in

blk_mq_tagset_busy_iter")

I think your mailer was a bit too eager to split lines here. I fixed it up.

Signature · 1 line
Jens Axboe
#2 lore raw
In reply to Ming Lei

block/blk-flush.c static void flush_end_io(struct request *flush_rq, blk_status_t error) · line 268

Quoted 4 lines · Ming Lei
block/blk-flush.c +4 -0
+bool is_flush_rq(struct request *rq)
+{
+	return rq->end_io == flush_end_io;
+}

My understanding is that calling is_flush_rq() is only safe if the caller guarantees that the request refcount >= 1. How about adding a comment that explains this?

Thanks,

Bart.

author
#3 lore raw
Quoted 9 lines · Bart Van Assche

Review

Quoted 4 lines · Ming Lei
(quoted diff) +4 -0
+bool is_flush_rq(struct request *rq)
+{
+	return rq->end_io == flush_end_io;
+}

My understanding is that calling is_flush_rq() is only safe if the caller guarantees that the request refcount >= 1. How about adding a comment that explains this?

Yeah, we can add the following words, but it isn't something urgent since is_flush_rq() is one block layer internal helper.

block/blk-flush.c +5 -0
index 4201728bf3a5..babf6262120e 100644
@@ -262,6 +262,11 @@ static void flush_end_io(struct request *flush_rq, blk_status_t error)
 	spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
 }
 
+/*
+ * Caller has to grab refcount of this request, otherwise the flush request
+ * may be re-cycled, then rq->end_io is cleared and kernel panic is caused,
+ * see blk_mq_put_rq_ref().
+ */
 bool is_flush_rq(struct request *rq)
 {
 	return rq->end_io == flush_end_io;

Thanks,
Ming

#4 lore raw
In reply to Ming Lei

Hi Ming,

Quoted 15 lines · Ming Lei

is_flush_rq() is called from bt_iter()/bt_tags_iter(), and runs the following check:

	hctx->fq->flush_rq == req

but the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:

1) memory re-order in blk_mq_rq_ctx_init():

	rq->mq_hctx = data->hctx;
	...
	refcount_set(&rq->ref, 1);

OR

Quoted 9 lines

2) tag re-use and ->rqs[] isn't updated with new request.

Fix the issue by re-writing is_flush_rq() as:

	return rq->end_io == flush_end_io;

which turns out simpler to follow and immune to data race since we have ordered WRITE rq->end_io and refcount_set(&rq->ref, 1).

Recently we've run into a similar crash due to NULL rq->mq_hctx in blk_mq_put_rq_ref() on ARM, and it is a normal write request. Since memory reorder truly exists, we may also risk other uninitialized member accessing after this commit, at least we have to be more careful in busy_iter_fn...
So here you don't use memory barrier before refcount_set() is for performance consideration?

Thanks,
Joseph

author
#5 lore raw
Quoted 27 lines · Joseph Qi

Hi Ming,

Quoted 15 lines · Ming Lei

is_flush_rq() is called from bt_iter()/bt_tags_iter(), and runs the following check:

	hctx->fq->flush_rq == req

but the passed hctx from bt_iter()/bt_tags_iter() may be NULL because:

1) memory re-order in blk_mq_rq_ctx_init():

	rq->mq_hctx = data->hctx;
	...
	refcount_set(&rq->ref, 1);

OR

Quoted 9 lines

2) tag re-use and ->rqs[] isn't updated with new request.

Fix the issue by re-writing is_flush_rq() as:

	return rq->end_io == flush_end_io;

which turns out simpler to follow and immune to data race since we have ordered WRITE rq->end_io and refcount_set(&rq->ref, 1).

Quoted 7 lines

Recently we've run into a similar crash due to NULL rq->mq_hctx in blk_mq_put_rq_ref() on ARM, and it is a normal write request. Since memory reorder truly exists, we may also risk other uninitialized member accessing after this commit, at least we have to be more careful in busy_iter_fn...
So here you don't use memory barrier before refcount_set() is for performance consideration?

Yes, also it is much simpler to check ->end_io in concept.

Thanks,
Ming