Skip to contentSkip to conversation

blk-mq: Replace tags->lock with SRCU for tag iterators

PATCH Applied Ming Lei 5 participants 13 messages Aug 30, 2025 – Oct 6, 2025

Subsystems (3)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

v2 cover blk-mq: Replace tags->lock with SRCU for tag iterators

Hello Jens,

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators.

Avoids scsi_host_busy() lockup during scsi host blocked in case of big cpu cores & deep queue depth.

Also the big tags lock isn't needed when reading disk sysfs attribute 'inflight' any more.

Take the following approach:

- clearing rq reference in tags->rqs[] and deferring freeing scheduler requests in SRCU callback

- replace tags->lock with srcu read lock in tags iterator.

V2:
- rebase on for-6.18/block
- add review tags

Thanks,
Ming

Ming Lei (5):
blk-mq: Move flush queue allocation into blk_mq_init_hctx()
blk-mq: Pass tag_set to blk_mq_free_rq_map/tags
blk-mq: Defer freeing of tags page_list to SRCU callback
blk-mq: Defer freeing flush queue to SRCU callback
blk-mq: Replace tags->lock with SRCU for tag iterators

 block/blk-mq-sysfs.c   |  1 -
 block/blk-mq-tag.c     | 38 +++++++++++++++---
 block/blk-mq.c         | 87 +++++++++++++++++++++---------------------
 block/blk-mq.h         |  4 +-
 block/blk.h            |  1 +
 include/linux/blk-mq.h |  2 +
 6 files changed, 80 insertions(+), 53 deletions(-)
Signature · 1 line
2.47.0
author
#1 lore raw

PATCH v2 1/5 blk-mq: Move flush queue allocation into blk_mq_init_hctx()

Move flush queue allocation into blk_mq_init_hctx() and its release into blk_mq_exit_hctx(), and prepare for replacing tags->lock with SRCU to draining inflight request walking. blk_mq_exit_hctx() is the last chance for us to get valid tag_set reference, and we need to add one SRCU to tag_set for freeing flush request via call_srcu().

It is safe to move flush queue & request release into blk_mq_exit_hctx(), because blk_mq_clear_flush_rq_mapping() clears the flush request reference int driver tags inflight request table, meantime inflight request walking is drained.

---
 block/blk-mq-sysfs.c |  1 -
 block/blk-mq.c       | 20 +++++++++++++-------
 2 files changed, 13 insertions(+), 8 deletions(-)
block/blk-mq-sysfs.c +0 -1
index 5c399ac562ea..58ec293373c6 100644
@@ -34,7 +34,6 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj)
 	struct blk_mq_hw_ctx *hctx = container_of(kobj, struct blk_mq_hw_ctx,
 						  kobj);
 
-	blk_free_flush_queue(hctx->fq);
 	sbitmap_free(&hctx->ctx_map);
 	free_cpumask_var(hctx->cpumask);
 	kfree(hctx->ctxs);
block/blk-mq.c +13 -7
index ba3a4b77f578..cfd4bbc161ac 100644
@@ -3939,6 +3939,9 @@ static void blk_mq_exit_hctx(struct request_queue *q,
 	if (set->ops->exit_hctx)
 		set->ops->exit_hctx(hctx, hctx_idx);
 
+	blk_free_flush_queue(hctx->fq);
+	hctx->fq = NULL;
+
 	xa_erase(&q->hctx_table, hctx_idx);
 
 	spin_lock(&q->unused_hctx_lock);
@@ -3964,13 +3967,19 @@ static int blk_mq_init_hctx(struct request_queue *q,
 		struct blk_mq_tag_set *set,
 		struct blk_mq_hw_ctx *hctx, unsigned hctx_idx)
 {
+	gfp_t gfp = GFP_NOIO | __GFP_NOWARN | __GFP_NORETRY;
+
+	hctx->fq = blk_alloc_flush_queue(hctx->numa_node, set->cmd_size, gfp);
+	if (!hctx->fq)
+		goto fail;
+
 	hctx->queue_num = hctx_idx;
 
 	hctx->tags = set->tags[hctx_idx];
 
 	if (set->ops->init_hctx &&
 	    set->ops->init_hctx(hctx, set->driver_data, hctx_idx))
-		goto fail;
+		goto fail_free_fq;
 
 	if (blk_mq_init_request(set, hctx->fq->flush_rq, hctx_idx,
 				hctx->numa_node))
@@ -3987,6 +3996,9 @@ static int blk_mq_init_hctx(struct request_queue *q,
  exit_hctx:
 	if (set->ops->exit_hctx)
 		set->ops->exit_hctx(hctx, hctx_idx);
+ fail_free_fq:
+	blk_free_flush_queue(hctx->fq);
+	hctx->fq = NULL;
  fail:
 	return -1;
 }
@@ -4038,16 +4050,10 @@ blk_mq_alloc_hctx(struct request_queue *q, struct blk_mq_tag_set *set,
 	init_waitqueue_func_entry(&hctx->dispatch_wait, blk_mq_dispatch_wake);
 	INIT_LIST_HEAD(&hctx->dispatch_wait.entry);
 
-	hctx->fq = blk_alloc_flush_queue(hctx->numa_node, set->cmd_size, gfp);
-	if (!hctx->fq)
-		goto free_bitmap;
-
 	blk_mq_hctx_kobj_init(hctx);
 
 	return hctx;
 
- free_bitmap:
-	sbitmap_free(&hctx->ctx_map);
  free_ctxs:
 	kfree(hctx->ctxs);
  free_cpumask:
Signature · 1 line
2.47.0
author
#2 lore raw

PATCH v2 2/5 blk-mq: Pass tag_set to blk_mq_free_rq_map/tags

In reply to Ming Lei

To prepare for converting the tag->rqs freeing to be SRCU-based, the tag_set is needed in the freeing helper functions.

This patch adds 'struct blk_mq_tag_set *' as the first parameter to blk_mq_free_rq_map() and blk_mq_free_tags(), and updates all their call sites.

This allows access to the tag_set's SRCU structure in the next step, which will be used to free the tag maps after a grace period.

No functional change is intended in this patch.

---
 block/blk-mq-tag.c |  2 +-
 block/blk-mq.c     | 10 +++++-----
 block/blk-mq.h     |  4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)
block/blk-mq-tag.c +1 -1
index d880c50629d6..6fce42851f03 100644
@@ -576,7 +576,7 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags,
 	return NULL;
 }
 
-void blk_mq_free_tags(struct blk_mq_tags *tags)
+void blk_mq_free_tags(struct blk_mq_tag_set *set, struct blk_mq_tags *tags)
 {
 	sbitmap_queue_free(&tags->bitmap_tags);
 	sbitmap_queue_free(&tags->breserved_tags);
block/blk-mq.c +5 -5
index cfd4bbc161ac..b8f13e321516 100644
@@ -3491,14 +3491,14 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,
 	}
 }
 
-void blk_mq_free_rq_map(struct blk_mq_tags *tags)
+void blk_mq_free_rq_map(struct blk_mq_tag_set *set, struct blk_mq_tags *tags)
 {
 	kfree(tags->rqs);
 	tags->rqs = NULL;
 	kfree(tags->static_rqs);
 	tags->static_rqs = NULL;
 
-	blk_mq_free_tags(tags);
+	blk_mq_free_tags(set, tags);
 }
 
 static enum hctx_type hctx_idx_to_type(struct blk_mq_tag_set *set,
@@ -3560,7 +3560,7 @@ static struct blk_mq_tags *blk_mq_alloc_rq_map(struct blk_mq_tag_set *set,
 err_free_rqs:
 	kfree(tags->rqs);
 err_free_tags:
-	blk_mq_free_tags(tags);
+	blk_mq_free_tags(set, tags);
 	return NULL;
 }
 
@@ -4107,7 +4107,7 @@ struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set,
 
 	ret = blk_mq_alloc_rqs(set, tags, hctx_idx, depth);
 	if (ret) {
-		blk_mq_free_rq_map(tags);
+		blk_mq_free_rq_map(set, tags);
 		return NULL;
 	}
 
@@ -4135,7 +4135,7 @@ void blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set,
 {
 	if (tags) {
 		blk_mq_free_rqs(set, tags, hctx_idx);
-		blk_mq_free_rq_map(tags);
+		blk_mq_free_rq_map(set, tags);
 	}
 }
 
block/blk-mq.h +2 -2
index affb2e14b56e..b96a753809ab 100644
@@ -59,7 +59,7 @@ void blk_mq_put_rq_ref(struct request *rq);
  */
 void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,
 		     unsigned int hctx_idx);
-void blk_mq_free_rq_map(struct blk_mq_tags *tags);
+void blk_mq_free_rq_map(struct blk_mq_tag_set *set, struct blk_mq_tags *tags);
 struct blk_mq_tags *blk_mq_alloc_map_and_rqs(struct blk_mq_tag_set *set,
 				unsigned int hctx_idx, unsigned int depth);
 void blk_mq_free_map_and_rqs(struct blk_mq_tag_set *set,
@@ -162,7 +162,7 @@ struct blk_mq_alloc_data {
 
 struct blk_mq_tags *blk_mq_init_tags(unsigned int nr_tags,
 		unsigned int reserved_tags, unsigned int flags, int node);
-void blk_mq_free_tags(struct blk_mq_tags *tags);
+void blk_mq_free_tags(struct blk_mq_tag_set *set, struct blk_mq_tags *tags);
 
 unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data);
 unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags,
Signature · 1 line
2.47.0
author
#3 lore raw

PATCH v2 3/5 blk-mq: Defer freeing of tags page_list to SRCU callback

In reply to Ming Lei

Tag iterators can race with the freeing of the request pages(tags->page_list), potentially leading to use-after-free issues.

Defer the freeing of the page list and the tags structure itself until after an SRCU grace period has passed. This ensures that any concurrent tag iterators have completed before the memory is released. With this way, we can replace the big tags->lock in tags iterator code path with srcu for solving the issue.

This is achieved by:
- Adding a new srcu_struct tags_srcu to blk_mq_tag_set to protect tag map iteration.
- Adding an rcu_head to struct blk_mq_tags to be used with call_srcu.
- Moving the page list freeing logic and the kfree(tags) call into a new callback function, blk_mq_free_tags_callback.
- In blk_mq_free_tags, invoking call_srcu to schedule the new callback for deferred execution.

The read-side protection for the tag iterators will be added in a subsequent patch.

---
 block/blk-mq-tag.c     | 24 +++++++++++++++++++++++-
 block/blk-mq.c         | 26 +++++++++++++-------------
 include/linux/blk-mq.h |  2 ++
 3 files changed, 38 insertions(+), 14 deletions(-)
block/blk-mq-tag.c +23 -1
index 6fce42851f03..6c2f5881e0de 100644
@@ -8,6 +8,9 @@
  */
 #include <linux/kernel.h>
 #include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/mm.h>
+#include <linux/kmemleak.h>
 
 #include <linux/delay.h>
 #include "blk.h"
@@ -576,11 +579,30 @@ struct blk_mq_tags *blk_mq_init_tags(unsigned int total_tags,
 	return NULL;
 }
 
+static void blk_mq_free_tags_callback(struct rcu_head *head)
+{
+	struct blk_mq_tags *tags = container_of(head, struct blk_mq_tags,
+						rcu_head);
+	struct page *page;
+
+	while (!list_empty(&tags->page_list)) {
+		page = list_first_entry(&tags->page_list, struct page, lru);
+		list_del_init(&page->lru);
+		/*
+		 * Remove kmemleak object previously allocated in
+		 * blk_mq_alloc_rqs().
+		 */
+		kmemleak_free(page_address(page));
+		__free_pages(page, page->private);
+	}
+	kfree(tags);
+}
+
 void blk_mq_free_tags(struct blk_mq_tag_set *set, struct blk_mq_tags *tags)
 {
 	sbitmap_queue_free(&tags->bitmap_tags);
 	sbitmap_queue_free(&tags->breserved_tags);
-	kfree(tags);
+	call_srcu(&set->tags_srcu, &tags->rcu_head, blk_mq_free_tags_callback);
 }
 
 int blk_mq_tag_update_depth(struct blk_mq_hw_ctx *hctx,
block/blk-mq.c +13 -13
index b8f13e321516..14bfdc6eadce 100644
@@ -3454,7 +3454,6 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,
 		     unsigned int hctx_idx)
 {
 	struct blk_mq_tags *drv_tags;
-	struct page *page;
 
 	if (list_empty(&tags->page_list))
 		return;
@@ -3478,17 +3477,10 @@ void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,
 	}
 
 	blk_mq_clear_rq_mapping(drv_tags, tags);
-
-	while (!list_empty(&tags->page_list)) {
-		page = list_first_entry(&tags->page_list, struct page, lru);
-		list_del_init(&page->lru);
-		/*
-		 * Remove kmemleak object previously allocated in
-		 * blk_mq_alloc_rqs().
-		 */
-		kmemleak_free(page_address(page));
-		__free_pages(page, page->private);
-	}
+	/*
+	 * Free request pages in SRCU callback, which is called from
+	 * blk_mq_free_tags().
+	 */
 }
 
 void blk_mq_free_rq_map(struct blk_mq_tag_set *set, struct blk_mq_tags *tags)
@@ -4834,6 +4826,9 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 		if (ret)
 			goto out_free_srcu;
 	}
+	ret = init_srcu_struct(&set->tags_srcu);
+	if (ret)
+		goto out_cleanup_srcu;
 
 	init_rwsem(&set->update_nr_hwq_lock);
 
@@ -4842,7 +4837,7 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 				 sizeof(struct blk_mq_tags *), GFP_KERNEL,
 				 set->numa_node);
 	if (!set->tags)
-		goto out_cleanup_srcu;
+		goto out_cleanup_tags_srcu;
 
 	for (i = 0; i < set->nr_maps; i++) {
 		set->map[i].mq_map = kcalloc_node(nr_cpu_ids,
@@ -4871,6 +4866,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 	}
 	kfree(set->tags);
 	set->tags = NULL;
+out_cleanup_tags_srcu:
+	cleanup_srcu_struct(&set->tags_srcu);
 out_cleanup_srcu:
 	if (set->flags & BLK_MQ_F_BLOCKING)
 		cleanup_srcu_struct(set->srcu);
@@ -4916,6 +4913,9 @@ void blk_mq_free_tag_set(struct blk_mq_tag_set *set)
 
 	kfree(set->tags);
 	set->tags = NULL;
+
+	srcu_barrier(&set->tags_srcu);
+	cleanup_srcu_struct(&set->tags_srcu);
 	if (set->flags & BLK_MQ_F_BLOCKING) {
 		cleanup_srcu_struct(set->srcu);
 		kfree(set->srcu);
include/linux/blk-mq.h +2 -0
index 2a5a828f19a0..1325ceeb743a 100644
@@ -531,6 +531,7 @@ struct blk_mq_tag_set {
 	struct mutex		tag_list_lock;
 	struct list_head	tag_list;
 	struct srcu_struct	*srcu;
+	struct srcu_struct	tags_srcu;
 
 	struct rw_semaphore	update_nr_hwq_lock;
 };
@@ -767,6 +768,7 @@ struct blk_mq_tags {
 	 * request pool
 	 */
 	spinlock_t lock;
+	struct rcu_head rcu_head;
 };
 
 static inline struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags,
Signature · 1 line
2.47.0
author
#4 lore raw

PATCH v2 4/5 blk-mq: Defer freeing flush queue to SRCU callback

In reply to Ming Lei

The freeing of the flush queue/request in blk_mq_exit_hctx() can race with tag iterators that may still be accessing it. To prevent a potential use-after-free, the deallocation should be deferred until after a grace period. With this way, we can replace the big tags->lock in tags iterator code path with srcu for solving the issue.

This patch introduces an SRCU-based deferred freeing mechanism for the flush queue.

The changes include:
- Adding a rcu_head to struct blk_flush_queue.
- Creating a new callback function, blk_free_flush_queue_callback, to handle the actual freeing.
- Replacing the direct call to blk_free_flush_queue() in blk_mq_exit_hctx() with call_srcu(), using the tags_srcu instance to ensure synchronization with tag iterators.

---
 block/blk-mq.c | 11 ++++++++++-
 block/blk.h    |  1 +
 2 files changed, 11 insertions(+), 1 deletion(-)
block/blk-mq.c +10 -1
index 14bfdc6eadce..c9c6e954bfbc 100644
@@ -3912,6 +3912,14 @@ static void blk_mq_clear_flush_rq_mapping(struct blk_mq_tags *tags,
 	spin_unlock_irqrestore(&tags->lock, flags);
 }
 
+static void blk_free_flush_queue_callback(struct rcu_head *head)
+{
+	struct blk_flush_queue *fq =
+		container_of(head, struct blk_flush_queue, rcu_head);
+
+	blk_free_flush_queue(fq);
+}
+
 /* hctx->ctxs will be freed in queue's release handler */
 static void blk_mq_exit_hctx(struct request_queue *q,
 		struct blk_mq_tag_set *set,
@@ -3931,7 +3939,8 @@ static void blk_mq_exit_hctx(struct request_queue *q,
 	if (set->ops->exit_hctx)
 		set->ops->exit_hctx(hctx, hctx_idx);
 
-	blk_free_flush_queue(hctx->fq);
+	call_srcu(&set->tags_srcu, &hctx->fq->rcu_head,
+			blk_free_flush_queue_callback);
 	hctx->fq = NULL;
 
 	xa_erase(&q->hctx_table, hctx_idx);
block/blk.h +1 -0
index 46f566f9b126..7d420c247d81 100644
@@ -41,6 +41,7 @@ struct blk_flush_queue {
 	struct list_head	flush_queue[2];
 	unsigned long		flush_data_in_flight;
 	struct request		*flush_rq;
+	struct rcu_head		rcu_head;
 };
 
 bool is_flush_rq(struct request *req);
Signature · 1 line
2.47.0
author
#5 lore raw

PATCH v2 5/5 blk-mq: Replace tags->lock with SRCU for tag iterators

In reply to Ming Lei

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators.

This is done by:

- Holding the SRCU read lock in blk_mq_queue_tag_busy_iter(), blk_mq_tagset_busy_iter(), and blk_mq_hctx_has_requests().

- Removing the now-redundant tags->lock from blk_mq_find_and_get_req().

This change fixes lockup issue in scsi_host_busy() in case of shost->host_blocked.

Also avoids big tags->lock when reading disk sysfs attribute inflight.

---
 block/blk-mq-tag.c | 12 ++++++++----
 block/blk-mq.c     | 24 ++++--------------------
 2 files changed, 12 insertions(+), 24 deletions(-)
block/blk-mq-tag.c +8 -4
index 6c2f5881e0de..7ae431077a32 100644
@@ -256,13 +256,10 @@ static struct request *blk_mq_find_and_get_req(struct blk_mq_tags *tags,
 		unsigned int bitnr)
 {
 	struct request *rq;
-	unsigned long flags;
 
-	spin_lock_irqsave(&tags->lock, flags);
 	rq = tags->rqs[bitnr];
 	if (!rq || rq->tag != bitnr || !req_ref_inc_not_zero(rq))
 		rq = NULL;
-	spin_unlock_irqrestore(&tags->lock, flags);
 	return rq;
 }
 
@@ -440,7 +437,9 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset,
 		busy_tag_iter_fn *fn, void *priv)
 {
 	unsigned int flags = tagset->flags;
-	int i, nr_tags;
+	int i, nr_tags, srcu_idx;
+
+	srcu_idx = srcu_read_lock(&tagset->tags_srcu);
 
 	nr_tags = blk_mq_is_shared_tags(flags) ? 1 : tagset->nr_hw_queues;
 
@@ -449,6 +448,7 @@ void blk_mq_tagset_busy_iter(struct blk_mq_tag_set *tagset,
 			__blk_mq_all_tag_iter(tagset->tags[i], fn, priv,
 					      BT_TAG_ITER_STARTED);
 	}
+	srcu_read_unlock(&tagset->tags_srcu, srcu_idx);
 }
 EXPORT_SYMBOL(blk_mq_tagset_busy_iter);
 
@@ -499,6 +499,8 @@ EXPORT_SYMBOL(blk_mq_tagset_wait_completed_request);
 void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn,
 		void *priv)
 {
+	int srcu_idx;
+
 	/*
 	 * __blk_mq_update_nr_hw_queues() updates nr_hw_queues and hctx_table
 	 * while the queue is frozen. So we can use q_usage_counter to avoid
@@ -507,6 +509,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn,
 	if (!percpu_ref_tryget(&q->q_usage_counter))
 		return;
 
+	srcu_idx = srcu_read_lock(&q->tag_set->tags_srcu);
 	if (blk_mq_is_shared_tags(q->tag_set->flags)) {
 		struct blk_mq_tags *tags = q->tag_set->shared_tags;
 		struct sbitmap_queue *bresv = &tags->breserved_tags;
@@ -536,6 +539,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn,
 			bt_for_each(hctx, q, btags, fn, priv, false);
 		}
 	}
+	srcu_read_unlock(&q->tag_set->tags_srcu, srcu_idx);
 	blk_queue_exit(q);
 }
 
block/blk-mq.c +4 -20
index c9c6e954bfbc..8191ffac998e 100644
@@ -3415,7 +3415,6 @@ static void blk_mq_clear_rq_mapping(struct blk_mq_tags *drv_tags,
 				    struct blk_mq_tags *tags)
 {
 	struct page *page;
-	unsigned long flags;
 
 	/*
 	 * There is no need to clear mapping if driver tags is not initialized
@@ -3439,15 +3438,6 @@ static void blk_mq_clear_rq_mapping(struct blk_mq_tags *drv_tags,
 			}
 		}
 	}
-
-	/*
-	 * Wait until all pending iteration is done.
-	 *
-	 * Request reference is cleared and it is guaranteed to be observed
-	 * after the ->lock is released.
-	 */
-	spin_lock_irqsave(&drv_tags->lock, flags);
-	spin_unlock_irqrestore(&drv_tags->lock, flags);
 }
 
 void blk_mq_free_rqs(struct blk_mq_tag_set *set, struct blk_mq_tags *tags,
@@ -3670,8 +3660,12 @@ static bool blk_mq_hctx_has_requests(struct blk_mq_hw_ctx *hctx)
 	struct rq_iter_data data = {
 		.hctx	= hctx,
 	};
+	int srcu_idx;
 
+	srcu_idx = srcu_read_lock(&hctx->queue->tag_set->tags_srcu);
 	blk_mq_all_tag_iter(tags, blk_mq_has_request, &data);
+	srcu_read_unlock(&hctx->queue->tag_set->tags_srcu, srcu_idx);
+
 	return data.has_rq;
 }
 
@@ -3891,7 +3885,6 @@ static void blk_mq_clear_flush_rq_mapping(struct blk_mq_tags *tags,
 		unsigned int queue_depth, struct request *flush_rq)
 {
 	int i;
-	unsigned long flags;
 
 	/* The hw queue may not be mapped yet */
 	if (!tags)
@@ -3901,15 +3894,6 @@ static void blk_mq_clear_flush_rq_mapping(struct blk_mq_tags *tags,
 
 	for (i = 0; i < queue_depth; i++)
 		cmpxchg(&tags->rqs[i], flush_rq, NULL);
-
-	/*
-	 * Wait until all pending iteration is done.
-	 *
-	 * Request reference is cleared and it is guaranteed to be observed
-	 * after the ->lock is released.
-	 */
-	spin_lock_irqsave(&tags->lock, flags);
-	spin_unlock_irqrestore(&tags->lock, flags);
 }
 
 static void blk_free_flush_queue_callback(struct rcu_head *head)
Signature · 1 line
2.47.0
#6 lore raw

Re: [PATCH V2 5/5] blk-mq: Replace tags->lock with SRCU for tag iterators - Rockchip UFS regression

Hi,

Quoted 10 lines · Ming Lei

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators.

This is done by:

- Holding the SRCU read lock in blk_mq_queue_tag_busy_iter(), blk_mq_tagset_busy_iter(), and blk_mq_hctx_has_requests().

- Removing the now-redundant tags->lock from blk_mq_find_and_get_req().

Quoted 7 lines

This change fixes lockup issue in scsi_host_busy() in case of shost->host_blocked.

Also avoids big tags->lock when reading disk sysfs attribute inflight.

---

This patch landed in the last few hours and I now see the following fatal error on the Radxa ROCK 4D (most likely Rockchip UFS in general). After reverting this patch the error is gone and things are working as expected:

[    2.713204] CPU: 7 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.17.0-g472ea195cdf3 #1 PREEMPT
[    2.713956] Hardware name: Radxa ROCK 4D (DT)
[    2.714342] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    2.714955] pc : __srcu_read_lock+0x30/0x80
[    2.715341] lr : blk_mq_tagset_busy_iter+0x44/0x300
[    2.715779] sp : ffff80008005b8b0
[    2.716073] x29: ffff80008005b8b0 x28: 0000000000000000 x27: 0000000000000000
[    2.716711] x26: ffff0000ca718898 x25: ffff0000c1595410 x24: ffff0000ca7180e0
[    2.717348] x23: ffffa77abec3f8e0 x22: ffffa77abf5dce68 x21: ffff0000c1595410
[    2.717984] x20: 0000000000000000 x19: 0000000000000000 x18: 000000000000000a
[    2.718620] x17: 000000040044ffff x16: 00500074b5503510 x15: 0000000000000000
[    2.719256] x14: 0000000000000000 x13: 303d657461747320 x12: 74736f4820534655
[    2.719892] x11: 0000000000000058 x10: 0000000000000018 x9 : ffffa77abf096cd0
[    2.720529] x8 : 0000000000057fa8 x7 : 000000000000010a x6 : ffffa77abf0eecd0
[    2.721165] x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff58893efc6000
[    2.721800] x2 : ffff0000c0268000 x1 : ffff58893efc6000 x0 : ffff0000ca718188
[    2.722437] Call trace:
[    2.722658]  __srcu_read_lock+0x30/0x80 (P)
[    2.723037]  blk_mq_tagset_busy_iter+0x44/0x300
[    2.723442]  scsi_host_busy+0x38/0x70
[    2.723774]  ufshcd_print_host_state+0x34/0x1bc
[    2.724178]  ufshcd_link_startup.constprop.0+0xe4/0x2e0
[    2.724644]  ufshcd_init+0x944/0xf80
[    2.724967]  ufshcd_pltfrm_init+0x504/0x820
[    2.725344]  ufs_rockchip_probe+0x2c/0x88
[    2.725706]  platform_probe+0x5c/0xa4
[    2.726039]  really_probe+0xc0/0x38c
[    2.726360]  __driver_probe_device+0x7c/0x150
[    2.726749]  driver_probe_device+0x40/0x120
[    2.727122]  __driver_attach+0xc8/0x1e0
[    2.727466]  bus_for_each_dev+0x7c/0xdc
[    2.727808]  driver_attach+0x24/0x30
[    2.728128]  bus_add_driver+0x110/0x230
[    2.728471]  driver_register+0x68/0x130
[    2.728815]  __platform_driver_register+0x20/0x2c
[    2.729236]  ufs_rockchip_pltform_init+0x1c/0x28
[    2.729654]  do_one_initcall+0x60/0x1e0
[    2.729999]  kernel_init_freeable+0x248/0x2c4
[    2.730394]  kernel_init+0x20/0x140
[    2.730712]  ret_from_fork+0x10/0x20
[    2.731038] Code: d2800024 aa0503e1 d538d083 8b030021 (c85f7c27)
[    2.731574] ---[ end trace 0000000000000000 ]---

Greetings,

-- Sebastian

author
#7 lore raw

Re: [PATCH V2 5/5] blk-mq: Replace tags->lock with SRCU for tag iterators - Rockchip UFS regression

Quoted 27 lines · Sebastian Reichel

Hi,

Quoted 10 lines · Ming Lei

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators.

This is done by:

- Holding the SRCU read lock in blk_mq_queue_tag_busy_iter(), blk_mq_tagset_busy_iter(), and blk_mq_hctx_has_requests().

- Removing the now-redundant tags->lock from blk_mq_find_and_get_req().

Quoted 7 lines

This change fixes lockup issue in scsi_host_busy() in case of shost->host_blocked.

Also avoids big tags->lock when reading disk sysfs attribute inflight.

---

This patch landed in the last few hours and I now see the following fatal error on the Radxa ROCK 4D (most likely Rockchip UFS in general). After reverting this patch the error is gone and things are working as expected:

Quoted 4 lines
[    2.713204] CPU: 7 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.17.0-g472ea195cdf3 #1 PREEMPT
[    2.713956] Hardware name: Radxa ROCK 4D (DT)
[    2.714342] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[    2.714955] pc : __srcu_read_lock+0x30/0x80

The problem: scsi_host_busy() tries to use tags_srcu before it's
initialized by init_srcu_struct().

The warning at __srcu_read_lock+0x30/0x80 indicates that the SRCU
structure (tags_srcu) has not been properly initialized yet. The code
at line 754 in srcutree.c tries to read ssp->srcu_ctrp, but this
pointer is likely NULL or contains garbage because init_srcu_struct()
hasn't been called yet.

Solution

The scsi_host_busy() call should only happen after the SCSI host has
been added and the tag_set has been initialized. The fix would be
to check if the tag_set is initialized before calling
scsi_host_busy(), or to defer the printing of host state until after the host is
fully initialized.

One possible fix is to check hba->scsi_host_added before calling
scsi_host_busy():

dev_err(hba->dev, "%d outstanding reqs, tasks=0x%lx\n",

      hba->scsi_host_added ? scsi_host_busy(hba->host) : 0,
      hba->outstanding_tasks);
#8 lore raw

Re: [PATCH V2 5/5] blk-mq: Replace tags->lock with SRCU for tag iterators - Rockchip UFS regression

Quoted 6 lines · Ming Lei

One possible fix is to check hba->scsi_host_added before calling
scsi_host_busy():

   dev_err(hba->dev, "%d outstanding reqs, tasks=0x%lx\n",
       hba->scsi_host_added ? scsi_host_busy(hba->host) : 0,
       hba->outstanding_tasks);

My preference is to add a check inside scsi_host_busy(), e.g. as follows (entirely untested):

drivers/scsi/hosts.c +3 -2
index cc5d05dc395c..17173239301e 100644
@@ -611,8 +611,9 @@ int scsi_host_busy(struct Scsi_Host *shost)
 {
 	int cnt = 0;
 
-	blk_mq_tagset_busy_iter(&shost->tag_set,
-				scsi_host_check_in_flight, &cnt);
+	if (shost->tag_set.ops)
+		blk_mq_tagset_busy_iter(&shost->tag_set,
+					scsi_host_check_in_flight, &cnt);
 	return cnt;
 }
 EXPORT_SYMBOL(scsi_host_busy);

Thanks,

Bart.

maintainer
#9 lore raw

Re: [PATCH V2 5/5] blk-mq: Replace tags->lock with SRCU for tag iterators - Rockchip UFS regression

Quoted 17 lines · Bart Van Assche
Quoted 6 lines · Ming Lei

   One possible fix is to check hba->scsi_host_added before calling scsi_host_busy():

   dev_err(hba->dev, "%d outstanding reqs, tasks=0x%lx\n", hba->scsi_host_added ? scsi_host_busy(hba->host) : 0, hba->outstanding_tasks);

My preference is to add a check inside scsi_host_busy(), e.g. as follows (entirely untested):

drivers/scsi/hosts.c +0 -0
index cc5d05dc395c..17173239301e 100644
@@ -611,8 +611,9 @@ int scsi_host_busy(struct Scsi_Host *shost)
  {

drivers/scsi/hosts.c int scsi_host_busy(struct Scsi_Host *shost) · line 619

Quoted 10 lines
drivers/scsi/hosts.c +3 -2
      int cnt = 0;
 
-    blk_mq_tagset_busy_iter(&shost->tag_set,
-                scsi_host_check_in_flight, &cnt);
+    if (shost->tag_set.ops)
+        blk_mq_tagset_busy_iter(&shost->tag_set,
+                    scsi_host_check_in_flight, &cnt);
      return cnt;
  }
  EXPORT_SYMBOL(scsi_host_busy);

Agree, that's a nicer way to solve it, putting that stuff outside of actual drivers.

Signature · 1 line
Jens Axboe
author
#11 lore raw
In reply to Ming Lei
Quoted 13 lines · Ming Lei

Hello Jens,

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators.

Avoids scsi_host_busy() lockup during scsi host blocked in case of big cpu cores & deep queue depth.

Also the big tags lock isn't needed when reading disk sysfs attribute 'inflight' any more.

Take the following approach:

Quoted 8 lines

- clearing rq reference in tags->rqs[] and deferring freeing scheduler requests in SRCU callback

- replace tags->lock with srcu read lock in tags iterator.

V2:
- rebase on for-6.18/block
- add review tags

Hello Jens,

Ping...

Thanks,
Ming

maintainer Applied
#12 lore raw
In reply to Ming Lei
Quoted 10 lines · Ming Lei

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators.

Avoids scsi_host_busy() lockup during scsi host blocked in case of big cpu cores & deep queue depth.

Also the big tags lock isn't needed when reading disk sysfs attribute 'inflight' any more.

[...]

Applied, thanks!

[1/5] blk-mq: Move flush queue allocation into blk_mq_init_hctx()

      commit: aba19ee71cd7c0253612d6a2a0b3a828ba9ece29
[2/5] blk-mq: Pass tag_set to blk_mq_free_rq_map/tags
      commit: 9ad8e5af327904dcc52e64ee5ab731c7018ffb0f
[3/5] blk-mq: Defer freeing of tags page_list to SRCU callback
      commit: ad0d05dbddc1bf86e92220fea873176de6b12f78
[4/5] blk-mq: Defer freeing flush queue to SRCU callback
      commit: 135b8521f21d4d4d4fde74e73b80d8e4d417e20a
[5/5] blk-mq: Replace tags->lock with SRCU for tag iterators
      commit: 995412e23bb2560a73db444e4c60bf5826b33aaa

Best regards,

Signature · 1 line
Jens Axboe