Skip to contentSkip to conversation

Submit zoned writes in order

PATCH In review Bart Van Assche 5 participants 24 messages Jun 21, 2023 – Jun 27, 2023

Subsystems (9)
SCSI Block layer s390 Device mapper +5 more subsystems
lore
Share & tools
lore
#0 lore raw

v4 cover Submit zoned writes in order

Hi Jens,

Tests with a zoned UFS prototype have shown that the block layer may reorder zoned writes (REQ_OP_WRITE). The UFS driver is more likely to trigger reordering than other SCSI drivers because it reports BLK_STS_DEV_RESOURCE more often, e.g. during clock scaling. This patch series makes sure that zoned writes are submitted in order without affecting other workloads significantly.

Please consider this patch series for the next merge window.

Thanks,

Bart.

Changes compared to v3:
- Dropped the changes for sending already dispatched requests back to the I/O scheduler if the block driver is busy.
- Dropped several patches that aren't needed to preserve the order of zoned writes.
- Only send requeued writes to the I/O scheduler. Send requeued reads straight to the dispatch queue.
- Changed the approach from one requeue list per request queue into one requeue list per hardware queue.

Changes compared to v2:
- Changed the approach from one requeue list per hctx into preserving one requeue list per request queue.
- Rebased on top of Jens' for-next branch. Left out the mq-deadline patches since these are already in the for-next branch.
- Modified patch "block: Requeue requests if a CPU is unplugged" such that it always uses the requeue list.
- Added a patch that removes blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list().
- Dropped patch "block: mq-deadline: Disable head insertion for zoned writes".
- Dropped patch "block: mq-deadline: Introduce a local variable".

Changes compared to v1:
- Fixed two issues detected by the kernel test robot.

Bart Van Assche (7):
block: Rename a local variable in blk_mq_requeue_work()
block: Simplify blk_mq_requeue_work()
block: Send requeued requests to the I/O scheduler
block: One requeue list per hctx
block: Preserve the order of requeued requests
dm: Inline __dm_mq_kick_requeue_list()
block: Inline blk_mq_{,delay_}kick_requeue_list()

 block/blk-flush.c            | 28 +++++++------
 block/blk-mq-debugfs.c       | 66 ++++++++++++++---------------
 block/blk-mq.c               | 81 ++++++++++++++----------------------
 drivers/block/ublk_drv.c     |  6 +--
 drivers/block/xen-blkfront.c |  1 -
 drivers/md/dm-rq.c           | 11 ++---
 drivers/nvme/host/core.c     |  2 +-
 drivers/s390/block/scm_blk.c |  2 +-
 drivers/scsi/scsi_lib.c      |  2 +-
 include/linux/blk-mq.h       | 11 +++--
 include/linux/blkdev.h       |  5 ---
 11 files changed, 96 insertions(+), 119 deletions(-)
#1 lore raw

PATCH v4 1/7 block: Rename a local variable in blk_mq_requeue_work()

Two data structures in blk_mq_requeue_work() represent request lists. Make it clear that rq_list holds requests that come from the requeue list by renaming that data structure.

---
 block/blk-mq.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -4
index 720b5061ffe8..41ee393c80a9 100644
@@ -1441,17 +1441,17 @@ static void blk_mq_requeue_work(struct work_struct *work)
 {
 	struct request_queue *q =
 		container_of(work, struct request_queue, requeue_work.work);
-	LIST_HEAD(rq_list);
+	LIST_HEAD(requeue_list);
 	LIST_HEAD(flush_list);
 	struct request *rq;
 
 	spin_lock_irq(&q->requeue_lock);
-	list_splice_init(&q->requeue_list, &rq_list);
+	list_splice_init(&q->requeue_list, &requeue_list);
 	list_splice_init(&q->flush_list, &flush_list);
 	spin_unlock_irq(&q->requeue_lock);
 
-	while (!list_empty(&rq_list)) {
-		rq = list_entry(rq_list.next, struct request, queuelist);
+	while (!list_empty(&requeue_list)) {
+		rq = list_entry(requeue_list.next, struct request, queuelist);
 		/*
 		 * If RQF_DONTPREP ist set, the request has been started by the
 		 * driver already and might have driver-specific data allocated
Reviewed-by on v4 1/7
#2 lore raw

Re: [PATCH v4 1/7] block: Rename a local variable in blk_mq_requeue_work()

Quoted 10 lines · Bart Van Assche

Two data structures in blk_mq_requeue_work() represent request lists. Make it clear that rq_list holds requests that come from the requeue list by renaming that data structure.

Signature · 2 lines
Damien Le Moal
Western Digital Research
#3 lore raw

PATCH v4 2/7 block: Simplify blk_mq_requeue_work()

In reply to Bart Van Assche

Move common code in front of the if-statement.

Cc Christoph Hellwig <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Ming Lei <[email protected]>
Cc Mike Snitzer <[email protected]>
---
 block/blk-mq.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
block/blk-mq.c +3 -5
index 41ee393c80a9..f440e4aaaae3 100644
@@ -1458,13 +1458,11 @@ static void blk_mq_requeue_work(struct work_struct *work)
 		 * already.  Insert it into the hctx dispatch list to avoid
 		 * block layer merges for the request.
 		 */
-		if (rq->rq_flags & RQF_DONTPREP) {
-			list_del_init(&rq->queuelist);
+		list_del_init(&rq->queuelist);
+		if (rq->rq_flags & RQF_DONTPREP)
 			blk_mq_request_bypass_insert(rq, 0);
-		} else {
-			list_del_init(&rq->queuelist);
+		else
 			blk_mq_insert_request(rq, BLK_MQ_INSERT_AT_HEAD);
-		}
 	}
 
 	while (!list_empty(&flush_list)) {
Reviewed-by on v4 2/7
#4 lore raw

Re: [PATCH v4 2/7] block: Simplify blk_mq_requeue_work()

Quoted 7 lines · Bart Van Assche

Move common code in front of the if-statement.

Cc Christoph Hellwig <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Ming Lei <[email protected]>
Cc Mike Snitzer <[email protected]>
Signature · 2 lines
Damien Le Moal
Western Digital Research
#6 lore raw

PATCH v4 3/7 block: Send requeued requests to the I/O scheduler

In reply to Bart Van Assche

Send requeued requests to the I/O scheduler if the dispatch order matters such that the I/O scheduler can control the order in which requests are dispatched.

This patch reworks commit aef1897cd36d ("blk-mq: insert rq with DONTPREP to hctx dispatch list when requeue"). Instead of sending DONTPREP requests to the dispatch list, send these to the I/O scheduler and prevent that the I/O scheduler merges these requests by adding RQF_DONTPREP to the list of flags that prevent merging (RQF_NOMERGE_FLAGS).

Cc Christoph Hellwig <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Ming Lei <[email protected]>
Cc Mike Snitzer <[email protected]>
---
 block/blk-mq.c         | 10 +++++-----
 include/linux/blk-mq.h |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)
block/blk-mq.c +5 -5
index f440e4aaaae3..453a90767f7a 100644
@@ -1453,13 +1453,13 @@ static void blk_mq_requeue_work(struct work_struct *work)
 	while (!list_empty(&requeue_list)) {
 		rq = list_entry(requeue_list.next, struct request, queuelist);
 		/*
-		 * If RQF_DONTPREP ist set, the request has been started by the
-		 * driver already and might have driver-specific data allocated
-		 * already.  Insert it into the hctx dispatch list to avoid
-		 * block layer merges for the request.
+		 * Only send those RQF_DONTPREP requests to the dispatch list
+		 * that may be reordered freely. If the request order matters,
+		 * send the request to the I/O scheduler.
 		 */
 		list_del_init(&rq->queuelist);
-		if (rq->rq_flags & RQF_DONTPREP)
+		if (rq->rq_flags & RQF_DONTPREP &&
+		    !op_needs_zoned_write_locking(req_op(rq)))
 			blk_mq_request_bypass_insert(rq, 0);
 		else
 			blk_mq_insert_request(rq, BLK_MQ_INSERT_AT_HEAD);
include/linux/blk-mq.h +2 -2
index f401067ac03a..2610b299ec77 100644
@@ -62,8 +62,8 @@ typedef __u32 __bitwise req_flags_t;
 #define RQF_RESV		((__force req_flags_t)(1 << 23))
 
 /* flags that prevent us from merging requests: */
-#define RQF_NOMERGE_FLAGS \
-	(RQF_STARTED | RQF_FLUSH_SEQ | RQF_SPECIAL_PAYLOAD)
+#define RQF_NOMERGE_FLAGS                                               \
+	(RQF_STARTED | RQF_FLUSH_SEQ | RQF_DONTPREP | RQF_SPECIAL_PAYLOAD)
 
 enum mq_rq_state {
 	MQ_RQ_IDLE		= 0,
#7 lore raw

Re: [PATCH v4 3/7] block: Send requeued requests to the I/O scheduler

Quoted 31 lines · Bart Van Assche

Send requeued requests to the I/O scheduler if the dispatch order matters such that the I/O scheduler can control the order in which requests are dispatched.

This patch reworks commit aef1897cd36d ("blk-mq: insert rq with DONTPREP to hctx dispatch list when requeue"). Instead of sending DONTPREP requests to the dispatch list, send these to the I/O scheduler and prevent that the I/O scheduler merges these requests by adding RQF_DONTPREP to the list of flags that prevent merging (RQF_NOMERGE_FLAGS).

Cc Christoph Hellwig <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Ming Lei <[email protected]>
Cc Mike Snitzer <[email protected]>
---
 block/blk-mq.c         | 10 +++++-----
 include/linux/blk-mq.h |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)
block/blk-mq.c +0 -2
index f440e4aaaae3..453a90767f7a 100644
@@ -1453,13 +1453,13 @@ static void blk_mq_requeue_work(struct work_struct *work)
 	while (!list_empty(&requeue_list)) {
 		rq = list_entry(requeue_list.next, struct request, queuelist);
 		/*
-		 * If RQF_DONTPREP ist set, the request has been started by the
-		 * driver already and might have driver-specific data allocated

block/blk-mq.c static void blk_mq_requeue_work(struct work_struct *work) · line 1462

Quoted 10 lines
block/blk-mq.c +5 -3
-		 * already.  Insert it into the hctx dispatch list to avoid
-		 * block layer merges for the request.
+		 * Only send those RQF_DONTPREP requests to the dispatch list
+		 * that may be reordered freely. If the request order matters,
+		 * send the request to the I/O scheduler.
 		 */
 		list_del_init(&rq->queuelist);
-		if (rq->rq_flags & RQF_DONTPREP)
+		if (rq->rq_flags & RQF_DONTPREP &&
+		    !op_needs_zoned_write_locking(req_op(rq)))

Why ? I still do not understand the need for this. There is always only a single in-flight write per sequential zone. Requeuing that in-flight write directly to the dispatch list will not reorder writes and it will be better for the command latency.

Quoted 18 lines
block/blk-mq.c +0 -0
 			blk_mq_request_bypass_insert(rq, 0);
 		else
 			blk_mq_insert_request(rq, BLK_MQ_INSERT_AT_HEAD);
include/linux/blk-mq.h +2 -2
index f401067ac03a..2610b299ec77 100644
@@ -62,8 +62,8 @@ typedef __u32 __bitwise req_flags_t;
 #define RQF_RESV		((__force req_flags_t)(1 << 23))
 
 /* flags that prevent us from merging requests: */
-#define RQF_NOMERGE_FLAGS \
-	(RQF_STARTED | RQF_FLUSH_SEQ | RQF_SPECIAL_PAYLOAD)
+#define RQF_NOMERGE_FLAGS                                               \
+	(RQF_STARTED | RQF_FLUSH_SEQ | RQF_DONTPREP | RQF_SPECIAL_PAYLOAD)
 
 enum mq_rq_state {
 	MQ_RQ_IDLE		= 0,
Signature · 2 lines
Damien Le Moal
Western Digital Research
#8 lore raw

Re: [PATCH v4 3/7] block: Send requeued requests to the I/O scheduler

Quoted 4 lines · Damien Le Moal

Why ? I still do not understand the need for this. There is always only a single in-flight write per sequential zone. Requeuing that in-flight write directly to the dispatch list will not reorder writes and it will be better for the command latency.

Hi Damien,

After having taken a closer look I see that blk_req_zone_write_unlock() is called from inside dd_insert_request() when requeuing a request. Hence, there is no reordering risk when requeuing a zoned write. I will drop this patch.

Do you agree with having one requeue list per hctx instead of per request queue? This change enables eliminating blk_mq_kick_requeue_list(). I think that's an interesting simplification of the block layer API.

Thanks,

Bart.

#9 lore raw

Re: [PATCH v4 3/7] block: Send requeued requests to the I/O scheduler

Quoted 11 lines · Bart Van Assche
Quoted 4 lines · Damien Le Moal

Why ? I still do not understand the need for this. There is always only a single in-flight write per sequential zone. Requeuing that in-flight write directly to the dispatch list will not reorder writes and it will be better for the command latency.

Hi Damien,

After having taken a closer look I see that blk_req_zone_write_unlock() is called from inside dd_insert_request() when requeuing a request. Hence, there is no reordering risk when requeuing a zoned write. I will drop this patch.

OK. Thanks.

Quoted 5 lines

Do you agree with having one requeue list per hctx instead of per request queue? This change enables eliminating blk_mq_kick_requeue_list(). I think that's an interesting simplification of the block layer API.

I do not see any issue with that. Indeed, it does simplify the code nicely. Reading patch 5, I wondered though if it is really worth keeping the helpers blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list(). May be calling blk_mq_run_hw_queues() and blk_mq_delay_run_hw_queues() is better ? No strong opinion though.

Signature · 2 lines
Damien Le Moal
Western Digital Research
#10 lore raw

PATCH v4 4/7 block: One requeue list per hctx

In reply to Bart Van Assche

Prepare for processing the requeue list from inside __blk_mq_run_hw_queue().

Cc Christoph Hellwig <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Ming Lei <[email protected]>
Cc Mike Snitzer <[email protected]>
---
 block/blk-flush.c      | 24 ++++++++--------
 block/blk-mq-debugfs.c | 64 +++++++++++++++++++++---------------------
 block/blk-mq.c         | 53 ++++++++++++++++++++--------------
 include/linux/blk-mq.h |  6 ++++
 include/linux/blkdev.h |  5 ----
 5 files changed, 83 insertions(+), 69 deletions(-)
block/blk-flush.c +13 -11
index dba392cf22be..4bfb92f58aa9 100644
@@ -91,7 +91,7 @@ enum {
 	FLUSH_PENDING_TIMEOUT	= 5 * HZ,
 };
 
-static void blk_kick_flush(struct request_queue *q,
+static void blk_kick_flush(struct blk_mq_hw_ctx *hctx,
 			   struct blk_flush_queue *fq, blk_opf_t flags);
 
 static inline struct blk_flush_queue *
@@ -165,6 +165,7 @@ static void blk_flush_complete_seq(struct request *rq,
 				   unsigned int seq, blk_status_t error)
 {
 	struct request_queue *q = rq->q;
+	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
 	struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
 	blk_opf_t cmd_flags;
 
@@ -188,9 +189,9 @@ static void blk_flush_complete_seq(struct request *rq,
 
 	case REQ_FSEQ_DATA:
 		list_move_tail(&rq->flush.list, &fq->flush_data_in_flight);
-		spin_lock(&q->requeue_lock);
-		list_add_tail(&rq->queuelist, &q->flush_list);
-		spin_unlock(&q->requeue_lock);
+		spin_lock(&hctx->requeue_lock);
+		list_add_tail(&rq->queuelist, &hctx->flush_list);
+		spin_unlock(&hctx->requeue_lock);
 		blk_mq_kick_requeue_list(q);
 		break;
 
@@ -210,7 +211,7 @@ static void blk_flush_complete_seq(struct request *rq,
 		BUG();
 	}
 
-	blk_kick_flush(q, fq, cmd_flags);
+	blk_kick_flush(hctx, fq, cmd_flags);
 }
 
 static enum rq_end_io_ret flush_end_io(struct request *flush_rq,
@@ -275,7 +276,7 @@ bool is_flush_rq(struct request *rq)
 
 /**
  * blk_kick_flush - consider issuing flush request
- * @q: request_queue being kicked
+ * @hctx: hwq being kicked
  * @fq: flush queue
  * @flags: cmd_flags of the original request
  *
@@ -286,9 +287,10 @@ bool is_flush_rq(struct request *rq)
  * spin_lock_irq(fq->mq_flush_lock)
  *
  */
-static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq,
-			   blk_opf_t flags)
+static void blk_kick_flush(struct blk_mq_hw_ctx *hctx,
+			   struct blk_flush_queue *fq, blk_opf_t flags)
 {
+	struct request_queue *q = hctx->queue;
 	struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
 	struct request *first_rq =
 		list_first_entry(pending, struct request, flush.list);
@@ -348,9 +350,9 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq,
 	smp_wmb();
 	req_ref_set(flush_rq, 1);
 
-	spin_lock(&q->requeue_lock);
-	list_add_tail(&flush_rq->queuelist, &q->flush_list);
-	spin_unlock(&q->requeue_lock);
+	spin_lock(&hctx->requeue_lock);
+	list_add_tail(&flush_rq->queuelist, &hctx->flush_list);
+	spin_unlock(&hctx->requeue_lock);
 
 	blk_mq_kick_requeue_list(q);
 }
block/blk-mq-debugfs.c +32 -32

88 lines not on this page · raw message

block/blk-mq.c +32 -21

110 lines not on this page · raw message

include/linux/blk-mq.h +6 -0
index 2610b299ec77..672e8880f9e2 100644
@@ -308,6 +308,12 @@ struct blk_mq_hw_ctx {
 		unsigned long		state;
 	} ____cacheline_aligned_in_smp;
 
+	struct list_head	flush_list;
+
+	struct list_head	requeue_list;
+	spinlock_t		requeue_lock;
+	struct delayed_work	requeue_work;
+
 	/**
 	 * @run_work: Used for scheduling a hardware queue run at a later time.
 	 */
include/linux/blkdev.h +0 -5
index ed44a997f629..ed4f89657f1f 100644
@@ -479,11 +479,6 @@ struct request_queue {
 	 * for flush operations
 	 */
 	struct blk_flush_queue	*fq;
-	struct list_head	flush_list;
-
-	struct list_head	requeue_list;
-	spinlock_t		requeue_lock;
-	struct delayed_work	requeue_work;
 
 	struct mutex		sysfs_lock;
 	struct mutex		sysfs_dir_lock;
maintainer Reviewed-by on v4 4/7
#11 lore raw

Re: [PATCH v4 4/7] block: One requeue list per hctx

Quoted 15 lines

void blk_mq_kick_requeue_list(struct request_queue *q) {

block/blk-mq.c +6 -1
-	kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND, &q->requeue_work, 0);
+	struct blk_mq_hw_ctx *hctx;
+	unsigned long i;
+
+	queue_for_each_hw_ctx(q, hctx, i)
+		kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND,
+					    &hctx->requeue_work, 0);
 }
 EXPORT_SYMBOL(blk_mq_kick_requeue_list);
 
 void blk_mq_delay_kick_requeue_list(struct request_queue *q,
 				    unsigned long msecs)
 {

block/blk-mq.c static void blk_mq_requeue_work(struct work_struct *work) · line 1499

Quoted 10 lines
block/blk-mq.c +7 -2
-	kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND, &q->requeue_work,
-				    msecs_to_jiffies(msecs));
+	struct blk_mq_hw_ctx *hctx;
+	unsigned long i;
+
+	queue_for_each_hw_ctx(q, hctx, i)
+		kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND,
+					    &hctx->requeue_work,
+					    msecs_to_jiffies(msecs));
 }

I'd iplement blk_mq_kick_requeue_list as a wrapper around blk_mq_delay_kick_requeue_list that passes a 0 msec argument.

Otherwise looks fine:

#12 lore raw

Re: [PATCH v4 4/7] block: One requeue list per hctx

In reply to Bart Van Assche
Quoted 1 line · Bart Van Assche

Prepare for processing the requeue list from inside __blk_mq_run_hw_queue().

Please add more details about the motivation for this kind of change.

IMO requeue is supposed to be run in slow code path, not see reason why it has to be moved to hw queue level.

Thanks,
Ming

#13 lore raw

Re: [PATCH v4 4/7] block: One requeue list per hctx

Quoted 7 lines · Ming Lei
Quoted 1 line · Bart Van Assche

Prepare for processing the requeue list from inside __blk_mq_run_hw_queue().

Please add more details about the motivation for this kind of change.

IMO requeue is supposed to be run in slow code path, not see reason why it has to be moved to hw queue level.

As Damien explained, the code for sending requeued requests to the I/O scheduler should be dropped because this is not necessary in order to prevent reordering of requeued zoned writes.

The remaining advantages of this patch series are:
* Simplification of the block layer API since the blk_mq_{,delay_}kick_requeue_list() functions are removed.
* Faster requeuing. Although I don't have any use case that needs faster requeuing, it is interesting to see that this patch series makes the blktest test that tests requeuing run twice as fast.

Is there agreement to proceed with this patch series if all review comments are addressed?

Thanks,

Bart.

#14 lore raw

Re: [PATCH v4 4/7] block: One requeue list per hctx

Quoted 13 lines · Bart Van Assche
Quoted 7 lines · Ming Lei
Quoted 1 line · Bart Van Assche

Prepare for processing the requeue list from inside __blk_mq_run_hw_queue().

Please add more details about the motivation for this kind of change.

IMO requeue is supposed to be run in slow code path, not see reason why it has to be moved to hw queue level.

As Damien explained, the code for sending requeued requests to the I/O scheduler should be dropped because this is not necessary in order to prevent reordering of requeued zoned writes.

Quoted 6 lines

The remaining advantages of this patch series are:
* Simplification of the block layer API since the blk_mq_{,delay_}kick_requeue_list() functions are removed.
* Faster requeuing. Although I don't have any use case that needs faster requeuing, it is interesting to see that this patch series makes the blktest test that tests requeuing run twice as fast.

Not sure if it is good, cause requeue is supposed to be slow.

Quoted 3 lines

Is there agreement to proceed with this patch series if all review comments are addressed?

One concern is that you added requeue stuff into hctx, which may add one extra L1 cache fetch in fast path.

Thank,
Ming

#15 lore raw

PATCH v4 5/7 block: Preserve the order of requeued requests

In reply to Bart Van Assche

If a queue is run before all requeued requests have been sent to the I/O scheduler, the I/O scheduler may dispatch the wrong request. Fix this by making blk_mq_run_hw_queue() process the requeue_list instead of blk_mq_requeue_work().

Cc Christoph Hellwig <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Ming Lei <[email protected]>
Cc Mike Snitzer <[email protected]>
---
 block/blk-mq.c         | 31 +++++++++----------------------
 include/linux/blk-mq.h |  1 -
 2 files changed, 9 insertions(+), 23 deletions(-)
block/blk-mq.c +9 -22
index c359a28d9b25..de39984d17c4 100644
@@ -68,6 +68,8 @@ static inline blk_qc_t blk_rq_to_qc(struct request *rq)
 static bool blk_mq_hctx_has_pending(struct blk_mq_hw_ctx *hctx)
 {
 	return !list_empty_careful(&hctx->dispatch) ||
+		!list_empty_careful(&hctx->requeue_list) ||
+		!list_empty_careful(&hctx->flush_list) ||
 		sbitmap_any_bit_set(&hctx->ctx_map) ||
 			blk_mq_sched_has_work(hctx);
 }
@@ -1438,10 +1440,8 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list)
 }
 EXPORT_SYMBOL(blk_mq_requeue_request);
 
-static void blk_mq_requeue_work(struct work_struct *work)
+static void blk_mq_process_requeue_list(struct blk_mq_hw_ctx *hctx)
 {
-	struct blk_mq_hw_ctx *hctx =
-		container_of(work, struct blk_mq_hw_ctx, requeue_work.work);
 	LIST_HEAD(requeue_list);
 	LIST_HEAD(flush_list);
 	struct request *rq;
@@ -1471,31 +1471,18 @@ static void blk_mq_requeue_work(struct work_struct *work)
 		list_del_init(&rq->queuelist);
 		blk_mq_insert_request(rq, 0);
 	}
-
-	blk_mq_run_hw_queue(hctx, false);
 }
 
 void blk_mq_kick_requeue_list(struct request_queue *q)
 {
-	struct blk_mq_hw_ctx *hctx;
-	unsigned long i;
-
-	queue_for_each_hw_ctx(q, hctx, i)
-		kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND,
-					    &hctx->requeue_work, 0);
+	blk_mq_run_hw_queues(q, true);
 }
 EXPORT_SYMBOL(blk_mq_kick_requeue_list);
 
 void blk_mq_delay_kick_requeue_list(struct request_queue *q,
 				    unsigned long msecs)
 {
-	struct blk_mq_hw_ctx *hctx;
-	unsigned long i;
-
-	queue_for_each_hw_ctx(q, hctx, i)
-		kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND,
-					    &hctx->requeue_work,
-					    msecs_to_jiffies(msecs));
+	blk_mq_delay_run_hw_queues(q, msecs);
 }
 EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list);
 
@@ -2248,6 +2235,7 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async)
 		return;
 	}
 
+	blk_mq_process_requeue_list(hctx);
 	blk_mq_run_dispatch_ops(hctx->queue,
 				blk_mq_sched_dispatch_requests(hctx));
 }
@@ -2296,7 +2284,7 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)
 		 * scheduler.
 		 */
 		if (!sq_hctx || sq_hctx == hctx ||
-		    !list_empty_careful(&hctx->dispatch))
+		    blk_mq_hctx_has_pending(hctx))
 			blk_mq_run_hw_queue(hctx, async);
 	}
 }
@@ -2332,7 +2320,7 @@ void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs)
 		 * scheduler.
 		 */
 		if (!sq_hctx || sq_hctx == hctx ||
-		    !list_empty_careful(&hctx->dispatch))
+		    blk_mq_hctx_has_pending(hctx))
 			blk_mq_delay_run_hw_queue(hctx, msecs);
 	}
 }
@@ -2417,6 +2405,7 @@ static void blk_mq_run_work_fn(struct work_struct *work)
 	struct blk_mq_hw_ctx *hctx =
 		container_of(work, struct blk_mq_hw_ctx, run_work.work);
 
+	blk_mq_process_requeue_list(hctx);
 	blk_mq_run_dispatch_ops(hctx->queue,
 				blk_mq_sched_dispatch_requests(hctx));
 }
@@ -3625,7 +3614,6 @@ static int blk_mq_init_hctx(struct request_queue *q,
 		struct blk_mq_tag_set *set,
 		struct blk_mq_hw_ctx *hctx, unsigned hctx_idx)
 {
-	INIT_DELAYED_WORK(&hctx->requeue_work, blk_mq_requeue_work);
 	INIT_LIST_HEAD(&hctx->flush_list);
 	INIT_LIST_HEAD(&hctx->requeue_list);
 	spin_lock_init(&hctx->requeue_lock);
@@ -4794,7 +4782,6 @@ void blk_mq_cancel_work_sync(struct request_queue *q)
 	unsigned long i;
 
 	queue_for_each_hw_ctx(q, hctx, i) {
-		cancel_delayed_work_sync(&hctx->requeue_work);
 		cancel_delayed_work_sync(&hctx->run_work);
 	}
 }
include/linux/blk-mq.h +0 -1
index 672e8880f9e2..b919de53dc28 100644
@@ -312,7 +312,6 @@ struct blk_mq_hw_ctx {
 
 	struct list_head	requeue_list;
 	spinlock_t		requeue_lock;
-	struct delayed_work	requeue_work;
 
 	/**
 	 * @run_work: Used for scheduling a hardware queue run at a later time.
#16 lore raw

Re: [PATCH v4 5/7] block: Preserve the order of requeued requests

Review

Quoted 1 line
(quoted diff) +1 -0
+	blk_mq_process_requeue_list(hctx);

Should this do list_empty_careful checks on ->requeue_list and ->flush_list so that we can avoid taking the requeue lock when these conditions aren't met before calling into blk_mq_process_requeue_list?

#17 lore raw

Re: [PATCH v4 5/7] block: Preserve the order of requeued requests

Quoted 6 lines · Christoph Hellwig

Review

Quoted 1 line
(quoted diff) +1 -0
+	blk_mq_process_requeue_list(hctx);

Should this do list_empty_careful checks on ->requeue_list and ->flush_list so that we can avoid taking the requeue lock when these conditions aren't met before calling into blk_mq_process_requeue_list?

Hi Christoph,

I agree that checks whether or not requeue_list and flush_list are empty should be added.

Does it matter in this context whether list_empty_careful() or list_empty() is used? If blk_mq_process_requeue_list() is called concurrently with code that adds an element to one of these two lists it is guaranteed that the queue will be run again. This is why I think that it is fine that the list checks in blk_mq_process_requeue_list() race with concurrent list additions.

Thanks,

Bart.

#18 lore raw

Re: [PATCH v4 5/7] block: Preserve the order of requeued requests

Quoted 6 lines · Bart Van Assche

Does it matter in this context whether list_empty_careful() or list_empty() is used? If blk_mq_process_requeue_list() is called concurrently with code that adds an element to one of these two lists it is guaranteed that the queue will be run again. This is why I think that it is fine that the list checks in blk_mq_process_requeue_list() race with concurrent list additions.

As far as I can tell we're not holding the relevant lock, so I think we need list_empty_careful. Or am I missing something?

#19 lore raw

Re: [PATCH v4 5/7] block: Preserve the order of requeued requests

In reply to Bart Van Assche
Quoted 2 lines · Bart Van Assche

If a queue is run before all requeued requests have been sent to the I/O scheduler, the I/O scheduler may dispatch the wrong request. Fix this by

Can you explain in more details about the issue? What is the wrong request? How?

If you mean write order for requeued write request, there isn't such issue, given new write request from same zone can be issued before the old requeued one is completed.

Thanks,
Ming

#20 lore raw

PATCH v4 6/7 dm: Inline __dm_mq_kick_requeue_list()

In reply to Bart Van Assche

Since commit 52d7f1b5c2f3 ("blk-mq: Avoid that requeueing starts stopped queues") the function __dm_mq_kick_requeue_list() is too short to keep it as a separate function. Hence, inline this function.

---
 drivers/md/dm-rq.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/md/dm-rq.c +2 -7
index f7e9a3632eb3..bbe1e2ea0aa4 100644
@@ -168,21 +168,16 @@ static void dm_end_request(struct request *clone, blk_status_t error)
 	rq_completed(md);
 }
 
-static void __dm_mq_kick_requeue_list(struct request_queue *q, unsigned long msecs)
-{
-	blk_mq_delay_kick_requeue_list(q, msecs);
-}
-
 void dm_mq_kick_requeue_list(struct mapped_device *md)
 {
-	__dm_mq_kick_requeue_list(md->queue, 0);
+	blk_mq_kick_requeue_list(md->queue);
 }
 EXPORT_SYMBOL(dm_mq_kick_requeue_list);
 
 static void dm_mq_delay_requeue_request(struct request *rq, unsigned long msecs)
 {
 	blk_mq_requeue_request(rq, false);
-	__dm_mq_kick_requeue_list(rq->q, msecs);
+	blk_mq_delay_kick_requeue_list(rq->q, msecs);
 }
 
 static void dm_requeue_original_request(struct dm_rq_target_io *tio, bool delay_requeue)
#21 lore raw

PATCH v4 7/7 block: Inline blk_mq_{,delay_}kick_requeue_list()

In reply to Bart Van Assche

Patch "block: Preserve the order of requeued requests" changed blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list() into blk_mq_run_hw_queues() and blk_mq_delay_run_hw_queues() calls respectively. Inline blk_mq_{,delay_}kick_requeue_list() because these functions are now too short to keep these as separate functions.

---
 block/blk-flush.c            |  4 ++--
 block/blk-mq-debugfs.c       |  2 +-
 block/blk-mq.c               | 15 +--------------
 drivers/block/ublk_drv.c     |  6 +++---
 drivers/block/xen-blkfront.c |  1 -
 drivers/md/dm-rq.c           |  6 +++---
 drivers/nvme/host/core.c     |  2 +-
 drivers/s390/block/scm_blk.c |  2 +-
 drivers/scsi/scsi_lib.c      |  2 +-
 include/linux/blk-mq.h       |  2 --
 10 files changed, 13 insertions(+), 29 deletions(-)
block/blk-flush.c +2 -2
index 4bfb92f58aa9..157b86fd9ccb 100644
@@ -192,7 +192,7 @@ static void blk_flush_complete_seq(struct request *rq,
 		spin_lock(&hctx->requeue_lock);
 		list_add_tail(&rq->queuelist, &hctx->flush_list);
 		spin_unlock(&hctx->requeue_lock);
-		blk_mq_kick_requeue_list(q);
+		blk_mq_run_hw_queues(q, true);
 		break;
 
 	case REQ_FSEQ_DONE:
@@ -354,7 +354,7 @@ static void blk_kick_flush(struct blk_mq_hw_ctx *hctx,
 	list_add_tail(&flush_rq->queuelist, &hctx->flush_list);
 	spin_unlock(&hctx->requeue_lock);
 
-	blk_mq_kick_requeue_list(q);
+	blk_mq_run_hw_queues(q, true);
 }
 
 static enum rq_end_io_ret mq_flush_data_end_io(struct request *rq,
block/blk-mq-debugfs.c +1 -1
index 787bdff3cc64..76792ebab935 100644
@@ -114,7 +114,7 @@ static ssize_t queue_state_write(void *data, const char __user *buf,
 	} else if (strcmp(op, "start") == 0) {
 		blk_mq_start_stopped_hw_queues(q, true);
 	} else if (strcmp(op, "kick") == 0) {
-		blk_mq_kick_requeue_list(q);
+		blk_mq_run_hw_queues(q, true);
 	} else {
 		pr_err("%s: unsupported operation '%s'\n", __func__, op);
 inval:
block/blk-mq.c +1 -14
index de39984d17c4..12fd8b65b930 100644
@@ -1436,7 +1436,7 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list)
 	spin_unlock_irqrestore(&hctx->requeue_lock, flags);
 
 	if (kick_requeue_list)
-		blk_mq_kick_requeue_list(q);
+		blk_mq_run_hw_queues(q, true);
 }
 EXPORT_SYMBOL(blk_mq_requeue_request);
 
@@ -1473,19 +1473,6 @@ static void blk_mq_process_requeue_list(struct blk_mq_hw_ctx *hctx)
 	}
 }
 
-void blk_mq_kick_requeue_list(struct request_queue *q)
-{
-	blk_mq_run_hw_queues(q, true);
-}
-EXPORT_SYMBOL(blk_mq_kick_requeue_list);
-
-void blk_mq_delay_kick_requeue_list(struct request_queue *q,
-				    unsigned long msecs)
-{
-	blk_mq_delay_run_hw_queues(q, msecs);
-}
-EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list);
-
 static bool blk_mq_rq_inflight(struct request *rq, void *priv)
 {
 	/*
drivers/block/ublk_drv.c +3 -3
index 1c823750c95a..cddbbdc9b199 100644
@@ -902,7 +902,7 @@ static inline void __ublk_rq_task_work(struct request *req,
 		 */
 		if (unlikely(!mapped_bytes)) {
 			blk_mq_requeue_request(req, false);
-			blk_mq_delay_kick_requeue_list(req->q,
+			blk_mq_delay_run_hw_queues(req->q,
 					UBLK_REQUEUE_DELAY_MS);
 			return;
 		}
@@ -1297,7 +1297,7 @@ static void ublk_unquiesce_dev(struct ublk_device *ub)
 
 	blk_mq_unquiesce_queue(ub->ub_disk->queue);
 	/* We may have requeued some rqs in ublk_quiesce_queue() */
-	blk_mq_kick_requeue_list(ub->ub_disk->queue);
+	blk_mq_run_hw_queues(ub->ub_disk->queue, true);
 }
 
 static void ublk_stop_dev(struct ublk_device *ub)
@@ -2341,7 +2341,7 @@ static int ublk_ctrl_end_recovery(struct ublk_device *ub,
 	blk_mq_unquiesce_queue(ub->ub_disk->queue);
 	pr_devel("%s: queue unquiesced, dev id %d.\n",
 			__func__, header->dev_id);
-	blk_mq_kick_requeue_list(ub->ub_disk->queue);
+	blk_mq_run_hw_queues(ub->ub_disk->queue, true);
 	ub->dev_info.state = UBLK_S_DEV_LIVE;
 	schedule_delayed_work(&ub->monitor_work, UBLK_DAEMON_MONITOR_PERIOD);
 	ret = 0;
drivers/block/xen-blkfront.c +0 -1
index 52e74adbaad6..b8ac217c92b6 100644
@@ -2040,7 +2040,6 @@ static int blkif_recover(struct blkfront_info *info)
 		blk_mq_requeue_request(req, false);
 	}
 	blk_mq_start_stopped_hw_queues(info->rq, true);
-	blk_mq_kick_requeue_list(info->rq);
 
 	while ((bio = bio_list_pop(&info->bio_list)) != NULL) {
 		/* Traverse the list of pending bios and re-queue them */
drivers/md/dm-rq.c +3 -3
index bbe1e2ea0aa4..6421cc2c9852 100644
@@ -64,7 +64,7 @@ int dm_request_based(struct mapped_device *md)
 void dm_start_queue(struct request_queue *q)
 {
 	blk_mq_unquiesce_queue(q);
-	blk_mq_kick_requeue_list(q);
+	blk_mq_run_hw_queues(q, true);
 }
 
 void dm_stop_queue(struct request_queue *q)
@@ -170,14 +170,14 @@ static void dm_end_request(struct request *clone, blk_status_t error)
 
 void dm_mq_kick_requeue_list(struct mapped_device *md)
 {
-	blk_mq_kick_requeue_list(md->queue);
+	blk_mq_run_hw_queues(md->queue, true);
 }
 EXPORT_SYMBOL(dm_mq_kick_requeue_list);
 
 static void dm_mq_delay_requeue_request(struct request *rq, unsigned long msecs)
 {
 	blk_mq_requeue_request(rq, false);
-	blk_mq_delay_kick_requeue_list(rq->q, msecs);
+	blk_mq_delay_run_hw_queues(rq->q, msecs);
 }
 
 static void dm_requeue_original_request(struct dm_rq_target_io *tio, bool delay_requeue)
drivers/nvme/host/core.c +1 -1
index f5dd6d8c7e1d..9b923d52e41c 100644
@@ -303,7 +303,7 @@ static void nvme_retry_req(struct request *req)
 
 	nvme_req(req)->retries++;
 	blk_mq_requeue_request(req, false);
-	blk_mq_delay_kick_requeue_list(req->q, delay);
+	blk_mq_delay_run_hw_queues(req->q, delay);
 }
 
 static void nvme_log_error(struct request *req)
drivers/s390/block/scm_blk.c +1 -1
index 0c1df1d5f1ac..fe5937d28fdc 100644
@@ -243,7 +243,7 @@ static void scm_request_requeue(struct scm_request *scmrq)
 
 	atomic_dec(&bdev->queued_reqs);
 	scm_request_done(scmrq);
-	blk_mq_kick_requeue_list(bdev->rq);
+	blk_mq_run_hw_queues(bdev->rq, true);
 }
 
 static void scm_request_finish(struct scm_request *scmrq)
drivers/scsi/scsi_lib.c +1 -1
index 0226c9279cef..2aa3c147e12f 100644
@@ -124,7 +124,7 @@ static void scsi_mq_requeue_cmd(struct scsi_cmnd *cmd, unsigned long msecs)
 
 	if (msecs) {
 		blk_mq_requeue_request(rq, false);
-		blk_mq_delay_kick_requeue_list(rq->q, msecs);
+		blk_mq_delay_run_hw_queues(rq->q, msecs);
 	} else
 		blk_mq_requeue_request(rq, true);
 }
include/linux/blk-mq.h +0 -2
index b919de53dc28..80761e7c4ea5 100644
@@ -871,8 +871,6 @@ static inline bool blk_mq_add_to_batch(struct request *req,
 }
 
 void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list);
-void blk_mq_kick_requeue_list(struct request_queue *q);
-void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs);
 void blk_mq_complete_request(struct request *rq);
 bool blk_mq_complete_request_remote(struct request *rq);
 void blk_mq_stop_hw_queue(struct blk_mq_hw_ctx *hctx);
Acked-by on v4 7/7
#22 lore raw

Re: [PATCH v4 7/7] block: Inline blk_mq_{,delay_}kick_requeue_list()

Quoted 6 lines · Bart Van Assche

Patch "block: Preserve the order of requeued requests" changed blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list() into blk_mq_run_hw_queues() and blk_mq_delay_run_hw_queues() calls respectively. Inline blk_mq_{,delay_}kick_requeue_list() because these functions are now too short to keep these as separate functions.

Quoted 6 lines

For the blkfront change:

Thanks, Roger.

#23 lore raw

Re: [PATCH v4 7/7] block: Inline blk_mq_{,delay_}kick_requeue_list()

In reply to Bart Van Assche
Quoted 5 lines · Bart Van Assche

Patch "block: Preserve the order of requeued requests" changed blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list() into blk_mq_run_hw_queues() and blk_mq_delay_run_hw_queues() calls respectively. Inline blk_mq_{,delay_}kick_requeue_list() because these functions are now too short to keep these as separate functions.

You're not inlining them, but you're removing them and open code the trivial logic in the callers.

The code change looks good to me, though.