Skip to contentSkip to conversation

mq-deadline: Improve support for zoned block devices

PATCH Superseded Bart Van Assche 2 participants 26 messages Apr 24, 2023 – May 1, 2023

Subsystems (2)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover mq-deadline: Improve support for zoned block devices

Hi Jens,

This patch series improves support for zoned block devices in the mq-deadline scheduler as follows:
* The order of requeued writes (REQ_OP_WRITE*) is preserved.
* The active zone limit is enforced.

Please consider this patch series for the next merge window.

Thanks,

Bart.

Changes compared to v2:
- In the patch that micro-optimizes blk_req_needs_zone_write_lock(), inline bdev_op_is_zoned_write() instead of modifying it.
- In patch "block: Introduce blk_rq_is_seq_zoned_write()", converted "case REQ_OP_ZONE_APPEND" into a source code comment.
- Reworked deadline_skip_seq_writes() as suggested by Christoph.
- Dropped the patch that disabled head insertion for zoned writes.
- Dropped patch "mq-deadline: Fix a race condition related to zoned writes".
- Reworked handling of requeued requests: the 'next_rq' pointer has been removed and instead the position of the most recently dispatched request is tracked.
- Dropped the patches for tracking zone capacity and for restricting the number of active zones.

Changes compared to v1:
- Left out the patches related to request insertion and requeuing since Christoph is busy with reworking these patches.
- Added a patch for enforcing the active zone limit.

Bart Van Assche (9):
block: Simplify blk_req_needs_zone_write_lock()
block: Micro-optimize blk_req_needs_zone_write_lock()
block: Introduce blk_rq_is_seq_zoned_write()
block: mq-deadline: Clean up deadline_check_fifo()
block: mq-deadline: Simplify deadline_skip_seq_writes()
block: mq-deadline: Improve deadline_skip_seq_writes()
block: mq-deadline: Track the dispatch position
block: mq-deadline: Handle requeued requests correctly
block: mq-deadline: Fix handling of at-head zoned writes

 block/blk-mq.h         |   4 +-
 block/blk-zoned.c      |  28 ++++++----
 block/mq-deadline.c    | 114 +++++++++++++++++++++++++++--------------
 include/linux/blk-mq.h |   6 +++
 include/linux/blkdev.h |   9 ----
 5 files changed, 102 insertions(+), 59 deletions(-)
#1 lore raw

PATCH v3 1/9 block: Simplify blk_req_needs_zone_write_lock()

Remove the blk_rq_is_passthrough() check because it is redundant:
blk_req_needs_zone_write_lock() also calls bdev_op_is_zoned_write() and the latter function returns false for pass-through requests.

---
 block/blk-zoned.c | 3 ---
 1 file changed, 3 deletions(-)
block/blk-zoned.c +0 -3
index fce9082384d6..835d9e937d4d 100644
@@ -57,9 +57,6 @@ EXPORT_SYMBOL_GPL(blk_zone_cond_str);
  */
 bool blk_req_needs_zone_write_lock(struct request *rq)
 {
-	if (blk_rq_is_passthrough(rq))
-		return false;
-
 	if (!rq->q->disk->seq_zones_wlock)
 		return false;
 
#2 lore raw

PATCH v3 2/9 block: Micro-optimize blk_req_needs_zone_write_lock()

In reply to Bart Van Assche

Micro-optimize blk_req_needs_zone_write_lock() by inlining bdev_op_is_zoned_write(). This is a micro-optimization because the number of pointers that is dereferenced while testing whether the request queue is associated with a zoned device is reduced.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/blk-mq.h         |  5 +++--
 block/blk-zoned.c      | 11 ++++-------
 include/linux/blkdev.h |  9 ---------
 3 files changed, 7 insertions(+), 18 deletions(-)
block/blk-mq.h +3 -2
index e876584d3516..6bb1281a61f2 100644
@@ -367,8 +367,9 @@ static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap)
 static inline struct blk_plug *blk_mq_plug( struct bio *bio)
 {
 	/* Zoned block device write operation case: do not plug the BIO */
-	if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
-	    bdev_op_is_zoned_write(bio->bi_bdev, bio_op(bio)))
+	if ((bio_op(bio) == REQ_OP_WRITE ||
+	     bio_op(bio) == REQ_OP_WRITE_ZEROES) &&
+	    disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector))
 		return NULL;
 
 	/*
block/blk-zoned.c +4 -7
index 835d9e937d4d..4640b75ae66f 100644
@@ -57,13 +57,10 @@ EXPORT_SYMBOL_GPL(blk_zone_cond_str);
  */
 bool blk_req_needs_zone_write_lock(struct request *rq)
 {
-	if (!rq->q->disk->seq_zones_wlock)
-		return false;
-
-	if (bdev_op_is_zoned_write(rq->q->disk->part0, req_op(rq)))
-		return blk_rq_zone_is_seq(rq);
-
-	return false;
+	return rq->q->disk->seq_zones_wlock &&
+		(req_op(rq) == REQ_OP_WRITE ||
+		 req_op(rq) == REQ_OP_WRITE_ZEROES) &&
+		blk_rq_zone_is_seq(rq);
 }
 EXPORT_SYMBOL_GPL(blk_req_needs_zone_write_lock);
 
include/linux/blkdev.h +0 -9
index e3242e67a8e3..b700c935e230 100644
@@ -1284,15 +1284,6 @@ static inline unsigned int bdev_zone_no(struct block_device *bdev, sector_t sec)
 	return disk_zone_no(bdev->bd_disk, sec);
 }
 
-static inline bool bdev_op_is_zoned_write(struct block_device *bdev,
-					  blk_opf_t op)
-{
-	if (!bdev_is_zoned(bdev))
-		return false;
-
-	return op == REQ_OP_WRITE || op == REQ_OP_WRITE_ZEROES;
-}
-
 static inline sector_t bdev_zone_sectors(struct block_device *bdev)
 {
 	struct request_queue *q = bdev_get_queue(bdev);
#3 lore raw

Re: [PATCH v3 2/9] block: Micro-optimize blk_req_needs_zone_write_lock()

block/blk-mq.h static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap) · line 373

Quoted 10 lines · Bart Van Assche
block/blk-mq.h +3 -2
@@ -367,8 +367,9 @@ static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap)
 static inline struct blk_plug *blk_mq_plug( struct bio *bio)
 {
 	/* Zoned block device write operation case: do not plug the BIO */
-	if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
-	    bdev_op_is_zoned_write(bio->bi_bdev, bio_op(bio)))
+	if ((bio_op(bio) == REQ_OP_WRITE ||
+	     bio_op(bio) == REQ_OP_WRITE_ZEROES) &&
+	    disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector))
 		return NULL;

I find this a bit hard to hard to read. Why not:

	if (disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector)) {
	  	/*
		 * Do not plug for writes that require zone locking.
		 */
		if (bio_op(bio) == REQ_OP_WRITE ||
		    bio_op(bio) == REQ_OP_WRITE_ZEROES)
			return NULL;
	}

block/blk-zoned.c EXPORT_SYMBOL_GPL(blk_zone_cond_str); · line 63

Quoted 13 lines
block/blk-zoned.c +4 -7
 bool blk_req_needs_zone_write_lock(struct request *rq)
 {
-	if (!rq->q->disk->seq_zones_wlock)
-		return false;
-
-	if (bdev_op_is_zoned_write(rq->q->disk->part0, req_op(rq)))
-		return blk_rq_zone_is_seq(rq);
-
-	return false;
+	return rq->q->disk->seq_zones_wlock &&
+		(req_op(rq) == REQ_OP_WRITE ||
+		 req_op(rq) == REQ_OP_WRITE_ZEROES) &&
+		blk_rq_zone_is_seq(rq);

Similaly here. The old version did flow much better, so I'd prefer something like:

	if (!rq->q->disk->seq_zones_wlock || !blk_rq_zone_is_seq(rq))
		return false;
	return req_op(rq) == REQ_OP_WRITE || req_op(rq) == REQ_OP_WRITE_ZEROES);

I also wonder if the check that and op is write or write zeroes, that is needs zone locking would be useful instead of dupliating it all over. That is instead of removing bdev_op_is_zoned_write keep a op_is_zoned_write without the bdev_is_zoned check.

#4 lore raw

Re: [PATCH v3 2/9] block: Micro-optimize blk_req_needs_zone_write_lock()

Quoted 14 lines · Christoph Hellwig

block/blk-mq.h static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap) · line 373

Quoted 10 lines · Bart Van Assche
block/blk-mq.h +3 -2
@@ -367,8 +367,9 @@ static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap)
  static inline struct blk_plug *blk_mq_plug( struct bio *bio)
  {
  	/* Zoned block device write operation case: do not plug the BIO */
-	if (IS_ENABLED(CONFIG_BLK_DEV_ZONED) &&
-	    bdev_op_is_zoned_write(bio->bi_bdev, bio_op(bio)))
+	if ((bio_op(bio) == REQ_OP_WRITE ||
+	     bio_op(bio) == REQ_OP_WRITE_ZEROES) &&
+	    disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector))
  		return NULL;

I find this a bit hard to hard to read. Why not:

Quoted 8 lines
	if (disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector)) {
	  	/*
		 * Do not plug for writes that require zone locking.
		 */
		if (bio_op(bio) == REQ_OP_WRITE ||
		    bio_op(bio) == REQ_OP_WRITE_ZEROES)
			return NULL;
	}

In the above alternative the expensive check happens before a check that is not expensive at all. Do you really want me to call disk_zone_is_seq() before checking the operation type?

Quoted 18 lines

block/blk-zoned.c EXPORT_SYMBOL_GPL(blk_zone_cond_str); · line 63

Quoted 13 lines
block/blk-zoned.c +4 -7
  bool blk_req_needs_zone_write_lock(struct request *rq)
  {
-	if (!rq->q->disk->seq_zones_wlock)
-		return false;
-
-	if (bdev_op_is_zoned_write(rq->q->disk->part0, req_op(rq)))
-		return blk_rq_zone_is_seq(rq);
-
-	return false;
+	return rq->q->disk->seq_zones_wlock &&
+		(req_op(rq) == REQ_OP_WRITE ||
+		 req_op(rq) == REQ_OP_WRITE_ZEROES) &&
+		blk_rq_zone_is_seq(rq);

Similaly here. The old version did flow much better, so I'd prefer something like:

Quoted 8 lines
	if (!rq->q->disk->seq_zones_wlock || !blk_rq_zone_is_seq(rq))
		return false;
	return req_op(rq) == REQ_OP_WRITE || req_op(rq) == REQ_OP_WRITE_ZEROES);

I also wonder if the check that and op is write or write zeroes, that is needs zone locking would be useful instead of dupliating it all over. That is instead of removing bdev_op_is_zoned_write keep a op_is_zoned_write without the bdev_is_zoned check.

I will introduce an op_is_zoned_write() helper function.

Thanks,

Bart.

#5 lore raw

Re: [PATCH v3 2/9] block: Micro-optimize blk_req_needs_zone_write_lock()

Quoted 16 lines · Bart Van Assche
Quoted 7 lines

block/blk-mq.h static inline void blk_mq_clear_mq_map(struct blk_mq_queue_map *qmap) · line 373

Quoted 4 lines
block/blk-mq.h +3 -0
+	if ((bio_op(bio) == REQ_OP_WRITE ||
+	     bio_op(bio) == REQ_OP_WRITE_ZEROES) &&
+	    disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector))
  		return NULL;

I find this a bit hard to hard to read. Why not:

Quoted 8 lines
	if (disk_zone_is_seq(bio->bi_bdev->bd_disk, bio->bi_iter.bi_sector)) {
	  	/*
		 * Do not plug for writes that require zone locking.
		 */
		if (bio_op(bio) == REQ_OP_WRITE ||
		    bio_op(bio) == REQ_OP_WRITE_ZEROES)
			return NULL;
	}
Quoted 3 lines

In the above alternative the expensive check happens before a check that is not expensive at all. Do you really want me to call disk_zone_is_seq() before checking the operation type?

What expensive check? The first check in disk_zone_is_seq is for a zoned device, avoiding any further check if it is not.

#6 lore raw

PATCH v3 3/9 block: Introduce blk_rq_is_seq_zoned_write()

In reply to Bart Van Assche

Introduce the function blk_rq_is_seq_zoned_write(). This function will be used in later patches to preserve the order of zoned writes for which the order needs to be preserved.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/blk-zoned.c      | 22 +++++++++++++++++++---
 include/linux/blk-mq.h |  6 ++++++
 2 files changed, 25 insertions(+), 3 deletions(-)
block/blk-zoned.c +19 -3
index 4640b75ae66f..78c39fc505bc 100644
@@ -52,15 +52,31 @@ const char *blk_zone_cond_str(enum blk_zone_cond zone_cond)
 }
 EXPORT_SYMBOL_GPL(blk_zone_cond_str);
 
+/**
+ * blk_rq_is_seq_zoned_write() - Check if @rq requires write serialization.
+ * @rq: Request to examine.
+ *
+ * Note: REQ_OP_ZONE_APPEND requests do not require serialization.
+ */
+bool blk_rq_is_seq_zoned_write(struct request *rq)
+{
+	switch (req_op(rq)) {
+	case REQ_OP_WRITE:
+	case REQ_OP_WRITE_ZEROES:
+		return blk_rq_zone_is_seq(rq);
+	default:
+		return false;
+	}
+}
+EXPORT_SYMBOL_GPL(blk_rq_is_seq_zoned_write);
+
 /*
  * Return true if a request is a write requests that needs zone write locking.
  */
 bool blk_req_needs_zone_write_lock(struct request *rq)
 {
 	return rq->q->disk->seq_zones_wlock &&
-		(req_op(rq) == REQ_OP_WRITE ||
-		 req_op(rq) == REQ_OP_WRITE_ZEROES) &&
-		blk_rq_zone_is_seq(rq);
+		blk_rq_is_seq_zoned_write(rq);
 }
 EXPORT_SYMBOL_GPL(blk_req_needs_zone_write_lock);
 
include/linux/blk-mq.h +6 -0
index 06caacd77ed6..e498b85bc470 100644
@@ -1164,6 +1164,7 @@ static inline unsigned int blk_rq_zone_is_seq(struct request *rq)
 	return disk_zone_is_seq(rq->q->disk, blk_rq_pos(rq));
 }
 
+bool blk_rq_is_seq_zoned_write(struct request *rq);
 bool blk_req_needs_zone_write_lock(struct request *rq);
 bool blk_req_zone_write_trylock(struct request *rq);
 void __blk_req_zone_write_lock(struct request *rq);
@@ -1194,6 +1195,11 @@ static inline bool blk_req_can_dispatch_to_zone(struct request *rq)
 	return !blk_req_zone_is_write_locked(rq);
 }
 #else /* CONFIG_BLK_DEV_ZONED */
+static inline bool blk_rq_is_seq_zoned_write(struct request *rq)
+{
+	return false;
+}
+
 static inline bool blk_req_needs_zone_write_lock(struct request *rq)
 {
 	return false;
#7 lore raw

Re: [PATCH v3 3/9] block: Introduce blk_rq_is_seq_zoned_write()

block/blk-zoned.c const char *blk_zone_cond_str(enum blk_zone_cond zone_cond) · line 66

Quoted 7 lines

+ */

block/blk-zoned.c +6 -0
+bool blk_rq_is_seq_zoned_write(struct request *rq)
+{
+	switch (req_op(rq)) {
+	case REQ_OP_WRITE:
+	case REQ_OP_WRITE_ZEROES:
+		return blk_rq_zone_is_seq(rq);

This would be another use for op_is_zoned_write..

block/blk-zoned.c const char *blk_zone_cond_str(enum blk_zone_cond zone_cond) · line 80

Quoted 8 lines

bool blk_req_needs_zone_write_lock(struct request *rq) {

block/blk-zoned.c +1 -3
 	return rq->q->disk->seq_zones_wlock &&
-		(req_op(rq) == REQ_OP_WRITE ||
-		 req_op(rq) == REQ_OP_WRITE_ZEROES) &&
-		blk_rq_zone_is_seq(rq);
+		blk_rq_is_seq_zoned_write(rq);
 }

.. and given that this just reshuffles the previous patch it might make sense to just move it before that.

#8 lore raw

Re: [PATCH v3 3/9] block: Introduce blk_rq_is_seq_zoned_write()

Quoted 11 lines · Christoph Hellwig

block/blk-zoned.c const char *blk_zone_cond_str(enum blk_zone_cond zone_cond) · line 80

Quoted 8 lines
  bool blk_req_needs_zone_write_lock(struct request *rq)
  {
  	return rq->q->disk->seq_zones_wlock &&
block/blk-zoned.c +1 -3
-		(req_op(rq) == REQ_OP_WRITE ||
-		 req_op(rq) == REQ_OP_WRITE_ZEROES) &&
-		blk_rq_zone_is_seq(rq);
+		blk_rq_is_seq_zoned_write(rq);
  }

.. and given that this just reshuffles the previous patch it might make sense to just move it before that.

Hi Christoph,

I will move this patch in front of the micro-optimization patch.

Thanks,

Bart.

#9 lore raw

PATCH v3 4/9 block: mq-deadline: Clean up deadline_check_fifo()

In reply to Bart Van Assche

Change the return type of deadline_check_fifo() from 'int' into 'bool'. Use time_is_before_eq_jiffies() instead of time_after_eq(). No functionality has been changed.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/mq-deadline.c | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)
block/mq-deadline.c +5 -11
index 5839a027e0f0..a016cafa54b3 100644
@@ -272,21 +272,15 @@ static u32 dd_queued(struct deadline_data *dd, enum dd_prio prio)
 }
 
 /*
- * deadline_check_fifo returns 0 if there are no expired requests on the fifo,
- * 1 otherwise. Requires !list_empty(&dd->fifo_list[data_dir])
+ * deadline_check_fifo returns true if and only if there are expired requests
+ * in the FIFO list. Requires !list_empty(&dd->fifo_list[data_dir]).
  */
-static inline int deadline_check_fifo(struct dd_per_prio *per_prio,
-				      enum dd_data_dir data_dir)
+static inline bool deadline_check_fifo(struct dd_per_prio *per_prio,
+				       enum dd_data_dir data_dir)
 {
 	struct request *rq = rq_entry_fifo(per_prio->fifo_list[data_dir].next);
 
-	/*
-	 * rq is expired!
-	 */
-	if (time_after_eq(jiffies, (unsigned long)rq->fifo_time))
-		return 1;
-
-	return 0;
+	return time_is_before_eq_jiffies((unsigned long)rq->fifo_time);
 }
 
 /*
Reviewed-by on v3 4/9
#10 lore raw

Re: [PATCH v3 4/9] block: mq-deadline: Clean up deadline_check_fifo()

Quoted 3 lines · Bart Van Assche

Change the return type of deadline_check_fifo() from 'int' into 'bool'. Use time_is_before_eq_jiffies() instead of time_after_eq(). No functionality has been changed.

Looks good:

#11 lore raw

PATCH v3 5/9 block: mq-deadline: Simplify deadline_skip_seq_writes()

In reply to Bart Van Assche

Make the deadline_skip_seq_writes() code shorter without changing its functionality.

---
 block/mq-deadline.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
block/mq-deadline.c +3 -6
index a016cafa54b3..6276afede9cd 100644
@@ -304,14 +304,11 @@ static struct request *deadline_skip_seq_writes(struct deadline_data *dd,
 						struct request *rq)
 {
 	sector_t pos = blk_rq_pos(rq);
-	sector_t skipped_sectors = 0;
 
-	while (rq) {
-		if (blk_rq_pos(rq) != pos + skipped_sectors)
-			break;
-		skipped_sectors += blk_rq_sectors(rq);
+	do {
+		pos += blk_rq_sectors(rq);
 		rq = deadline_latter_request(rq);
-	}
+	} while (rq && blk_rq_pos(rq) == pos);
 
 	return rq;
 }
Reviewed-by on v3 5/9
#12 lore raw

Re: [PATCH v3 5/9] block: mq-deadline: Simplify deadline_skip_seq_writes()

Quoted 20 lines · Bart Van Assche

Make the deadline_skip_seq_writes() code shorter without changing its functionality.

---
 block/mq-deadline.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
block/mq-deadline.c +0 -1
index a016cafa54b3..6276afede9cd 100644
@@ -304,14 +304,11 @@ static struct request *deadline_skip_seq_writes(struct deadline_data *dd,
 						struct request *rq)
 {
 	sector_t pos = blk_rq_pos(rq);
-	sector_t skipped_sectors = 0;

block/mq-deadline.c static struct request *deadline_skip_seq_writes(struct deadline_data *dd, · line 311

Quoted 10 lines
block/mq-deadline.c +3 -5
 
-	while (rq) {
-		if (blk_rq_pos(rq) != pos + skipped_sectors)
-			break;
-		skipped_sectors += blk_rq_sectors(rq);
+	do {
+		pos += blk_rq_sectors(rq);
 		rq = deadline_latter_request(rq);
-	}
+	} while (rq && blk_rq_pos(rq) == pos);

Looks good, although I'd maybe rename pos to next_pos for clarity.

#13 lore raw

PATCH v3 6/9 block: mq-deadline: Improve deadline_skip_seq_writes()

In reply to Bart Van Assche

Make deadline_skip_seq_writes() do what its name suggests, namely to skip sequential writes.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/mq-deadline.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/mq-deadline.c +1 -1
index 6276afede9cd..dbc0feca963e 100644
@@ -308,7 +308,7 @@ static struct request *deadline_skip_seq_writes(struct deadline_data *dd,
 	do {
 		pos += blk_rq_sectors(rq);
 		rq = deadline_latter_request(rq);
-	} while (rq && blk_rq_pos(rq) == pos);
+	} while (rq && blk_rq_pos(rq) == pos && blk_rq_is_seq_zoned_write(rq));
 
 	return rq;
 }
Reviewed-by on v3 6/9
#14 lore raw

Re: [PATCH v3 6/9] block: mq-deadline: Improve deadline_skip_seq_writes()

Quoted 2 lines · Bart Van Assche

Make deadline_skip_seq_writes() do what its name suggests, namely to skip sequential writes.

Looks good:

#15 lore raw

PATCH v3 7/9 block: mq-deadline: Track the dispatch position

In reply to Bart Van Assche

Track the position (sector_t) of the most recently dispatched request instead of tracking a pointer to the next request to dispatch. This patch simplifies the code for inserting and removing a request from the red-black tree.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/mq-deadline.c | 45 +++++++++++++++++++++++++++++++--------------
 1 file changed, 31 insertions(+), 14 deletions(-)
block/mq-deadline.c +31 -14
index dbc0feca963e..bf1dfe9fe9c9 100644
@@ -74,8 +74,8 @@ struct dd_per_prio {
 	struct list_head dispatch;
 	struct rb_root sort_list[DD_DIR_COUNT];
 	struct list_head fifo_list[DD_DIR_COUNT];
-	/* Next request in FIFO order. Read, write or both are NULL. */
-	struct request *next_rq[DD_DIR_COUNT];
+	/* Position of the most recently dispatched request. */
+	sector_t latest_pos[DD_DIR_COUNT];
 	struct io_stats_per_prio stats;
 };
 
@@ -156,6 +156,25 @@ deadline_latter_request(struct request *rq)
 	return NULL;
 }
 
+/* Return the first request for which blk_rq_pos() >= pos. */
+static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio,
+				enum dd_data_dir data_dir, sector_t pos)
+{
+	struct rb_node *node = per_prio->sort_list[data_dir].rb_node;
+	struct request *rq, *res = NULL;
+
+	while (node) {
+		rq = rb_entry_rq(node);
+		if (blk_rq_pos(rq) >= pos) {
+			res = rq;
+			node = node->rb_left;
+		} else {
+			node = node->rb_right;
+		}
+	}
+	return res;
+}
+
 static void
 deadline_add_rq_rb(struct dd_per_prio *per_prio, struct request *rq)
 {
@@ -167,11 +186,6 @@ deadline_add_rq_rb(struct dd_per_prio *per_prio, struct request *rq)
 static inline void
 deadline_del_rq_rb(struct dd_per_prio *per_prio, struct request *rq)
 {
-	const enum dd_data_dir data_dir = rq_data_dir(rq);
-
-	if (per_prio->next_rq[data_dir] == rq)
-		per_prio->next_rq[data_dir] = deadline_latter_request(rq);
-
 	elv_rb_del(deadline_rb_root(per_prio, rq), rq);
 }
 
@@ -251,10 +265,6 @@ static void
 deadline_move_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 		      struct request *rq)
 {
-	const enum dd_data_dir data_dir = rq_data_dir(rq);
-
-	per_prio->next_rq[data_dir] = deadline_latter_request(rq);
-
 	/*
 	 * take it off the sort and fifo list
 	 */
@@ -363,7 +373,8 @@ deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 	struct request *rq;
 	unsigned long flags;
 
-	rq = per_prio->next_rq[data_dir];
+	rq = deadline_from_pos(per_prio, data_dir,
+			       per_prio->latest_pos[data_dir]);
 	if (!rq)
 		return NULL;
 
@@ -426,6 +437,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
 		if (started_after(dd, rq, latest_start))
 			return NULL;
 		list_del_init(&rq->queuelist);
+		data_dir = rq_data_dir(rq);
 		goto done;
 	}
 
@@ -433,9 +445,11 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
 	 * batches are currently reads XOR writes
 	 */
 	rq = deadline_next_request(dd, per_prio, dd->last_dir);
-	if (rq && dd->batching < dd->fifo_batch)
+	if (rq && dd->batching < dd->fifo_batch) {
 		/* we have a next request are still entitled to batch */
+		data_dir = rq_data_dir(rq);
 		goto dispatch_request;
+	}
 
 	/*
 	 * at this point we are not running a batch. select the appropriate
@@ -513,6 +527,7 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
 done:
 	ioprio_class = dd_rq_ioclass(rq);
 	prio = ioprio_class_to_prio[ioprio_class];
+	dd->per_prio[prio].latest_pos[data_dir] = blk_rq_pos(rq);
 	dd->per_prio[prio].stats.dispatched++;
 	/*
 	 * If the request needs its target zone locked, do it.
@@ -1026,8 +1041,10 @@ static int deadline_##name##_next_rq_show(void *data,			\
 	struct request_queue *q = data;					\
 	struct deadline_data *dd = q->elevator->elevator_data;		\
 	struct dd_per_prio *per_prio = &dd->per_prio[prio];		\
-	struct request *rq = per_prio->next_rq[data_dir];		\
+	struct request *rq;						\
 									\
+	rq = deadline_from_pos(per_prio, data_dir,			\
+			       per_prio->latest_pos[data_dir]);		\
 	if (rq)								\
 		__blk_mq_debugfs_rq_show(m, rq);			\
 	return 0;							\
#16 lore raw

Re: [PATCH v3 7/9] block: mq-deadline: Track the dispatch position

Quoted 4 lines · Bart Van Assche

Track the position (sector_t) of the most recently dispatched request instead of tracking a pointer to the next request to dispatch. This patch simplifies the code for inserting and removing a request from the red-black tree.

Can you explain what the simplification is? As of this patch the code looks more complicated to me..

#17 lore raw

Re: [PATCH v3 7/9] block: mq-deadline: Track the dispatch position

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Bart Van Assche

Track the position (sector_t) of the most recently dispatched request instead of tracking a pointer to the next request to dispatch. This patch simplifies the code for inserting and removing a request from the red-black tree.

Can you explain what the simplification is? As of this patch the code looks more complicated to me..

Hi Christoph,

I will rewrite the patch description.

Thanks,

Bart.

#18 lore raw

Re: [PATCH v3 7/9] block: mq-deadline: Track the dispatch position

In reply to Christoph Hellwig
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Bart Van Assche

Track the position (sector_t) of the most recently dispatched request instead of tracking a pointer to the next request to dispatch. This patch simplifies the code for inserting and removing a request from the red-black tree.

Can you explain what the simplification is? As of this patch the code looks more complicated to me..

How about changing the patch description into the following text?

"Track the position (sector_t) of the most recently dispatched request instead of tracking a pointer to the next request to dispatch. This patch is the basis for patch "Handle requeued requests correctly". Without this patch it would me significantly more complicated to start searching for a zoned write from the start of a zone."

Thanks,

Bart.

#19 lore raw

Re: [PATCH v3 7/9] block: mq-deadline: Track the dispatch position

Quoted 4 lines · Bart Van Assche

"Track the position (sector_t) of the most recently dispatched request instead of tracking a pointer to the next request to dispatch. This patch is the basis for patch "Handle requeued requests correctly". Without this patch it would me significantly more complicated to start

s/me/be/

#20 lore raw

PATCH v3 8/9 block: mq-deadline: Handle requeued requests correctly

In reply to Bart Van Assche

Start dispatching from the start of a zone instead of from the starting position of the most recently dispatched request.

If a zoned write is requeued with an LBA that is lower than already inserted zoned writes, make sure that it is submitted first.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/mq-deadline.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
block/mq-deadline.c +23 -2
index bf1dfe9fe9c9..2de6c63190d8 100644
@@ -156,13 +156,23 @@ deadline_latter_request(struct request *rq)
 	return NULL;
 }
 
-/* Return the first request for which blk_rq_pos() >= pos. */
+/*
+ * Return the first request for which blk_rq_pos() >= @pos. For zoned devices,
+ * return the first request after the highest zone start <= @pos.
+ */
 static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio,
 				enum dd_data_dir data_dir, sector_t pos)
 {
 	struct rb_node *node = per_prio->sort_list[data_dir].rb_node;
 	struct request *rq, *res = NULL;
 
+	if (!node)
+		return NULL;
+
+	rq = rb_entry_rq(node);
+	if (blk_rq_is_seq_zoned_write(rq))
+		pos -= bdev_offset_from_zone_start(rq->q->disk->part0, pos);
+
 	while (node) {
 		rq = rb_entry_rq(node);
 		if (blk_rq_pos(rq) >= pos) {
@@ -809,6 +819,8 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
 		list_add(&rq->queuelist, &per_prio->dispatch);
 		rq->fifo_time = jiffies;
 	} else {
+		struct list_head *insert_before;
+
 		deadline_add_rq_rb(per_prio, rq);
 
 		if (rq_mergeable(rq)) {
@@ -821,7 +833,16 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
 		 * set expire time and add to fifo list
 		 */
 		rq->fifo_time = jiffies + dd->fifo_expire[data_dir];
-		list_add_tail(&rq->queuelist, &per_prio->fifo_list[data_dir]);
+		insert_before = &per_prio->fifo_list[data_dir];
+#ifdef CONFIG_BLK_DEV_ZONED
+		if (blk_rq_is_seq_zoned_write(rq)) {
+			struct request *rq2 = deadline_latter_request(rq);
+
+			if (rq2 && blk_rq_zone_no(rq2) == blk_rq_zone_no(rq))
+				insert_before = &rq2->queuelist;
+		}
+#endif
+		list_add_tail(&rq->queuelist, insert_before);
 	}
 }
 
#21 lore raw

Re: [PATCH v3 8/9] block: mq-deadline: Handle requeued requests correctly

Quoted 29 lines · Bart Van Assche

Start dispatching from the start of a zone instead of from the starting position of the most recently dispatched request.

If a zoned write is requeued with an LBA that is lower than already inserted zoned writes, make sure that it is submitted first.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/mq-deadline.c | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)
block/mq-deadline.c +4 -1
index bf1dfe9fe9c9..2de6c63190d8 100644
@@ -156,13 +156,23 @@ deadline_latter_request(struct request *rq)
 	return NULL;
 }
 
-/* Return the first request for which blk_rq_pos() >= pos. */
+/*
+ * Return the first request for which blk_rq_pos() >= @pos. For zoned devices,
+ * return the first request after the highest zone start <= @pos.
+ */
 static inline struct request *deadline_from_pos(struct dd_per_prio *per_prio,
 				enum dd_data_dir data_dir, sector_t pos)

block/mq-deadline.c deadline_latter_request(struct request *rq) · line 174

Quoted 10 lines
block/mq-deadline.c +6 -0
 {
 	struct rb_node *node = per_prio->sort_list[data_dir].rb_node;
 	struct request *rq, *res = NULL;
 
+	if (!node)
+		return NULL;
+
+	rq = rb_entry_rq(node);
+	if (blk_rq_is_seq_zoned_write(rq))
+		pos -= bdev_offset_from_zone_start(rq->q->disk->part0, pos);

This looks a bit odd. I'd write this as:

	if (blk_rq_is_seq_zoned_write(rq))
		pos = round_down(pos, rq->q->limits.chunk_sectors);
Quoted 4 lines
block/mq-deadline.c +0 -0
@@ -821,7 +833,16 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
 		 * set expire time and add to fifo list
 		 */
 		rq->fifo_time = jiffies + dd->fifo_expire[data_dir];

block/mq-deadline.c static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, · line 844

Quoted 10 lines
block/mq-deadline.c +9 -1
-		list_add_tail(&rq->queuelist, &per_prio->fifo_list[data_dir]);
+		insert_before = &per_prio->fifo_list[data_dir];
+#ifdef CONFIG_BLK_DEV_ZONED
+		if (blk_rq_is_seq_zoned_write(rq)) {
+			struct request *rq2 = deadline_latter_request(rq);
+
+			if (rq2 && blk_rq_zone_no(rq2) == blk_rq_zone_no(rq))
+				insert_before = &rq2->queuelist;
+		}
+#endif

Why does this need an ifdef?

Also can you please always add comments for these special cases? Same for the one above.

block/mq-deadline.c static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, · line 848

Quoted 4 lines
block/mq-deadline.c +1 -0
+		list_add_tail(&rq->queuelist, insert_before);
 	}
 }
 

---end quoted text---

#22 lore raw

Re: [PATCH v3 8/9] block: mq-deadline: Handle requeued requests correctly

Quoted 17 lines · Christoph Hellwig
Quoted 4 lines · Bart Van Assche
block/mq-deadline.c +0 -0
@@ -821,7 +833,16 @@ static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq,
  		 * set expire time and add to fifo list
  		 */
  		rq->fifo_time = jiffies + dd->fifo_expire[data_dir];

block/mq-deadline.c static void dd_insert_request(struct blk_mq_hw_ctx *hctx, struct request *rq, · line 844

Quoted 10 lines
block/mq-deadline.c +9 -1
-		list_add_tail(&rq->queuelist, &per_prio->fifo_list[data_dir]);
+		insert_before = &per_prio->fifo_list[data_dir];
+#ifdef CONFIG_BLK_DEV_ZONED
+		if (blk_rq_is_seq_zoned_write(rq)) {
+			struct request *rq2 = deadline_latter_request(rq);
+
+			if (rq2 && blk_rq_zone_no(rq2) == blk_rq_zone_no(rq))
+				insert_before = &rq2->queuelist;
+		}
+#endif

Why does this need an ifdef?

Because the blk_rq_zone_no() definition is surrounded with #ifdef CONFIG_BLK_DEV_ZONED / #endif. Without the #ifdef the above code would trigger a compilation error with CONFIG_BLK_DEV_ZONED=n. I have considered to add a definition of blk_rq_zone_no() for CONFIG_BLK_DEV_ZONED=n. I prefer not to do this because I think it's better to cause a compiler error if blk_rq_zone_no() is used in code that is also used for the CONFIG_BLK_DEV_ZONED=n case.

Quoted 1 line

Also can you please always add comments for these special cases?

Will do.

Thanks,

Bart.

#23 lore raw

Re: [PATCH v3 8/9] block: mq-deadline: Handle requeued requests correctly

Quoted 7 lines · Bart Van Assche

Because the blk_rq_zone_no() definition is surrounded with #ifdef CONFIG_BLK_DEV_ZONED / #endif. Without the #ifdef the above code would trigger a compilation error with CONFIG_BLK_DEV_ZONED=n. I have considered to add a definition of blk_rq_zone_no() for CONFIG_BLK_DEV_ZONED=n. I prefer not to do this because I think it's better to cause a compiler error if blk_rq_zone_no() is used in code that is also used for the CONFIG_BLK_DEV_ZONED=n case.

Ok.

#24 lore raw

PATCH v3 9/9 block: mq-deadline: Fix handling of at-head zoned writes

In reply to Bart Van Assche

Before dispatching a zoned write from the FIFO list, check whether there are any zoned writes in the RB-tree for a lower LBA for the same zone. This patch ensures that zoned writes happen in order even if at_head is set for some writes for a zone and not for others.

Cc Damien Le Moal <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Ming Lei <[email protected]>
---
 block/mq-deadline.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
block/mq-deadline.c +12 -6
index 2de6c63190d8..ed0da04c51ec 100644
@@ -341,7 +341,7 @@ static struct request *
 deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 		      enum dd_data_dir data_dir)
 {
-	struct request *rq;
+	struct request *rq, *rb_rq, *next;
 	unsigned long flags;
 
 	if (list_empty(&per_prio->fifo_list[data_dir]))
@@ -353,13 +353,19 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 
 	/*
 	 * Look for a write request that can be dispatched, that is one with
-	 * an unlocked target zone. For some HDDs, breaking a sequential
-	 * write stream can lead to lower throughput, so make sure to preserve
-	 * sequential write streams, even if that stream crosses into the next
-	 * zones and these zones are unlocked.
+	 * an unlocked target zone. For each write request from the FIFO list,
+	 * check whether an earlier write request exists in the RB tree. For
+	 * some HDDs, breaking a sequential write stream can lead to lower
+	 * throughput, so make sure to preserve sequential write streams, even
+	 * if that stream crosses into the next zones and these zones are
+	 * unlocked.
 	 */
 	spin_lock_irqsave(&dd->zone_lock, flags);
-	list_for_each_entry(rq, &per_prio->fifo_list[DD_WRITE], queuelist) {
+	list_for_each_entry_safe(rq, next, &per_prio->fifo_list[DD_WRITE],
+				 queuelist) {
+		rb_rq = deadline_from_pos(per_prio, data_dir, blk_rq_pos(rq));
+		if (rb_rq && blk_rq_pos(rb_rq) < blk_rq_pos(rq))
+			rq = rb_rq;
 		if (blk_req_can_dispatch_to_zone(rq) &&
 		    (blk_queue_nonrot(rq->q) ||
 		     !deadline_is_seq_write(dd, rq)))
#25 lore raw

Re: [PATCH v3 9/9] block: mq-deadline: Fix handling of at-head zoned writes

block/mq-deadline.c deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,

Quoted 4 lines
block/mq-deadline.c +2 -0
 	/*
 	 * Look for a write request that can be dispatched, that is one with
+	 * an unlocked target zone. For each write request from the FIFO list,
+	 * check whether an earlier write request exists in the RB tree.

I can see that you are checking the RB tree, so the comment is a bit useless. It should explain why. And preferably right next to the code instead of hidden in this long and fairly hard to read above the loop comment.