Skip to contentSkip to conversation

Enable zoned write pipelining for UFS devices

PATCH In review Bart Van Assche 2 participants 19 messages Jul 10, 2023 – Jul 24, 2023

Subsystems (4)
lore
Share & tools
lore

Bart Van Assche

author reviewer
#0 lore raw

v2 cover Enable zoned write pipelining for UFS devices

Hi Jens,

This patch series improves write performance for zoned UFS devices. Please consider these patches for the next merge window.

Thank you,

Bart.

Changes compared to v1:
- Left out the patches that are already upstream.
- Switched the approach in patch "scsi: Retry unaligned zoned writes" from retrying immediately to sending unaligned write commands to the SCSI error handler.

Bart Van Assche (5):
block: Introduce a request queue flag for pipelining zoned writes block/mq-deadline: Only use zone locking if necessary block/null_blk: Add support for pipelining zoned writes
scsi: Retry unaligned zoned writes
scsi: ufs: Enable zoned write pipelining

 block/blk-zoned.c                 |  3 ++-
 block/mq-deadline.c               | 14 +++++++-----
 drivers/block/null_blk/main.c     |  2 ++
 drivers/block/null_blk/null_blk.h |  1 +
 drivers/block/null_blk/zoned.c    |  3 +++
 drivers/scsi/scsi_error.c         | 37 +++++++++++++++++++++++++++++++
 drivers/scsi/scsi_lib.c           |  1 +
 drivers/scsi/sd.c                 |  3 +++
 drivers/ufs/core/ufshcd.c         |  1 +
 include/linux/blkdev.h            |  7 ++++++
 include/scsi/scsi.h               |  1 +
 11 files changed, 67 insertions(+), 6 deletions(-)
author reviewer
#1 lore raw

PATCH v2 1/5 block: Introduce a request queue flag for pipelining zoned writes

Writes in sequential write required zones must happen at the write pointer. Even if the submitter of the write commands (e.g. a filesystem) submits writes for sequential write required zones in order, the block layer or the storage controller may reorder these write commands.

The zone locking mechanism in the mq-deadline I/O scheduler serializes write commands for sequential zones. Some but not all storage controllers require this serialization. Introduce a new flag such that block drivers can request pipelining of writes for sequential write required zones.

An example of a storage controller standard that requires write serialization is AHCI (Advanced Host Controller Interface). Submitting commands to AHCI controllers happens by writing a bit pattern into a register. Each set bit corresponds to an active command. This mechanism does not preserve command ordering information.

---
 include/linux/blkdev.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/blkdev.h +7 -0
index ed44a997f629..805012c5a6ab 100644
@@ -534,6 +534,8 @@ struct request_queue {
 #define QUEUE_FLAG_NONROT	6	/* non-rotational device (SSD) */
 #define QUEUE_FLAG_VIRT		QUEUE_FLAG_NONROT /* paravirt device */
 #define QUEUE_FLAG_IO_STAT	7	/* do disk/partitions IO accounting */
+/* Writes for sequential write required zones may be pipelined. */
+#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8
 #define QUEUE_FLAG_NOXMERGES	9	/* No extended merges */
 #define QUEUE_FLAG_ADD_RANDOM	10	/* Contributes to random pool */
 #define QUEUE_FLAG_SYNCHRONOUS	11	/* always completes in submit context */
@@ -596,6 +598,11 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
 #define blk_queue_skip_tagset_quiesce(q) \
 	test_bit(QUEUE_FLAG_SKIP_TAGSET_QUIESCE, &(q)->queue_flags)
 
+static inline bool blk_queue_pipeline_zoned_writes(struct request_queue *q)
+{
+	return test_bit(QUEUE_FLAG_PIPELINE_ZONED_WRITES, &q->queue_flags);
+}
+
 extern void blk_set_pm_only(struct request_queue *q);
 extern void blk_clear_pm_only(struct request_queue *q);
 
maintainer
#2 lore raw

Re: [PATCH v2 1/5] block: Introduce a request queue flag for pipelining zoned writes

Quoted 27 lines · Bart Van Assche

Writes in sequential write required zones must happen at the write pointer. Even if the submitter of the write commands (e.g. a filesystem) submits writes for sequential write required zones in order, the block layer or the storage controller may reorder these write commands.

The zone locking mechanism in the mq-deadline I/O scheduler serializes write commands for sequential zones. Some but not all storage controllers require this serialization. Introduce a new flag such that block drivers can request pipelining of writes for sequential write required zones.

An example of a storage controller standard that requires write serialization is AHCI (Advanced Host Controller Interface). Submitting commands to AHCI controllers happens by writing a bit pattern into a register. Each set bit corresponds to an active command. This mechanism does not preserve command ordering information.

---
 include/linux/blkdev.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/blkdev.h +0 -0
index ed44a997f629..805012c5a6ab 100644

include/linux/blkdev.h struct request_queue { · line 538

Quoted 5 lines
include/linux/blkdev.h +2 -0
@@ -534,6 +534,8 @@ struct request_queue {
 #define QUEUE_FLAG_NONROT	6	/* non-rotational device (SSD) */
 #define QUEUE_FLAG_VIRT		QUEUE_FLAG_NONROT /* paravirt device */
 #define QUEUE_FLAG_IO_STAT	7	/* do disk/partitions IO accounting */
+/* Writes for sequential write required zones may be pipelined. */
+#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8

I am not a big fan of this name as "pipeline" does not necessarily imply "high queue depth write". What about simply calling this QUEUE_FLAG_NO_ZONE_WRITE_LOCK, indicating that there is no need to write-lock zones ?

Quoted 15 lines
include/linux/blkdev.h +5 -0
 #define QUEUE_FLAG_NOXMERGES	9	/* No extended merges */
 #define QUEUE_FLAG_ADD_RANDOM	10	/* Contributes to random pool */
 #define QUEUE_FLAG_SYNCHRONOUS	11	/* always completes in submit context */
@@ -596,6 +598,11 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);
 #define blk_queue_skip_tagset_quiesce(q) \
 	test_bit(QUEUE_FLAG_SKIP_TAGSET_QUIESCE, &(q)->queue_flags)
 
+static inline bool blk_queue_pipeline_zoned_writes(struct request_queue *q)
+{
+	return test_bit(QUEUE_FLAG_PIPELINE_ZONED_WRITES, &q->queue_flags);
+}
+
 extern void blk_set_pm_only(struct request_queue *q);
 extern void blk_clear_pm_only(struct request_queue *q);
 
Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#3 lore raw

Re: [PATCH v2 1/5] block: Introduce a request queue flag for pipelining zoned writes

Quoted 8 lines · Damien Le Moal

include/linux/blkdev.h struct request_queue { · line 538

Quoted 2 lines · Bart Van Assche
include/linux/blkdev.h +2 -0
+/* Writes for sequential write required zones may be pipelined. */
+#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8

I am not a big fan of this name as "pipeline" does not necessarily imply "high queue depth write". What about simply calling this QUEUE_FLAG_NO_ZONE_WRITE_LOCK, indicating that there is no need to write-lock zones ?

Hi Damien,

I'm not a big fan of names with negative words like "no" or "not" embedded. Isn't pipelining standard computer science terminology? See also
https://en.wikipedia.org/wiki/Pipeline_(computing).

Thanks,

Bart.

maintainer
#4 lore raw

Re: [PATCH v2 1/5] block: Introduce a request queue flag for pipelining zoned writes

Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Damien Le Moal

include/linux/blkdev.h struct request_queue { · line 538

Quoted 2 lines · Bart Van Assche
include/linux/blkdev.h +2 -0
+/* Writes for sequential write required zones may be pipelined. */
+#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8

I am not a big fan of this name as "pipeline" does not necessarily imply "high
queue depth write". What about simply calling this QUEUE_FLAG_NO_ZONE_WRITE_LOCK, indicating that there is no need to write-lock
zones ?

Quoted 5 lines

Hi Damien,

I'm not a big fan of names with negative words like "no" or "not" embedded. Isn't pipelining standard computer science terminology? See also
https://en.wikipedia.org/wiki/Pipeline_(computing).

Sure, pipeline is a well known term. But I do not think it is synonymous with "high queue depth write" :) A "pipeline" for zoned write may still operate at write qd=1 per zone...

Given that the default is using zone write locking, I would prefer a flag name that indicates a change to this default. What about something like QUEUE_FLAG_UNRESTRICTED_ZONE_WRITE ?

But tht is beside the point as I still have reservations on this approach anyway. See my reply to patch 4.

Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#5 lore raw

PATCH v2 2/5 block/mq-deadline: Only use zone locking if necessary

In reply to Bart Van Assche

Measurements have shown that limiting the queue depth to one for zoned writes has a significant negative performance impact on zoned UFS devices. Hence this patch that disables zone locking from the mq-deadline scheduler for storage controllers that support pipelining zoned writes. This patch is based on the following assumptions:
- Applications submit write requests to sequential write required zones in order.
- It happens infrequently that zoned write requests are reordered by the block layer.
- The storage controller does not reorder write requests that have been submitted to the same hardware queue. This is the case for UFS: the UFSHCI specification requires that UFS controllers process requests in order per hardware queue.
- The I/O priority of all pipelined write requests is the same per zone.
- Either no I/O scheduler is used or an I/O scheduler is used that submits write requests per zone in LBA order.

---
 block/blk-zoned.c   |  3 ++-
 block/mq-deadline.c | 14 +++++++++-----
 2 files changed, 11 insertions(+), 6 deletions(-)
block/blk-zoned.c +2 -1
index 0f9f97cdddd9..59560d1657e3 100644
@@ -504,7 +504,8 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,
 		break;
 	case BLK_ZONE_TYPE_SEQWRITE_REQ:
 	case BLK_ZONE_TYPE_SEQWRITE_PREF:
-		if (!args->seq_zones_wlock) {
+		if (!blk_queue_pipeline_zoned_writes(q) &&
+		    !args->seq_zones_wlock) {
 			args->seq_zones_wlock =
 				blk_alloc_zone_bitmap(q->node, args->nr_zones);
 			if (!args->seq_zones_wlock)
block/mq-deadline.c +9 -5
index 6aa5daf7ae32..0bed2bdeed89 100644
@@ -353,7 +353,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 		return NULL;
 
 	rq = rq_entry_fifo(per_prio->fifo_list[data_dir].next);
-	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q))
+	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q) ||
+	    blk_queue_pipeline_zoned_writes(rq->q))
 		return rq;
 
 	/*
@@ -398,7 +399,8 @@ deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 	if (!rq)
 		return NULL;
 
-	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q))
+	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q) ||
+	    blk_queue_pipeline_zoned_writes(rq->q))
 		return rq;
 
 	/*
@@ -526,8 +528,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
 	}
 
 	/*
-	 * For a zoned block device, if we only have writes queued and none of
-	 * them can be dispatched, rq will be NULL.
+	 * For a zoned block device that requires write serialization, if we
+	 * only have writes queued and none of them can be dispatched, rq will
+	 * be NULL.
 	 */
 	if (!rq)
 		return NULL;
@@ -933,7 +936,8 @@ static void dd_finish_request(struct request *rq)
 
 	atomic_inc(&per_prio->stats.completed);
 
-	if (blk_queue_is_zoned(q)) {
+	if (blk_queue_is_zoned(rq->q) &&
+	    !blk_queue_pipeline_zoned_writes(q)) {
 		unsigned long flags;
 
 		spin_lock_irqsave(&dd->zone_lock, flags);
maintainer
#6 lore raw

Re: [PATCH v2 2/5] block/mq-deadline: Only use zone locking if necessary

Quoted 29 lines · Bart Van Assche

Measurements have shown that limiting the queue depth to one for zoned writes has a significant negative performance impact on zoned UFS devices. Hence this patch that disables zone locking from the mq-deadline scheduler for storage controllers that support pipelining zoned writes. This patch is based on the following assumptions:
- Applications submit write requests to sequential write required zones in order.
- It happens infrequently that zoned write requests are reordered by the block layer.
- The storage controller does not reorder write requests that have been submitted to the same hardware queue. This is the case for UFS: the UFSHCI specification requires that UFS controllers process requests in order per hardware queue.
- The I/O priority of all pipelined write requests is the same per zone.
- Either no I/O scheduler is used or an I/O scheduler is used that submits write requests per zone in LBA order.

---
 block/blk-zoned.c   |  3 ++-
 block/mq-deadline.c | 14 +++++++++-----
 2 files changed, 11 insertions(+), 6 deletions(-)
block/blk-zoned.c +0 -0
index 0f9f97cdddd9..59560d1657e3 100644

block/blk-zoned.c static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx, · line 508

Quoted 6 lines
block/blk-zoned.c +2 -1
@@ -504,7 +504,8 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,
 		break;
 	case BLK_ZONE_TYPE_SEQWRITE_REQ:
 	case BLK_ZONE_TYPE_SEQWRITE_PREF:
-		if (!args->seq_zones_wlock) {
+		if (!blk_queue_pipeline_zoned_writes(q) &&
+		    !args->seq_zones_wlock) {

I think that this change should go into the first patch, no ?

Quoted 8 lines
block/blk-zoned.c +0 -0
 			args->seq_zones_wlock =
 				blk_alloc_zone_bitmap(q->node, args->nr_zones);
 			if (!args->seq_zones_wlock)
block/mq-deadline.c +0 -0
index 6aa5daf7ae32..0bed2bdeed89 100644

block/mq-deadline.c deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio, · line 357

Quoted 6 lines
block/mq-deadline.c +2 -1
@@ -353,7 +353,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 		return NULL;
 
 	rq = rq_entry_fifo(per_prio->fifo_list[data_dir].next);
-	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q))
+	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q) ||
+	    blk_queue_pipeline_zoned_writes(rq->q))

What about using blk_req_needs_zone_write_lock() ?

block/mq-deadline.c deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio, · line 403

Quoted 10 lines
block/mq-deadline.c +2 -1
 		return rq;
 
 	/*
@@ -398,7 +399,8 @@ deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
 	if (!rq)
 		return NULL;
 
-	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q))
+	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q) ||
+	    blk_queue_pipeline_zoned_writes(rq->q))

Same.

Quoted 16 lines
block/mq-deadline.c +3 -2
 		return rq;
 
 	/*
@@ -526,8 +528,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd,
 	}
 
 	/*
-	 * For a zoned block device, if we only have writes queued and none of
-	 * them can be dispatched, rq will be NULL.
+	 * For a zoned block device that requires write serialization, if we
+	 * only have writes queued and none of them can be dispatched, rq will
+	 * be NULL.
 	 */
 	if (!rq)
 		return NULL;

block/mq-deadline.c static void dd_finish_request(struct request *rq) · line 940

Quoted 6 lines
block/mq-deadline.c +2 -1
@@ -933,7 +936,8 @@ static void dd_finish_request(struct request *rq)
 
 	atomic_inc(&per_prio->stats.completed);
 
-	if (blk_queue_is_zoned(q)) {
+	if (blk_queue_is_zoned(rq->q) &&
+	    !blk_queue_pipeline_zoned_writes(q)) {

And again here.

Quoted 3 lines
block/mq-deadline.c +0 -0
 		unsigned long flags;
 
 		spin_lock_irqsave(&dd->zone_lock, flags);
Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#7 lore raw

Re: [PATCH v2 2/5] block/mq-deadline: Only use zone locking if necessary

Quoted 14 lines · Damien Le Moal

block/blk-zoned.c static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx, · line 508

Quoted 11 lines · Bart Van Assche
block/blk-zoned.c +2 -1
index 0f9f97cdddd9..59560d1657e3 100644
@@ -504,7 +504,8 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,
  		break;
  	case BLK_ZONE_TYPE_SEQWRITE_REQ:
  	case BLK_ZONE_TYPE_SEQWRITE_PREF:
-		if (!args->seq_zones_wlock) {
+		if (!blk_queue_pipeline_zoned_writes(q) &&
+		    !args->seq_zones_wlock) {

I think that this change should go into the first patch, no ?

That's a good point. I will move this change into the first patch.

Thanks,

Bart.

author reviewer
#8 lore raw

Re: [PATCH v2 2/5] block/mq-deadline: Only use zone locking if necessary

In reply to Damien Le Moal
Quoted 8 lines · Damien Le Moal
Quoted 7 lines · Bart Van Assche
block/mq-deadline.c +0 -0
index 6aa5daf7ae32..0bed2bdeed89 100644
@@ -353,7 +353,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,
  		return NULL;
 
Quoted 6 lines

block/mq-deadline.c deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio, · line 357

Quoted 4 lines
block/mq-deadline.c +2 -1
  	rq = rq_entry_fifo(per_prio->fifo_list[data_dir].next);
-	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q))
+	if (data_dir == DD_READ || !blk_queue_is_zoned(rq->q) ||
+	    blk_queue_pipeline_zoned_writes(rq->q))

What about using blk_req_needs_zone_write_lock() ?

Hmm ... how would using blk_req_needs_zone_write_lock() improve the generated code? blk_queue_pipeline_zoned_writes() can be inlined and only tests a single bit (a request queue flag) while blk_req_needs_zone_write_lock() cannot be inlined by the compiler because it has been defined in a .c file. Additionally, blk_req_needs_zone_write_lock() has to dereference more pointers than blk_queue_pipeline_zoned_writes(). From block/blk-zoned.c:

bool blk_req_needs_zone_write_lock(struct request *rq)

{
	if (!rq->q->disk->seq_zones_wlock)
		return false;

	return blk_rq_is_seq_zoned_write(rq);
}

EXPORT_SYMBOL_GPL(blk_req_needs_zone_write_lock);

Thanks,

Bart.

author reviewer
#9 lore raw

PATCH v2 3/5 block/null_blk: Add support for pipelining zoned writes

In reply to Bart Van Assche

Add a new configfs attribute for enabling pipelining of zoned writes. The test script below reports 250 K IOPS with no I/O scheduler, 6 K IOPS with mq-deadline and pipelining disabled and 123 K IOPS with mq-deadline and pipelining enabled. This shows that pipelining results in about 20 times more IOPS for this particular test case.

    #!/bin/bash

    for mode in "none 0" "mq-deadline 0" "mq-deadline 1"; do
        set +e
        for d in /sys/kernel/config/nullb/*; do
            [ -d "$d" ] && rmdir "$d"
        done
        modprobe -r null_blk
        set -e
        read -r iosched pipelining <<<"$mode"
        modprobe null_blk nr_devices=0
        (
            cd /sys/kernel/config/nullb
            mkdir nullb0
            cd nullb0
            params=(
                completion_nsec=100000
                hw_queue_depth=64
                irqmode=2                # NULL_IRQ_TIMER
                max_sectors=$((4096/512))
                memory_backed=1
                pipeline_zoned_writes="${pipelining}"
                size=1
                submit_queues=1
                zone_size=1
                zoned=1
                power=1
            )
            for p in "${params[@]}"; do
                echo "${p//*=}" > "${p//=*}"
            done
        )
        udevadm settle
        dev=/dev/nullb0
        [ -b "${dev}" ]
        params=(
            --direct=1
            --filename="${dev}"
            --iodepth=64
            --iodepth_batch=16
            --ioengine=io_uring
            --ioscheduler="${iosched}"
            --gtod_reduce=1
            --hipri=0
            --name=nullb0
            --runtime=30
            --rw=write
            --time_based=1
            --zonemode=zbd
        )
        fio "${params[@]}"
    done
---
 drivers/block/null_blk/main.c     | 2 ++
 drivers/block/null_blk/null_blk.h | 1 +
 drivers/block/null_blk/zoned.c    | 3 +++
 3 files changed, 6 insertions(+)
drivers/block/null_blk/main.c +2 -0
index 864013019d6b..1cd6eb4e2c16 100644
@@ -424,6 +424,7 @@ NULLB_DEVICE_ATTR(zone_capacity, ulong, NULL);
 NULLB_DEVICE_ATTR(zone_nr_conv, uint, NULL);
 NULLB_DEVICE_ATTR(zone_max_open, uint, NULL);
 NULLB_DEVICE_ATTR(zone_max_active, uint, NULL);
+NULLB_DEVICE_ATTR(pipeline_zoned_writes, bool, NULL);
 NULLB_DEVICE_ATTR(virt_boundary, bool, NULL);
 NULLB_DEVICE_ATTR(no_sched, bool, NULL);
 NULLB_DEVICE_ATTR(shared_tag_bitmap, bool, NULL);
@@ -569,6 +570,7 @@ static struct configfs_attribute *nullb_device_attrs[] = {
 	&nullb_device_attr_zone_max_active,
 	&nullb_device_attr_zone_readonly,
 	&nullb_device_attr_zone_offline,
+	&nullb_device_attr_pipeline_zoned_writes,
 	&nullb_device_attr_virt_boundary,
 	&nullb_device_attr_no_sched,
 	&nullb_device_attr_shared_tag_bitmap,
drivers/block/null_blk/null_blk.h +1 -0
index 929f659dd255..248acf288a8e 100644
@@ -117,6 +117,7 @@ struct nullb_device {
 	bool memory_backed; /* if data is stored in memory */
 	bool discard; /* if support discard */
 	bool zoned; /* if device is zoned */
+	bool pipeline_zoned_writes;
 	bool virt_boundary; /* virtual boundary on/off for the device */
 	bool no_sched; /* no IO scheduler for the device */
 	bool shared_tag_bitmap; /* use hostwide shared tags */
drivers/block/null_blk/zoned.c +3 -0
index 635ce0648133..2bfd5f7cee67 100644
@@ -96,6 +96,9 @@ int null_init_zoned_dev(struct nullb_device *dev, struct request_queue *q)
 
 	spin_lock_init(&dev->zone_res_lock);
 
+	if (dev->pipeline_zoned_writes)
+		blk_queue_flag_set(QUEUE_FLAG_PIPELINE_ZONED_WRITES, q);
+
 	if (dev->zone_nr_conv >= dev->nr_zones) {
 		dev->zone_nr_conv = dev->nr_zones - 1;
 		pr_info("changed the number of conventional zones to %u",
author reviewer
#10 lore raw

PATCH v2 4/5 scsi: Retry unaligned zoned writes

In reply to Bart Van Assche

From ZBC-2: "The device server terminates with CHECK CONDITION status, with the sense key set to ILLEGAL REQUEST, and the additional sense code set to UNALIGNED WRITE COMMAND a write command, other than an entire medium write same command, that specifies: a) the starting LBA in a sequential write required zone set to a value that is not equal to the write pointer for that sequential write required zone; or b) an ending LBA that is not equal to the last logical block within a physical block (see SBC-5)."

I am not aware of any other conditions that may trigger the UNALIGNED WRITE COMMAND response.

Send commands that failed with an unaligned write error to the SCSI error handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.

Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.

Cc Martin K. Petersen <[email protected]>
Cc Damien Le Moal <[email protected]>
---
 drivers/scsi/scsi_error.c | 37 +++++++++++++++++++++++++++++++++++++
 drivers/scsi/scsi_lib.c   |  1 +
 drivers/scsi/sd.c         |  3 +++
 include/scsi/scsi.h       |  1 +
 4 files changed, 42 insertions(+)
drivers/scsi/scsi_error.c +37 -0
index 3ec8bfd4090f..a6d562f3a085 100644
@@ -27,6 +27,7 @@
 #include <linux/blkdev.h>
 #include <linux/delay.h>
 #include <linux/jiffies.h>
+#include <linux/list_sort.h>
 
 #include <scsi/scsi.h>
 #include <scsi/scsi_cmnd.h>
@@ -681,6 +682,17 @@ enum scsi_disposition scsi_check_sense(struct scsi_cmnd *scmd)
 		fallthrough;
 
 	case ILLEGAL_REQUEST:
+		/*
+		 * Unaligned write command. This indicates that zoned writes
+		 * have been received by the device in the wrong order. If zoned
+		 * write pipelining is enabled, retry after all pending commands
+		 * have completed.
+		 */
+		if (sshdr.asc == 0x21 && sshdr.ascq == 0x04 &&
+		    blk_queue_pipeline_zoned_writes(sdev->request_queue) &&
+		    !scsi_noretry_cmd(scmd) && scsi_cmd_retry_allowed(scmd))
+			return NEEDS_DELAYED_RETRY;
+
 		if (sshdr.asc == 0x20 || /* Invalid command operation code */
 		    sshdr.asc == 0x21 || /* Logical block address out of range */
 		    sshdr.asc == 0x22 || /* Invalid function */
@@ -2177,6 +2189,25 @@ void scsi_eh_flush_done_q(struct list_head *done_q)
 }
 EXPORT_SYMBOL(scsi_eh_flush_done_q);
 
+/*
+ * Returns a negative value if @_a has a lower LBA than @_b, zero if
+ * both have the same LBA and a positive value otherwise.
+ */
+static int scsi_cmp_lba(void *priv, const struct list_head *_a,
+			const struct list_head *_b)
+{
+	struct scsi_cmnd *a = list_entry(_a, typeof(*a), eh_entry);
+	struct scsi_cmnd *b = list_entry(_b, typeof(*b), eh_entry);
+	const sector_t pos_a = blk_rq_pos(scsi_cmd_to_rq(a));
+	const sector_t pos_b = blk_rq_pos(scsi_cmd_to_rq(b));
+
+	if (pos_a < pos_b)
+		return -1;
+	if (pos_a > pos_b)
+		return 1;
+	return 0;
+}
+
 /**
  * scsi_unjam_host - Attempt to fix a host which has a cmd that failed.
  * @shost:	Host to unjam.
@@ -2212,6 +2243,12 @@ static void scsi_unjam_host(struct Scsi_Host *shost)
 
 	SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(shost, &eh_work_q));
 
+	/*
+	 * Sort pending SCSI commands in LBA order. This is important if write
+	 * pipelining is enabled for a zoned SCSI device.
+	 */
+	list_sort(NULL, &eh_work_q, scsi_cmp_lba);
+
 	if (!scsi_eh_get_sense(&eh_work_q, &eh_done_q))
 		scsi_eh_ready_devs(shost, &eh_work_q, &eh_done_q);
 
drivers/scsi/scsi_lib.c +1 -0
index 0226c9279cef..a6cfdc4bfbf1 100644
@@ -1445,6 +1445,7 @@ static void scsi_complete(struct request *rq)
 	case ADD_TO_MLQUEUE:
 		scsi_queue_insert(cmd, SCSI_MLQUEUE_DEVICE_BUSY);
 		break;
+	case NEEDS_DELAYED_RETRY:
 	default:
 		scsi_eh_scmd_add(cmd);
 		break;
drivers/scsi/sd.c +3 -0
index ab216976dbdc..6cf7495c6b56 100644
@@ -1206,6 +1206,9 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd)
 	cmd->transfersize = sdp->sector_size;
 	cmd->underflow = nr_blocks << 9;
 	cmd->allowed = sdkp->max_retries;
+	if (blk_queue_pipeline_zoned_writes(rq->q) &&
+	    blk_rq_is_seq_zoned_write(rq))
+		cmd->allowed += rq->q->nr_requests;
 	cmd->sdb.length = nr_blocks * sdp->sector_size;
 
 	SCSI_LOG_HLQUEUE(1,
include/scsi/scsi.h +1 -0
index ec093594ba53..6600db046227 100644
@@ -93,6 +93,7 @@ static inline int scsi_status_is_check_condition(int status)
  * Internal return values.
  */
 enum scsi_disposition {
+	NEEDS_DELAYED_RETRY	= 0x2000,
 	NEEDS_RETRY		= 0x2001,
 	SUCCESS			= 0x2002,
 	FAILED			= 0x2003,
maintainer
#11 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 10 lines · Bart Van Assche

From ZBC-2: "The device server terminates with CHECK CONDITION status, with the sense key set to ILLEGAL REQUEST, and the additional sense code set to UNALIGNED WRITE COMMAND a write command, other than an entire medium write same command, that specifies: a) the starting LBA in a sequential write required zone set to a value that is not equal to the write pointer for that sequential write required zone; or b) an ending LBA that is not equal to the last logical block within a physical block (see SBC-5)."

I am not aware of any other conditions that may trigger the UNALIGNED WRITE COMMAND response.

Trying to write less than 4KB on a 4Kn drive. But that should not ever happen for kernel issued commands.

Quoted 7 lines

Send commands that failed with an unaligned write error to the SCSI error handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.

Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.

I think I mentioned this before. When we started btrfs work, we did something similar (but at the IO scheduler level) to try to avoid adding a big lock in btrfs to serialize (and thus order) writes. What we discovered is that it was extremely easy to fall into a situation were the maximum number of possible outstanding request is already issued, but they all are behind a "hole" and indefinitely delayed because the missing request cannot be issued due to the max nr request limit being reached. No forward progress and deadlock.

I do not see how your change addresses this problem. The same will happen with this and I do not have any suggestion how to solve this. For btrfs, we ended up using cone append emulation for scsi to avoid the big lock and avoid the FS from having to order writes. That solution guarantees forward progress. Delaying already issued writes that are not sequential has no such guarantees.

Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#12 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 16 lines · Damien Le Moal
Quoted 6 lines · Bart Van Assche

Send commands that failed with an unaligned write error to the SCSI error handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.

Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.

I think I mentioned this before. When we started btrfs work, we did something similar (but at the IO scheduler level) to try to avoid adding a big lock in btrfs to serialize (and thus order) writes. What we discovered is that it was extremely easy to fall into a situation were the maximum number of possible outstanding request is already issued, but they all are behind a "hole" and indefinitely delayed because the missing request cannot be issued due to the max nr request limit being reached. No forward progress and deadlock.

Quoted 5 lines

I do not see how your change addresses this problem. The same will happen with this and I do not have any suggestion how to solve this. For btrfs, we ended up using cone append emulation for scsi to avoid the big lock and avoid the FS from having to order writes. That solution guarantees forward progress. Delaying already issued writes that are not sequential has no such guarantees.

Hi Damien,

Thank you for having explained in detail the scenario that you ran into.

I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order per zone? I think that this is how F2FS supports zoned storage.

Thanks,

Bart.

maintainer
#13 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 36 lines · Bart Van Assche
Quoted 23 lines · Damien Le Moal
Quoted 7 lines · Bart Van Assche

Send commands that failed with an unaligned write error to the SCSI error
handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.

Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.

I think I mentioned this before. When we started btrfs work, we did something
similar (but at the IO scheduler level) to try to avoid adding a big lock in
btrfs to serialize (and thus order) writes. What we discovered is that it was
extremely easy to fall into a situation were the maximum number of possible
outstanding request is already issued, but they all are behind a "hole" and
indefinitely delayed because the missing request cannot be issued due to the max
nr request limit being reached. No forward progress and deadlock.

Quoted 9 lines

I do not see how your change addresses this problem. The same will happen with
this and I do not have any suggestion how to solve this. For btrfs, we ended up
using cone append emulation for scsi to avoid the big lock and avoid the FS from
having to order writes. That solution guarantees forward progress. Delaying
already issued writes that are not sequential has no such guarantees.

Hi Damien,

Quoted 7 lines

Thank you for having explained in detail the scenario that you ran into.

I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order
per zone? I think that this is how F2FS supports zoned storage.

Sure. But what if an application uses the drive directly ? You loose guarantees of forward progress then. Given that an application has to use direct IO for writes to sequential zones, this is unlikely to happen in a "good" scenario, but it also would not be hard to write an application that can deadlock the drive forever by simply missing one write in a sequence of writes for a zone... That is my concern. While f2fs would likely be OK, the delay approach is not solid enough for all cases.

Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#14 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 7 lines · Damien Le Moal
Quoted 5 lines · Bart Van Assche

I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order
per zone? I think that this is how F2FS supports zoned storage.

Quoted 8 lines

Sure. But what if an application uses the drive directly ? You loose guarantees of forward progress then. Given that an application has to use direct IO for writes to sequential zones, this is unlikely to happen in a "good" scenario, but it also would not be hard to write an application that can deadlock the drive forever by simply missing one write in a sequence of writes for a zone... That is my concern. While f2fs would likely be OK, the delay approach is not solid enough for all cases.

Hi Damien,

This patch series increases the number of retries for zoned writes submitted by a filesystem. Direct I/O from user space is not affected since the code that increases the number of retries occurs in sd_setup_read_write_cmnd(). As you know scsi_prepare_cmd() only calls sd_setup_read_write_cmnd() for requests that come from a filesystem and not for pass-through requests.

Does this address your concern?

Thanks,

Bart.

maintainer
#15 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 19 lines · Bart Van Assche
Quoted 7 lines · Damien Le Moal
Quoted 5 lines · Bart Van Assche

I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order
per zone? I think that this is how F2FS supports zoned storage.

Quoted 8 lines

Sure. But what if an application uses the drive directly ? You loose guarantees of forward progress then. Given that an application has to use direct IO for writes to sequential zones, this is unlikely to happen in a "good" scenario, but it also would not be hard to write an application that can deadlock the drive forever by simply missing one write in a sequence of writes for a zone... That is my concern. While f2fs would likely be OK, the delay approach is not solid enough for all cases.

Hi Damien,

Quoted 8 lines

This patch series increases the number of retries for zoned writes submitted by a filesystem. Direct I/O from user space is not affected since the code that increases the number of retries occurs in sd_setup_read_write_cmnd(). As you know scsi_prepare_cmd() only calls sd_setup_read_write_cmnd() for requests that come from a filesystem and not for pass-through requests.

Does this address your concern?

Not really. I was not thinking about passthrough requests but rather write syscalls to /dev/sdX by applications. sd_setup_read_write_cmnd() is used for these too. After all, /dev/sdX *is* a file system too, albeit the simplest possible.

Quoted 5 lines

Thanks,

Bart.

Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#16 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 3 lines · Damien Le Moal

Not really. I was not thinking about passthrough requests but rather write syscalls to /dev/sdX by applications. sd_setup_read_write_cmnd() is used for these too. After all, /dev/sdX *is* a file system too, albeit the simplest possible.

How about introducing a new request flag (REQ_*) such that f2fs can disable zone locking while it remains enabled for all other users (BTRFS and direct I/O)?

Thanks,

Bart.

maintainer
#17 lore raw

Re: [PATCH v2 4/5] scsi: Retry unaligned zoned writes

Quoted 8 lines · Bart Van Assche
Quoted 3 lines · Damien Le Moal

Not really. I was not thinking about passthrough requests but rather write syscalls to /dev/sdX by applications. sd_setup_read_write_cmnd() is used for these too. After all, /dev/sdX *is* a file system too, albeit the simplest possible.

How about introducing a new request flag (REQ_*) such that f2fs can disable zone locking while it remains enabled for all other users (BTRFS and direct I/O)?

That could be a safer approach. Still a little worried about any DM used between the FS and the device though. The DMs that support zoned devices should be OK but only under the assumption that the bio issuer (f2fs) guarantees that writes to the same zone are not issued in parallel when that REQ_NO_ZONE_WRITE (or whatever the name) is used. This flag definition will need to have a big fat warning mentioning this constraint, that is, make it clear that it is not a replacement for zone append (in which case we do not write lock zones and do not care about ordering).

Signature · 2 lines
Damien Le Moal
Western Digital Research
author reviewer
#18 lore raw

PATCH v2 5/5 scsi: ufs: Enable zoned write pipelining

In reply to Bart Van Assche

From the UFSHCI 4.0 specification, about the legacy (single queue) mode:
"The host controller always process transfer requests in-order according to the order submitted to the list. In case of multiple commands with single doorbell register ringing (batch mode), The dispatch order for these transfer requests by host controller will base on their index in the List. A transfer request with lower index value will be executed before a transfer request with higher index value."

From the UFSHCI 4.0 specification, about the MCQ mode:
"Command Submission
1. Host SW writes an Entry to SQ
2. Host SW updates SQ doorbell tail pointer

Command Processing
3. After fetching the Entry, Host Controller updates SQ doorbell head pointer
4. Host controller sends COMMAND UPIU to UFS device"

In other words, for both legacy and MCQ mode, UFS controllers are required to forward commands to the UFS device in the order these commands have been received from the host.

Note: for legacy mode this is only correct if the host submits one command at a time. The UFS driver does this.

This patch improves small write IOPS by about 150% on my test setup.

Cc Martin K. Petersen <[email protected]>
Cc Damien Le Moal <[email protected]>
Cc Avri Altman <[email protected]>
---
 drivers/ufs/core/ufshcd.c | 1 +
 1 file changed, 1 insertion(+)
drivers/ufs/core/ufshcd.c +1 -0
index e7e79f515e14..8d0e495ae6fa 100644
@@ -5146,6 +5146,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)
 
 	ufshcd_hpb_configure(hba, sdev);
 
+	blk_queue_flag_set(QUEUE_FLAG_PIPELINE_ZONED_WRITES, q);
 	blk_queue_update_dma_pad(q, PRDT_DATA_BYTE_COUNT_PAD - 1);
 	if (hba->quirks & UFSHCD_QUIRK_4KB_DMA_ALIGNMENT)
 		blk_queue_update_dma_alignment(q, 4096 - 1);