Enable zoned write pipelining for UFS devices
Subsystems (4)
Bart Van Assche
reviewerv2 cover Enable zoned write pipelining for UFS devices
Hi Jens,
This patch series improves write performance for zoned UFS devices. Please consider these patches for the next merge window.
Thank you,
Bart.
Changes compared to v1:
- Left out the patches that are already upstream.
- Switched the approach in patch "scsi: Retry unaligned zoned writes" from retrying immediately to sending unaligned write commands to the SCSI error handler.
Bart Van Assche (5):
block: Introduce a request queue flag for pipelining zoned writes block/mq-deadline: Only use zone locking if necessary block/null_blk: Add support for pipelining zoned writes
scsi: Retry unaligned zoned writes
scsi: ufs: Enable zoned write pipelining
block/blk-zoned.c | 3 ++- block/mq-deadline.c | 14 +++++++----- drivers/block/null_blk/main.c | 2 ++ drivers/block/null_blk/null_blk.h | 1 + drivers/block/null_blk/zoned.c | 3 +++ drivers/scsi/scsi_error.c | 37 +++++++++++++++++++++++++++++++ drivers/scsi/scsi_lib.c | 1 + drivers/scsi/sd.c | 3 +++ drivers/ufs/core/ufshcd.c | 1 + include/linux/blkdev.h | 7 ++++++ include/scsi/scsi.h | 1 + 11 files changed, 67 insertions(+), 6 deletions(-)
Bart Van Assche
reviewerPATCH v2 1/5 block: Introduce a request queue flag for pipelining zoned writes
Writes in sequential write required zones must happen at the write pointer. Even if the submitter of the write commands (e.g. a filesystem) submits writes for sequential write required zones in order, the block layer or the storage controller may reorder these write commands.
The zone locking mechanism in the mq-deadline I/O scheduler serializes write commands for sequential zones. Some but not all storage controllers require this serialization. Introduce a new flag such that block drivers can request pipelining of writes for sequential write required zones.
An example of a storage controller standard that requires write serialization is AHCI (Advanced Host Controller Interface). Submitting commands to AHCI controllers happens by writing a bit pattern into a register. Each set bit corresponds to an active command. This mechanism does not preserve command ordering information.
--- include/linux/blkdev.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/linux/blkdev.h +7 -0
@@ -534,6 +534,8 @@ struct request_queue {#define QUEUE_FLAG_NONROT 6/* non-rotational device (SSD) */ #define QUEUE_FLAG_VIRT QUEUE_FLAG_NONROT/* paravirt device */ #define QUEUE_FLAG_IO_STAT 7/* do disk/partitions IO accounting */ +/* Writes for sequential write required zones may be pipelined. */ +#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8#define QUEUE_FLAG_NOXMERGES 9/* No extended merges */ #define QUEUE_FLAG_ADD_RANDOM 10/* Contributes to random pool */ #define QUEUE_FLAG_SYNCHRONOUS 11/* always completes in submit context */
@@ -596,6 +598,11 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);#define blk_queue_skip_tagset_quiesce (q) \ test_bit(QUEUE_FLAG_SKIP_TAGSET_QUIESCE, &(q)->queue_flags) +static inline bool blk_queue_pipeline_zoned_writes (struct request_queue *q) +{ +return test_bit (QUEUE_FLAG_PIPELINE_ZONED_WRITES , &q->queue_flags); +} +extern void blk_set_pm_only (struct request_queue *q);extern void blk_clear_pm_only (struct request_queue *q);
Damien Le Moal
maintainerRe: [PATCH v2 1/5] block: Introduce a request queue flag for pipelining zoned writes
Quoted 27 lines · Bart Van Assche
Writes in sequential write required zones must happen at the write pointer. Even if the submitter of the write commands (e.g. a filesystem) submits writes for sequential write required zones in order, the block layer or the storage controller may reorder these write commands.
The zone locking mechanism in the mq-deadline I/O scheduler serializes write commands for sequential zones. Some but not all storage controllers require this serialization. Introduce a new flag such that block drivers can request pipelining of writes for sequential write required zones.
An example of a storage controller standard that requires write serialization is AHCI (Advanced Host Controller Interface). Submitting commands to AHCI controllers happens by writing a bit pattern into a register. Each set bit corresponds to an active command. This mechanism does not preserve command ordering information.
Cc Damien Le Moal <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- include/linux/blkdev.h | 7 +++++++ 1 file changed, 7 insertions(+)include/linux/blkdev.h +0
-0
include/linux/blkdev.h struct request_queue { · line 538
Quoted 5 lines
include/linux/blkdev.h +2
-0@@ -534,6 +534,8 @@ struct request_queue {#define QUEUE_FLAG_NONROT 6/* non-rotational device (SSD) */ #define QUEUE_FLAG_VIRT QUEUE_FLAG_NONROT/* paravirt device */ #define QUEUE_FLAG_IO_STAT 7/* do disk/partitions IO accounting */ +/* Writes for sequential write required zones may be pipelined. */ +#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8
I am not a big fan of this name as "pipeline" does not necessarily imply "high queue depth write". What about simply calling this QUEUE_FLAG_NO_ZONE_WRITE_LOCK, indicating that there is no need to write-lock zones ?
Quoted 15 lines
include/linux/blkdev.h +5
-0#define QUEUE_FLAG_NOXMERGES 9/* No extended merges */ #define QUEUE_FLAG_ADD_RANDOM 10/* Contributes to random pool */ #define QUEUE_FLAG_SYNCHRONOUS 11/* always completes in submit context */ @@ -596,6 +598,11 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q);#define blk_queue_skip_tagset_quiesce (q) \ test_bit(QUEUE_FLAG_SKIP_TAGSET_QUIESCE, &(q)->queue_flags) +static inline bool blk_queue_pipeline_zoned_writes (struct request_queue *q) +{ +return test_bit (QUEUE_FLAG_PIPELINE_ZONED_WRITES , &q->queue_flags); +} +extern void blk_set_pm_only (struct request_queue *q);extern void blk_clear_pm_only (struct request_queue *q);
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerRe: [PATCH v2 1/5] block: Introduce a request queue flag for pipelining zoned writes
Quoted 8 lines · Damien Le Moal
include/linux/blkdev.h struct request_queue { · line 538
Quoted 2 lines · Bart Van Assche
include/linux/blkdev.h +2
-0+/* Writes for sequential write required zones may be pipelined. */ +#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8I am not a big fan of this name as "pipeline" does not necessarily imply "high queue depth write". What about simply calling this
QUEUE_FLAG_NO_ZONE_WRITE_LOCK, indicating that there is no need to write-lock zones ?
Hi Damien,
I'm not a big fan of names with negative words like "no" or "not" embedded. Isn't pipelining standard computer science terminology? See also
https://en.wikipedia.org/wiki/Pipeline_(computing).
Thanks,
Bart.
Damien Le Moal
maintainerRe: [PATCH v2 1/5] block: Introduce a request queue flag for pipelining zoned writes
Quoted 12 lines · Bart Van Assche
Quoted 10 lines · Damien Le Moal
include/linux/blkdev.h struct request_queue { · line 538
Quoted 2 lines · Bart Van Assche
include/linux/blkdev.h +2
-0+/* Writes for sequential write required zones may be pipelined. */ +#define QUEUE_FLAG_PIPELINE_ZONED_WRITES 8I am not a big fan of this name as "pipeline" does not necessarily imply "high
queue depth write". What about simply calling thisQUEUE_FLAG_NO_ZONE_WRITE_LOCK, indicating that there is no need to write-lock
zones ?
Quoted 5 lines
Hi Damien,
I'm not a big fan of names with negative words like "no" or "not" embedded. Isn't pipelining standard computer science terminology? See also
https://en.wikipedia.org/wiki/Pipeline_(computing).
Sure, pipeline is a well known term. But I do not think it is synonymous with "high queue depth write" :) A "pipeline" for zoned write may still operate at write qd=1 per zone...
Given that the default is using zone write locking, I would prefer a flag name that indicates a change to this default. What about something like QUEUE_FLAG_UNRESTRICTED_ZONE_WRITE ?
But tht is beside the point as I still have reservations on this approach anyway. See my reply to patch 4.
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerPATCH v2 2/5 block/mq-deadline: Only use zone locking if necessary
In reply to Bart Van AsscheMeasurements have shown that limiting the queue depth to one for zoned writes has a significant negative performance impact on zoned UFS devices. Hence this patch that disables zone locking from the mq-deadline scheduler for storage controllers that support pipelining zoned writes. This patch is based on the following assumptions:
- Applications submit write requests to sequential write required zones in order.
- It happens infrequently that zoned write requests are reordered by the block layer.
- The storage controller does not reorder write requests that have been submitted to the same hardware queue. This is the case for UFS: the UFSHCI specification requires that UFS controllers process requests in order per hardware queue.
- The I/O priority of all pipelined write requests is the same per zone.
- Either no I/O scheduler is used or an I/O scheduler is used that submits write requests per zone in LBA order.
--- block/blk-zoned.c | 3 ++- block/mq-deadline.c | 14 +++++++++----- 2 files changed, 11 insertions(+), 6 deletions(-)
block/blk-zoned.c +2 -1
@@ -504,7 +504,8 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,break ;case BLK_ZONE_TYPE_SEQWRITE_REQ :case BLK_ZONE_TYPE_SEQWRITE_PREF : -if (!args->seq_zones_wlock) { +if (!blk_queue_pipeline_zoned_writes (q) && + !args->seq_zones_wlock) { args->seq_zones_wlock =blk_alloc_zone_bitmap (q->node, args->nr_zones);if (!args->seq_zones_wlock)
block/mq-deadline.c +9 -5
@@ -353,7 +353,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,return NULL ; rq =rq_entry_fifo (per_prio->fifo_list[data_dir].next); -if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q)) +if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q) || +blk_queue_pipeline_zoned_writes (rq->q))return rq;/*
@@ -398,7 +399,8 @@ deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio,if (!rq)return NULL ; -if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q)) +if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q) || +blk_queue_pipeline_zoned_writes (rq->q))return rq;/*
@@ -526,8 +528,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, }/* -* For a zoned block device, if we only have writes queued and none of -* them can be dispatched, rq will be NULL. +* For a zoned block device that requires write serialization, if we +* only have writes queued and none of them can be dispatched, rq will +* be NULL. */ if (!rq)return NULL ;
@@ -933,7 +936,8 @@ static void dd_finish_request(struct request *rq)atomic_inc (&per_prio->stats.completed); -if (blk_queue_is_zoned (q)) { +if (blk_queue_is_zoned (rq->q) && + !blk_queue_pipeline_zoned_writes (q)) {unsigned long flags;spin_lock_irqsave (&dd->zone_lock, flags);
Damien Le Moal
maintainerRe: [PATCH v2 2/5] block/mq-deadline: Only use zone locking if necessary
Quoted 29 lines · Bart Van Assche
Measurements have shown that limiting the queue depth to one for zoned writes has a significant negative performance impact on zoned UFS devices. Hence this patch that disables zone locking from the mq-deadline scheduler for storage controllers that support pipelining zoned writes. This patch is based on the following assumptions:
- Applications submit write requests to sequential write required zones in order.
- It happens infrequently that zoned write requests are reordered by the block layer.
- The storage controller does not reorder write requests that have been submitted to the same hardware queue. This is the case for UFS: the UFSHCI specification requires that UFS controllers process requests in order per hardware queue.
- The I/O priority of all pipelined write requests is the same per zone.
- Either no I/O scheduler is used or an I/O scheduler is used that submits write requests per zone in LBA order.Cc Damien Le Moal <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- block/blk-zoned.c | 3 ++- block/mq-deadline.c | 14 +++++++++----- 2 files changed, 11 insertions(+), 6 deletions(-)block/blk-zoned.c +0
-0
block/blk-zoned.c static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx, · line 508
Quoted 6 lines
block/blk-zoned.c +2
-1@@ -504,7 +504,8 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,break ;case BLK_ZONE_TYPE_SEQWRITE_REQ :case BLK_ZONE_TYPE_SEQWRITE_PREF : -if (!args->seq_zones_wlock) { +if (!blk_queue_pipeline_zoned_writes (q) && + !args->seq_zones_wlock) {
I think that this change should go into the first patch, no ?
Quoted 8 lines
block/blk-zoned.c +0
-0args->seq_zones_wlock =blk_alloc_zone_bitmap (q->node, args->nr_zones);if (!args->seq_zones_wlock)block/mq-deadline.c +0
-0
block/mq-deadline.c deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio, · line 357
Quoted 6 lines
block/mq-deadline.c +2
-1@@ -353,7 +353,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,return NULL ; rq =rq_entry_fifo (per_prio->fifo_list[data_dir].next); -if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q)) +if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q) || +blk_queue_pipeline_zoned_writes (rq->q))
What about using blk_req_needs_zone_write_lock() ?
block/mq-deadline.c deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio, · line 403
Quoted 10 lines
block/mq-deadline.c +2
-1return rq;/* @@ -398,7 +399,8 @@ deadline_next_request(struct deadline_data *dd, struct dd_per_prio *per_prio,if (!rq)return NULL ; -if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q)) +if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q) || +blk_queue_pipeline_zoned_writes (rq->q))
Same.
Quoted 16 lines
block/mq-deadline.c +3
-2return rq;/* @@ -526,8 +528,9 @@ static struct request *__dd_dispatch_request(struct deadline_data *dd, }/* -* For a zoned block device, if we only have writes queued and none of -* them can be dispatched, rq will be NULL. +* For a zoned block device that requires write serialization, if we +* only have writes queued and none of them can be dispatched, rq will +* be NULL. */ if (!rq)return NULL ;
block/mq-deadline.c static void dd_finish_request(struct request *rq) · line 940
Quoted 6 lines
block/mq-deadline.c +2
-1@@ -933,7 +936,8 @@ static void dd_finish_request(struct request *rq)atomic_inc (&per_prio->stats.completed); -if (blk_queue_is_zoned (q)) { +if (blk_queue_is_zoned (rq->q) && + !blk_queue_pipeline_zoned_writes (q)) {
And again here.
Quoted 3 lines
block/mq-deadline.c +0
-0unsigned long flags;spin_lock_irqsave (&dd->zone_lock, flags);
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerRe: [PATCH v2 2/5] block/mq-deadline: Only use zone locking if necessary
Quoted 14 lines · Damien Le Moal
block/blk-zoned.c static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx, · line 508
Quoted 11 lines · Bart Van Assche
block/blk-zoned.c +2
-1@@ -504,7 +504,8 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,break ;case BLK_ZONE_TYPE_SEQWRITE_REQ :case BLK_ZONE_TYPE_SEQWRITE_PREF : -if (!args->seq_zones_wlock) { +if (!blk_queue_pipeline_zoned_writes (q) && + !args->seq_zones_wlock) {I think that this change should go into the first patch, no ?
That's a good point. I will move this change into the first patch.
Thanks,
Bart.
Bart Van Assche
reviewerRe: [PATCH v2 2/5] block/mq-deadline: Only use zone locking if necessary
In reply to Damien Le MoalQuoted 8 lines · Damien Le Moal
Quoted 7 lines · Bart Van Assche
block/mq-deadline.c +0
-0@@ -353,7 +353,8 @@ deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio,return NULL ;
Quoted 6 lines
block/mq-deadline.c deadline_fifo_request(struct deadline_data *dd, struct dd_per_prio *per_prio, · line 357
Quoted 4 lines
block/mq-deadline.c +2
-1rq =rq_entry_fifo (per_prio->fifo_list[data_dir].next); -if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q)) +if (data_dir ==DD_READ || !blk_queue_is_zoned (rq->q) || +blk_queue_pipeline_zoned_writes (rq->q))What about using
blk_req_needs_zone_write_lock()?
Hmm ... how would using blk_req_needs_zone_write_lock() improve the generated code? blk_queue_pipeline_zoned_writes() can be inlined and only tests a single bit (a request queue flag) while blk_req_needs_zone_write_lock() cannot be inlined by the compiler because it has been defined in a .c file. Additionally, blk_req_needs_zone_write_lock() has to dereference more pointers than blk_queue_pipeline_zoned_writes(). From block/blk-zoned.c:
bool blk_req_needs_zone_write_lock(struct request *rq)
{
if (!rq->q->disk->seq_zones_wlock)
return false;
return blk_rq_is_seq_zoned_write(rq);
}EXPORT_SYMBOL_GPL(blk_req_needs_zone_write_lock);
Thanks,
Bart.
Bart Van Assche
reviewerPATCH v2 3/5 block/null_blk: Add support for pipelining zoned writes
In reply to Bart Van AsscheAdd a new configfs attribute for enabling pipelining of zoned writes. The test script below reports 250 K IOPS with no I/O scheduler, 6 K IOPS with mq-deadline and pipelining disabled and 123 K IOPS with mq-deadline and pipelining enabled. This shows that pipelining results in about 20 times more IOPS for this particular test case.
#!/bin/bash
for mode in "none 0" "mq-deadline 0" "mq-deadline 1"; do
set +e
for d in /sys/kernel/config/nullb/*; do
[ -d "$d" ] && rmdir "$d"
done
modprobe -r null_blk
set -e
read -r iosched pipelining <<<"$mode"
modprobe null_blk nr_devices=0
(
cd /sys/kernel/config/nullb
mkdir nullb0
cd nullb0
params=(
completion_nsec=100000
hw_queue_depth=64
irqmode=2 # NULL_IRQ_TIMER
max_sectors=$((4096/512))
memory_backed=1
pipeline_zoned_writes="${pipelining}"
size=1
submit_queues=1
zone_size=1
zoned=1
power=1
)
for p in "${params[@]}"; do
echo "${p//*=}" > "${p//=*}"
done
)
udevadm settle
dev=/dev/nullb0
[ -b "${dev}" ]
params=(
--direct=1
--filename="${dev}"
--iodepth=64
--iodepth_batch=16
--ioengine=io_uring
--ioscheduler="${iosched}"
--gtod_reduce=1
--hipri=0
--name=nullb0
--runtime=30
--rw=write
--time_based=1
--zonemode=zbd
)
fio "${params[@]}"
done--- drivers/block/null_blk/main.c | 2 ++ drivers/block/null_blk/null_blk.h | 1 + drivers/block/null_blk/zoned.c | 3 +++ 3 files changed, 6 insertions(+)
drivers/block/null_blk/main.c +2 -0
@@ -424,6 +424,7 @@ NULLB_DEVICE_ATTR(zone_capacity, ulong, NULL);NULLB_DEVICE_ATTR (zone_nr_conv, uint,NULL );NULLB_DEVICE_ATTR (zone_max_open, uint,NULL );NULLB_DEVICE_ATTR (zone_max_active, uint,NULL ); +NULLB_DEVICE_ATTR (pipeline_zoned_writes, bool,NULL );NULLB_DEVICE_ATTR (virt_boundary, bool,NULL );NULLB_DEVICE_ATTR (no_sched, bool,NULL );NULLB_DEVICE_ATTR (shared_tag_bitmap, bool,NULL );
@@ -569,6 +570,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_zone_max_active, &nullb_device_attr_zone_readonly, &nullb_device_attr_zone_offline, + &nullb_device_attr_pipeline_zoned_writes, &nullb_device_attr_virt_boundary, &nullb_device_attr_no_sched, &nullb_device_attr_shared_tag_bitmap,
drivers/block/null_blk/null_blk.h +1 -0
@@ -117,6 +117,7 @@ struct nullb_device {bool memory_backed;/* if data is stored in memory */ bool discard;/* if support discard */ bool zoned;/* if device is zoned */ +bool pipeline_zoned_writes;bool virt_boundary;/* virtual boundary on/off for the device */ bool no_sched;/* no IO scheduler for the device */ bool shared_tag_bitmap;/* use hostwide shared tags */
drivers/block/null_blk/zoned.c +3 -0
@@ -96,6 +96,9 @@ int null_init_zoned_dev(struct nullb_device *dev, struct request_queue *q)spin_lock_init (&dev->zone_res_lock); +if (dev->pipeline_zoned_writes) +blk_queue_flag_set (QUEUE_FLAG_PIPELINE_ZONED_WRITES , q); +if (dev->zone_nr_conv >= dev->nr_zones) { dev->zone_nr_conv = dev->nr_zones -1 ; pr_info("changed the number of conventional zones to %u",
Bart Van Assche
reviewerPATCH v2 4/5 scsi: Retry unaligned zoned writes
In reply to Bart Van AsscheFrom ZBC-2: "The device server terminates with CHECK CONDITION status, with the sense key set to ILLEGAL REQUEST, and the additional sense code set to UNALIGNED WRITE COMMAND a write command, other than an entire medium write same command, that specifies: a) the starting LBA in a sequential write required zone set to a value that is not equal to the write pointer for that sequential write required zone; or b) an ending LBA that is not equal to the last logical block within a physical block (see SBC-5)."
I am not aware of any other conditions that may trigger the UNALIGNED WRITE COMMAND response.
Send commands that failed with an unaligned write error to the SCSI error handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.
Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.
--- drivers/scsi/scsi_error.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/scsi/scsi_lib.c | 1 + drivers/scsi/sd.c | 3 +++ include/scsi/scsi.h | 1 + 4 files changed, 42 insertions(+)
drivers/scsi/scsi_error.c +37 -0
@@ -27,6 +27,7 @@#include <linux/blkdev.h> #include <linux/delay.h> #include <linux/jiffies.h> +#include <linux/list_sort.h> #include <scsi/scsi.h> #include <scsi/scsi_cmnd.h>
@@ -681,6 +682,17 @@ enum scsi_disposition scsi_check_sense(struct scsi_cmnd *scmd)fallthrough ;case ILLEGAL_REQUEST : +/* +* Unaligned write command. This indicates that zoned writes +* have been received by the device in the wrong order. If zoned +* write pipelining is enabled, retry after all pending commands +* have completed. +*/ +if (sshdr.asc ==0x21 && sshdr.ascq ==0x04 && +blk_queue_pipeline_zoned_writes (sdev->request_queue) && + !scsi_noretry_cmd (scmd) &&scsi_cmd_retry_allowed (scmd)) +return NEEDS_DELAYED_RETRY ; + if (sshdr.asc == 0x20 || /* Invalid command operation code */ sshdr.asc == 0x21 || /* Logical block address out of range */ sshdr.asc == 0x22 || /* Invalid function */
@@ -2177,6 +2189,25 @@ void scsi_eh_flush_done_q(struct list_head *done_q) }EXPORT_SYMBOL (scsi_eh_flush_done_q); +/* +* Returns a negative value if @_a has a lower LBA than @_b, zero if +* both have the same LBA and a positive value otherwise. +*/ +static int scsi_cmp_lba (void *priv,const struct list_head *_a, +const struct list_head *_b) +{ +struct scsi_cmnd *a =list_entry (_a,typeof (*a), eh_entry); +struct scsi_cmnd *b =list_entry (_b,typeof (*b), eh_entry); +const sector_t pos_a =blk_rq_pos (scsi_cmd_to_rq (a)); +const sector_t pos_b =blk_rq_pos (scsi_cmd_to_rq (b)); + +if (pos_a < pos_b) +return -1 ; +if (pos_a > pos_b) +return 1 ; +return 0 ; +} +/** * scsi_unjam_host - Attempt to fix a host which has a cmd that failed. * @shost: Host to unjam.
@@ -2212,6 +2243,12 @@ static void scsi_unjam_host(struct Scsi_Host *shost)SCSI_LOG_ERROR_RECOVERY (1 ,scsi_eh_prt_fail_stats (shost, &eh_work_q)); +/* +* Sort pending SCSI commands in LBA order. This is important if write +* pipelining is enabled for a zoned SCSI device. +*/ +list_sort (NULL , &eh_work_q, scsi_cmp_lba); +if (!scsi_eh_get_sense (&eh_work_q, &eh_done_q))scsi_eh_ready_devs (shost, &eh_work_q, &eh_done_q);
drivers/scsi/scsi_lib.c +1 -0
@@ -1445,6 +1445,7 @@ static void scsi_complete(struct request *rq)case ADD_TO_MLQUEUE :scsi_queue_insert (cmd,SCSI_MLQUEUE_DEVICE_BUSY );break ; +case NEEDS_DELAYED_RETRY :default :scsi_eh_scmd_add (cmd);break ;
drivers/scsi/sd.c +3 -0
@@ -1206,6 +1206,9 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd) cmd->transfersize = sdp->sector_size; cmd->underflow = nr_blocks <<9 ; cmd->allowed = sdkp->max_retries; +if (blk_queue_pipeline_zoned_writes (rq->q) && +blk_rq_is_seq_zoned_write (rq)) + cmd->allowed += rq->q->nr_requests; cmd->sdb.length = nr_blocks * sdp->sector_size; SCSI_LOG_HLQUEUE(1,
include/scsi/scsi.h +1 -0
@@ -93,6 +93,7 @@ static inline int scsi_status_is_check_condition(int status)* Internal return values. */ enum scsi_disposition { +NEEDS_DELAYED_RETRY =0x2000 ,NEEDS_RETRY =0x2001 ,SUCCESS =0x2002 ,FAILED =0x2003 ,
Damien Le Moal
maintainerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 10 lines · Bart Van Assche
From ZBC-2: "The device server terminates with CHECK CONDITION status, with the sense key set to ILLEGAL REQUEST, and the additional sense code set to UNALIGNED WRITE COMMAND a write command, other than an entire medium write same command, that specifies: a) the starting LBA in a sequential write required zone set to a value that is not equal to the write pointer for that sequential write required zone; or b) an ending LBA that is not equal to the last logical block within a physical block (see SBC-5)."
I am not aware of any other conditions that may trigger the UNALIGNED WRITE COMMAND response.
Trying to write less than 4KB on a 4Kn drive. But that should not ever happen for kernel issued commands.
Quoted 7 lines
Send commands that failed with an unaligned write error to the SCSI error handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.
Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.
I think I mentioned this before. When we started btrfs work, we did something similar (but at the IO scheduler level) to try to avoid adding a big lock in btrfs to serialize (and thus order) writes. What we discovered is that it was extremely easy to fall into a situation were the maximum number of possible outstanding request is already issued, but they all are behind a "hole" and indefinitely delayed because the missing request cannot be issued due to the max nr request limit being reached. No forward progress and deadlock.
I do not see how your change addresses this problem. The same will happen with this and I do not have any suggestion how to solve this. For btrfs, we ended up using cone append emulation for scsi to avoid the big lock and avoid the FS from having to order writes. That solution guarantees forward progress. Delaying already issued writes that are not sequential has no such guarantees.
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 16 lines · Damien Le Moal
Quoted 6 lines · Bart Van Assche
Send commands that failed with an unaligned write error to the SCSI error handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.
Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.
I think I mentioned this before. When we started btrfs work, we did something similar (but at the IO scheduler level) to try to avoid adding a big lock in btrfs to serialize (and thus order) writes. What we discovered is that it was extremely easy to fall into a situation were the maximum number of possible outstanding request is already issued, but they all are behind a "hole" and indefinitely delayed because the missing request cannot be issued due to the max nr request limit being reached. No forward progress and deadlock.
Quoted 5 lines
I do not see how your change addresses this problem. The same will happen with this and I do not have any suggestion how to solve this. For btrfs, we ended up using cone append emulation for scsi to avoid the big lock and avoid the FS from having to order writes. That solution guarantees forward progress. Delaying already issued writes that are not sequential has no such guarantees.
Hi Damien,
Thank you for having explained in detail the scenario that you ran into.
I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order per zone? I think that this is how F2FS supports zoned storage.
Thanks,
Bart.
Damien Le Moal
maintainerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 36 lines · Bart Van Assche
Quoted 23 lines · Damien Le Moal
Quoted 7 lines · Bart Van Assche
Send commands that failed with an unaligned write error to the SCSI error
handler. Let the SCSI error handler sort SCSI commands per LBA before resubmitting these.Increase the number of retries for write commands sent to a sequential zone to the maximum number of outstanding commands.
I think I mentioned this before. When we started btrfs work, we did something
similar (but at the IO scheduler level) to try to avoid adding a big lock in
btrfs to serialize (and thus order) writes. What we discovered is that it was
extremely easy to fall into a situation were the maximum number of possible
outstanding request is already issued, but they all are behind a "hole" and
indefinitely delayed because the missing request cannot be issued due to the max
nr request limit being reached. No forward progress and deadlock.Quoted 9 lines
I do not see how your change addresses this problem. The same will happen with
this and I do not have any suggestion how to solve this. For btrfs, we ended up
using cone append emulation for scsi to avoid the big lock and avoid the FS from
having to order writes. That solution guarantees forward progress. Delaying
already issued writes that are not sequential has no such guarantees.Hi Damien,
Quoted 7 lines
Thank you for having explained in detail the scenario that you ran into.
I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order
per zone? I think that this is how F2FS supports zoned storage.
Sure. But what if an application uses the drive directly ? You loose guarantees of forward progress then. Given that an application has to use direct IO for writes to sequential zones, this is unlikely to happen in a "good" scenario, but it also would not be hard to write an application that can deadlock the drive forever by simply missing one write in a sequence of writes for a zone... That is my concern. While f2fs would likely be OK, the delay approach is not solid enough for all cases.
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 7 lines · Damien Le Moal
Quoted 5 lines · Bart Van Assche
I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order
per zone? I think that this is how F2FS supports zoned storage.
Quoted 8 lines
Sure. But what if an application uses the drive directly ? You loose guarantees of forward progress then. Given that an application has to use direct IO for writes to sequential zones, this is unlikely to happen in a "good" scenario, but it also would not be hard to write an application that can deadlock the drive forever by simply missing one write in a sequence of writes for a zone... That is my concern. While f2fs would likely be OK, the delay approach is not solid enough for all cases.
Hi Damien,
This patch series increases the number of retries for zoned writes submitted by a filesystem. Direct I/O from user space is not affected since the code that increases the number of retries occurs in sd_setup_read_write_cmnd(). As you know scsi_prepare_cmd() only calls sd_setup_read_write_cmnd() for requests that come from a filesystem and not for pass-through requests.
Does this address your concern?
Thanks,
Bart.
Damien Le Moal
maintainerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 19 lines · Bart Van Assche
Quoted 7 lines · Damien Le Moal
Quoted 5 lines · Bart Van Assche
I think what has been explained above is a scenario in which the filesystem allocates requests per zone in another order than the LBA order. How about requiring that the filesystem allocates and submits zoned writes in LBA order
per zone? I think that this is how F2FS supports zoned storage.Quoted 8 lines
Sure. But what if an application uses the drive directly ? You loose guarantees of forward progress then. Given that an application has to use direct IO for writes to sequential zones, this is unlikely to happen in a "good" scenario, but it also would not be hard to write an application that can deadlock the drive forever by simply missing one write in a sequence of writes for a zone... That is my concern. While f2fs would likely be OK, the delay approach is not solid enough for all cases.
Hi Damien,
Quoted 8 lines
This patch series increases the number of retries for zoned writes submitted by a filesystem. Direct I/O from user space is not affected since the code that increases the number of retries occurs in
sd_setup_read_write_cmnd(). As you knowscsi_prepare_cmd()only callssd_setup_read_write_cmnd()for requests that come from a filesystem and not for pass-through requests.Does this address your concern?
Not really. I was not thinking about passthrough requests but rather write syscalls to /dev/sdX by applications. sd_setup_read_write_cmnd() is used for these too. After all, /dev/sdX *is* a file system too, albeit the simplest possible.
Quoted 5 lines
Thanks,
Bart.
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 3 lines · Damien Le Moal
Not really. I was not thinking about passthrough requests but rather write syscalls to
/dev/sdXby applications.sd_setup_read_write_cmnd()is used for these too. After all,/dev/sdX*is* a file system too, albeit the simplest possible.
How about introducing a new request flag (REQ_*) such that f2fs can disable zone locking while it remains enabled for all other users (BTRFS and direct I/O)?
Thanks,
Bart.
Damien Le Moal
maintainerRe: [PATCH v2 4/5] scsi: Retry unaligned zoned writes
Quoted 8 lines · Bart Van Assche
Quoted 3 lines · Damien Le Moal
Not really. I was not thinking about passthrough requests but rather write syscalls to
/dev/sdXby applications.sd_setup_read_write_cmnd()is used for these too. After all,/dev/sdX*is* a file system too, albeit the simplest possible.How about introducing a new request flag (REQ_*) such that f2fs can disable zone locking while it remains enabled for all other users (BTRFS and direct I/O)?
That could be a safer approach. Still a little worried about any DM used between the FS and the device though. The DMs that support zoned devices should be OK but only under the assumption that the bio issuer (f2fs) guarantees that writes to the same zone are not issued in parallel when that REQ_NO_ZONE_WRITE (or whatever the name) is used. This flag definition will need to have a big fat warning mentioning this constraint, that is, make it clear that it is not a replacement for zone append (in which case we do not write lock zones and do not care about ordering).
Signature · 2 lines
Damien Le Moal Western Digital Research
Bart Van Assche
reviewerPATCH v2 5/5 scsi: ufs: Enable zoned write pipelining
In reply to Bart Van AsscheFrom the UFSHCI 4.0 specification, about the legacy (single queue) mode:
"The host controller always process transfer requests in-order according to the order submitted to the list. In case of multiple commands with single doorbell register ringing (batch mode), The dispatch order for these transfer requests by host controller will base on their index in the List. A transfer request with lower index value will be executed before a transfer request with higher index value."
From the UFSHCI 4.0 specification, about the MCQ mode:
"Command Submission
1. Host SW writes an Entry to SQ
2. Host SW updates SQ doorbell tail pointer
Command Processing
3. After fetching the Entry, Host Controller updates SQ doorbell head pointer
4. Host controller sends COMMAND UPIU to UFS device"
In other words, for both legacy and MCQ mode, UFS controllers are required to forward commands to the UFS device in the order these commands have been received from the host.
Note: for legacy mode this is only correct if the host submits one command at a time. The UFS driver does this.
This patch improves small write IOPS by about 150% on my test setup.
--- drivers/ufs/core/ufshcd.c | 1 + 1 file changed, 1 insertion(+)
drivers/ufs/core/ufshcd.c +1 -0
@@ -5146,6 +5146,7 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)ufshcd_hpb_configure (hba, sdev); +blk_queue_flag_set (QUEUE_FLAG_PIPELINE_ZONED_WRITES , q);blk_queue_update_dma_pad (q,PRDT_DATA_BYTE_COUNT_PAD -1 );if (hba->quirks &UFSHCD_QUIRK_4KB_DMA_ALIGNMENT )blk_queue_update_dma_alignment (q,4096 -1 );