Submit zoned writes in order
Subsystems (9)
v4 cover Submit zoned writes in order
Hi Jens,
Tests with a zoned UFS prototype have shown that the block layer may reorder zoned writes (REQ_OP_WRITE). The UFS driver is more likely to trigger reordering than other SCSI drivers because it reports BLK_STS_DEV_RESOURCE more often, e.g. during clock scaling. This patch series makes sure that zoned writes are submitted in order without affecting other workloads significantly.
Please consider this patch series for the next merge window.
Thanks,
Bart.
Changes compared to v3:
- Dropped the changes for sending already dispatched requests back to the I/O scheduler if the block driver is busy.
- Dropped several patches that aren't needed to preserve the order of zoned writes.
- Only send requeued writes to the I/O scheduler. Send requeued reads straight to the dispatch queue.
- Changed the approach from one requeue list per request queue into one requeue list per hardware queue.
Changes compared to v2:
- Changed the approach from one requeue list per hctx into preserving one requeue list per request queue.
- Rebased on top of Jens' for-next branch. Left out the mq-deadline patches since these are already in the for-next branch.
- Modified patch "block: Requeue requests if a CPU is unplugged" such that it always uses the requeue list.
- Added a patch that removes blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list().
- Dropped patch "block: mq-deadline: Disable head insertion for zoned writes".
- Dropped patch "block: mq-deadline: Introduce a local variable".
Changes compared to v1:
- Fixed two issues detected by the kernel test robot.
Bart Van Assche (7):
block: Rename a local variable in blk_mq_requeue_work()
block: Simplify blk_mq_requeue_work()
block: Send requeued requests to the I/O scheduler
block: One requeue list per hctx
block: Preserve the order of requeued requests
dm: Inline __dm_mq_kick_requeue_list()
block: Inline blk_mq_{,delay_}kick_requeue_list()
block/blk-flush.c | 28 +++++++------ block/blk-mq-debugfs.c | 66 ++++++++++++++--------------- block/blk-mq.c | 81 ++++++++++++++---------------------- drivers/block/ublk_drv.c | 6 +-- drivers/block/xen-blkfront.c | 1 - drivers/md/dm-rq.c | 11 ++--- drivers/nvme/host/core.c | 2 +- drivers/s390/block/scm_blk.c | 2 +- drivers/scsi/scsi_lib.c | 2 +- include/linux/blk-mq.h | 11 +++-- include/linux/blkdev.h | 5 --- 11 files changed, 96 insertions(+), 119 deletions(-)
PATCH v4 1/7 block: Rename a local variable in blk_mq_requeue_work()
Two data structures in blk_mq_requeue_work() represent request lists. Make it clear that rq_list holds requests that come from the requeue list by renaming that data structure.
--- block/blk-mq.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -4
@@ -1441,17 +1441,17 @@ static void blk_mq_requeue_work(struct work_struct *work) {struct request_queue *q = container_of(work, struct request_queue, requeue_work.work); -LIST_HEAD (rq_list); +LIST_HEAD (requeue_list);LIST_HEAD (flush_list);struct request *rq;spin_lock_irq (&q->requeue_lock); -list_splice_init (&q->requeue_list, &rq_list); +list_splice_init (&q->requeue_list, &requeue_list);list_splice_init (&q->flush_list, &flush_list);spin_unlock_irq (&q->requeue_lock); -while (!list_empty (&rq_list)) { - rq = list_entry(rq_list.next, struct request, queuelist); +while (!list_empty (&requeue_list)) { + rq = list_entry(requeue_list.next, struct request, queuelist);/* * If RQF_DONTPREP ist set, the request has been started by the * driver already and might have driver-specific data allocated
Re: [PATCH v4 1/7] block: Rename a local variable in blk_mq_requeue_work()
Quoted 10 lines · Bart Van Assche
Two data structures in
blk_mq_requeue_work()represent request lists. Make it clear thatrq_listholds requests that come from the requeue list by renaming that data structure.Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>Cc Damien Le Moal <[email protected]>Cc Ming Lei <[email protected]>Cc Mike Snitzer <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
Signature · 2 lines
Damien Le Moal Western Digital Research
PATCH v4 2/7 block: Simplify blk_mq_requeue_work()
In reply to Bart Van AsscheMove common code in front of the if-statement.
--- block/blk-mq.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
block/blk-mq.c +3 -5
@@ -1458,13 +1458,11 @@ static void blk_mq_requeue_work(struct work_struct *work)* already. Insert it into the hctx dispatch list to avoid * block layer merges for the request. */ -if (rq->rq_flags &RQF_DONTPREP ) { -list_del_init (&rq->queuelist); +list_del_init (&rq->queuelist); +if (rq->rq_flags &RQF_DONTPREP )blk_mq_request_bypass_insert (rq,0 ); - }else { -list_del_init (&rq->queuelist); +else blk_mq_insert_request (rq,BLK_MQ_INSERT_AT_HEAD ); - } }while (!list_empty (&flush_list)) {
Re: [PATCH v4 2/7] block: Simplify blk_mq_requeue_work()
Quoted 7 lines · Bart Van Assche
Move common code in front of the if-statement.
Cc Christoph Hellwig <[email protected]>Cc Damien Le Moal <[email protected]>Cc Ming Lei <[email protected]>Cc Mike Snitzer <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v4 2/7] block: Simplify blk_mq_requeue_work()
In reply to Bart Van AsscheQuoted 1 line · Bart Van Assche
Move common code in front of the if-statement.
This mechanical change looks ok to me, it's more the existing ad prevailing logic here I'm worried about..
PATCH v4 3/7 block: Send requeued requests to the I/O scheduler
In reply to Bart Van AsscheSend requeued requests to the I/O scheduler if the dispatch order matters such that the I/O scheduler can control the order in which requests are dispatched.
This patch reworks commit aef1897cd36d ("blk-mq: insert rq with DONTPREP to hctx dispatch list when requeue"). Instead of sending DONTPREP requests to the dispatch list, send these to the I/O scheduler and prevent that the I/O scheduler merges these requests by adding RQF_DONTPREP to the list of flags that prevent merging (RQF_NOMERGE_FLAGS).
--- block/blk-mq.c | 10 +++++----- include/linux/blk-mq.h | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-)
block/blk-mq.c +5 -5
@@ -1453,13 +1453,13 @@ static void blk_mq_requeue_work(struct work_struct *work)while (!list_empty (&requeue_list)) { rq = list_entry(requeue_list.next, struct request, queuelist);/* -* If RQF_DONTPREP ist set, the request has been started by the -* driver already and might have driver-specific data allocated -* already. Insert it into the hctx dispatch list to avoid -* block layer merges for the request. +* Only send those RQF_DONTPREP requests to the dispatch list +* that may be reordered freely. If the request order matters, +* send the request to the I/O scheduler. */ list_del_init (&rq->queuelist); -if (rq->rq_flags &RQF_DONTPREP ) +if (rq->rq_flags &RQF_DONTPREP && + !op_needs_zoned_write_locking (req_op (rq)))blk_mq_request_bypass_insert (rq,0 );else blk_mq_insert_request (rq,BLK_MQ_INSERT_AT_HEAD );
include/linux/blk-mq.h +2 -2
@@ -62,8 +62,8 @@ typedef __u32 __bitwise req_flags_t;#define RQF_RESV ((__force req_flags_t)(1 << 23))/* flags that prevent us from merging requests: */ -#define RQF_NOMERGE_FLAGS \ - (RQF_STARTED | RQF_FLUSH_SEQ | RQF_SPECIAL_PAYLOAD) +#define RQF_NOMERGE_FLAGS \ + (RQF_STARTED | RQF_FLUSH_SEQ | RQF_DONTPREP | RQF_SPECIAL_PAYLOAD)enum mq_rq_state {MQ_RQ_IDLE =0 ,
Re: [PATCH v4 3/7] block: Send requeued requests to the I/O scheduler
Quoted 31 lines · Bart Van Assche
Send requeued requests to the I/O scheduler if the dispatch order matters such that the I/O scheduler can control the order in which requests are dispatched.
This patch reworks commit
aef1897cd36d("blk-mq: insert rq with DONTPREP to hctx dispatch list when requeue"). Instead of sending DONTPREP requests to the dispatch list, send these to the I/O scheduler and prevent that the I/O scheduler merges these requests by addingRQF_DONTPREPto the list of flags that prevent merging (RQF_NOMERGE_FLAGS).Cc Christoph Hellwig <[email protected]>Cc Damien Le Moal <[email protected]>Cc Ming Lei <[email protected]>Cc Mike Snitzer <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- block/blk-mq.c | 10 +++++----- include/linux/blk-mq.h | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-)block/blk-mq.c +0
-2@@ -1453,13 +1453,13 @@ static void blk_mq_requeue_work(struct work_struct *work)while (!list_empty (&requeue_list)) { rq = list_entry(requeue_list.next, struct request, queuelist);/* -* If RQF_DONTPREP ist set, the request has been started by the -* driver already and might have driver-specific data allocated
block/blk-mq.c static void blk_mq_requeue_work(struct work_struct *work) · line 1462
Quoted 10 lines
block/blk-mq.c +5
-3-* already. Insert it into the hctx dispatch list to avoid -* block layer merges for the request. +* Only send those RQF_DONTPREP requests to the dispatch list +* that may be reordered freely. If the request order matters, +* send the request to the I/O scheduler. */ list_del_init (&rq->queuelist); -if (rq->rq_flags &RQF_DONTPREP ) +if (rq->rq_flags &RQF_DONTPREP && + !op_needs_zoned_write_locking (req_op (rq)))
Why ? I still do not understand the need for this. There is always only a single in-flight write per sequential zone. Requeuing that in-flight write directly to the dispatch list will not reorder writes and it will be better for the command latency.
Quoted 18 lines
block/blk-mq.c +0
-0blk_mq_request_bypass_insert (rq,0 ); elseblk_mq_insert_request (rq ,BLK_MQ_INSERT_AT_HEAD );include/linux/blk-mq.h +2
-2@@ -62,8 +62,8 @@ typedef __u32 __bitwise req_flags_t;#define RQF_RESV ((__force req_flags_t)(1 << 23))/* flags that prevent us from merging requests: */ -#define RQF_NOMERGE_FLAGS \ - (RQF_STARTED | RQF_FLUSH_SEQ | RQF_SPECIAL_PAYLOAD) +#define RQF_NOMERGE_FLAGS \ + (RQF_STARTED | RQF_FLUSH_SEQ | RQF_DONTPREP | RQF_SPECIAL_PAYLOAD)enum mq_rq_state {MQ_RQ_IDLE =0 ,
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v4 3/7] block: Send requeued requests to the I/O scheduler
Quoted 4 lines · Damien Le Moal
Why ? I still do not understand the need for this. There is always only a single in-flight write per sequential zone. Requeuing that in-flight write directly to the dispatch list will not reorder writes and it will be better for the command latency.
Hi Damien,
After having taken a closer look I see that blk_req_zone_write_unlock() is called from inside dd_insert_request() when requeuing a request. Hence, there is no reordering risk when requeuing a zoned write. I will drop this patch.
Do you agree with having one requeue list per hctx instead of per request queue? This change enables eliminating blk_mq_kick_requeue_list(). I think that's an interesting simplification of the block layer API.
Thanks,
Bart.
Re: [PATCH v4 3/7] block: Send requeued requests to the I/O scheduler
Quoted 11 lines · Bart Van Assche
Quoted 4 lines · Damien Le Moal
Why ? I still do not understand the need for this. There is always only a single in-flight write per sequential zone. Requeuing that in-flight write directly to the dispatch list will not reorder writes and it will be better for the command latency.
Hi Damien,
After having taken a closer look I see that
blk_req_zone_write_unlock()is called from insidedd_insert_request()when requeuing a request. Hence, there is no reordering risk when requeuing a zoned write. I will drop this patch.
OK. Thanks.
Quoted 5 lines
Do you agree with having one requeue list per hctx instead of per request queue? This change enables eliminating
blk_mq_kick_requeue_list(). I think that's an interesting simplification of the block layer API.
I do not see any issue with that. Indeed, it does simplify the code nicely. Reading patch 5, I wondered though if it is really worth keeping the helpers blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list(). May be calling blk_mq_run_hw_queues() and blk_mq_delay_run_hw_queues() is better ? No strong opinion though.
Signature · 2 lines
Damien Le Moal Western Digital Research
PATCH v4 4/7 block: One requeue list per hctx
In reply to Bart Van AsschePrepare for processing the requeue list from inside __blk_mq_run_hw_queue().
--- block/blk-flush.c | 24 ++++++++-------- block/blk-mq-debugfs.c | 64 +++++++++++++++++++++--------------------- block/blk-mq.c | 53 ++++++++++++++++++++-------------- include/linux/blk-mq.h | 6 ++++ include/linux/blkdev.h | 5 ---- 5 files changed, 83 insertions(+), 69 deletions(-)
block/blk-flush.c +13 -11
@@ -91,7 +91,7 @@ enum { FLUSH_PENDING_TIMEOUT = 5 * HZ, }; -static void blk_kick_flush (struct request_queue *q, +static void blk_kick_flush (struct blk_mq_hw_ctx *hctx,struct blk_flush_queue *fq,blk_opf_t flags); static inline struct blk_flush_queue *
@@ -165,6 +165,7 @@ static void blk_flush_complete_seq(struct request *rq, unsigned int seq, blk_status_t error) {struct request_queue *q = rq->q; +struct blk_mq_hw_ctx *hctx = rq->mq_hctx;struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];blk_opf_t cmd_flags;
@@ -188,9 +189,9 @@ static void blk_flush_complete_seq(struct request *rq,case REQ_FSEQ_DATA :list_move_tail (&rq->flush.list, &fq->flush_data_in_flight); -spin_lock (&q->requeue_lock); -list_add_tail (&rq->queuelist, &q->flush_list); -spin_unlock (&q->requeue_lock); +spin_lock (&hctx->requeue_lock); +list_add_tail (&rq->queuelist, &hctx->flush_list); +spin_unlock (&hctx->requeue_lock);blk_mq_kick_requeue_list (q);break ;
@@ -210,7 +211,7 @@ static void blk_flush_complete_seq(struct request *rq,BUG (); } -blk_kick_flush (q, fq, cmd_flags); +blk_kick_flush (hctx, fq, cmd_flags); } static enum rq_end_io_ret flush_end_io(struct request *flush_rq,
@@ -275,7 +276,7 @@ bool is_flush_rq(struct request *rq)/** * blk_kick_flush - consider issuing flush request -* @q: request_queue being kicked +* @hctx: hwq being kicked * @fq: flush queue * @flags: cmd_flags of the original request *
@@ -286,9 +287,10 @@ bool is_flush_rq(struct request *rq)* spin_lock_irq(fq->mq_flush_lock) * */ -static void blk_kick_flush (struct request_queue *q,struct blk_flush_queue *fq, -blk_opf_t flags) +static void blk_kick_flush (struct blk_mq_hw_ctx *hctx, +struct blk_flush_queue *fq,blk_opf_t flags) { +struct request_queue *q = hctx->queue;struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];struct request *first_rq = list_first_entry(pending, struct request, flush.list);
@@ -348,9 +350,9 @@ static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq,smp_wmb ();req_ref_set (flush_rq,1 ); -spin_lock (&q->requeue_lock); -list_add_tail (&flush_rq->queuelist, &q->flush_list); -spin_unlock (&q->requeue_lock); +spin_lock (&hctx->requeue_lock); +list_add_tail (&flush_rq->queuelist, &hctx->flush_list); +spin_unlock (&hctx->requeue_lock);blk_mq_kick_requeue_list (q); }
block/blk-mq-debugfs.c +32 -32
88 lines not on this page · raw message
block/blk-mq.c +32 -21
110 lines not on this page · raw message
include/linux/blk-mq.h +6 -0
@@ -308,6 +308,12 @@ struct blk_mq_hw_ctx {unsigned long state; }____cacheline_aligned_in_smp ; +struct list_head flush_list; + +struct list_head requeue_list; +spinlock_t requeue_lock; +struct delayed_work requeue_work; +/** * @run_work: Used for scheduling a hardware queue run at a later time. */
include/linux/blkdev.h +0 -5
@@ -479,11 +479,6 @@ struct request_queue {* for flush operations */ struct blk_flush_queue *fq; -struct list_head flush_list; - -struct list_head requeue_list; -spinlock_t requeue_lock; -struct delayed_work requeue_work;struct mutex sysfs_lock;struct mutex sysfs_dir_lock;
Re: [PATCH v4 4/7] block: One requeue list per hctx
Quoted 15 lines
void
blk_mq_kick_requeue_list(struct request_queue *q){block/blk-mq.c +6
-1-kblockd_mod_delayed_work_on (WORK_CPU_UNBOUND , &q->requeue_work,0 ); +struct blk_mq_hw_ctx *hctx; +unsigned long i; + +queue_for_each_hw_ctx (q, hctx, i) +kblockd_mod_delayed_work_on (WORK_CPU_UNBOUND , + &hctx->requeue_work,0 ); }EXPORT_SYMBOL (blk_mq_kick_requeue_list);void blk_mq_delay_kick_requeue_list (struct request_queue *q,unsigned long msecs) {
block/blk-mq.c static void blk_mq_requeue_work(struct work_struct *work) · line 1499
Quoted 10 lines
block/blk-mq.c +7
-2-kblockd_mod_delayed_work_on (WORK_CPU_UNBOUND , &q->requeue_work, -msecs_to_jiffies (msecs)); +struct blk_mq_hw_ctx *hctx; +unsigned long i; + +queue_for_each_hw_ctx (q, hctx, i) +kblockd_mod_delayed_work_on (WORK_CPU_UNBOUND , + &hctx->requeue_work, +msecs_to_jiffies (msecs)); }
I'd iplement blk_mq_kick_requeue_list as a wrapper around blk_mq_delay_kick_requeue_list that passes a 0 msec argument.
Otherwise looks fine:
Re: [PATCH v4 4/7] block: One requeue list per hctx
In reply to Bart Van AsscheQuoted 1 line · Bart Van Assche
Prepare for processing the requeue list from inside
__blk_mq_run_hw_queue().
Please add more details about the motivation for this kind of change.
IMO requeue is supposed to be run in slow code path, not see reason why it has to be moved to hw queue level.
Thanks,
Ming
Re: [PATCH v4 4/7] block: One requeue list per hctx
Quoted 7 lines · Ming Lei
Quoted 1 line · Bart Van Assche
Prepare for processing the requeue list from inside
__blk_mq_run_hw_queue().Please add more details about the motivation for this kind of change.
IMO requeue is supposed to be run in slow code path, not see reason why it has to be moved to hw queue level.
As Damien explained, the code for sending requeued requests to the I/O scheduler should be dropped because this is not necessary in order to prevent reordering of requeued zoned writes.
The remaining advantages of this patch series are:
* Simplification of the block layer API since the blk_mq_{,delay_}kick_requeue_list() functions are removed.
* Faster requeuing. Although I don't have any use case that needs faster requeuing, it is interesting to see that this patch series makes the blktest test that tests requeuing run twice as fast.
Is there agreement to proceed with this patch series if all review comments are addressed?
Thanks,
Bart.
Re: [PATCH v4 4/7] block: One requeue list per hctx
Quoted 13 lines · Bart Van Assche
Quoted 7 lines · Ming Lei
Quoted 1 line · Bart Van Assche
Prepare for processing the requeue list from inside
__blk_mq_run_hw_queue().Please add more details about the motivation for this kind of change.
IMO requeue is supposed to be run in slow code path, not see reason why it has to be moved to hw queue level.
As Damien explained, the code for sending requeued requests to the I/O scheduler should be dropped because this is not necessary in order to prevent reordering of requeued zoned writes.
Quoted 6 lines
The remaining advantages of this patch series are:
* Simplification of the block layer API since the blk_mq_{,delay_}kick_requeue_list() functions are removed.
* Faster requeuing. Although I don't have any use case that needs faster requeuing, it is interesting to see that this patch series makes the blktest test that tests requeuing run twice as fast.
Not sure if it is good, cause requeue is supposed to be slow.
Quoted 3 lines
Is there agreement to proceed with this patch series if all review comments are addressed?
One concern is that you added requeue stuff into hctx, which may add one extra L1 cache fetch in fast path.
Thank,
Ming
PATCH v4 5/7 block: Preserve the order of requeued requests
In reply to Bart Van AsscheIf a queue is run before all requeued requests have been sent to the I/O scheduler, the I/O scheduler may dispatch the wrong request. Fix this by making blk_mq_run_hw_queue() process the requeue_list instead of blk_mq_requeue_work().
--- block/blk-mq.c | 31 +++++++++---------------------- include/linux/blk-mq.h | 1 - 2 files changed, 9 insertions(+), 23 deletions(-)
block/blk-mq.c +9 -22
@@ -68,6 +68,8 @@ static inline blk_qc_t blk_rq_to_qc(struct request *rq)static bool blk_mq_hctx_has_pending (struct blk_mq_hw_ctx *hctx) {return !list_empty_careful (&hctx->dispatch) || + !list_empty_careful (&hctx->requeue_list) || + !list_empty_careful (&hctx->flush_list) ||sbitmap_any_bit_set (&hctx->ctx_map) ||blk_mq_sched_has_work (hctx); }
@@ -1438,10 +1440,8 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list) }EXPORT_SYMBOL (blk_mq_requeue_request); -static void blk_mq_requeue_work (struct work_struct *work) +static void blk_mq_process_requeue_list (struct blk_mq_hw_ctx *hctx) { -struct blk_mq_hw_ctx *hctx = - container_of(work, struct blk_mq_hw_ctx, requeue_work.work);LIST_HEAD (requeue_list);LIST_HEAD (flush_list);struct request *rq;
@@ -1471,31 +1471,18 @@ static void blk_mq_requeue_work(struct work_struct *work)list_del_init (&rq->queuelist);blk_mq_insert_request (rq,0 ); } - -blk_mq_run_hw_queue (hctx, false); }void blk_mq_kick_requeue_list (struct request_queue *q) { -struct blk_mq_hw_ctx *hctx; -unsigned long i; - -queue_for_each_hw_ctx (q, hctx, i) -kblockd_mod_delayed_work_on (WORK_CPU_UNBOUND , - &hctx->requeue_work,0 ); +blk_mq_run_hw_queues (q, true); }EXPORT_SYMBOL (blk_mq_kick_requeue_list);void blk_mq_delay_kick_requeue_list (struct request_queue *q,unsigned long msecs) { -struct blk_mq_hw_ctx *hctx; -unsigned long i; - -queue_for_each_hw_ctx (q, hctx, i) -kblockd_mod_delayed_work_on (WORK_CPU_UNBOUND , - &hctx->requeue_work, -msecs_to_jiffies (msecs)); +blk_mq_delay_run_hw_queues (q, msecs); }EXPORT_SYMBOL (blk_mq_delay_kick_requeue_list);
@@ -2248,6 +2235,7 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async)return ; } +blk_mq_process_requeue_list (hctx);blk_mq_run_dispatch_ops (hctx->queue,blk_mq_sched_dispatch_requests (hctx)); }
@@ -2296,7 +2284,7 @@ void blk_mq_run_hw_queues(struct request_queue *q, bool async)* scheduler. */ if (!sq_hctx || sq_hctx == hctx || - !list_empty_careful (&hctx->dispatch)) +blk_mq_hctx_has_pending (hctx))blk_mq_run_hw_queue (hctx, async); } }
@@ -2332,7 +2320,7 @@ void blk_mq_delay_run_hw_queues(struct request_queue *q, unsigned long msecs)* scheduler. */ if (!sq_hctx || sq_hctx == hctx || - !list_empty_careful (&hctx->dispatch)) +blk_mq_hctx_has_pending (hctx))blk_mq_delay_run_hw_queue (hctx, msecs); } }
@@ -2417,6 +2405,7 @@ static void blk_mq_run_work_fn(struct work_struct *work)struct blk_mq_hw_ctx *hctx = container_of(work, struct blk_mq_hw_ctx, run_work.work); +blk_mq_process_requeue_list (hctx);blk_mq_run_dispatch_ops (hctx->queue,blk_mq_sched_dispatch_requests (hctx)); }
@@ -3625,7 +3614,6 @@ static int blk_mq_init_hctx(struct request_queue *q,struct blk_mq_tag_set *set, struct blk_mq_hw_ctx *hctx, unsigned hctx_idx) { - INIT_DELAYED_WORK(&hctx->requeue_work, blk_mq_requeue_work); INIT_LIST_HEAD(&hctx->flush_list);INIT_LIST_HEAD (&hctx->requeue_list);spin_lock_init (&hctx->requeue_lock);
@@ -4794,7 +4782,6 @@ void blk_mq_cancel_work_sync(struct request_queue *q)unsigned long i;queue_for_each_hw_ctx (q, hctx, i) { -cancel_delayed_work_sync (&hctx->requeue_work);cancel_delayed_work_sync (&hctx->run_work); } }
include/linux/blk-mq.h +0 -1
@@ -312,7 +312,6 @@ struct blk_mq_hw_ctx {struct list_head requeue_list;spinlock_t requeue_lock; -struct delayed_work requeue_work;/** * @run_work: Used for scheduling a hardware queue run at a later time.
Christoph Hellwig
maintainerRe: [PATCH v4 5/7] block: Preserve the order of requeued requests
Review
Quoted 1 line
(quoted diff) +1
-0+ blk_mq_process_requeue_list(hctx);
Should this do list_empty_careful checks on ->requeue_list and ->flush_list so that we can avoid taking the requeue lock when these conditions aren't met before calling into blk_mq_process_requeue_list?
Re: [PATCH v4 5/7] block: Preserve the order of requeued requests
Quoted 6 lines · Christoph Hellwig
Review
Quoted 1 line
(quoted diff) +1
-0+ blk_mq_process_requeue_list(hctx);Should this do
list_empty_carefulchecks on ->requeue_list and ->flush_list so that we can avoid taking the requeue lock when these conditions aren't met before calling intoblk_mq_process_requeue_list?
Hi Christoph,
I agree that checks whether or not requeue_list and flush_list are empty should be added.
Does it matter in this context whether list_empty_careful() or list_empty() is used? If blk_mq_process_requeue_list() is called concurrently with code that adds an element to one of these two lists it is guaranteed that the queue will be run again. This is why I think that it is fine that the list checks in blk_mq_process_requeue_list() race with concurrent list additions.
Thanks,
Bart.
Christoph Hellwig
maintainerRe: [PATCH v4 5/7] block: Preserve the order of requeued requests
Quoted 6 lines · Bart Van Assche
Does it matter in this context whether
list_empty_careful()orlist_empty()is used? Ifblk_mq_process_requeue_list()is called concurrently with code that adds an element to one of these two lists it is guaranteed that the queue will be run again. This is why I think that it is fine that the list checks inblk_mq_process_requeue_list()race with concurrent list additions.
As far as I can tell we're not holding the relevant lock, so I think we need list_empty_careful. Or am I missing something?
Re: [PATCH v4 5/7] block: Preserve the order of requeued requests
In reply to Bart Van AsscheQuoted 2 lines · Bart Van Assche
If a queue is run before all requeued requests have been sent to the I/O scheduler, the I/O scheduler may dispatch the wrong request. Fix this by
Can you explain in more details about the issue? What is the wrong request? How?
If you mean write order for requeued write request, there isn't such issue, given new write request from same zone can be issued before the old requeued one is completed.
Thanks,
Ming
PATCH v4 6/7 dm: Inline __dm_mq_kick_requeue_list()
In reply to Bart Van AsscheSince commit 52d7f1b5c2f3 ("blk-mq: Avoid that requeueing starts stopped queues") the function __dm_mq_kick_requeue_list() is too short to keep it as a separate function. Hence, inline this function.
--- drivers/md/dm-rq.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/md/dm-rq.c +2 -7
@@ -168,21 +168,16 @@ static void dm_end_request(struct request *clone, blk_status_t error)rq_completed (md); } -static void __dm_mq_kick_requeue_list (struct request_queue *q,unsigned long msecs) -{ -blk_mq_delay_kick_requeue_list (q, msecs); -} -void dm_mq_kick_requeue_list (struct mapped_device *md) { -__dm_mq_kick_requeue_list (md->queue,0 ); +blk_mq_kick_requeue_list (md->queue); }EXPORT_SYMBOL (dm_mq_kick_requeue_list);static void dm_mq_delay_requeue_request (struct request *rq,unsigned long msecs) {blk_mq_requeue_request (rq, false); -__dm_mq_kick_requeue_list (rq->q, msecs); +blk_mq_delay_kick_requeue_list (rq->q, msecs); }static void dm_requeue_original_request (struct dm_rq_target_io *tio,bool delay_requeue)
PATCH v4 7/7 block: Inline blk_mq_{,delay_}kick_requeue_list()
In reply to Bart Van AsschePatch "block: Preserve the order of requeued requests" changed blk_mq_kick_requeue_list() and blk_mq_delay_kick_requeue_list() into blk_mq_run_hw_queues() and blk_mq_delay_run_hw_queues() calls respectively. Inline blk_mq_{,delay_}kick_requeue_list() because these functions are now too short to keep these as separate functions.
--- block/blk-flush.c | 4 ++-- block/blk-mq-debugfs.c | 2 +- block/blk-mq.c | 15 +-------------- drivers/block/ublk_drv.c | 6 +++--- drivers/block/xen-blkfront.c | 1 - drivers/md/dm-rq.c | 6 +++--- drivers/nvme/host/core.c | 2 +- drivers/s390/block/scm_blk.c | 2 +- drivers/scsi/scsi_lib.c | 2 +- include/linux/blk-mq.h | 2 -- 10 files changed, 13 insertions(+), 29 deletions(-)
block/blk-flush.c +2 -2
@@ -192,7 +192,7 @@ static void blk_flush_complete_seq(struct request *rq,spin_lock (&hctx->requeue_lock);list_add_tail (&rq->queuelist, &hctx->flush_list);spin_unlock (&hctx->requeue_lock); -blk_mq_kick_requeue_list (q); +blk_mq_run_hw_queues (q, true);break ;case REQ_FSEQ_DONE :
@@ -354,7 +354,7 @@ static void blk_kick_flush(struct blk_mq_hw_ctx *hctx,list_add_tail (&flush_rq->queuelist, &hctx->flush_list);spin_unlock (&hctx->requeue_lock); -blk_mq_kick_requeue_list (q); +blk_mq_run_hw_queues (q, true); } static enum rq_end_io_ret mq_flush_data_end_io(struct request *rq,
block/blk-mq-debugfs.c +1 -1
@@ -114,7 +114,7 @@ static ssize_t queue_state_write(void *data, const char __user *buf, }else if (strcmp (op,"start" ) ==0 ) {blk_mq_start_stopped_hw_queues (q, true); }else if (strcmp (op,"kick" ) ==0 ) { -blk_mq_kick_requeue_list (q); +blk_mq_run_hw_queues (q, true); }else {pr_err ("%s: unsupported operation '%s'\n" , __func__, op); inval:
block/blk-mq.c +1 -14
@@ -1436,7 +1436,7 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list)spin_unlock_irqrestore (&hctx->requeue_lock, flags);if (kick_requeue_list) -blk_mq_kick_requeue_list (q); +blk_mq_run_hw_queues (q, true); }EXPORT_SYMBOL (blk_mq_requeue_request);
@@ -1473,19 +1473,6 @@ static void blk_mq_process_requeue_list(struct blk_mq_hw_ctx *hctx) } } -void blk_mq_kick_requeue_list (struct request_queue *q) -{ -blk_mq_run_hw_queues (q, true); -} -EXPORT_SYMBOL (blk_mq_kick_requeue_list); - -void blk_mq_delay_kick_requeue_list (struct request_queue *q, -unsigned long msecs) -{ -blk_mq_delay_run_hw_queues (q, msecs); -} -EXPORT_SYMBOL (blk_mq_delay_kick_requeue_list); -static bool blk_mq_rq_inflight (struct request *rq,void *priv) {/*
drivers/block/ublk_drv.c +3 -3
@@ -902,7 +902,7 @@ static inline void __ublk_rq_task_work(struct request *req,*/ if (unlikely (!mapped_bytes)) {blk_mq_requeue_request (req, false); -blk_mq_delay_kick_requeue_list (req->q, +blk_mq_delay_run_hw_queues (req->q,UBLK_REQUEUE_DELAY_MS );return ; }
@@ -1297,7 +1297,7 @@ static void ublk_unquiesce_dev(struct ublk_device *ub)blk_mq_unquiesce_queue (ub->ub_disk->queue);/* We may have requeued some rqs in ublk_quiesce_queue() */ -blk_mq_kick_requeue_list (ub->ub_disk->queue); +blk_mq_run_hw_queues (ub->ub_disk->queue, true); }static void ublk_stop_dev (struct ublk_device *ub)
@@ -2341,7 +2341,7 @@ static int ublk_ctrl_end_recovery(struct ublk_device *ub,blk_mq_unquiesce_queue (ub->ub_disk->queue);pr_devel ("%s: queue unquiesced, dev id %d.\n" , __func__, header->dev_id); -blk_mq_kick_requeue_list (ub->ub_disk->queue); +blk_mq_run_hw_queues (ub->ub_disk->queue, true); ub->dev_info.state =UBLK_S_DEV_LIVE ;schedule_delayed_work (&ub->monitor_work,UBLK_DAEMON_MONITOR_PERIOD ); ret =0 ;
drivers/block/xen-blkfront.c +0 -1
@@ -2040,7 +2040,6 @@ static int blkif_recover(struct blkfront_info *info)blk_mq_requeue_request (req, false); }blk_mq_start_stopped_hw_queues (info->rq, true); -blk_mq_kick_requeue_list (info->rq);while ((bio =bio_list_pop (&info->bio_list)) !=NULL ) {/* Traverse the list of pending bios and re-queue them */
drivers/md/dm-rq.c +3 -3
@@ -64,7 +64,7 @@ int dm_request_based(struct mapped_device *md)void dm_start_queue (struct request_queue *q) {blk_mq_unquiesce_queue (q); -blk_mq_kick_requeue_list (q); +blk_mq_run_hw_queues (q, true); }void dm_stop_queue (struct request_queue *q)
@@ -170,14 +170,14 @@ static void dm_end_request(struct request *clone, blk_status_t error)void dm_mq_kick_requeue_list (struct mapped_device *md) { -blk_mq_kick_requeue_list (md->queue); +blk_mq_run_hw_queues (md->queue, true); }EXPORT_SYMBOL (dm_mq_kick_requeue_list);static void dm_mq_delay_requeue_request (struct request *rq,unsigned long msecs) {blk_mq_requeue_request (rq, false); -blk_mq_delay_kick_requeue_list (rq->q, msecs); +blk_mq_delay_run_hw_queues (rq->q, msecs); }static void dm_requeue_original_request (struct dm_rq_target_io *tio,bool delay_requeue)
drivers/nvme/host/core.c +1 -1
@@ -303,7 +303,7 @@ static void nvme_retry_req(struct request *req)nvme_req (req)->retries++;blk_mq_requeue_request (req, false); -blk_mq_delay_kick_requeue_list (req->q, delay); +blk_mq_delay_run_hw_queues (req->q, delay); }static void nvme_log_error (struct request *req)
drivers/s390/block/scm_blk.c +1 -1
@@ -243,7 +243,7 @@ static void scm_request_requeue(struct scm_request *scmrq)atomic_dec (&bdev->queued_reqs);scm_request_done (scmrq); -blk_mq_kick_requeue_list (bdev->rq); +blk_mq_run_hw_queues (bdev->rq, true); }static void scm_request_finish (struct scm_request *scmrq)
drivers/scsi/scsi_lib.c +1 -1
@@ -124,7 +124,7 @@ static void scsi_mq_requeue_cmd(struct scsi_cmnd *cmd, unsigned long msecs)if (msecs) {blk_mq_requeue_request (rq, false); -blk_mq_delay_kick_requeue_list (rq->q, msecs); +blk_mq_delay_run_hw_queues (rq->q, msecs); }else blk_mq_requeue_request (rq, true); }
include/linux/blk-mq.h +0 -2
@@ -871,8 +871,6 @@ static inline bool blk_mq_add_to_batch(struct request *req, }void blk_mq_requeue_request (struct request *rq,bool kick_requeue_list); -void blk_mq_kick_requeue_list (struct request_queue *q); -void blk_mq_delay_kick_requeue_list (struct request_queue *q,unsigned long msecs);void blk_mq_complete_request (struct request *rq);bool blk_mq_complete_request_remote (struct request *rq);void blk_mq_stop_hw_queue (struct blk_mq_hw_ctx *hctx);
Re: [PATCH v4 7/7] block: Inline blk_mq_{,delay_}kick_requeue_list()
Quoted 6 lines · Bart Van Assche
Patch "block: Preserve the order of requeued requests" changed
blk_mq_kick_requeue_list()andblk_mq_delay_kick_requeue_list()intoblk_mq_run_hw_queues()andblk_mq_delay_run_hw_queues()calls respectively. Inline blk_mq_{,delay_}kick_requeue_list() because these functions are now too short to keep these as separate functions.
Quoted 6 lines
Cc Christoph Hellwig <[email protected]>Cc Damien Le Moal <[email protected]>Cc Ming Lei <[email protected]>Cc Mike Snitzer <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
For the blkfront change:
Thanks, Roger.
Christoph Hellwig
maintainerRe: [PATCH v4 7/7] block: Inline blk_mq_{,delay_}kick_requeue_list()
In reply to Bart Van AsscheQuoted 5 lines · Bart Van Assche
Patch "block: Preserve the order of requeued requests" changed
blk_mq_kick_requeue_list()andblk_mq_delay_kick_requeue_list()intoblk_mq_run_hw_queues()andblk_mq_delay_run_hw_queues()calls respectively. Inline blk_mq_{,delay_}kick_requeue_list() because these functions are now too short to keep these as separate functions.
You're not inlining them, but you're removing them and open code the trivial logic in the callers.
The code change looks good to me, though.