blk_mq_complete_request overhaul
Subsystems (15)
Christoph Hellwig
maintainerHi Jens,
Peters recent inquiry made me dust off various bits of unfinished work around blk_mq_complete_request and massage it into a coherent series.
This does three different things all touching the same area:
- merge the softirq based single queue completion into the main blk-mq completion mechanism, as there is a lot of duplicate logic between the two
- move the error injection that just fails to complete requests out into the drivers. The fact that blk_mq_complete_request just wouldn't complete when called from the error handle has been causing all kinds of pain
- optimize the fast path by allowing drivers to avoid the indirect call with a little more work for polled or per-CPU IRQ completions. With this a polled block device I/O only has two indirect calls let, one for the file operation that ends up in the block device code, and another one for disptching to the nvme driver
Christoph Hellwig
maintainerPATCH 1/12 blk-mq: merge blk-softirq.c into blk-mq.c
__blk_complete_request is only called from the blk-mq code, and duplicates a lot of code from blk-mq.c. Move it there to prepare for better code sharing and simplifications.
--- block/Makefile | 2 +- block/blk-mq.c | 135 +++++++++++++++++++++++++++++++++++ block/blk-softirq.c | 156 ----------------------------------------- include/linux/blkdev.h | 1 - 4 files changed, 136 insertions(+), 158 deletions(-) delete mode 100644 block/blk-softirq.c
block/Makefile +1 -1
@@ -5,7 +5,7 @@ obj-$(CONFIG_BLOCK) := bio.o elevator.o blk-core.o blk-sysfs.o \ blk-flush.o blk-settings.o blk-ioc.o blk-map.o \ - blk-exec.o blk-merge.o blk-softirq.o blk-timeout.o \ + blk-exec.o blk-merge.o blk-timeout.o \ blk-lib.o blk-mq.o blk-mq-tag.o blk-stat.o \ blk-mq-sysfs.o blk-mq-cpumap.o blk-mq-sched.o ioctl.o \ genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o
block/blk-mq.c +135 -0
153 lines not on this page · raw message
block/blk-softirq.c deleted +0 -156
156 lines not on this page · raw message
include/linux/blkdev.h +0 -1
@@ -1078,7 +1078,6 @@ void blk_steal_bios(struct bio_list *list, struct request *rq);extern bool blk_update_request (struct request *rq,blk_status_t error,unsigned int nr_bytes); -extern void __blk_complete_request (struct request *);extern void blk_abort_request (struct request *);/*
Signature · 1 line
2.26.2
Re: [PATCH 01/12] blk-mq: merge blk-softirq.c into blk-mq.c
Quoted 5 lines · Christoph Hellwig
__blk_complete_requestis only called from the blk-mq code, and duplicates a lot of code fromblk-mq.c. Move it there to prepare for better code sharing and simplifications.Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 2/12 blk-mq: factor out a helper to reise the block softirq
In reply to Christoph HellwigAdd a helper to deduplicate the logic that raises the block softirq.
--- block/blk-mq.c | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-)
block/blk-mq.c +14 -17
@@ -598,19 +598,27 @@ static __latent_entropy void blk_done_softirq(struct softirq_action *h) } } -#ifdef CONFIG_SMP -static void trigger_softirq (void *data) +static void blk_mq_trigger_softirq (struct request *rq) { -struct request *rq = data; -struct list_head *list; +struct list_head *list =this_cpu_ptr (&blk_cpu_done); - list =this_cpu_ptr (&blk_cpu_done);list_add_tail (&rq->ipi_list, list); +/* +* If the list only contains our just added request, signal a raise of +* the softirq. If there are already entries there, someone already +* raised the irq but it hasn't run yet. +*/ if (list->next == &rq->ipi_list)raise_softirq_irqoff (BLOCK_SOFTIRQ ); } +#ifdef CONFIG_SMP +static void trigger_softirq (void *data) +{ +blk_mq_trigger_softirq (data); +} +/* * Setup and invoke a run of 'trigger_softirq' on the given cpu. */
@@ -681,19 +689,8 @@ static void __blk_complete_request(struct request *req)* avoids IPI sending from current CPU to the first CPU of a group. */ if (ccpu == cpu || shared) { -struct list_head *list; do_local: - list =this_cpu_ptr (&blk_cpu_done); -list_add_tail (&req->ipi_list, list); - -/* -* if the list only contains our just added request, -* signal a raise of the softirq. If there are already -* entries there, someone already raised the irq but it -* hasn't run yet. -*/ -if (list->next == &req->ipi_list) -raise_softirq_irqoff (BLOCK_SOFTIRQ ); +blk_mq_trigger_softirq (req); }else if (raise_blk_irq (ccpu, req)) goto do_local;
Signature · 1 line
2.26.2
Re: [PATCH 02/12] blk-mq: factor out a helper to reise the block softirq
Quoted 13 lines · Christoph Hellwig
/*
* Setup and invoke a run of 'trigger_softirq' on the given cpu. */block/blk-mq.c +0
-5@@ -681,19 +689,8 @@ static void __blk_complete_request(struct request *req)* avoids IPI sending from current CPU to the first CPU of a group. */ if (ccpu == cpu || shared) { -struct list_head *list; do_local: - list =this_cpu_ptr (&blk_cpu_done); -list_add_tail (&req->ipi_list, list); - -/*
block/blk-mq.c static void __blk_complete_request(struct request *req) · line 695
Quoted 10 lines
block/blk-mq.c +1
-7-* if the list only contains our just added request, -* signal a raise of the softirq. If there are already -* entries there, someone already raised the irq but it -* hasn't run yet. -*/ -if (list->next == &req->ipi_list) -raise_softirq_irqoff (BLOCK_SOFTIRQ ); +blk_mq_trigger_softirq (req); }else if (raise_blk_irq (ccpu, req)) goto do_local;
Couldn't this be folded into the previous condition, e.g
if (ccpu == cpu || shared || raised_blk_irq(ccpu, req))
?
Re: [PATCH 02/12] blk-mq: factor out a helper to reise the block softirq
Quoted 3 lines · Daniel Wagner
Couldn't this be folded into the previous condition, e.g
if (ccpu == cpu || shared || raised_blk_irq(ccpu, req))
to answer my own question, patch #3 does addresses this :)
Re: [PATCH 02/12] blk-mq: factor out a helper to reise the block softirq
In reply to Christoph HellwigQuoted 3 lines · Christoph Hellwig
Add a helper to deduplicate the logic that raises the block softirq.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 3/12 blk-mq: remove raise_blk_irq
In reply to Christoph HellwigBy open coding raise_blk_irq in the only caller, and replacing the ifdef CONFIG_SMP with an IS_ENABLED check the flow in the caller can be significantly simplified.
--- block/blk-mq.c | 40 ++++++++++------------------------------ 1 file changed, 10 insertions(+), 30 deletions(-)
block/blk-mq.c +10 -30
@@ -613,37 +613,11 @@ static void blk_mq_trigger_softirq(struct request *rq)raise_softirq_irqoff (BLOCK_SOFTIRQ ); } -#ifdef CONFIG_SMP static void trigger_softirq (void *data) {blk_mq_trigger_softirq (data); } -/* -* Setup and invoke a run of 'trigger_softirq' on the given cpu. -*/ -static int raise_blk_irq (int cpu,struct request *rq) -{ -if (cpu_online (cpu)) { -call_single_data_t *data = &rq->csd; - - data->func = trigger_softirq; - data->info = rq; - data->flags =0 ; - -smp_call_function_single_async (cpu, data); -return 0 ; - } - -return 1 ; -} -#else /* CONFIG_SMP */ -static int raise_blk_irq (int cpu,struct request *rq) -{ -return 1 ; -} -#endif -static int blk_softirq_cpu_dead (unsigned int cpu) {/*
@@ -688,11 +662,17 @@ static void __blk_complete_request(struct request *req)* support multiple interrupts, so current CPU is unique actually. This * avoids IPI sending from current CPU to the first CPU of a group. */ -if (ccpu == cpu || shared) { -do_local: +if (IS_ENABLED (CONFIG_SMP ) && + ccpu != cpu && !shared &&cpu_online (ccpu)) { +call_single_data_t *data = &req->csd; + + data->func = trigger_softirq; + data->info = req; + data->flags =0 ; +smp_call_function_single_async (cpu, data); + }else {blk_mq_trigger_softirq (req); - }else if (raise_blk_irq (ccpu, req)) - goto do_local; + }local_irq_restore (flags); }
Signature · 1 line
2.26.2
Re: [PATCH 03/12] blk-mq: remove raise_blk_irq
Quoted 5 lines · Christoph Hellwig
By open coding
raise_blk_irqin the only caller, and replacing the ifdefCONFIG_SMPwith anIS_ENABLEDcheck the flow in the caller can be significantly simplified.Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 4/12 blk-mq: complete polled requests directly
In reply to Christoph HellwigEven for single queue devices there is no point in offloading a polled completion to the softirq, given that blk_mq_force_complete_rq is called from the polling thread in that case and thus there are no starvation issues.
--- block/blk-mq.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
block/blk-mq.c +11 -6
@@ -703,6 +703,16 @@ void blk_mq_force_complete_rq(struct request *rq)int cpu;WRITE_ONCE (rq->state,MQ_RQ_COMPLETE ); + +/* +* For a polled request, always complete locallly, it's pointless +* to redirect the completion. +*/ +if (rq->cmd_flags &REQ_HIPRI ) { + q->mq_ops->complete (rq); +return ; + } +/* * Most of single queue controllers, there is only one irq vector * for handling IO completion, and the only irq's affinity is set
@@ -717,12 +727,7 @@ void blk_mq_force_complete_rq(struct request *rq)return ; } -/* -* For a polled request, always complete locallly, it's pointless -* to redirect the completion. -*/ -if ((rq->cmd_flags &REQ_HIPRI ) || - !test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags)) { +if (!test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags)) { q->mq_ops->complete (rq);return ; }
Signature · 1 line
2.26.2
Re: [PATCH 04/12] blk-mq: complete polled requests directly
Quoted 6 lines · Christoph Hellwig
Even for single queue devices there is no point in offloading a polled completion to the softirq, given that
blk_mq_force_complete_rqis called from the polling thread in that case and thus there are no starvation issues.Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 5/12 blk-mq: short cut the IPI path in blk_mq_force_complete_rq for !SMP
In reply to Christoph HellwigLet the compile optimize out the entire IPI path, given that we are obviously not going to use it.
--- block/blk-mq.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-mq.c +2 -1
@@ -727,7 +727,8 @@ void blk_mq_force_complete_rq(struct request *rq)return ; } -if (!test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags)) { +if (!IS_ENABLED (CONFIG_SMP ) || + !test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags)) { q->mq_ops->complete (rq);return ; }
Signature · 1 line
2.26.2
Re: [PATCH 05/12] blk-mq: short cut the IPI path in blk_mq_force_complete_rq for !SMP
Quoted 4 lines · Christoph Hellwig
Let the compile optimize out the entire IPI path, given that we are obviously not going to use it.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 6/12 blk-mq: merge the softirq vs non-softirq IPI logic
In reply to Christoph HellwigBoth the softirq path for single queue devices and the multi-queue completion handler share the same logic to figure out if we need an IPI for the completion and eventually issue it. Merge the two versions into a single unified code path.
--- block/blk-mq.c | 85 ++++++++++++-------------------------------------- 1 file changed, 20 insertions(+), 65 deletions(-)
block/blk-mq.c +20 -65
@@ -600,8 +600,11 @@ static __latent_entropy void blk_done_softirq(struct softirq_action *h)static void blk_mq_trigger_softirq (struct request *rq) { -struct list_head *list =this_cpu_ptr (&blk_cpu_done); +struct list_head *list; +unsigned long flags; +local_irq_save (flags); + list =this_cpu_ptr (&blk_cpu_done);list_add_tail (&rq->ipi_list, list);/*
@@ -611,11 +614,7 @@ static void blk_mq_trigger_softirq(struct request *rq)*/ if (list->next == &rq->ipi_list)raise_softirq_irqoff (BLOCK_SOFTIRQ ); -} - -static void trigger_softirq (void *data) -{ -blk_mq_trigger_softirq (data); +local_irq_restore (flags); }static int blk_softirq_cpu_dead (unsigned int cpu)
@@ -633,56 +632,26 @@ static int blk_softirq_cpu_dead(unsigned int cpu)return 0 ; } -static void __blk_complete_request (struct request *req) +static void __blk_mq_complete_request (struct request *rq) { -struct request_queue *q = req->q; -int cpu, ccpu = req->mq_ctx->cpu; -unsigned long flags; -bool shared = false; - -BUG_ON (!q->mq_ops->complete); - -local_irq_save (flags); - cpu =smp_processor_id (); - -/* -* Select completion CPU -*/ -if (test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags) && ccpu !=-1 ) { -if (!test_bit (QUEUE_FLAG_SAME_FORCE , &q->queue_flags)) - shared =cpus_share_cache (cpu, ccpu); - }else - ccpu = cpu; -/* -* If current CPU and requested CPU share a cache, run the softirq on -* the current CPU. One might concern this is just like -* QUEUE_FLAG_SAME_FORCE, but actually not. blk_complete_request() is -* running in interrupt handler, and currently I/O controller doesn't -* support multiple interrupts, so current CPU is unique actually. This -* avoids IPI sending from current CPU to the first CPU of a group. +* For most of single queue controllers, there is only one irq vector +* for handling I/O completion, and the only irq's affinity is set +* to all possible CPUs. On most of ARCHs, this affinity means the irq +* is handled on one specific CPU. +* +* So complete I/O requests in softirq context in case of single queue +* devices to avoid degrading I/O performance due to irqsoff latency. */ -if (IS_ENABLED (CONFIG_SMP ) && - ccpu != cpu && !shared &&cpu_online (ccpu)) { -call_single_data_t *data = &req->csd; - - data->func = trigger_softirq; - data->info = req; - data->flags =0 ; -smp_call_function_single_async (cpu, data); - }else { -blk_mq_trigger_softirq (req); - } - -local_irq_restore (flags); +if (rq->q->nr_hw_queues ==1 ) +blk_mq_trigger_softirq (rq); +else + rq->q->mq_ops->complete (rq); }static void __blk_mq_complete_request_remote (void *data) { -struct request *rq = data; -struct request_queue *q = rq->q; - - q->mq_ops->complete (rq); +__blk_mq_complete_request (data); }/**
@@ -713,23 +682,9 @@ void blk_mq_force_complete_rq(struct request *rq)return ; } -/* -* Most of single queue controllers, there is only one irq vector -* for handling IO completion, and the only irq's affinity is set -* as all possible CPUs. On most of ARCHs, this affinity means the -* irq is handled on one specific CPU. -* -* So complete IO reqeust in softirq context in case of single queue -* for not degrading IO performance by irqsoff latency. -*/ -if (q->nr_hw_queues ==1 ) { -__blk_complete_request (rq); -return ; - } -if (!IS_ENABLED (CONFIG_SMP ) || !test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags)) { - q->mq_ops->complete (rq); +__blk_mq_complete_request (rq);return ; }
@@ -743,7 +698,7 @@ void blk_mq_force_complete_rq(struct request *rq) rq->csd.flags =0 ;smp_call_function_single_async (ctx->cpu, &rq->csd); }else { - q->mq_ops->complete (rq); +__blk_mq_complete_request (rq); }put_cpu (); }
Signature · 1 line
2.26.2
Re: [PATCH 06/12] blk-mq: merge the softirq vs non-softirq IPI logic
Quoted 6 lines · Christoph Hellwig
Both the softirq path for single queue devices and the multi-queue completion handler share the same logic to figure out if we need an IPI for the completion and eventually issue it. Merge the two versions into a single unified code path.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 7/12 blk-mq: move failure injection out of blk_mq_complete_request
In reply to Christoph HellwigMove the call to blk_should_fake_timeout out of blk_mq_complete_request and into the drivers, skipping call sites that are obvious error handlers, and remove the now superflous blk_mq_force_complete_rq helper. This ensures we don't keep injecting errors into completions that just terminate the Linux request after the hardware has been reset or the command has been aborted.
--- block/blk-mq.c | 34 +++++++------------------------ block/blk-timeout.c | 6 ++---- block/blk.h | 9 -------- block/bsg-lib.c | 5 ++++- drivers/block/loop.c | 6 ++++-- drivers/block/mtip32xx/mtip32xx.c | 3 ++- drivers/block/nbd.c | 5 ++++- drivers/block/null_blk_main.c | 5 +++-- drivers/block/skd_main.c | 9 +++++--- drivers/block/virtio_blk.c | 3 ++- drivers/block/xen-blkfront.c | 3 ++- drivers/md/dm-rq.c | 3 ++- drivers/mmc/core/block.c | 8 ++++---- drivers/nvme/host/core.c | 2 +- drivers/nvme/host/nvme.h | 3 ++- drivers/s390/block/dasd.c | 2 +- drivers/s390/block/scm_blk.c | 3 ++- drivers/scsi/scsi_lib.c | 12 +++-------- include/linux/blk-mq.h | 12 +++++++++-- 19 files changed, 61 insertions(+), 72 deletions(-)
block/blk-mq.c +7 -27
@@ -655,16 +655,13 @@ static void __blk_mq_complete_request_remote(void *data) }/** -* blk_mq_force_complete_rq() - Force complete the request, bypassing any error -* injection that could drop the completion. -* @rq: Request to be force completed +* blk_mq_complete_request - end I/O on a request +* @rq: the request being processed * -* Drivers should use blk_mq_complete_request() to complete requests in their -* normal IO path. For timeout error recovery, drivers may call this forced -* completion routine after they've reclaimed timed out requests to bypass -* potentially subsequent fake timeouts. -*/ -void blk_mq_force_complete_rq (struct request *rq) +* Description: +* Complete a request by scheduling the ->complete_rq operation. +**/ +void blk_mq_complete_request (struct request *rq) {struct blk_mq_ctx *ctx = rq->mq_ctx;struct request_queue *q = rq->q;
@@ -702,7 +699,7 @@ void blk_mq_force_complete_rq(struct request *rq) }put_cpu (); } -EXPORT_SYMBOL_GPL (blk_mq_force_complete_rq); +EXPORT_SYMBOL (blk_mq_complete_request); static void hctx_unlock(struct blk_mq_hw_ctx *hctx, int srcu_idx) __releases(hctx->srcu)
@@ -724,23 +721,6 @@ static void hctx_lock(struct blk_mq_hw_ctx *hctx, int *srcu_idx) *srcu_idx =srcu_read_lock (hctx->srcu); } -/** -* blk_mq_complete_request - end I/O on a request -* @rq: the request being processed -* -* Description: -* Ends all I/O on a request. It does not handle partial completions. -* The actual completion happens out-of-order, through a IPI handler. -**/ -bool blk_mq_complete_request (struct request *rq) -{ -if (unlikely (blk_should_fake_timeout (rq->q))) -return false; -blk_mq_force_complete_rq (rq); -return true; -} -EXPORT_SYMBOL (blk_mq_complete_request); -/** * blk_mq_start_request - Start processing a request * @rq: Pointer to request to be started
block/blk-timeout.c +2 -4
@@ -20,13 +20,11 @@ static int __init setup_fail_io_timeout(char *str) }__setup ("fail_io_timeout=" , setup_fail_io_timeout); -int blk_should_fake_timeout (struct request_queue *q) +bool __blk_should_fake_timeout (struct request_queue *q) { -if (!test_bit (QUEUE_FLAG_FAIL_IO , &q->queue_flags)) -return 0 ; -return should_fail (&fail_io_timeout,1 ); } +EXPORT_SYMBOL_GPL (__blk_should_fake_timeout);static int __init fail_io_timeout_debugfs (void ) {
block/blk.h +0 -9
@@ -223,18 +223,9 @@ ssize_t part_fail_show(struct device *dev, struct device_attribute *attr, char *buf);ssize_t part_fail_store (struct device *dev,struct device_attribute *attr,const char *buf,size_t count); - -#ifdef CONFIG_FAIL_IO_TIMEOUT -int blk_should_fake_timeout (struct request_queue *);ssize_t part_timeout_show (struct device *,struct device_attribute *,char *);ssize_t part_timeout_store (struct device *,struct device_attribute *,const char *,size_t ); -#else -static inline int blk_should_fake_timeout (struct request_queue *q) -{ -return 0 ; -} -#endif void __blk_queue_split (struct request_queue *q,struct bio **bio,unsigned int *nr_segs);
block/bsg-lib.c +4 -1
@@ -181,9 +181,12 @@ EXPORT_SYMBOL_GPL(bsg_job_get);void bsg_job_done (struct bsg_job *job,int result,unsigned int reply_payload_rcv_len) { +struct request *rq =blk_mq_rq_from_pdu (job); + job->result = result; job->reply_payload_rcv_len = reply_payload_rcv_len; -blk_mq_complete_request (blk_mq_rq_from_pdu (job)); +if (likely (!blk_should_fake_timeout (rq->q))) +blk_mq_complete_request (rq); }EXPORT_SYMBOL_GPL (bsg_job_done);
drivers/block/loop.c +4 -2
@@ -509,7 +509,8 @@ static void lo_rw_aio_do_completion(struct loop_cmd *cmd)return ;kfree (cmd->bvec); cmd->bvec =NULL ; -blk_mq_complete_request (rq); +if (likely (!blk_should_fake_timeout (rq->q))) +blk_mq_complete_request (rq); }static void lo_rw_aio_complete (struct kiocb *iocb,long ret,long ret2)
@@ -2048,7 +2049,8 @@ static void loop_handle_cmd(struct loop_cmd *cmd) cmd->ret = ret; else cmd->ret = ret ? -EIO : 0; -blk_mq_complete_request (rq); +if (likely (!blk_should_fake_timeout (rq->q))) +blk_mq_complete_request (rq); } }
drivers/block/mtip32xx/mtip32xx.c +2 -1
@@ -492,7 +492,8 @@ static void mtip_complete_command(struct mtip_cmd *cmd, blk_status_t status)struct request *req =blk_mq_rq_from_pdu (cmd); cmd->status = status; -blk_mq_complete_request (req); +if (likely (!blk_should_fake_timeout (req->q))) +blk_mq_complete_request (req); }/*
drivers/block/nbd.c +4 -1
@@ -784,6 +784,7 @@ static void recv_work(struct work_struct *work)struct nbd_device *nbd = args->nbd;struct nbd_config *config = nbd->config;struct nbd_cmd *cmd; +struct request *rq;while (1 ) { cmd =nbd_read_stat (nbd, args->index);
@@ -796,7 +797,9 @@ static void recv_work(struct work_struct *work)break ; } -blk_mq_complete_request (blk_mq_rq_from_pdu (cmd)); + rq =blk_mq_rq_from_pdu (cmd); +if (likely (!blk_should_fake_timeout (rq->q))) +blk_mq_complete_request (rq); }atomic_dec (&config->recv_threads);wake_up (&config->recv_wq);
drivers/block/null_blk_main.c +3 -2
@@ -1283,7 +1283,8 @@ static inline void nullb_complete_cmd(struct nullb_cmd *cmd)case NULL_IRQ_SOFTIRQ :switch (cmd->nq->dev->queue_mode) {case NULL_Q_MQ : -blk_mq_complete_request (cmd->rq); +if (likely (!blk_should_fake_timeout (cmd->rq->q))) +blk_mq_complete_request (cmd->rq);break ;case NULL_Q_BIO :/*
@@ -1423,7 +1424,7 @@ static bool should_requeue_request(struct request *rq)static enum blk_eh_timer_return null_timeout_rq (struct request *rq,bool res) {pr_info ("rq %p timed out\n" , rq); -blk_mq_force_complete_rq (rq); +blk_mq_complete_request (rq);return BLK_EH_DONE ; }
drivers/block/skd_main.c +6 -3
@@ -1417,7 +1417,8 @@ static void skd_resolve_req_exception(struct skd_device *skdev,case SKD_CHECK_STATUS_REPORT_GOOD :case SKD_CHECK_STATUS_REPORT_SMART_ALERT : skreq->status =BLK_STS_OK ; -blk_mq_complete_request (req); +if (likely (!blk_should_fake_timeout (req->q))) +blk_mq_complete_request (req);break ;case SKD_CHECK_STATUS_BUSY_IMMINENT :
@@ -1440,7 +1441,8 @@ static void skd_resolve_req_exception(struct skd_device *skdev,case SKD_CHECK_STATUS_REPORT_ERROR :default : skreq->status =BLK_STS_IOERR ; -blk_mq_complete_request (req); +if (likely (!blk_should_fake_timeout (req->q))) +blk_mq_complete_request (req);break ; } }
@@ -1560,7 +1562,8 @@ static int skd_isr_completion_posted(struct skd_device *skdev,*/ if (likely (cmp_status ==SAM_STAT_GOOD )) { skreq->status =BLK_STS_OK ; -blk_mq_complete_request (rq); +if (likely (!blk_should_fake_timeout (rq->q))) +blk_mq_complete_request (rq); }else {skd_resolve_req_exception (skdev, skreq, rq); }
drivers/block/virtio_blk.c +2 -1
@@ -171,7 +171,8 @@ static void virtblk_done(struct virtqueue *vq)while ((vbr =virtqueue_get_buf (vblk->vqs[qid].vq, &len)) !=NULL ) {struct request *req =blk_mq_rq_from_pdu (vbr); -blk_mq_complete_request (req); +if (likely (!blk_should_fake_timeout (req->q))) +blk_mq_complete_request (req); req_done = true; }if (unlikely (virtqueue_is_broken (vq)))
drivers/block/xen-blkfront.c +2 -1
@@ -1655,7 +1655,8 @@ static irqreturn_t blkif_interrupt(int irq, void *dev_id)BUG (); } -blk_mq_complete_request (req); +if (likely (!blk_should_fake_timeout (req->q))) +blk_mq_complete_request (req); } rinfo->ring.rsp_cons = i;
drivers/md/dm-rq.c +2 -1
@@ -288,7 +288,8 @@ static void dm_complete_request(struct request *rq, blk_status_t error)struct dm_rq_target_io *tio =tio_from_request (rq); tio->error = error; -blk_mq_complete_request (rq); +if (likely (!blk_should_fake_timeout (rq->q))) +blk_mq_complete_request (rq); }/*
drivers/mmc/core/block.c +4 -4
@@ -1446,7 +1446,7 @@ static void mmc_blk_cqe_req_done(struct mmc_request *mrq)*/ if (mq->in_recovery)mmc_blk_cqe_complete_rq (mq, req); -else +else if (likely (!blk_should_fake_timeout (req->q)))blk_mq_complete_request (req); }
@@ -1926,7 +1926,7 @@ static void mmc_blk_hsq_req_done(struct mmc_request *mrq)*/ if (mq->in_recovery)mmc_blk_cqe_complete_rq (mq, req); -else +else if (likely (!blk_should_fake_timeout (req->q)))blk_mq_complete_request (req); }
@@ -1936,7 +1936,7 @@ void mmc_blk_mq_complete(struct request *req)if (mq->use_cqe)mmc_blk_cqe_complete_rq (mq, req); -else +else if (likely (!blk_should_fake_timeout (req->q)))mmc_blk_mq_complete_rq (mq, req); }
@@ -1988,7 +1988,7 @@ static void mmc_blk_mq_post_req(struct mmc_queue *mq, struct request *req)*/ if (mq->in_recovery)mmc_blk_mq_complete_rq (mq, req); -else +else if (likely (!blk_should_fake_timeout (req->q)))blk_mq_complete_request (req);mmc_blk_mq_dec_in_flight (mq, req);
drivers/nvme/host/core.c +1 -1
@@ -304,7 +304,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)return true;nvme_req (req)->status =NVME_SC_HOST_ABORTED_CMD ; -blk_mq_force_complete_rq (req); +blk_mq_complete_request (req);return true; }EXPORT_SYMBOL_GPL (nvme_cancel_request);
drivers/nvme/host/nvme.h +2 -1
@@ -481,7 +481,8 @@ static inline void nvme_end_request(struct request *req, __le16 status, rq->result = result;/* inject error when permitted by fault injection framework */ nvme_should_fail (req); -blk_mq_complete_request (req); +if (likely (!blk_should_fake_timeout (req->q))) +blk_mq_complete_request (req); }static inline void nvme_get_ctrl (struct nvme_ctrl *ctrl)
drivers/s390/block/dasd.c +1 -1
@@ -2802,7 +2802,7 @@ static void __dasd_cleanup_cqr(struct dasd_ccw_req *cqr)blk_update_request (req,BLK_STS_OK ,blk_rq_bytes (req) - proc_bytes);blk_mq_requeue_request (req, true); - }else { + }else if (likely (!blk_should_fake_timeout (req->q))) {blk_mq_complete_request (req); } }
drivers/s390/block/scm_blk.c +2 -1
@@ -256,7 +256,8 @@ static void scm_request_finish(struct scm_request *scmrq)for (i =0 ; i < nr_requests_per_io && scmrq->request[i]; i++) { error =blk_mq_rq_to_pdu (scmrq->request[i]); *error = scmrq->error; -blk_mq_complete_request (scmrq->request[i]); +if (likely (!blk_should_fake_timeout (scmrq->request[i]->q))) +blk_mq_complete_request (scmrq->request[i]); }atomic_dec (&bdev->queued_reqs);
drivers/scsi/scsi_lib.c +3 -9
@@ -1589,18 +1589,12 @@ static blk_status_t scsi_mq_prep_fn(struct request *req)static void scsi_mq_done (struct scsi_cmnd *cmd) { +if (unlikely (blk_should_fake_timeout (cmd->request->q))) +return ;if (unlikely (test_and_set_bit (SCMD_STATE_COMPLETE , &cmd->state)))return ;trace_scsi_dispatch_cmd_done (cmd); - -/* -* If the block layer didn't complete the request due to a timeout -* injection, scsi must clear its internal completed state so that the -* timeout handler will see it needs to escalate its own error -* recovery. -*/ -if (unlikely (!blk_mq_complete_request (cmd->request))) -clear_bit (SCMD_STATE_COMPLETE , &cmd->state); +blk_mq_complete_request (cmd->request); }static void scsi_mq_put_budget (struct blk_mq_hw_ctx *hctx)
include/linux/blk-mq.h +10 -2
@@ -503,8 +503,7 @@ void __blk_mq_end_request(struct request *rq, blk_status_t error);void blk_mq_requeue_request (struct request *rq,bool kick_requeue_list);void blk_mq_kick_requeue_list (struct request_queue *q);void blk_mq_delay_kick_requeue_list (struct request_queue *q,unsigned long msecs); -bool blk_mq_complete_request (struct request *rq); -void blk_mq_force_complete_rq (struct request *rq); +void blk_mq_complete_request (struct request *rq);bool blk_mq_bio_list_merge (struct request_queue *q,struct list_head *list,struct bio *bio,unsigned int nr_segs);bool blk_mq_queue_stopped (struct request_queue *q);
@@ -537,6 +536,15 @@ void blk_mq_quiesce_queue_nowait(struct request_queue *q);unsigned int blk_mq_rq_cpu (struct request *rq); +bool __blk_should_fake_timeout (struct request_queue *q); +static inline bool blk_should_fake_timeout (struct request_queue *q) +{ +if (IS_ENABLED (CONFIG_FAIL_IO_TIMEOUT ) && +test_bit (QUEUE_FLAG_FAIL_IO , &q->queue_flags)) +return __blk_should_fake_timeout (q); +return false; +} +/** * blk_mq_rq_from_pdu - cast a PDU to a request * @pdu: the PDU (Protocol Data Unit) to be casted
Signature · 1 line
2.26.2
Re: [PATCH 07/12] blk-mq: move failure injection out of blk_mq_complete_request
Quoted 8 lines · Christoph Hellwig
Move the call to
blk_should_fake_timeoutout ofblk_mq_complete_requestand into the drivers, skipping call sites that are obvious error handlers, and remove the now superflousblk_mq_force_complete_rqhelper. This ensures we don't keep injecting errors into completions that just terminate the Linux request after the hardware has been reset or the command has been aborted.Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 8/12 blk-mq: remove the get_cpu/put_cpu pair in blk_mq_complete_request
In reply to Christoph HellwigWe don't really care if we get migrated during the I/O completion. In the worth case we either perform an IPI that wasn't required, or complete the request on a CPU which we just migrated off.
--- block/blk-mq.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
block/blk-mq.c +1 -2
@@ -685,7 +685,7 @@ void blk_mq_complete_request(struct request *rq)return ; } - cpu =get_cpu (); + cpu =raw_smp_processor_id ();if (!test_bit (QUEUE_FLAG_SAME_FORCE , &q->queue_flags)) shared =cpus_share_cache (cpu, ctx->cpu);
@@ -697,7 +697,6 @@ void blk_mq_complete_request(struct request *rq) }else {__blk_mq_complete_request (rq); } -put_cpu (); }EXPORT_SYMBOL (blk_mq_complete_request);
Signature · 1 line
2.26.2
Re: [PATCH 08/12] blk-mq: remove the get_cpu/put_cpu pair in blk_mq_complete_request
Quoted 5 lines · Christoph Hellwig
We don't really care if we get migrated during the I/O completion. In the worth case we either perform an IPI that wasn't required, or complete the request on a CPU which we just migrated off.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 9/12 blk-mq: factor out a blk_mq_complete_need_ipi helper
In reply to Christoph HellwigAdd a helper to decide if we can complete locally or need an IPI.
--- block/blk-mq.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)
block/blk-mq.c +21 -18
@@ -654,6 +654,24 @@ static void __blk_mq_complete_request_remote(void *data)__blk_mq_complete_request (data); } +static inline bool blk_mq_complete_need_ipi (struct request *rq) +{ +int cpu =raw_smp_processor_id (); + +if (!IS_ENABLED (CONFIG_SMP ) || + !test_bit (QUEUE_FLAG_SAME_COMP , &rq->q->queue_flags)) +return false; + +/* same CPU or cache domain? Complete locally */ +if (cpu == rq->mq_ctx->cpu || + (!test_bit (QUEUE_FLAG_SAME_FORCE , &rq->q->queue_flags) && +cpus_share_cache (cpu, rq->mq_ctx->cpu))) +return false; + +/* don't try to IPI to an offline CPU */ +return cpu_online (rq->mq_ctx->cpu); +} +/** * blk_mq_complete_request - end I/O on a request * @rq: the request being processed
@@ -663,11 +681,6 @@ static void __blk_mq_complete_request_remote(void *data)**/ void blk_mq_complete_request (struct request *rq) { -struct blk_mq_ctx *ctx = rq->mq_ctx; -struct request_queue *q = rq->q; -bool shared = false; -int cpu; -WRITE_ONCE (rq->state,MQ_RQ_COMPLETE );/*
@@ -675,25 +688,15 @@ void blk_mq_complete_request(struct request *rq)* to redirect the completion. */ if (rq->cmd_flags &REQ_HIPRI ) { - q->mq_ops->complete (rq); -return ; - } - -if (!IS_ENABLED (CONFIG_SMP ) || - !test_bit (QUEUE_FLAG_SAME_COMP , &q->queue_flags)) { -__blk_mq_complete_request (rq); + rq->q->mq_ops->complete (rq);return ; } - cpu =raw_smp_processor_id (); -if (!test_bit (QUEUE_FLAG_SAME_FORCE , &q->queue_flags)) - shared =cpus_share_cache (cpu, ctx->cpu); - -if (cpu != ctx->cpu && !shared &&cpu_online (ctx->cpu)) { +if (blk_mq_complete_need_ipi (rq)) { rq->csd.func = __blk_mq_complete_request_remote; rq->csd.info = rq; rq->csd.flags =0 ; -smp_call_function_single_async (ctx->cpu, &rq->csd); +smp_call_function_single_async (rq->mq_ctx->cpu, &rq->csd); }else {__blk_mq_complete_request (rq); }
Signature · 1 line
2.26.2
Re: [PATCH 09/12] blk-mq: factor out a blk_mq_complete_need_ipi helper
Quoted 3 lines · Christoph Hellwig
Add a helper to decide if we can complete locally or need an IPI.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 10/12 blk-mq: add a new blk_mq_complete_request_remote API
In reply to Christoph HellwigThis is a variant of blk_mq_complete_request_remote that only completes the request if it needs to be bounced to another CPU or a softirq. If the request can be completed locally the function returns false and lets the driver complete it without requring and indirect function call.
Suggestions for a better name welcome.
--- block/blk-mq.c | 45 ++++++++++++++++++++++++------------------ include/linux/blk-mq.h | 1 + 2 files changed, 27 insertions(+), 19 deletions(-)
block/blk-mq.c +26 -19
@@ -632,8 +632,11 @@ static int blk_softirq_cpu_dead(unsigned int cpu)return 0 ; } -static void __blk_mq_complete_request (struct request *rq) + +static void __blk_mq_complete_request_remote (void *data) { +struct request *rq = data; +/* * For most of single queue controllers, there is only one irq vector * for handling I/O completion, and the only irq's affinity is set
@@ -649,11 +652,6 @@ static void __blk_mq_complete_request(struct request *rq) rq->q->mq_ops->complete (rq); } -static void __blk_mq_complete_request_remote (void *data) -{ -__blk_mq_complete_request (data); -} -static inline bool blk_mq_complete_need_ipi (struct request *rq) {int cpu =raw_smp_processor_id ();
@@ -672,14 +670,7 @@ static inline bool blk_mq_complete_need_ipi(struct request *rq)return cpu_online (rq->mq_ctx->cpu); } -/** -* blk_mq_complete_request - end I/O on a request -* @rq: the request being processed -* -* Description: -* Complete a request by scheduling the ->complete_rq operation. -**/ -void blk_mq_complete_request (struct request *rq) +bool blk_mq_complete_request_remote (struct request *rq) {WRITE_ONCE (rq->state,MQ_RQ_COMPLETE );
@@ -687,10 +678,8 @@ void blk_mq_complete_request(struct request *rq)* For a polled request, always complete locallly, it's pointless * to redirect the completion. */ -if (rq->cmd_flags &REQ_HIPRI ) { - rq->q->mq_ops->complete (rq); -return ; - } +if (rq->cmd_flags &REQ_HIPRI ) +return false;if (blk_mq_complete_need_ipi (rq)) { rq->csd.func = __blk_mq_complete_request_remote;
@@ -698,8 +687,26 @@ void blk_mq_complete_request(struct request *rq) rq->csd.flags =0 ;smp_call_function_single_async (rq->mq_ctx->cpu, &rq->csd); }else { -__blk_mq_complete_request (rq); +if (rq->q->nr_hw_queues >1 ) +return false; +blk_mq_trigger_softirq (rq); } + +return true; +} +EXPORT_SYMBOL_GPL (blk_mq_complete_request_remote); + +/** +* blk_mq_complete_request - end I/O on a request +* @rq: the request being processed +* +* Description: +* Complete a request by scheduling the ->complete_rq operation. +**/ +void blk_mq_complete_request (struct request *rq) +{ +if (!blk_mq_complete_request_remote (rq)) + rq->q->mq_ops->complete (rq); }EXPORT_SYMBOL (blk_mq_complete_request);
include/linux/blk-mq.h +1 -0
@@ -504,6 +504,7 @@ void blk_mq_requeue_request(struct request *rq, bool kick_requeue_list);void blk_mq_kick_requeue_list (struct request_queue *q);void blk_mq_delay_kick_requeue_list (struct request_queue *q,unsigned long msecs);void blk_mq_complete_request (struct request *rq); +bool blk_mq_complete_request_remote (struct request *rq);bool blk_mq_bio_list_merge (struct request_queue *q,struct list_head *list,struct bio *bio,unsigned int nr_segs);bool blk_mq_queue_stopped (struct request_queue *q);
Signature · 1 line
2.26.2
Re: [PATCH 10/12] blk-mq: add a new blk_mq_complete_request_remote API
Quoted 4 lines · Christoph Hellwig
This is a variant of
blk_mq_complete_request_remotethat only completes the request if it needs to be bounced to another CPU or a softirq. If the request can be completed locally the function returns false and lets the driver complete it without requring and indirect function call.
s/requring/requiring/
Quoted 3 lines
Suggestions for a better name welcome.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 11/12 nvme-rdma: factor out a nvme_rdma_end_request helper
In reply to Christoph HellwigFactor a small sniplet of duplicated code into a new helper in preparation for making this sniplet a little bit less trivial.
--- drivers/nvme/host/rdma.c | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-)
drivers/nvme/host/rdma.c +17 -19
@@ -1149,6 +1149,15 @@ static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl)queue_work (nvme_reset_wq, &ctrl->err_work); } +static void nvme_rdma_end_request (struct nvme_rdma_request *req) +{ +struct request *rq =blk_mq_rq_from_pdu (req); + +if (!refcount_dec_and_test (&req->ref)) +return ; +nvme_end_request (rq, req->status, req->result); +} +static void nvme_rdma_wr_error (struct ib_cq *cq,struct ib_wc *wc,const char *op) {
@@ -1173,16 +1182,11 @@ static void nvme_rdma_inv_rkey_done(struct ib_cq *cq, struct ib_wc *wc) {struct nvme_rdma_request *req = container_of(wc->wr_cqe, struct nvme_rdma_request, reg_cqe); -struct request *rq =blk_mq_rq_from_pdu (req); -if (unlikely (wc->status !=IB_WC_SUCCESS )) { +if (unlikely (wc->status !=IB_WC_SUCCESS ))nvme_rdma_wr_error (cq, wc,"LOCAL_INV" ); -return ; - } - -if (refcount_dec_and_test (&req->ref)) -nvme_end_request (rq, req->status, req->result); - +else +nvme_rdma_end_request (req); } static int nvme_rdma_inv_rkey(struct nvme_rdma_queue *queue,
@@ -1547,15 +1551,11 @@ static void nvme_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc) container_of(wc->wr_cqe, struct nvme_rdma_qe, cqe);struct nvme_rdma_request *req = container_of(qe, struct nvme_rdma_request, sqe); -struct request *rq =blk_mq_rq_from_pdu (req); -if (unlikely (wc->status !=IB_WC_SUCCESS )) { +if (unlikely (wc->status !=IB_WC_SUCCESS ))nvme_rdma_wr_error (cq, wc,"SEND" ); -return ; - } - -if (refcount_dec_and_test (&req->ref)) -nvme_end_request (rq, req->status, req->result); +else +nvme_rdma_end_request (req); } static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
@@ -1694,11 +1694,9 @@ static void nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue,nvme_rdma_error_recovery (queue->ctrl); }/* the local invalidation completion will end the request */ -return ; + }else { +nvme_rdma_end_request (req); } - -if (refcount_dec_and_test (&req->ref)) -nvme_end_request (rq, req->status, req->result); }static void nvme_rdma_recv_done (struct ib_cq *cq,struct ib_wc *wc)
Signature · 1 line
2.26.2
Re: [PATCH 11/12] nvme-rdma: factor out a nvme_rdma_end_request helper
Quoted 4 lines · Christoph Hellwig
Factor a small sniplet of duplicated code into a new helper in preparation for making this sniplet a little bit less trivial.
Signed-off-by Christoph Hellwig <[email protected]>
Christoph Hellwig
maintainerPATCH 12/12 nvme: use blk_mq_complete_request_remote to avoid an indirect function call
In reply to Christoph HellwigUse the new blk_mq_complete_request_remote helper to avoid an indirect function call in the completion fast path.
--- drivers/nvme/host/fc.c | 4 +++- drivers/nvme/host/nvme.h | 7 ++++--- drivers/nvme/host/pci.c | 3 ++- drivers/nvme/host/rdma.c | 4 +++- drivers/nvme/host/tcp.c | 6 ++++-- drivers/nvme/target/loop.c | 3 ++- 6 files changed, 18 insertions(+), 9 deletions(-)
drivers/nvme/host/fc.c +3 -1
@@ -227,6 +227,7 @@ static DECLARE_COMPLETION(nvme_fc_unload_proceed);*/ static struct device *fc_udev_device; +static void nvme_fc_complete_rq (struct request *rq);/* *********************** FC-NVME Port Management ************************ */
@@ -2033,7 +2034,8 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) }__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate); -nvme_end_request (rq, status, result); +if (!nvme_end_request (rq, status, result)) +nvme_fc_complete_rq (rq); check_error:if (terminate_assoc)
drivers/nvme/host/nvme.h +4 -3
@@ -472,7 +472,7 @@ static inline u32 nvme_bytes_to_numd(size_t len)return (len >>2 ) -1 ; } -static inline void nvme_end_request (struct request *req,__le16 status, +static inline bool nvme_end_request (struct request *req,__le16 status,union nvme_result result) {struct nvme_request *rq =nvme_req (req);
@@ -481,8 +481,9 @@ static inline void nvme_end_request(struct request *req, __le16 status, rq->result = result;/* inject error when permitted by fault injection framework */ nvme_should_fail (req); -if (likely (!blk_should_fake_timeout (req->q))) -blk_mq_complete_request (req); +if (unlikely (blk_should_fake_timeout (req->q))) +return true; +return blk_mq_complete_request_remote (req); }static inline void nvme_get_ctrl (struct nvme_ctrl *ctrl)
drivers/nvme/host/pci.c +2 -1
@@ -963,7 +963,8 @@ static inline void nvme_handle_cqe(struct nvme_queue *nvmeq, u16 idx) req =blk_mq_tag_to_rq (nvme_queue_tagset (nvmeq), cqe->command_id);trace_nvme_sq (req, cqe->sq_head, nvmeq->sq_tail); -nvme_end_request (req, cqe->status, cqe->result); +if (!nvme_end_request (req, cqe->status, cqe->result)) +nvme_pci_complete_rq (req); }static inline void nvme_update_cq_head (struct nvme_queue *nvmeq)
drivers/nvme/host/rdma.c +3 -1
@@ -149,6 +149,7 @@ MODULE_PARM_DESC(register_always,static int nvme_rdma_cm_handler (struct rdma_cm_id *cm_id,struct rdma_cm_event *event);static void nvme_rdma_recv_done (struct ib_cq *cq,struct ib_wc *wc); +static void nvme_rdma_complete_rq (struct request *rq);static const struct blk_mq_ops nvme_rdma_mq_ops;static const struct blk_mq_ops nvme_rdma_admin_mq_ops;
@@ -1155,7 +1156,8 @@ static void nvme_rdma_end_request(struct nvme_rdma_request *req)if (!refcount_dec_and_test (&req->ref))return ; -nvme_end_request (rq, req->status, req->result); +if (!nvme_end_request (rq, req->status, req->result)) +nvme_rdma_complete_rq (rq); } static void nvme_rdma_wr_error(struct ib_cq *cq, struct ib_wc *wc,
drivers/nvme/host/tcp.c +4 -2
@@ -464,7 +464,8 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,return -EINVAL ; } -nvme_end_request (rq, cqe->status, cqe->result); +if (!nvme_end_request (rq, cqe->status, cqe->result)) +nvme_complete_rq (rq); queue->nr_cqe++;return 0 ;
@@ -654,7 +655,8 @@ static inline void nvme_tcp_end_request(struct request *rq, u16 status) {union nvme_result res = {}; -nvme_end_request (rq,cpu_to_le16 (status <<1 ), res); +if (!nvme_end_request (rq,cpu_to_le16 (status <<1 ), res)) +nvme_complete_rq (rq); } static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,
drivers/nvme/target/loop.c +2 -1
@@ -116,7 +116,8 @@ static void nvme_loop_queue_response(struct nvmet_req *req)return ; } -nvme_end_request (rq, cqe->status, cqe->result); +if (!nvme_end_request (rq, cqe->status, cqe->result)) +nvme_loop_complete_rq (rq); } }
Signature · 1 line
2.26.2
Re: [PATCH 12/12] nvme: use blk_mq_complete_request_remote to avoid an indirect function call
Quoted 4 lines · Christoph Hellwig
Use the new
blk_mq_complete_request_remotehelper to avoid an indirect function call in the completion fast path.Signed-off-by Christoph Hellwig <[email protected]>
Any comments?
(and yes, I misspelled the nvme list address, but the interested parties should be on linux-block anyway)
Quoted 7 lines · Christoph Hellwig
Hi Jens,
Peters recent inquiry made me dust off various bits of unfinished work around
blk_mq_complete_requestand massage it into a coherent series.This does three different things all touching the same area:
Quoted 12 lines
- merge the softirq based single queue completion into the main blk-mq completion mechanism, as there is a lot of duplicate logic between the two
- move the error injection that just fails to complete requests out into the drivers. The fact thatblk_mq_complete_requestjust wouldn't complete when called from the error handle has been causing all kinds of pain
- optimize the fast path by allowing drivers to avoid the indirect call with a little more work for polled or per-CPU IRQ completions. With this a polled block device I/O only has two indirect calls let, one for the file operation that ends up in the block device code, and another one for disptching to the nvme driver
---end quoted text---
Jens Axboe
maintainerQuoted 4 lines · Christoph Hellwig
Any comments?
(and yes, I misspelled the nvme list address, but the interested parties should be on linux-block anyway)
I've looked over it, and it looks fine to me. Haven't done a full test pass yet, but assuming that's good, I'll queue this up for 5.9.
Signature · 1 line
Jens Axboe