Skip to contentSkip to conversation

blk-mq: complete req in softirq context in case of single queue

PATCH In review Ming Lei 2 participants 3 messages Sep 28, 2018 – Oct 8, 2018 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH v2 1/1 blk-mq: complete req in softirq context in case of single queue

Lot of controllers may have only one irq vector for completing IO request. And usually affinity of the only irq vector is all possible CPUs, however, on most of ARCH, there may be only one specific CPU for handling this interrupt.

So if all IOs are completed in hardirq context, it is inevitable to degrade IO performance because of increased irq latency.

This patch tries to address this issue by allowing to complete request in softirq context, like the legacy IO path.

IOPS is observed as ~13%+ in the following randread test on raid0 over virtio-scsi.

mdadm --create --verbose /dev/md0 --level=0 --chunk=1024 --raid-devices=8 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

fio --time_based --name=benchmark --runtime=30 --filename=/dev/md0 --nrfiles=1 --ioengine=libaio --iodepth=32 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=32 --rw=randread --blocksize=4k

Cc Dongli Zhang <[email protected]>
Cc Zach Marano <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Jianchao Wang <[email protected]>
---
 block/blk-mq.c      | 14 ++++++++++++++
 block/blk-softirq.c |  5 ++---
 2 files changed, 16 insertions(+), 3 deletions(-)
block/blk-mq.c +14 -0
index 85a1c1a59c72..d4792c3ac983 100644
@@ -565,6 +565,20 @@ static void __blk_mq_complete_request(struct request *rq)
 	if (rq->internal_tag != -1)
 		blk_mq_sched_completed_request(rq);
 
+	/*
+	 * Most of single queue controllers, there is only one irq vector
+	 * for handling IO completion, and the only irq's affinity is set
+	 * as all possible CPUs. On most of ARCHs, this affinity means the
+	 * irq is handled on one specific CPU.
+	 *
+	 * So complete IO reqeust in softirq context in case of single queue
+	 * for not degrading IO performance by irqsoff latency.
+	 */
+	if (rq->q->nr_hw_queues == 1) {
+		__blk_complete_request(rq);
+		return;
+	}
+
 	if (!test_bit(QUEUE_FLAG_SAME_COMP, &rq->q->queue_flags)) {
 		rq->q->softirq_done_fn(rq);
 		return;
block/blk-softirq.c +2 -3
index 15c1f5e12eb8..e47a2f751884 100644
@@ -97,8 +97,8 @@ static int blk_softirq_cpu_dead(unsigned int cpu)
 
 void __blk_complete_request(struct request *req)
 {
-	int ccpu, cpu;
 	struct request_queue *q = req->q;
+	int cpu, ccpu = q->mq_ops ? req->mq_ctx->cpu : req->cpu;
 	unsigned long flags;
 	bool shared = false;
 
@@ -110,8 +110,7 @@ void __blk_complete_request(struct request *req)
 	/*
 	 * Select completion CPU
 	 */
-	if (req->cpu != -1) {
-		ccpu = req->cpu;
+	if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags) && ccpu != -1) {
 		if (!test_bit(QUEUE_FLAG_SAME_FORCE, &q->queue_flags))
 			shared = cpus_share_cache(cpu, ccpu);
 	} else
Signature · 1 line
2.9.5
author
#1 lore raw
Quoted 81 lines · Ming Lei

Lot of controllers may have only one irq vector for completing IO request. And usually affinity of the only irq vector is all possible CPUs, however, on most of ARCH, there may be only one specific CPU for handling this interrupt.

So if all IOs are completed in hardirq context, it is inevitable to degrade IO performance because of increased irq latency.

This patch tries to address this issue by allowing to complete request in softirq context, like the legacy IO path.

IOPS is observed as ~13%+ in the following randread test on raid0 over virtio-scsi.

mdadm --create --verbose /dev/md0 --level=0 --chunk=1024 --raid-devices=8 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

fio --time_based --name=benchmark --runtime=30 --filename=/dev/md0 --nrfiles=1 --ioengine=libaio --iodepth=32 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=32 --rw=randread --blocksize=4k

Cc Dongli Zhang <[email protected]>
Cc Zach Marano <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Jianchao Wang <[email protected]>
---
 block/blk-mq.c      | 14 ++++++++++++++
 block/blk-softirq.c |  5 ++---
 2 files changed, 16 insertions(+), 3 deletions(-)
block/blk-mq.c +14 -0
index 85a1c1a59c72..d4792c3ac983 100644
@@ -565,6 +565,20 @@ static void __blk_mq_complete_request(struct request *rq)
 	if (rq->internal_tag != -1)
 		blk_mq_sched_completed_request(rq);
 
+	/*
+	 * Most of single queue controllers, there is only one irq vector
+	 * for handling IO completion, and the only irq's affinity is set
+	 * as all possible CPUs. On most of ARCHs, this affinity means the
+	 * irq is handled on one specific CPU.
+	 *
+	 * So complete IO reqeust in softirq context in case of single queue
+	 * for not degrading IO performance by irqsoff latency.
+	 */
+	if (rq->q->nr_hw_queues == 1) {
+		__blk_complete_request(rq);
+		return;
+	}
+
 	if (!test_bit(QUEUE_FLAG_SAME_COMP, &rq->q->queue_flags)) {
 		rq->q->softirq_done_fn(rq);
 		return;
block/blk-softirq.c +2 -3
index 15c1f5e12eb8..e47a2f751884 100644
@@ -97,8 +97,8 @@ static int blk_softirq_cpu_dead(unsigned int cpu)
 
 void __blk_complete_request(struct request *req)
 {
-	int ccpu, cpu;
 	struct request_queue *q = req->q;
+	int cpu, ccpu = q->mq_ops ? req->mq_ctx->cpu : req->cpu;
 	unsigned long flags;
 	bool shared = false;
 
@@ -110,8 +110,7 @@ void __blk_complete_request(struct request *req)
 	/*
 	 * Select completion CPU
 	 */
-	if (req->cpu != -1) {
-		ccpu = req->cpu;
+	if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags) && ccpu != -1) {
 		if (!test_bit(QUEUE_FLAG_SAME_FORCE, &q->queue_flags))
 			shared = cpus_share_cache(cpu, ccpu);
 	} else
Signature · 1 line
2.9.5

Ping...

Thanks,
Ming

maintainer
#2 lore raw
In reply to Ming Lei
Quoted 11 lines · Ming Lei

Lot of controllers may have only one irq vector for completing IO request. And usually affinity of the only irq vector is all possible CPUs, however, on most of ARCH, there may be only one specific CPU for handling this interrupt.

So if all IOs are completed in hardirq context, it is inevitable to degrade IO performance because of increased irq latency.

This patch tries to address this issue by allowing to complete request in softirq context, like the legacy IO path.

Quoted 6 lines

IOPS is observed as ~13%+ in the following randread test on raid0 over virtio-scsi.

mdadm --create --verbose /dev/md0 --level=0 --chunk=1024 --raid-devices=8 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

fio --time_based --name=benchmark --runtime=30 --filename=/dev/md0 --nrfiles=1 --ioengine=libaio --iodepth=32 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 --numjobs=32 --rw=randread --blocksize=4k

Looks fine to me, and imho it makes a lot of sense to apply the same logic for remote completions to single queue as we did for the legacy parts. I have applied it for 4.20.

Signature · 1 line
Jens Axboe