Skip to contentSkip to conversation

block: rq_affinity default and reserved tag limits

PATCH New Robert Elliott 3 participants 7 messages Sep 10, 2014 Block layer

lore
Share & tools
lore
#0 lore raw

cover block: rq_affinity default and reserved tag limits

The following series changes the default blk-mq rq_affinity to handling completions on the submitting CPU, and handles requests for too many reserved tags.

---

Robert Elliott (2):
block: default to rq_affinity=2 for blk-mq
block: return error if too many reserved tags are requested

 block/blk-mq.c         |    8 +++++---
 include/linux/blkdev.h |    3 ++-
 2 files changed, 7 insertions(+), 4 deletions(-)
Signature · 1 line
Robert Elliott, HP Server Storage
#1 lore raw

PATCH 1/2 block: default to rq_affinity=2 for blk-mq

From: Robert Elliott <[email protected]>

One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.

On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itself

Example with CPU5 receiving all interrupts:

   CPU usage:   CPU0   CPU1   CPU2   CPU3   CPU4   CPU5
        %usr:   0.00   3.03   1.01   2.02   2.00   0.00
        %sys:  14.58  75.76  14.14   4.04  78.00   0.00
        %irq:   0.00   0.00   0.00   1.01   0.00 100.00
       %soft:   0.00   0.00   0.00   0.00   0.00   0.00
%iowait idle:  85.42  21.21  84.85  92.93  20.00   0.00
       %idle:   0.00   0.00   0.00   0.00   0.00   0.00

When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.

Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.

The SCSI LLD can set affinity_hint for each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.

Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.

rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).

Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.

---
 include/linux/blkdev.h |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
include/linux/blkdev.h +2 -1
index 518b465..9f41a02 100644
@@ -522,7 +522,8 @@ struct request_queue {
 				 (1 << QUEUE_FLAG_ADD_RANDOM))
 
 #define QUEUE_FLAG_MQ_DEFAULT	((1 << QUEUE_FLAG_IO_STAT) |		\
-				 (1 << QUEUE_FLAG_SAME_COMP))
+				 (1 << QUEUE_FLAG_SAME_COMP)	|	\
+				 (1 << QUEUE_FLAG_SAME_FORCE))
 
 static inline void queue_lockdep_assert_held(struct request_queue *q)
 {
maintainer
#2 lore raw

Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq

Quoted 47 lines · Robert Elliott

From: Robert Elliott <[email protected]>

One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.

On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itself

Example with CPU5 receiving all interrupts:

   CPU usage:   CPU0   CPU1   CPU2   CPU3   CPU4   CPU5
        %usr:   0.00   3.03   1.01   2.02   2.00   0.00
        %sys:  14.58  75.76  14.14   4.04  78.00   0.00
        %irq:   0.00   0.00   0.00   1.01   0.00 100.00
       %soft:   0.00   0.00   0.00   0.00   0.00   0.00
%iowait idle:  85.42  21.21  84.85  92.93  20.00   0.00
       %idle:   0.00   0.00   0.00   0.00   0.00   0.00

When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.

Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.

The SCSI LLD can set affinity_hint for each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.

Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.

Quoted 7 lines

rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).

Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.

I don't think we should do this generically. For "sane" devices with multiple completion queues, and with proper affinity setting in the driver, this is going to be a loss.

So lets not add it to QUEUE_FLAG_MQ_DEFAULT, but we can make it default for nr_hw_queues == 1. I think that would be way saner.

Signature · 1 line
Jens Axboe
#3 lore raw

RE: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq

Quoted 69 lines

-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, 10 September, 2014 1:15 PM
To: Robert Elliott; Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq

Quoted 47 lines · Robert Elliott

From: Robert Elliott <[email protected]>

One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.

On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itself

Example with CPU5 receiving all interrupts:

   CPU usage:   CPU0   CPU1   CPU2   CPU3   CPU4   CPU5
        %usr:   0.00   3.03   1.01   2.02   2.00   0.00
        %sys:  14.58  75.76  14.14   4.04  78.00   0.00
        %irq:   0.00   0.00   0.00   1.01   0.00 100.00
       %soft:   0.00   0.00   0.00   0.00   0.00   0.00
%iowait idle:  85.42  21.21  84.85  92.93  20.00   0.00
       %idle:   0.00   0.00   0.00   0.00   0.00   0.00

When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.

Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.

The SCSI LLD can set affinity_hint for each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.

Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.

Quoted 7 lines

rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).

Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.

I don't think we should do this generically. For "sane" devices with multiple completion queues, and with proper affinity setting in the driver, this is going to be a loss.

Quoted 6 lines

So lets not add it to QUEUE_FLAG_MQ_DEFAULT, but we can make it default
for nr_hw_queues == 1. I think that would be way saner.

--
Jens Axboe

If the interrupt does arrive on the submitting CPU, then it meets the criteria for all the cases:
* 1: complete on any CPU
* 2: complete on submitting CPU's node (QUEUE_FLAG_SAME_COMP)
* 3: complete on submitting CPU (QUEUE_FLAG_SAME_FORCE)

and _blk_complete_request handles it locally rather than sending an IPI.

        if (req->cpu != -1) {
                ccpu = req->cpu;
                if (!test_bit(QUEUE_FLAG_SAME_FORCE, &q->queue_flags))
                        shared = cpus_share_cache(cpu, ccpu);
        } else
                ccpu = cpu;
        ...
        if (ccpu == cpu || shared) {
                struct list_head *list;
do_local:
        ...
        } else if (raise_blk_irq(ccpu, req))
                goto do_local;

Are you saying you want the blk_queue_bio submission to not even set the req->cpu field (which defaulted to -1):

        if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags))
                req->cpu = raw_smp_processor_id();

when you expect the interrupt routing is good so that _blk_complete_request can avoid the test_bit and cpus_share_cache calls?

With irqbalance no longer honoring affinity_hint by default, I'm worried that most LLDs will not find their interrupts routed that way anymore. That's how we ran into this; scsi-mq + kernel-3.17 on an up-to-date RHEL 6.5 distro (which now carries the new irqbalance).

We plan to create a policyscript for the new irqbalance for hpsa devices, but other high-IOPS drivers will hit the same problem.

---
Rob Elliott HP Server Storage

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

maintainer
#4 lore raw

Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq

Quoted 85 lines · Robert (Server Storage)
Quoted 69 lines

-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, 10 September, 2014 1:15 PM
To: Robert Elliott; Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq

Quoted 47 lines · Robert Elliott

From: Robert Elliott <[email protected]>

One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.

On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itself

Example with CPU5 receiving all interrupts:

   CPU usage:   CPU0   CPU1   CPU2   CPU3   CPU4   CPU5
        %usr:   0.00   3.03   1.01   2.02   2.00   0.00
        %sys:  14.58  75.76  14.14   4.04  78.00   0.00
        %irq:   0.00   0.00   0.00   1.01   0.00 100.00
       %soft:   0.00   0.00   0.00   0.00   0.00   0.00
%iowait idle:  85.42  21.21  84.85  92.93  20.00   0.00
       %idle:   0.00   0.00   0.00   0.00   0.00   0.00

When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.

Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.

The SCSI LLD can set affinity_hint for each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.

Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.

Quoted 7 lines

rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).

Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.

I don't think we should do this generically. For "sane" devices with multiple completion queues, and with proper affinity setting in the driver, this is going to be a loss.

Quoted 6 lines

So lets not add it to QUEUE_FLAG_MQ_DEFAULT, but we can make it default
for nr_hw_queues == 1. I think that would be way saner.

--
Jens Axboe

If the interrupt does arrive on the submitting CPU, then it meets the criteria for all the cases:
* 1: complete on any CPU
* 2: complete on submitting CPU's node (QUEUE_FLAG_SAME_COMP)
* 3: complete on submitting CPU (QUEUE_FLAG_SAME_FORCE)

and _blk_complete_request handles it locally rather than sending an IPI.

Quoted 13 lines
        if (req->cpu != -1) {
                ccpu = req->cpu;
                if (!test_bit(QUEUE_FLAG_SAME_FORCE, &q->queue_flags))
                        shared = cpus_share_cache(cpu, ccpu);
        } else
                ccpu = cpu;
        ...
        if (ccpu == cpu || shared) {
                struct list_head *list;
do_local:
        ...
        } else if (raise_blk_irq(ccpu, req))
                goto do_local;

I forgot about the shared case being handled appropriately, so that should probably be fine to do. My primary concern here is a performance penalty on sync IO, I'll run a few test on a single IRQ case (like the mtip32xx) and see how that performs. But you are right, it might not be a bad thing to do by default.

Quoted 8 lines

Are you saying you want the blk_queue_bio submission to not even set the req->cpu field (which defaulted to -1):

        if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags))
                req->cpu = raw_smp_processor_id();

when you expect the interrupt routing is good so that _blk_complete_request can avoid the test_bit and cpus_share_cache calls?

No, and since those are non-serializing tests, I suspect if we start adding a branch to avoid that we will negate any potential win on that. The flags should basically never get dirtied. Well I guess they could for heavy uses of start/stop queue, but that might be something that's worthwhile to tackle separately.

Quoted 10 lines

With irqbalance no longer honoring affinity_hint by default, I'm worried that most LLDs will not find their interrupts routed that way anymore. That's how we ran into this; scsi-mq + kernel-3.17 on an up-to-date RHEL 6.5 distro (which now carries the new irqbalance).

We plan to create a policyscript for the new irqbalance for hpsa devices, but other high-IOPS drivers will hit the same problem.

irqbalance has _always_ been a pain in the butt... Suboptimal or changing behaviour from release to release, it's been one of the most annoying parts of performance tuning.

Signature · 1 line
Jens Axboe
#5 lore raw

PATCH 2/2 block: return error if too many reserved tags are requested

In reply to Robert Elliott

From: Robert Elliott <[email protected]>

Make blk_mq_alloc_tag_set return an error if set->reserved_tags is greater than BLK_MQ_MAX_DEPTH minus the minimum number of tags, since:
* set->queue_depth is truncated to that value
* set->reserved_tags needs to be less than set->queue_depth

---
 block/blk-mq.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)
block/blk-mq.c +5 -3
index c49fe00..dc2970d 100644
@@ -1936,16 +1936,18 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 		return -EINVAL;
 	if (!set->queue_depth)
 		return -EINVAL;
+	if (set->reserved_tags > BLK_MQ_MAX_DEPTH - BLK_MQ_TAG_MIN)
+		return -EINVAL;
 	if (set->queue_depth < set->reserved_tags + BLK_MQ_TAG_MIN)
 		return -EINVAL;
 
 	if (!set->nr_hw_queues || !set->ops->queue_rq || !set->ops->map_queue)
 		return -EINVAL;
 
-	if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
+	if (set->queue_depth > BLK_MQ_MAX_DEPTH - set->reserved_tags) {
+		set->queue_depth = BLK_MQ_MAX_DEPTH - set->reserved_tags;
 		pr_info("blk-mq: reduced tag depth to %u\n",
-			BLK_MQ_MAX_DEPTH);
-		set->queue_depth = BLK_MQ_MAX_DEPTH;
+			set->queue_depth);
 	}
 
 	set->tags = kmalloc_node(set->nr_hw_queues *
maintainer
#6 lore raw

Re: [PATCH 2/2] block: return error if too many reserved tags are requested

Quoted 21 lines · Robert Elliott

From: Robert Elliott <[email protected]>

Make blk_mq_alloc_tag_set return an error if set->reserved_tags is greater than BLK_MQ_MAX_DEPTH minus the minimum number of tags, since:
* set->queue_depth is truncated to that value
* set->reserved_tags needs to be less than set->queue_depth

---
 block/blk-mq.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)
block/blk-mq.c +0 -0
index c49fe00..dc2970d 100644
@@ -1936,16 +1936,18 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 		return -EINVAL;
 	if (!set->queue_depth)
 		return -EINVAL;

block/blk-mq.c int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) · line 1947

Quoted 10 lines
block/blk-mq.c +3 -1
+	if (set->reserved_tags > BLK_MQ_MAX_DEPTH - BLK_MQ_TAG_MIN)
+		return -EINVAL;
 	if (set->queue_depth < set->reserved_tags + BLK_MQ_TAG_MIN)
 		return -EINVAL;
 
 	if (!set->nr_hw_queues || !set->ops->queue_rq || !set->ops->map_queue)
 		return -EINVAL;
 
-	if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
+	if (set->queue_depth > BLK_MQ_MAX_DEPTH - set->reserved_tags) {

This is harder to read than:

	if (set->queue_depth + set->reserved_tags > BLK_MQ_MAX_DEPTH) {

which more clearly expresses the same thing. But set->queue_depth is the total pool, reserved tags come out of that pool. So I don't think this is correct.

Signature · 1 line
Jens Axboe