block: rq_affinity default and reserved tag limits
cover block: rq_affinity default and reserved tag limits
The following series changes the default blk-mq rq_affinity to handling completions on the submitting CPU, and handles requests for too many reserved tags.
---
Robert Elliott (2):
block: default to rq_affinity=2 for blk-mq
block: return error if too many reserved tags are requested
block/blk-mq.c | 8 +++++--- include/linux/blkdev.h | 3 ++- 2 files changed, 7 insertions(+), 4 deletions(-)
Signature · 1 line
Robert Elliott, HP Server Storage
PATCH 1/2 block: default to rq_affinity=2 for blk-mq
From: Robert Elliott <[email protected]>
One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.
On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itself
Example with CPU5 receiving all interrupts:
CPU usage: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
%usr: 0.00 3.03 1.01 2.02 2.00 0.00
%sys: 14.58 75.76 14.14 4.04 78.00 0.00
%irq: 0.00 0.00 0.00 1.01 0.00 100.00
%soft: 0.00 0.00 0.00 0.00 0.00 0.00
%iowait idle: 85.42 21.21 84.85 92.93 20.00 0.00
%idle: 0.00 0.00 0.00 0.00 0.00 0.00When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.
Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.
The SCSI LLD can set affinity_hint for each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.
Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.
rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).
Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.
--- include/linux/blkdev.h | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
include/linux/blkdev.h +2 -1
@@ -522,7 +522,8 @@ struct request_queue { (1 << QUEUE_FLAG_ADD_RANDOM))#define QUEUE_FLAG_MQ_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ - (1 << QUEUE_FLAG_SAME_COMP)) + (1 << QUEUE_FLAG_SAME_COMP) | \ + (1 << QUEUE_FLAG_SAME_FORCE))static inline void queue_lockdep_assert_held (struct request_queue *q) {
Jens Axboe
maintainerRe: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq
Quoted 47 lines · Robert Elliott
From: Robert Elliott <[email protected]>
One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.
On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itselfExample with CPU5 receiving all interrupts:
CPU usage: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 %usr: 0.00 3.03 1.01 2.02 2.00 0.00 %sys: 14.58 75.76 14.14 4.04 78.00 0.00 %irq: 0.00 0.00 0.00 1.01 0.00 100.00 %soft: 0.00 0.00 0.00 0.00 0.00 0.00 %iowait idle: 85.42 21.21 84.85 92.93 20.00 0.00 %idle: 0.00 0.00 0.00 0.00 0.00 0.00When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.
Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.The SCSI LLD can set
affinity_hintfor each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.
Quoted 7 lines
rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).
Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.
I don't think we should do this generically. For "sane" devices with multiple completion queues, and with proper affinity setting in the driver, this is going to be a loss.
So lets not add it to QUEUE_FLAG_MQ_DEFAULT, but we can make it default for nr_hw_queues == 1. I think that would be way saner.
Signature · 1 line
Jens Axboe
RE: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq
Quoted 69 lines
-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, 10 September, 2014 1:15 PM
To: Robert Elliott; Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mqQuoted 47 lines · Robert Elliott
From: Robert Elliott <[email protected]>
One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.
On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itselfExample with CPU5 receiving all interrupts:
CPU usage: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 %usr: 0.00 3.03 1.01 2.02 2.00 0.00 %sys: 14.58 75.76 14.14 4.04 78.00 0.00 %irq: 0.00 0.00 0.00 1.01 0.00 100.00 %soft: 0.00 0.00 0.00 0.00 0.00 0.00 %iowait idle: 85.42 21.21 84.85 92.93 20.00 0.00 %idle: 0.00 0.00 0.00 0.00 0.00 0.00When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.
Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.The SCSI LLD can set
affinity_hintfor each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.
Quoted 7 lines
rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).
Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.
I don't think we should do this generically. For "sane" devices with multiple completion queues, and with proper affinity setting in the driver, this is going to be a loss.
Quoted 6 lines
So lets not add it to
QUEUE_FLAG_MQ_DEFAULT, but we can make it default
fornr_hw_queues== 1. I think that would be way saner.--
Jens Axboe
If the interrupt does arrive on the submitting CPU, then it meets the criteria for all the cases:
* 1: complete on any CPU
* 2: complete on submitting CPU's node (QUEUE_FLAG_SAME_COMP)
* 3: complete on submitting CPU (QUEUE_FLAG_SAME_FORCE)
and _blk_complete_request handles it locally rather than sending an IPI.
if (req->cpu != -1) {
ccpu = req->cpu;
if (!test_bit(QUEUE_FLAG_SAME_FORCE, &q->queue_flags))
shared = cpus_share_cache(cpu, ccpu);
} else
ccpu = cpu;
...
if (ccpu == cpu || shared) {
struct list_head *list;
do_local:
...
} else if (raise_blk_irq(ccpu, req))
goto do_local;Are you saying you want the blk_queue_bio submission to not even set the req->cpu field (which defaulted to -1):
if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags))
req->cpu = raw_smp_processor_id();when you expect the interrupt routing is good so that _blk_complete_request can avoid the test_bit and cpus_share_cache calls?
With irqbalance no longer honoring affinity_hint by default, I'm worried that most LLDs will not find their interrupts routed that way anymore. That's how we ran into this; scsi-mq + kernel-3.17 on an up-to-date RHEL 6.5 distro (which now carries the new irqbalance).
We plan to create a policyscript for the new irqbalance for hpsa devices, but other high-IOPS drivers will hit the same problem.
---
Rob Elliott HP Server Storage
���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m����������?�I�
Jens Axboe
maintainerRe: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq
Quoted 85 lines · Robert (Server Storage)
Quoted 69 lines
-----Original Message-----
From: Jens Axboe [mailto:[email protected]]
Sent: Wednesday, 10 September, 2014 1:15 PM
To: Robert Elliott; Elliott, Robert (Server Storage); [email protected]; [email protected]
Subject: Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mqQuoted 47 lines · Robert Elliott
From: Robert Elliott <[email protected]>
One change introduced by blk-mq is that it does all the completion work in hard irq context rather than soft irq context.
On a 6 core system, if all interrupts are routed to one CPU, then you can easily run into this:
* 5 CPUs submitting IOs
* 1 CPU spending 100% of its time in hard irq context processing IO completions, not able to submit anything itselfExample with CPU5 receiving all interrupts:
CPU usage: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 %usr: 0.00 3.03 1.01 2.02 2.00 0.00 %sys: 14.58 75.76 14.14 4.04 78.00 0.00 %irq: 0.00 0.00 0.00 1.01 0.00 100.00 %soft: 0.00 0.00 0.00 0.00 0.00 0.00 %iowait idle: 85.42 21.21 84.85 92.93 20.00 0.00 %idle: 0.00 0.00 0.00 0.00 0.00 0.00When the submitting CPUs are forced to process their own completion interrupts, this steals time from new submissions and self-throttles them.
Without that, there is no direct feedback to the submitters to slow down. The only feedback is:
* reaching max queue depth
* lots of timeouts, resulting in aborts, resets, soft lockups and self-detected stalls on CPU5, bogus clocksource tsc unstable reports, network drop-offs, etc.The SCSI LLD can set
affinity_hintfor each of its interrupts to request that a program like irqbalance route the interrupts back to the submitting CPU. The latest version of irqbalance ignores those hints, though, instead offering an option to run a policy script that could honor them. Otherwise, it balances them based on its own algorithms. So, we cannot rely on this.Hardware might perform interrupt coalescing to help, but it cannot help 1 CPU keep up with the work generated by many other CPUs.
Quoted 7 lines
rq_affinity=2 helps by pushing most of the block layer and SCSI midlayer completion work back to the submitting CPU (via an IPI).
Change the default rq_affinity=2 under blk-mq so there's at least some feedback to slow down the submitters.
I don't think we should do this generically. For "sane" devices with multiple completion queues, and with proper affinity setting in the driver, this is going to be a loss.
Quoted 6 lines
So lets not add it to
QUEUE_FLAG_MQ_DEFAULT, but we can make it default
fornr_hw_queues== 1. I think that would be way saner.--
Jens AxboeIf the interrupt does arrive on the submitting CPU, then it meets the criteria for all the cases:
* 1: complete on any CPU
* 2: complete on submitting CPU's node (QUEUE_FLAG_SAME_COMP)
* 3: complete on submitting CPU (QUEUE_FLAG_SAME_FORCE)and
_blk_complete_requesthandles it locally rather than sending an IPI.
Quoted 13 lines
if (req->cpu != -1) {
ccpu = req->cpu;
if (!test_bit(QUEUE_FLAG_SAME_FORCE, &q->queue_flags))
shared = cpus_share_cache(cpu, ccpu);
} else
ccpu = cpu;
...
if (ccpu == cpu || shared) {
struct list_head *list;
do_local:
...
} else if (raise_blk_irq(ccpu, req))
goto do_local;I forgot about the shared case being handled appropriately, so that should probably be fine to do. My primary concern here is a performance penalty on sync IO, I'll run a few test on a single IRQ case (like the mtip32xx) and see how that performs. But you are right, it might not be a bad thing to do by default.
Quoted 8 lines
Are you saying you want the
blk_queue_biosubmission to not even set thereq->cpufield (which defaulted to -1):if (test_bit(QUEUE_FLAG_SAME_COMP, &q->queue_flags)) req->cpu = raw_smp_processor_id();when you expect the interrupt routing is good so that
_blk_complete_requestcan avoid thetest_bitandcpus_share_cachecalls?
No, and since those are non-serializing tests, I suspect if we start adding a branch to avoid that we will negate any potential win on that. The flags should basically never get dirtied. Well I guess they could for heavy uses of start/stop queue, but that might be something that's worthwhile to tackle separately.
Quoted 10 lines
With irqbalance no longer honoring
affinity_hintby default, I'm worried that most LLDs will not find their interrupts routed that way anymore. That's how we ran into this; scsi-mq + kernel-3.17 on an up-to-date RHEL 6.5 distro (which now carries the new irqbalance).We plan to create a policyscript for the new irqbalance for hpsa devices, but other high-IOPS drivers will hit the same problem.
irqbalance has _always_ been a pain in the butt... Suboptimal or changing behaviour from release to release, it's been one of the most annoying parts of performance tuning.
Signature · 1 line
Jens Axboe
PATCH 2/2 block: return error if too many reserved tags are requested
In reply to Robert ElliottFrom: Robert Elliott <[email protected]>
Make blk_mq_alloc_tag_set return an error if set->reserved_tags is greater than BLK_MQ_MAX_DEPTH minus the minimum number of tags, since:
* set->queue_depth is truncated to that value
* set->reserved_tags needs to be less than set->queue_depth
--- block/blk-mq.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)
block/blk-mq.c +5 -3
@@ -1936,16 +1936,18 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)return -EINVAL ;if (!set->queue_depth)return -EINVAL ; +if (set->reserved_tags >BLK_MQ_MAX_DEPTH -BLK_MQ_TAG_MIN ) +return -EINVAL ;if (set->queue_depth < set->reserved_tags +BLK_MQ_TAG_MIN )return -EINVAL ;if (!set->nr_hw_queues || !set->ops->queue_rq || !set->ops->map_queue)return -EINVAL ; -if (set->queue_depth >BLK_MQ_MAX_DEPTH ) { +if (set->queue_depth >BLK_MQ_MAX_DEPTH - set->reserved_tags) { + set->queue_depth =BLK_MQ_MAX_DEPTH - set->reserved_tags;pr_info ("blk-mq: reduced tag depth to %u\n" , -BLK_MQ_MAX_DEPTH ); - set->queue_depth =BLK_MQ_MAX_DEPTH ; + set->queue_depth); } set->tags = kmalloc_node(set->nr_hw_queues *
Jens Axboe
maintainerRe: [PATCH 2/2] block: return error if too many reserved tags are requested
Quoted 21 lines · Robert Elliott
From: Robert Elliott <[email protected]>
Make
blk_mq_alloc_tag_setreturn an error ifset->reserved_tagsis greater thanBLK_MQ_MAX_DEPTHminus the minimum number of tags, since:
*set->queue_depthis truncated to that value
*set->reserved_tagsneeds to be less thanset->queue_depthSigned-off-by Robert Elliott <[email protected]>--- block/blk-mq.c | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-)block/blk-mq.c +0
-0@@ -1936,16 +1936,18 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)return -EINVAL ;if (!set->queue_depth)return -EINVAL ;
block/blk-mq.c int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) · line 1947
Quoted 10 lines
block/blk-mq.c +3
-1+if (set->reserved_tags >BLK_MQ_MAX_DEPTH -BLK_MQ_TAG_MIN ) +return -EINVAL ;if (set->queue_depth < set->reserved_tags +BLK_MQ_TAG_MIN )return -EINVAL ;if (!set->nr_hw_queues || !set->ops->queue_rq || !set->ops->map_queue)return -EINVAL ; -if (set->queue_depth >BLK_MQ_MAX_DEPTH ) { +if (set->queue_depth >BLK_MQ_MAX_DEPTH - set->reserved_tags) {
This is harder to read than:
if (set->queue_depth + set->reserved_tags > BLK_MQ_MAX_DEPTH) {which more clearly expresses the same thing. But set->queue_depth is the total pool, reserved tags come out of that pool. So I don't think this is correct.
Signature · 1 line
Jens Axboe