Skip to contentSkip to conversation

blk-mq: test QUEUE_FLAG_HCTX_ACTIVE for sbitmap_shared in hctx_may_queue

PATCH Applied Ming Lei 3 participants 10 messages Dec 27, 2020 – Jan 25, 2021 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 blk-mq: test QUEUE_FLAG_HCTX_ACTIVE for sbitmap_shared in hctx_may_queue

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Cc John Garry <[email protected]>
Cc Kashyap Desai <[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")
---
 block/blk-mq.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-mq.h +1 -1
index c1458d9502f1..3616453ca28c 100644
@@ -304,7 +304,7 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx,
 		struct request_queue *q = hctx->queue;
 		struct blk_mq_tag_set *set = q->tag_set;
 
-		if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &q->queue_flags))
+		if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags))
 			return true;
 		users = atomic_read(&set->active_queues_shared_sbitmap);
 	} else {
Signature · 1 line
2.28.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 9 lines · Ming Lei

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Cc John Garry <[email protected]>
Cc Kashyap Desai <[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")
Quoted 9 lines
---
  block/blk-mq.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
block/blk-mq.h +0 -0
index c1458d9502f1..3616453ca28c 100644

block/blk-mq.h static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, · line 307

Quoted 5 lines
block/blk-mq.h +1 -1
@@ -304,7 +304,7 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx,
  		struct request_queue *q = hctx->queue;
  		struct blk_mq_tag_set *set = q->tag_set;
 
-		if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &q->queue_flags))
+		if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags))

I wonder how this ever worked properly, as BLK_MQ_S_TAG_ACTIVE is bit index 1, and for q->queue_flags that means QUEUE_FLAG_DYING bit, which I figure is not set normally..

Quoted 4 lines
block/blk-mq.h +0 -0
  			return true;
  		users = atomic_read(&set->active_queues_shared_sbitmap);
  	} else {
author
#2 lore raw
Quoted 17 lines · John Garry
Quoted 9 lines · Ming Lei

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Cc John Garry <[email protected]>
Cc Kashyap Desai <[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")
Quoted 4 lines
---
  block/blk-mq.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines

block/blk-mq.h static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, · line 306

Quoted 9 lines
block/blk-mq.h +1 -1
index c1458d9502f1..3616453ca28c 100644
@@ -304,7 +304,7 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx,
  		struct request_queue *q = hctx->queue;
  		struct blk_mq_tag_set *set = q->tag_set;
-		if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &q->queue_flags))
+		if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags))

I wonder how this ever worked properly, as BLK_MQ_S_TAG_ACTIVE is bit index 1, and for q->queue_flags that means QUEUE_FLAG_DYING bit, which I figure is not set normally..

It always return true, and might just take a bit more CPU especially the tag queue depth of magsas_raid and hisi_sas_v3 is quite high.

Thanks,
Ming

#3 lore raw
Quoted 29 lines · Ming Lei
Quoted 16 lines · John Garry
Quoted 9 lines · Ming Lei

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Cc John Garry<[email protected]>
Cc Kashyap Desai<[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")
Quoted 4 lines
---
   block/blk-mq.h | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 12 lines

block/blk-mq.h static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, · line 306

Quoted 9 lines
block/blk-mq.h +1 -1
index c1458d9502f1..3616453ca28c 100644
@@ -304,7 +304,7 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx,
   		struct request_queue *q = hctx->queue;
   		struct blk_mq_tag_set *set = q->tag_set;
-		if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &q->queue_flags))
+		if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags))

I wonder how this ever worked properly, as BLK_MQ_S_TAG_ACTIVE is bit index 1, and for q->queue_flags that means QUEUE_FLAG_DYING bit, which I figure is not set normally..

Quoted 2 lines

It always return true, and might just take a bit more CPU especially the tag queue depth of magsas_raid and hisi_sas_v3 is quite high.

Hi Ming,

Right, but we actually tested by hacking the host tag queue depth to be lower such that we should have tag contention, here is an extract from the original series cover letter for my results:

Tag depth 4000 (default) 260**

Baseline (v5.9-rc1):
none sched: 2094K IOPS 513K
mq-deadline sched: 2145K IOPS 1336K

Final, host_tagset=0 in LLDD *, ***:
none sched: 2120K IOPS 550K
mq-deadline sched: 2121K IOPS 1309K

Final ***:
none sched: 2132K IOPS 1185
mq-deadline sched: 2145K IOPS 2097

Maybe my test did not expose the issue. Kashyap also tested this and reported the original issue such that we needed this feature, so I'm confused.

Thanks,
John

author
#4 lore raw
Quoted 49 lines · John Garry
Quoted 29 lines · Ming Lei
Quoted 16 lines · John Garry
Quoted 9 lines · Ming Lei

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Cc John Garry<[email protected]>
Cc Kashyap Desai<[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")
Quoted 4 lines
---
   block/blk-mq.h | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 12 lines

block/blk-mq.h static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx, · line 306

Quoted 9 lines
block/blk-mq.h +1 -1
index c1458d9502f1..3616453ca28c 100644
@@ -304,7 +304,7 @@ static inline bool hctx_may_queue(struct blk_mq_hw_ctx *hctx,
   		struct request_queue *q = hctx->queue;
   		struct blk_mq_tag_set *set = q->tag_set;
-		if (!test_bit(BLK_MQ_S_TAG_ACTIVE, &q->queue_flags))
+		if (!test_bit(QUEUE_FLAG_HCTX_ACTIVE, &q->queue_flags))

I wonder how this ever worked properly, as BLK_MQ_S_TAG_ACTIVE is bit index 1, and for q->queue_flags that means QUEUE_FLAG_DYING bit, which I figure is not set normally..

Quoted 2 lines

It always return true, and might just take a bit more CPU especially the tag queue depth of magsas_raid and hisi_sas_v3 is quite high.

Hi Ming,

Right, but we actually tested by hacking the host tag queue depth to be lower such that we should have tag contention, here is an extract from the original series cover letter for my results:

Tag depth 4000 (default) 260**

Baseline (v5.9-rc1):
none sched: 2094K IOPS 513K
mq-deadline sched: 2145K IOPS 1336K

Final, host_tagset=0 in LLDD *, ***:
none sched: 2120K IOPS 550K
mq-deadline sched: 2121K IOPS 1309K

Quoted 7 lines

Final ***:
none sched: 2132K IOPS 1185
mq-deadline sched: 2145K IOPS 2097

Maybe my test did not expose the issue. Kashyap also tested this and reported the original issue such that we needed this feature, so I'm confused.

How many LUNs are involved in above test with 260 depth?

Thanks,
Ming

#5 lore raw
Quoted 19 lines · Ming Lei
Quoted 19 lines
Quoted 3 lines
Quoted 1 line

ot set normally..

It always return true, and might just take a bit more CPU especially the tag queue depth of magsas_raid and hisi_sas_v3 is quite high.

Hi Ming,

Right, but we actually tested by hacking the host tag queue depth to be lower such that we should have tag contention, here is an extract from the original series cover letter for my results:

Tag depth 4000 (default) 260**

Baseline (v5.9-rc1):
none sched: 2094K IOPS 513K
mq-deadline sched: 2145K IOPS 1336K

Final, host_tagset=0 in LLDD *, ***:
none sched: 2120K IOPS 550K
mq-deadline sched: 2121K IOPS 1309K

Quoted 7 lines
Quoted 7 lines

Final ***:
none sched: 2132K IOPS 1185
mq-deadline sched: 2145K IOPS 2097

Maybe my test did not expose the issue. Kashyap also tested this and reported the original issue such that we needed this feature, so I'm confused.

Hi Ming,

Quoted 1 line

How many LUNs are involved in above test with 260 depth?

For me, there was 12 SAS SSDs; for convenience here is the cover letter with details:
https://lore.kernel.org/linux-block/[email protected]/

IIRC, for megaraid sas, Kashyap used many more LUNs for testing (64) and high fio depth (128) but did not reduce .can_queue, topic originally raised here:
https://lore.kernel.org/linux-block/[email protected]/

Thanks,
John

author
#6 lore raw
Quoted 32 lines · John Garry
Quoted 19 lines · Ming Lei
Quoted 19 lines
Quoted 3 lines
Quoted 1 line

ot set normally..

It always return true, and might just take a bit more CPU especially the tag queue depth of magsas_raid and hisi_sas_v3 is quite high.

Hi Ming,

Right, but we actually tested by hacking the host tag queue depth to be lower such that we should have tag contention, here is an extract from the original series cover letter for my results:

Tag depth 4000 (default) 260**

Baseline (v5.9-rc1):
none sched: 2094K IOPS 513K
mq-deadline sched: 2145K IOPS 1336K

Final, host_tagset=0 in LLDD *, ***:
none sched: 2120K IOPS 550K
mq-deadline sched: 2121K IOPS 1309K

Quoted 7 lines
Quoted 7 lines

Final ***:
none sched: 2132K IOPS 1185
mq-deadline sched: 2145K IOPS 2097

Maybe my test did not expose the issue. Kashyap also tested this and reported the original issue such that we needed this feature, so I'm confused.

Hi Ming,

Quoted 1 line

How many LUNs are involved in above test with 260 depth?

Quoted 8 lines

For me, there was 12 SAS SSDs; for convenience here is the cover letter with details:
https://lore.kernel.org/linux-block/[email protected]/

IIRC, for megaraid sas, Kashyap used many more LUNs for testing (64) and high fio depth (128) but did not reduce .can_queue, topic originally raised here:
https://lore.kernel.org/linux-block/[email protected]/

OK, in both tests, nr_luns are big enough wrt. 260 depth. Maybe that is why very low IOPS is observed in 'Final(hosttag=1)' with 260 depth.

I'd suggest to run your previous test again after applying this patch, and see if difference can be observed.

Signature · 1 line
Ming
#7 lore raw
Quoted 9 lines · Ming Lei
Quoted 9 lines
Quoted 1 line

How many LUNs are involved in above test with 260 depth?

For me, there was 12 SAS SSDs; for convenience here is the cover letter with details:
https://lore.kernel.org/linux-block/[email protected]/

IIRC, for megaraid sas, Kashyap used many more LUNs for testing (64) and high fio depth (128) but did not reduce .can_queue, topic originally raised here:
https://lore.kernel.org/linux-block/[email protected]/

Quoted 5 lines

OK, in both tests, nr_luns are big enough wrt. 260 depth. Maybe that is why very low IOPS is observed in 'Final(hosttag=1)' with 260 depth.

I'd suggest to run your previous test again after applying this patch, and see if difference can be observed.

Hi Ming,

I tested and didn't see a noticeable difference with the fix when using the reducing tag queue depth. I got ~500K IOPs with tag queue depth of 260, as opposed to 2M with full tag queue depth. However I was doubtful on this test method before. Regardless, your change and this feature still look proper.

@Kashyap, it would be great if you guys could test this also on that same setup you described previously:

https://lore.kernel.org/linux-block/[email protected]/

Thanks,
John

author
#8 lore raw
In reply to Ming Lei
Quoted 9 lines · Ming Lei

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Cc John Garry <[email protected]>
Cc Kashyap Desai <[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")

Hello Jens,

This one fixes one v5.11 issue, can you queue it?

Thanks,
Ming

maintainer Applied
#9 lore raw
Quoted 6 lines · Ming Lei
Quoted 5 lines · Ming Lei

In case of blk_mq_is_sbitmap_shared(), we should test QUEUE_FLAG_HCTX_ACTIVE against q->queue_flags instead of BLK_MQ_S_TAG_ACTIVE.

So fix it.

Quoted 8 lines
Quoted 4 lines
Cc John Garry <[email protected]>
Cc Kashyap Desai <[email protected]>
Fixes f1b49fdc1c64 ("blk-mq: Record active_queues_shared_sbitmap per tag_set for when using shared sbitmap")

Hello Jens,

This one fixes one v5.11 issue, can you queue it?

Queued up, thanks.

Signature · 1 line
Jens Axboe