block: Do not pull requests from the scheduler when we cannot dispatch them
PATCH v2 1/1 block: Do not pull requests from the scheduler when we cannot dispatch them
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in __blk_mq_do_dispatch_sched() pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.
For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because __blk_mq_do_dispatch_sched() always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.
So grab request tag for a request pulled out of the IO scheduler already in __blk_mq_do_dispatch_sched() and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.
--- block/blk-mq-sched.c | 12 +++++++++++- block/blk-mq.c | 2 +- block/blk-mq.h | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-)
Jens, can you please merge the patch? Thanks!
block/blk-mq-sched.c +11 -1
@@ -168,9 +168,19 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)* in blk_mq_dispatch_rq_list(). */ list_add_tail (&rq->queuelist, &rq_list); + count++;if (rq->mq_hctx != hctx) multi_hctxs = true; - }while (++count < max_dispatch); + +/* +* If we cannot get tag for the request, stop dequeueing +* requests from the IO scheduler. We are unlikely to be able +* to submit them anyway and it creates false impression for +* scheduling heuristics that the device can take more IO. +*/ +if (!blk_mq_get_driver_tag (rq)) +break ; + }while (count < max_dispatch);if (!count) {if (run_queue)
block/blk-mq.c +1 -1
@@ -1100,7 +1100,7 @@ static bool __blk_mq_get_driver_tag(struct request *rq)return true; } -static bool blk_mq_get_driver_tag (struct request *rq) +bool blk_mq_get_driver_tag (struct request *rq) {struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
block/blk-mq.h +2 -0
@@ -259,6 +259,8 @@ static inline void blk_mq_put_driver_tag(struct request *rq)__blk_mq_put_driver_tag (rq->mq_hctx, rq); } +bool blk_mq_get_driver_tag (struct request *rq); +static inline void blk_mq_clear_mq_map (struct blk_mq_queue_map *qmap) {int cpu;
Signature · 1 line
2.26.2
Quoted 17 lines · Jan Kara
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in
__blk_mq_do_dispatch_sched()pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because
__blk_mq_do_dispatch_sched()always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.
Quoted 5 lines
So grab request tag for a request pulled out of the IO scheduler already in
__blk_mq_do_dispatch_sched()and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.
Applied to for-5.14/block, thanks.
Signature · 1 line
Jens Axboe
Quoted 48 lines · Jan Kara
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in
__blk_mq_do_dispatch_sched()pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because
__blk_mq_do_dispatch_sched()always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.So grab request tag for a request pulled out of the IO scheduler already in
__blk_mq_do_dispatch_sched()and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.Reviewed-by Ming Lei <[email protected]>Signed-off-by Jan Kara <[email protected]>--- block/blk-mq-sched.c | 12 +++++++++++- block/blk-mq.c | 2 +- block/blk-mq.h | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-)Jens, can you please merge the patch? Thanks!
block/blk-mq-sched.c +4
-1@@ -168,9 +168,19 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)* in blk_mq_dispatch_rq_list(). */ list_add_tail (&rq->queuelist, &rq_list); + count++;if (rq->mq_hctx != hctx) multi_hctxs = true; - }while (++count < max_dispatch); + +/* +* If we cannot get tag for the request, stop dequeueing
block/blk-mq-sched.c static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) · line 186
Quoted 10 lines
block/blk-mq-sched.c +7
-0+* requests from the IO scheduler. We are unlikely to be able +* to submit them anyway and it creates false impression for +* scheduling heuristics that the device can take more IO. +*/ +if (!blk_mq_get_driver_tag (rq)) +break ; + }while (count < max_dispatch);if (!count) {if (run_queue)
Doesn't this lead to a double accounting of the allocated tags? From what I can see we don't really check if the tag is already allocated in blk_mq_get_driver_tag() ...
Hmm?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, 90409 Nürnberg GF: F. Imendörffer, HRB 36809 (AG Nürnberg)
Quoted 57 lines · Hannes Reinecke
Quoted 56 lines · Jan Kara
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in
__blk_mq_do_dispatch_sched()pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because
__blk_mq_do_dispatch_sched()always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.So grab request tag for a request pulled out of the IO scheduler already in
__blk_mq_do_dispatch_sched()and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.Reviewed-by Ming Lei <[email protected]>Signed-off-by Jan Kara <[email protected]>--- block/blk-mq-sched.c | 12 +++++++++++- block/blk-mq.c | 2 +- block/blk-mq.h | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-)Jens, can you please merge the patch? Thanks!
block/blk-mq-sched.c +11
-1@@ -168,9 +168,19 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)* in blk_mq_dispatch_rq_list(). */ list_add_tail (&rq->queuelist, &rq_list); + count++;if (rq->mq_hctx != hctx) multi_hctxs = true; - }while (++count < max_dispatch); + +/* +* If we cannot get tag for the request, stop dequeueing +* requests from the IO scheduler. We are unlikely to be able +* to submit them anyway and it creates false impression for +* scheduling heuristics that the device can take more IO. +*/ +if (!blk_mq_get_driver_tag (rq)) +break ; + }while (count < max_dispatch);
Quoted 6 lines
block/blk-mq-sched.c static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx) · line 186
Quoted 2 lines
block/blk-mq-sched.c +0
-0if (!count) {if (run_queue)Doesn't this lead to a double accounting of the allocated tags? From what I can see we don't really check if the tag is already allocated in
blk_mq_get_driver_tag()...
I think we do check. blk_mq_get_driver_tag() has:
if (rq->tag == BLK_MQ_NO_TAG && !__blk_mq_get_driver_tag(rq))
return false;
if ((hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED) &&
!(rq->rq_flags & RQF_MQ_INFLIGHT)) {
rq->rq_flags |= RQF_MQ_INFLIGHT;
__blk_mq_inc_active_requests(hctx);
}
hctx->tags->rqs[rq->tag] = rq;So once we call it, rq->tag will be != BLK_MQ_NO_TAG and RQF_MQ_INFLIGHT will be set. So neither __blk_mq_get_driver_tag() nor __blk_mq_inc_active_requests() will get repeated.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 25 lines · Jan Kara
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in
__blk_mq_do_dispatch_sched()pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because
__blk_mq_do_dispatch_sched()always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.So grab request tag for a request pulled out of the IO scheduler already in
__blk_mq_do_dispatch_sched()and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.Reviewed-by Ming Lei <[email protected]>Signed-off-by Jan Kara <[email protected]>
Quoted 7 lines
--- block/blk-mq-sched.c | 12 +++++++++++- block/blk-mq.c | 2 +- block/blk-mq.h | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-)Jens, can you please merge the patch? Thanks!
Ping Jens?
Honza
Quoted 55 lines
block/blk-mq-sched.c +11
-1@@ -168,9 +168,19 @@ static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)* in blk_mq_dispatch_rq_list(). */ list_add_tail (&rq->queuelist, &rq_list); + count++;if (rq->mq_hctx != hctx) multi_hctxs = true; - }while (++count < max_dispatch); + +/* +* If we cannot get tag for the request, stop dequeueing +* requests from the IO scheduler. We are unlikely to be able +* to submit them anyway and it creates false impression for +* scheduling heuristics that the device can take more IO. +*/ +if (!blk_mq_get_driver_tag (rq)) +break ; + }while (count < max_dispatch);if (!count) {if (run_queue)block/blk-mq.c +1
-1@@ -1100,7 +1100,7 @@ static bool __blk_mq_get_driver_tag(struct request *rq)return true; } -static bool blk_mq_get_driver_tag (struct request *rq) +bool blk_mq_get_driver_tag (struct request *rq) {struct blk_mq_hw_ctx *hctx = rq->mq_hctx;block/blk-mq.h +2
-0@@ -259,6 +259,8 @@ static inline void blk_mq_put_driver_tag(struct request *rq)__blk_mq_put_driver_tag (rq->mq_hctx, rq); } +bool blk_mq_get_driver_tag (struct request *rq); +static inline void blk_mq_clear_mq_map (struct blk_mq_queue_map *qmap) {int cpu;Signature · 1 line
2.26.2
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jens Axboe
maintainerQuoted 32 lines · Jan Kara
Quoted 31 lines · Jan Kara
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in
__blk_mq_do_dispatch_sched()pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because
__blk_mq_do_dispatch_sched()always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.So grab request tag for a request pulled out of the IO scheduler already in
__blk_mq_do_dispatch_sched()and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.Reviewed-by Ming Lei <[email protected]>Signed-off-by Jan Kara <[email protected]>--- block/blk-mq-sched.c | 12 +++++++++++- block/blk-mq.c | 2 +- block/blk-mq.h | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-)
Quoted 3 lines
Quoted 1 line
Jens, can you please merge the patch? Thanks!
Ping Jens?
Didn't I reply? It's merged for a while:
Signature · 1 line
Jens Axboe
Quoted 33 lines · Jens Axboe
Quoted 32 lines · Jan Kara
Quoted 31 lines · Jan Kara
Provided the device driver does not implement dispatch budget accounting (which only SCSI does) the loop in
__blk_mq_do_dispatch_sched()pulls requests from the IO scheduler as long as it is willing to give out any. That defeats scheduling heuristics inside the scheduler by creating false impression that the device can take more IO when it in fact cannot.For example with BFQ IO scheduler on top of virtio-blk device setting blkio cgroup weight has barely any impact on observed throughput of async IO because
__blk_mq_do_dispatch_sched()always sucks out all the IO queued in BFQ. BFQ first submits IO from higher weight cgroups but when that is all dispatched, it will give out IO of lower weight cgroups as well. And then we have to wait for all this IO to be dispatched to the disk (which means lot of it actually has to complete) before the IO scheduler is queried again for dispatching more requests. This completely destroys any service differentiation.So grab request tag for a request pulled out of the IO scheduler already in
__blk_mq_do_dispatch_sched()and do not pull any more requests if we cannot get it because we are unlikely to be able to dispatch it. That way only single request is going to wait in the dispatch list for some tag to free.Reviewed-by Ming Lei <[email protected]>Signed-off-by Jan Kara <[email protected]>--- block/blk-mq-sched.c | 12 +++++++++++- block/blk-mq.c | 2 +- block/blk-mq.h | 2 ++ 3 files changed, 14 insertions(+), 2 deletions(-)
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Jens, can you please merge the patch? Thanks!
Ping Jens?
Didn't I reply? It's merged for a while:
No, I didn't get email back and didn't notice the patch is already in your tree. Sorry for the noise.
Honza
Quoted 6 lines
Signature · 1 line
Jens Axboe
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR