bfq: Improve waker detection
Subsystems (2)
cover bfq: Improve waker detection
Hello,
this patch series restores regression in dbench for large number of processes that was introduced by c65e6fd460b4 ("bfq: Do not let waker requests skip proper accounting"). The detailed explanation is in the first patch but the culprit in the end was that with large number of dbench clients it often happened that flush worker bfqq replaced jbd2 bfqq as a waker of the bfqq shared by dbench clients and that resulted in lot of idling and reduced throughput.
The first two patches in this series improve the waker detection, the other two are just cleanups I've spotted when working on the code.
I've tested the patches and they don't seem to cause regression for other workloads.
Honza
PATCH 1/4 bfq: Relax waker detection for shared queues
Currently we look for waker only if current queue has no requests. This makes sense for bfq queues with a single process however for shared queues when there is a larger number of processes the condition that queue has no requests is difficult to meet because often at least one process has some request in flight although all the others are waiting for the waker to do the work and this harms throughput. Relax the "no queued request for bfq queue" condition to "the current task has no queued requests yet". For this, we also need to start tracking number of requests in flight for each task.
This patch (together with the following one) restores the performance for dbench with 128 clients that regressed with commit c65e6fd460b4 ("bfq: Do not let waker requests skip proper accounting") because this commit makes requests of wakers properly enter BFQ queues and thus these queues become ineligible for the old waker detection logic. Dbench results:
Vanilla 5.18-rc3 5.18-rc3 + revert 5.18-rc3 patched Mean 1237.36 ( 0.00%) 950.16 * 23.21%* 988.35 * 20.12%*
Numbers are time to complete workload so lower is better.
--- block/bfq-iosched.c | 5 +++-- block/bfq-iosched.h | 1 + 2 files changed, 4 insertions(+), 2 deletions(-)
block/bfq-iosched.c +3 -2
@@ -2127,7 +2127,6 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq,if (!bfqd->last_completed_rq_bfqq || bfqd->last_completed_rq_bfqq == bfqq ||bfq_bfqq_has_short_ttime (bfqq) || - bfqq->dispatched >0 || now_ns - bfqd->last_completion >=4 *NSEC_PER_MSEC || bfqd->last_completed_rq_bfqq == bfqq->waker_bfqq)return ;
@@ -2204,7 +2203,7 @@ static void bfq_add_request(struct request *rq) bfqq->queued[rq_is_sync (rq)]++; bfqd->queued++; -if (RB_EMPTY_ROOT (&bfqq->sort_list) &&bfq_bfqq_sync (bfqq)) { +if (bfq_bfqq_sync (bfqq) &&RQ_BIC (rq)->requests <=1 ) {bfq_check_waker (bfqd, bfqq, now_ns);/*
@@ -6557,6 +6556,7 @@ static void bfq_finish_requeue_request(struct request *rq)bfq_completed_request (bfqq, bfqd); }bfq_finish_requeue_request_body (bfqq); +RQ_BIC (rq)->requests--;spin_unlock_irqrestore (&bfqd->lock, flags);/*
@@ -6790,6 +6790,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq)bfqq_request_allocated (bfqq); bfqq->ref++; + bic->requests++;bfq_log_bfqq (bfqd, bfqq,"get_request %p: bfqq %p, %d" , rq, bfqq, bfqq->ref);
block/bfq-iosched.h +1 -0
@@ -468,6 +468,7 @@ struct bfq_io_cq {struct bfq_queue *stable_merge_bfqq;bool stably_merged;/* non splittable if true */ +unsigned int requests;/* Number of requests this process has in flight */ };/**
Signature · 1 line
2.35.3
PATCH 2/4 bfq: Allow current waker to defend against a tentative one
In reply to Jan KaraThe code in bfq_check_waker() ignores wake up events from the current waker. This makes it more likely we select a new tentative waker although the current one is generating more wake up events. Treat current waker the same way as any other process and allow it to reset the waker detection logic.
--- block/bfq-iosched.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
block/bfq-iosched.c +1 -2
@@ -2127,8 +2127,7 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq,if (!bfqd->last_completed_rq_bfqq || bfqd->last_completed_rq_bfqq == bfqq ||bfq_bfqq_has_short_ttime (bfqq) || - now_ns - bfqd->last_completion >=4 *NSEC_PER_MSEC || - bfqd->last_completed_rq_bfqq == bfqq->waker_bfqq) + now_ns - bfqd->last_completion >=4 *NSEC_PER_MSEC )return ;/*
Signature · 1 line
2.35.3
PATCH 3/4 bfq: Remove superfluous conversion from RQ_BIC()
In reply to Jan KaraWe store struct bfq_io_cq pointer in rq->elv.priv[0] in bfq_init_rq(). Thus a call to icq_to_bic() in RQ_BIC() is wrong. Luckily it does no harm currently because struct io_iq is the first one in struct bfq_io_cq.
--- block/bfq-iosched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
block/bfq-iosched.c +1 -1
@@ -374,7 +374,7 @@ static const unsigned long bfq_activation_stable_merging = 600;*/ static const unsigned long bfq_late_stable_merging =600 ; -#define RQ_BIC (rq) icq_to_bic((rq)->elv.priv[0]) +#define RQ_BIC (rq) ((struct bfq_io_cq *)((rq)->elv.priv[0]))#define RQ_BFQQ (rq) ((rq)->elv.priv[1])struct bfq_queue *bic_to_bfqq (struct bfq_io_cq *bic,bool is_sync)
Signature · 1 line
2.35.3
PATCH 4/4 bfq: Remove bfq_requeue_request_body()
In reply to Jan KaraThe function has only a single caller and two lines. Just remove it since it is pointless and just harming readability.
--- block/bfq-iosched.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
block/bfq-iosched.c +2 -7
@@ -6352,12 +6352,6 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)bfq_schedule_dispatch (bfqd); } -static void bfq_finish_requeue_request_body (struct bfq_queue *bfqq) -{ -bfqq_request_freed (bfqq); -bfq_put_queue (bfqq); -} -/* * The processes associated with bfqq may happen to generate their * cumulative I/O at a lower rate than the rate at which the device
@@ -6554,7 +6548,8 @@ static void bfq_finish_requeue_request(struct request *rq)bfq_completed_request (bfqq, bfqd); } -bfq_finish_requeue_request_body (bfqq); +bfqq_request_freed (bfqq); +bfq_put_queue (bfqq);RQ_BIC (rq)->requests--;spin_unlock_irqrestore (&bfqd->lock, flags);
Signature · 1 line
2.35.3
Quoted 9 lines · Jan Kara
this patch series restores regression in dbench for large number of processes that was introduced by
c65e6fd460b4("bfq: Do not let waker requests skip proper accounting"). The detailed explanation is in the first patch but the culprit in the end was that with large number of dbench clients it often happened that flush worker bfqq replaced jbd2 bfqq as a waker of the bfqq shared by dbench clients and that resulted in lot of idling and reduced throughput.[...]
Applied, thanks!
[1/4] bfq: Relax waker detection for shared queues
commit: f950667356ce90a41b446b726d4595a10cb65415
[2/4] bfq: Allow current waker to defend against a tentative one
commit: c5ac56bb6110e42e79d3106866658376b2e48ab9
[3/4] bfq: Remove superfluous conversion from RQ_BIC()
commit: e79cf8892e332b9dafc99aef02189a2897eced24
[4/4] bfq: Remove bfq_requeue_request_body()
commit: a249ca7dfbce1eb82bcd3a5a6bb21daeade20469Best regards,
Signature · 1 line
Jens Axboe