Skip to contentSkip to conversation

bfq: Improve waker detection

PATCH Applied Jan Kara 2 participants 6 messages May 19, 2022

Subsystems (2)
lore
Share & tools
lore

Jan Kara

author
#0 lore raw

cover bfq: Improve waker detection

Hello,

this patch series restores regression in dbench for large number of processes that was introduced by c65e6fd460b4 ("bfq: Do not let waker requests skip proper accounting"). The detailed explanation is in the first patch but the culprit in the end was that with large number of dbench clients it often happened that flush worker bfqq replaced jbd2 bfqq as a waker of the bfqq shared by dbench clients and that resulted in lot of idling and reduced throughput.

The first two patches in this series improve the waker detection, the other two are just cleanups I've spotted when working on the code.

I've tested the patches and they don't seem to cause regression for other workloads.

Honza

author
#1 lore raw

PATCH 1/4 bfq: Relax waker detection for shared queues

Currently we look for waker only if current queue has no requests. This makes sense for bfq queues with a single process however for shared queues when there is a larger number of processes the condition that queue has no requests is difficult to meet because often at least one process has some request in flight although all the others are waiting for the waker to do the work and this harms throughput. Relax the "no queued request for bfq queue" condition to "the current task has no queued requests yet". For this, we also need to start tracking number of requests in flight for each task.

This patch (together with the following one) restores the performance for dbench with 128 clients that regressed with commit c65e6fd460b4 ("bfq: Do not let waker requests skip proper accounting") because this commit makes requests of wakers properly enter BFQ queues and thus these queues become ineligible for the old waker detection logic. Dbench results:

         Vanilla 5.18-rc3        5.18-rc3 + revert      5.18-rc3 patched
Mean     1237.36 (   0.00%)      950.16 *  23.21%*      988.35 *  20.12%*

Numbers are time to complete workload so lower is better.

Fixes c65e6fd460b4 ("bfq: Do not let waker requests skip proper accounting")
---
 block/bfq-iosched.c | 5 +++--
 block/bfq-iosched.h | 1 +
 2 files changed, 4 insertions(+), 2 deletions(-)
block/bfq-iosched.c +3 -2
index 2e0dd68a3cbe..397f6be3c8fe 100644
@@ -2127,7 +2127,6 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 	if (!bfqd->last_completed_rq_bfqq ||
 	    bfqd->last_completed_rq_bfqq == bfqq ||
 	    bfq_bfqq_has_short_ttime(bfqq) ||
-	    bfqq->dispatched > 0 ||
 	    now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC ||
 	    bfqd->last_completed_rq_bfqq == bfqq->waker_bfqq)
 		return;
@@ -2204,7 +2203,7 @@ static void bfq_add_request(struct request *rq)
 	bfqq->queued[rq_is_sync(rq)]++;
 	bfqd->queued++;
 
-	if (RB_EMPTY_ROOT(&bfqq->sort_list) && bfq_bfqq_sync(bfqq)) {
+	if (bfq_bfqq_sync(bfqq) && RQ_BIC(rq)->requests <= 1) {
 		bfq_check_waker(bfqd, bfqq, now_ns);
 
 		/*
@@ -6557,6 +6556,7 @@ static void bfq_finish_requeue_request(struct request *rq)
 		bfq_completed_request(bfqq, bfqd);
 	}
 	bfq_finish_requeue_request_body(bfqq);
+	RQ_BIC(rq)->requests--;
 	spin_unlock_irqrestore(&bfqd->lock, flags);
 
 	/*
@@ -6790,6 +6790,7 @@ static struct bfq_queue *bfq_init_rq(struct request *rq)
 
 	bfqq_request_allocated(bfqq);
 	bfqq->ref++;
+	bic->requests++;
 	bfq_log_bfqq(bfqd, bfqq, "get_request %p: bfqq %p, %d",
 		     rq, bfqq, bfqq->ref);
 
block/bfq-iosched.h +1 -0
index 3b83e3d1c2e5..25fada961bc9 100644
@@ -468,6 +468,7 @@ struct bfq_io_cq {
 	struct bfq_queue *stable_merge_bfqq;
 
 	bool stably_merged;	/* non splittable if true */
+	unsigned int requests;	/* Number of requests this process has in flight */
 };
 
 /**
Signature · 1 line
2.35.3
author
#2 lore raw

PATCH 2/4 bfq: Allow current waker to defend against a tentative one

In reply to Jan Kara

The code in bfq_check_waker() ignores wake up events from the current waker. This makes it more likely we select a new tentative waker although the current one is generating more wake up events. Treat current waker the same way as any other process and allow it to reset the waker detection logic.

Fixes 71217df39dc6 ("block, bfq: make waker-queue detection more robust")
---
 block/bfq-iosched.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
block/bfq-iosched.c +1 -2
index 397f6be3c8fe..1cc242c90fb6 100644
@@ -2127,8 +2127,7 @@ static void bfq_check_waker(struct bfq_data *bfqd, struct bfq_queue *bfqq,
 	if (!bfqd->last_completed_rq_bfqq ||
 	    bfqd->last_completed_rq_bfqq == bfqq ||
 	    bfq_bfqq_has_short_ttime(bfqq) ||
-	    now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC ||
-	    bfqd->last_completed_rq_bfqq == bfqq->waker_bfqq)
+	    now_ns - bfqd->last_completion >= 4 * NSEC_PER_MSEC)
 		return;
 
 	/*
Signature · 1 line
2.35.3
author
#3 lore raw

PATCH 3/4 bfq: Remove superfluous conversion from RQ_BIC()

In reply to Jan Kara

We store struct bfq_io_cq pointer in rq->elv.priv[0] in bfq_init_rq(). Thus a call to icq_to_bic() in RQ_BIC() is wrong. Luckily it does no harm currently because struct io_iq is the first one in struct bfq_io_cq.

---
 block/bfq-iosched.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/bfq-iosched.c +1 -1
index 1cc242c90fb6..904fe56495ce 100644
@@ -374,7 +374,7 @@ static const unsigned long bfq_activation_stable_merging = 600;
  */
 static const unsigned long bfq_late_stable_merging = 600;
 
-#define RQ_BIC(rq)		icq_to_bic((rq)->elv.priv[0])
+#define RQ_BIC(rq)		((struct bfq_io_cq *)((rq)->elv.priv[0]))
 #define RQ_BFQQ(rq)		((rq)->elv.priv[1])
 
 struct bfq_queue *bic_to_bfqq(struct bfq_io_cq *bic, bool is_sync)
Signature · 1 line
2.35.3
author
#4 lore raw

PATCH 4/4 bfq: Remove bfq_requeue_request_body()

In reply to Jan Kara

The function has only a single caller and two lines. Just remove it since it is pointless and just harming readability.

---
 block/bfq-iosched.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
block/bfq-iosched.c +2 -7
index 904fe56495ce..a1127dfe647e 100644
@@ -6352,12 +6352,6 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd)
 		bfq_schedule_dispatch(bfqd);
 }
 
-static void bfq_finish_requeue_request_body(struct bfq_queue *bfqq)
-{
-	bfqq_request_freed(bfqq);
-	bfq_put_queue(bfqq);
-}
-
 /*
  * The processes associated with bfqq may happen to generate their
  * cumulative I/O at a lower rate than the rate at which the device
@@ -6554,7 +6548,8 @@ static void bfq_finish_requeue_request(struct request *rq)
 
 		bfq_completed_request(bfqq, bfqd);
 	}
-	bfq_finish_requeue_request_body(bfqq);
+	bfqq_request_freed(bfqq);
+	bfq_put_queue(bfqq);
 	RQ_BIC(rq)->requests--;
 	spin_unlock_irqrestore(&bfqd->lock, flags);
 
Signature · 1 line
2.35.3
maintainer Applied
#5 lore raw
In reply to Jan Kara
Quoted 9 lines · Jan Kara

this patch series restores regression in dbench for large number of processes that was introduced by c65e6fd460b4 ("bfq: Do not let waker requests skip proper accounting"). The detailed explanation is in the first patch but the culprit in the end was that with large number of dbench clients it often happened that flush worker bfqq replaced jbd2 bfqq as a waker of the bfqq shared by dbench clients and that resulted in lot of idling and reduced throughput.

[...]

Applied, thanks!

[1/4] bfq: Relax waker detection for shared queues

      commit: f950667356ce90a41b446b726d4595a10cb65415
[2/4] bfq: Allow current waker to defend against a tentative one
      commit: c5ac56bb6110e42e79d3106866658376b2e48ab9
[3/4] bfq: Remove superfluous conversion from RQ_BIC()
      commit: e79cf8892e332b9dafc99aef02189a2897eced24
[4/4] bfq: Remove bfq_requeue_request_body()
      commit: a249ca7dfbce1eb82bcd3a5a6bb21daeade20469

Best regards,

Signature · 1 line
Jens Axboe