Skip to contentSkip to conversation

Reduce overhead of CONFIG_BLK_CGROUP

PATCH In review Jens Axboe 2 participants 4 messages Oct 5, 2021

Subsystems (2)
Block layer cgroup blkio
lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

cover Reduce overhead of CONFIG_BLK_CGROUP

Hi,

As some are probably aware, I've been doing efficiency chasing for block devices using io_uring. Currently we can do about 5.1M IOPS using a single logical core of a single physical core, but that's with a kernel config that's shaved down. Enabling BLK_CGROUP and performance drops to ~3.6M for the same test.

These two patches bring us to 3.9M, which is a nice improvement. More work to be done here.

Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH 1/2 block: move blk-throtl fast path inline

Even if no policies are defined, we spend ~2% of the total IO time checking. Move the fast path inline.

Signed-off-by Jens Axboe <[email protected]>
---
 block/blk-cgroup.c   |   1 +
 block/blk-core.c     |   1 +
 block/blk-merge.c    |   1 +
 block/blk-sysfs.c    |   1 +
 block/blk-throttle.c | 161 +-------------------------------------
 block/blk-throttle.h | 182 +++++++++++++++++++++++++++++++++++++++++++
 block/blk.h          |  16 ----
 7 files changed, 189 insertions(+), 174 deletions(-)
 create mode 100644 block/blk-throttle.h
block/blk-cgroup.c +1 -0
index 38b9f7684952..eb48090eefce 100644
@@ -32,6 +32,7 @@
 #include <linux/psi.h>
 #include "blk.h"
 #include "blk-ioprio.h"
+#include "blk-throttle.h"
 
 /*
  * blkcg_pol_mutex protects blkcg_policy[] and policy [de]activation.
block/blk-core.c +1 -0
index 532c817525de..d83e56b2f64e 100644
@@ -51,6 +51,7 @@
 #include "blk-mq-sched.h"
 #include "blk-pm.h"
 #include "blk-rq-qos.h"
+#include "blk-throttle.h"
 
 struct dentry *blk_debugfs_root;
block/blk-merge.c +1 -0
index 5b4f23014df8..bf7aedaad8f8 100644
@@ -13,6 +13,7 @@
 
 #include "blk.h"
 #include "blk-rq-qos.h"
+#include "blk-throttle.h"
 
 static inline bool bio_will_gap(struct request_queue *q,
 		struct request *prev_rq, struct bio *prev, struct bio *next)
block/blk-sysfs.c +1 -0
index 614d9d47de36..da883efcba33 100644
@@ -17,6 +17,7 @@
 #include "blk-mq.h"
 #include "blk-mq-debugfs.h"
 #include "blk-wbt.h"
+#include "blk-throttle.h"
 
 struct queue_sysfs_entry {
 	struct attribute attr;
block/blk-throttle.c +3 -158

218 lines not on this page · raw message

block/blk-throttle.h new file +182 -0

182 lines not on this page · raw message

block/blk.h +0 -16
index 762af963b302..21283541a99f 100644
@@ -323,22 +323,6 @@ void ioc_clear_queue(struct request_queue *q);
 
 int create_task_io_context(struct task_struct *task, gfp_t gfp_mask, int node);
 
-/*
- * Internal throttling interface
- */
-#ifdef CONFIG_BLK_DEV_THROTTLING
-extern int blk_throtl_init(struct request_queue *q);
-extern void blk_throtl_exit(struct request_queue *q);
-extern void blk_throtl_register_queue(struct request_queue *q);
-extern void blk_throtl_charge_bio_split(struct bio *bio);
-bool blk_throtl_bio(struct bio *bio);
-#else /* CONFIG_BLK_DEV_THROTTLING */
-static inline int blk_throtl_init(struct request_queue *q) { return 0; }
-static inline void blk_throtl_exit(struct request_queue *q) { }
-static inline void blk_throtl_register_queue(struct request_queue *q) { }
-static inline void blk_throtl_charge_bio_split(struct bio *bio) { }
-static inline bool blk_throtl_bio(struct bio *bio) { return false; }
-#endif /* CONFIG_BLK_DEV_THROTTLING */
 #ifdef CONFIG_BLK_DEV_THROTTLING_LOW
 extern ssize_t blk_throtl_sample_time_show(struct request_queue *q, char *page);
 extern ssize_t blk_throtl_sample_time_store(struct request_queue *q,
Signature · 1 line
2.33.0
author maintainer
#2 lore raw

PATCH 2/2 block: inherit request start time from bio for BLK_CGROUP

In reply to Jens Axboe

Doing high IOPS testing with blk-cgroups enabled spends ~15-20% of the time just doing ktime_get_ns() -> readtsc. We essentially read and set the start time twice, one for the bio and then again when that bio is mapped to a request.

Given that the time between the two is very short, inherit the bio start time instead of reading it again. This cuts 1/3rd of the overhead of the time keeping.

Signed-off-by Jens Axboe <[email protected]>
---
 block/blk-mq.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
block/blk-mq.c +8 -1
index a8c437afc2c3..a40c94505680 100644
@@ -718,7 +718,14 @@ void blk_mq_start_request(struct request *rq)
 	trace_block_rq_issue(rq);
 
 	if (test_bit(QUEUE_FLAG_STATS, &q->queue_flags)) {
-		rq->io_start_time_ns = ktime_get_ns();
+		u64 start_time;
+#ifdef CONFIG_BLK_CGROUP
+		if (rq->bio)
+			start_time = bio_issue_time(&rq->bio->bi_issue);
+		else
+#endif
+			start_time = ktime_get_ns();
+		rq->io_start_time_ns = start_time;
 		rq->stats_sectors = blk_rq_sectors(rq);
 		rq->rq_flags |= RQF_STATS;
 		rq_qos_issue(q, rq);
Signature · 1 line
2.33.0
maintainer Acked-by on the v1 series
#3 lore raw
In reply to Jens Axboe
Quoted 10 lines · Jens Axboe

Hi,

As some are probably aware, I've been doing efficiency chasing for block devices using io_uring. Currently we can do about 5.1M IOPS using a single logical core of a single physical core, but that's with a kernel config that's shaved down. Enabling BLK_CGROUP and performance drops to ~3.6M for the same test.

These two patches bring us to 3.9M, which is a nice improvement. More work to be done here.

Acked-by Tejun Heo <[email protected]>

Thanks.

Signature · 1 line
tejun