Skip to contentSkip to conversation

block, bfq: set default slice_idle to zero for SSDs

PATCH In review Pradeep P V K 2 participants 2 messages Nov 13, 2019 – Nov 18, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 block, bfq: set default slice_idle to zero for SSDs

With default 8ms as a slice idle time, we seen few time bounded applications(sensors) on v4.19 kernel are getting timedout during multimedia tests (audio, video playbacks etc) with Reboots and leading to crash. The timeout configured for these applications (sensors) are 20sec.

In crash dumps, we seen few synchronous requests from sensors/other applications were in their bfq_queues for more than 12-20sec.

Idling due to anticipation of future near-by IO requests and wait on completion of submitted requests, will effect in choosing the next bfq-queue and its scheduling. There by it effecting some time bounded applications.

After making the slice idle to zero, we didn't seen any crash during our 72hrs of testing and also it increases the IO throughput.

Following FIO benchmark results were taken on a local SSD run:

RandomReads that were taken on v4.19 kernel:

Idling iops avg-lat(us) stddev bw
----------------------------------------------------

 On       4136    1189.07        17221.65    16.9MB/s
 Off      7246     670.11        1054.76     29.7MB/s
    fio --name=temp --size=5G --time_based --ioengine=sync \
	--randrepeat=0 --direct=1 --invalidate=1 --verify=0 \
	--verify_fatal=0 --rw=randread --blocksize=4k \
	--group_reporting=1 --directory=/data --runtime=10 \
	--iodepth=64 --numjobs=5

Following code changes were made based on [1],[2] and [3].

[1] https://lkml.org/lkml/2018/11/1/1285
[2] Commit 41c0126b3f22 ("block: Make CFQ default to IOPS mode on

    SSDs")
[3] Commit 0bb979472a74 ("cfq-iosched: fix the setting of IOPS mode on
    SSDs")
---
 Documentation/block/bfq-iosched.rst |  7 ++++---
 block/bfq-iosched.c                 | 13 +++++++++++++
 block/elevator.c                    |  2 ++
 include/linux/elevator.h            |  1 +
 4 files changed, 20 insertions(+), 3 deletions(-)
Documentation/block/bfq-iosched.rst +4 -3
index 0d237d4..244f4ca 100644
@@ -329,9 +329,10 @@ slice_idle
 
 This parameter specifies how long BFQ should idle for next I/O
 request, when certain sync BFQ queues become empty. By default
-slice_idle is a non-zero value. Idling has a double purpose: boosting
-throughput and making sure that the desired throughput distribution is
-respected (see the description of how BFQ works, and, if needed, the
+slice_idle is a non-zero value for rotational devices.
+Idling has a double purpose: boosting throughput and making
+sure that the desired throughput distribution is respected
+(see the description of how BFQ works, and, if needed, the
 papers referred there).
 
 As for throughput, idling can be very helpful on highly seeky media
block/bfq-iosched.c +13 -0
index 0319d63..9c994d1 100644
@@ -6514,6 +6514,18 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e)
 	return -ENOMEM;
 }
 
+static void bfq_registered_queue(struct request_queue *q)
+{
+	struct elevator_queue *e = q->elevator;
+	struct bfq_data *bfqd = e->elevator_data;
+
+	/*
+	 * Default to IOPS mode with no idling for SSDs
+	 */
+	if (blk_queue_nonrot(q))
+		bfqd->bfq_slice_idle = 0;
+}
+
 static void bfq_slab_kill(void)
 {
 	kmem_cache_destroy(bfq_pool);
@@ -6761,6 +6773,7 @@ static ssize_t bfq_low_latency_store(struct elevator_queue *e,
 		.init_hctx		= bfq_init_hctx,
 		.init_sched		= bfq_init_queue,
 		.exit_sched		= bfq_exit_queue,
+		.elevator_registered_fn = bfq_registered_queue,
 	},
 
 	.icq_size =		sizeof(struct bfq_io_cq),
block/elevator.c +2 -0
index 076ba73..b882d25 100644
@@ -504,6 +504,8 @@ int elv_register_queue(struct request_queue *q, bool uevent)
 			kobject_uevent(&e->kobj, KOBJ_ADD);
 
 		e->registered = 1;
+		if (e->type->ops.elevator_registered_fn)
+			e->type->ops.elevator_registered_fn(q);
 	}
 	return error;
 }
include/linux/elevator.h +1 -0
index 901bda3..23dcc35 100644
@@ -50,6 +50,7 @@ struct elevator_mq_ops {
 	struct request *(*next_request)(struct request_queue *, struct request *);
 	void (*init_icq)(struct io_cq *);
 	void (*exit_icq)(struct io_cq *);
+	void (*elevator_registered_fn)(struct request_queue *q);
 };
 
 #define ELV_NAME_MAX	(16)
Signature · 1 line
1.9.1
#1 lore raw
Quoted 13 lines

Il giorno 13 nov 2019, alle ore 15:53, Pradeep P V K <[email protected]> ha scritto:

With default 8ms as a slice idle time, we seen few time bounded applications(sensors) on v4.19 kernel are getting timedout during multimedia tests (audio, video playbacks etc) with Reboots and leading to crash. The timeout configured for these applications (sensors) are 20sec.

In crash dumps, we seen few synchronous requests from sensors/other applications were in their bfq_queues for more than 12-20sec.

Quoted 6 lines

Idling due to anticipation of future near-by IO requests and wait on completion of submitted requests, will effect in choosing the next bfq-queue and its scheduling. There by it effecting some time bounded applications.

After making the slice idle to zero,

As written in the comments that your patch modifies, idling is essential for controlling I/O. So your change is unacceptable unfortunately.

I would recommend you to analyze carefully why this anomaly occurs with non-zero slice idle. I'd be willing to help in that.

Alternatively, if you don't want to waste your time finding out the cause of this problem, then just set slice_idle to 0 manually for your application.

Quoted 12 lines

we didn't seen any crash during
our 72hrs of testing and also it increases the IO throughput.

Following FIO benchmark results were taken on a local SSD run:

RandomReads that were taken on v4.19 kernel:

Idling iops avg-lat(us) stddev bw
----------------------------------------------------

On       4136    1189.07        17221.65    16.9MB/s
Off      7246     670.11        1054.76     29.7MB/s

This is anomalous too. Probably the kernel you are using lacks commits made after 4.19 (around one hundred commits IIRC).

Thanks,
Paolo

Quoted 99 lines
   fio --name=temp --size=5G --time_based --ioengine=sync \
	--randrepeat=0 --direct=1 --invalidate=1 --verify=0 \
	--verify_fatal=0 --rw=randread --blocksize=4k \
	--group_reporting=1 --directory=/data --runtime=10 \
	--iodepth=64 --numjobs=5

Following code changes were made based on [1],[2] and [3].

[1] https://lkml.org/lkml/2018/11/1/1285
[2] Commit 41c0126b3f22 ("block: Make CFQ default to IOPS mode on SSDs")
[3] Commit 0bb979472a74 ("cfq-iosched: fix the setting of IOPS mode on SSDs")

---

Documentation/block/bfq-iosched.rst |  7 ++++---
block/bfq-iosched.c                 | 13 +++++++++++++
block/elevator.c                    |  2 ++
include/linux/elevator.h            |  1 +
4 files changed, 20 insertions(+), 3 deletions(-)
Documentation/block/bfq-iosched.rst +0 -0
index 0d237d4..244f4ca 100644
@@ -329,9 +329,10 @@ slice_idle
 

This parameter specifies how long BFQ should idle for next I/O request, when certain sync BFQ queues become empty. By default

Documentation/block/bfq-iosched.rst +4 -3
-slice_idle is a non-zero value. Idling has a double purpose: boosting
-throughput and making sure that the desired throughput distribution is
-respected (see the description of how BFQ works, and, if needed, the
+slice_idle is a non-zero value for rotational devices.
+Idling has a double purpose: boosting throughput and making
+sure that the desired throughput distribution is respected
+(see the description of how BFQ works, and, if needed, the

papers referred there).

As for throughput, idling can be very helpful on highly seeky media

block/bfq-iosched.c +13 -0
index 0319d63..9c994d1 100644
@@ -6514,6 +6514,18 @@ static int bfq_init_queue(struct request_queue *q, struct elevator_type *e)
 	return -ENOMEM;
 }
 
+static void bfq_registered_queue(struct request_queue *q)
+{
+	struct elevator_queue *e = q->elevator;
+	struct bfq_data *bfqd = e->elevator_data;
+
+	/*
+	 * Default to IOPS mode with no idling for SSDs
+	 */
+	if (blk_queue_nonrot(q))
+		bfqd->bfq_slice_idle = 0;
+}
+
 static void bfq_slab_kill(void)
 {
 	kmem_cache_destroy(bfq_pool);
@@ -6761,6 +6773,7 @@ static ssize_t bfq_low_latency_store(struct elevator_queue *e,
 		.init_hctx		= bfq_init_hctx,
 		.init_sched		= bfq_init_queue,
 		.exit_sched		= bfq_exit_queue,
+		.elevator_registered_fn = bfq_registered_queue,
 	},
 
 	.icq_size =		sizeof(struct bfq_io_cq),
block/elevator.c +2 -0
index 076ba73..b882d25 100644
@@ -504,6 +504,8 @@ int elv_register_queue(struct request_queue *q, bool uevent)
 			kobject_uevent(&e->kobj, KOBJ_ADD);
 
 		e->registered = 1;
+		if (e->type->ops.elevator_registered_fn)
+			e->type->ops.elevator_registered_fn(q);
 	}
 	return error;
 }
include/linux/elevator.h +1 -0
index 901bda3..23dcc35 100644
@@ -50,6 +50,7 @@ struct elevator_mq_ops {
 	struct request *(*next_request)(struct request_queue *, struct request *);
 	void (*init_icq)(struct io_cq *);
 	void (*exit_icq)(struct io_cq *);
+	void (*elevator_registered_fn)(struct request_queue *q);
 };
 
 #define ELV_NAME_MAX	(16)
Signature · 1 line
1.9.1