blk-mq and I/O scheduling

Hi Andreas,

I don't understand the time slicing algorithm to well, but from the blk-mq integration perspective this looks nice, and anything that helps improving blk-mq for spinning rust is useful.

As a nitpick some of the larger "if (use_time_slice)" blocks should be moved into separate helper functions‥

Hi,

I've looked into blk-mq and possible support for I/O scheduling.

The reason for this is to minimize performance degradation with rotational devices when scsi_mod.use_blk_mq=1 is switched on.

I think that the degradation is well reflected with fio measurements. With an increasing number of jobs you'll encounter a significant performance drop for sequential reads and writes with blk-mq in contrast to CFQ. blk-mq ensures that requests from different processes (CPUs) are "perfectly shuffled" in a hardware queue. This is no problem for non-rotational devices for which blk-mq is aimed for but not so nice for rotational disks.

(i) I've done some tests with patch c2ed2f2dcf92 (blk-mq: first cut deadline scheduling) from branch mq-deadline of linux-block repository. I've not seen a significant performance impact when enabling it (neither for non-rotational nor for rotational disks).

(ii) I've played with code to enable sorting/merging of requests. I
did this in flush_busy_ctxs. This didn't have a performance impact either. On a closer look this was due to high frequency of calls to __blk_mq_run_hw_queue. There was almost nothing to sort (too few requests). I guess that's also the reason why (i) had not much impact.

(iii) With CFQ I've observed similar performance patterns to blk-mq if
slice_idle was set to 0.

(iv) I thought about introducing a per software queue time slice
during which blk-mq will service only one software queue (one CPU) and not flush all software queues. This could help to enqueue multiple requests belonging to the same process (as long as it runs on same CPU) into a hardware queue. A minimal patch to implement this is attached below.

The latter helped to improve performance for sequential reads and writes. But it's not on a par with CFQ. Increasing the time slice is suboptimal (as shown with the 2ms results, see below). It might be possible to get better performance when further reducing the initial time slice and adapting it up to a maximum value if there are repeatedly pending requests for a CPU.

After these observations and assuming that non-rotational devices are most likely fine using blk-mq without I/O scheduling support I wonder whether

- it's really a good idea to re-implement scheduling support for blk-mq that eventually behaves like CFQ for rotational devices.

- it's technical possible to support both blk-mq and CFQ for different devices on the same host adapter. This would allow to use "good old" code for "good old" rotational devices. (But this might not be a choice if in the long run a goal is to get rid of non-blk-mq code -- not sure what the plans are.)

What do you think about this?

On 11/19/2015 05:02 AM, Andreas Herrmann wrote:

Quoted 14 lines

The latter helped to improve performance for sequential reads and writes. But it's not on a par with CFQ. Increasing the time slice is suboptimal (as shown with the 2ms results, see below). It might be possible to get better performance when further reducing the initial time slice and adapting it up to a maximum value if there are repeatedly pending requests for a CPU.

After these observations and assuming that non-rotational devices are most likely fine using blk-mq without I/O scheduling support I wonder whether

- it's really a good idea to re-implement scheduling support for blk-mq that eventually behaves like CFQ for rotational devices.

Quoted 7 lines

- it's technical possible to support both blk-mq and CFQ for different devices on the same host adapter. This would allow to use "good old" code for "good old" rotational devices. (But this might not be a choice if in the long run a goal is to get rid of non-blk-mq code -- not sure what the plans are.)

What do you think about this?

Sorry I did not get around to properly looking at this this week, I'll tend to it next week. I think the concept of tying the idling to a specific CPU is likely fine, though I wonder if there are cases where we preempt more heavily and subsequently miss breaking the idling properly. I don't think we want/need cfq for blk-mq, but basic idling could potentially be enough. That's still a far cry from a full cfq implementation. The long term plans are still to move away from the legacy IO path, though with things like scheduling, that's sure to take some time.

That is actually where the mq-deadline work comes in. The mq-deadline work is missing a test patch to limit tag allocations, and a bunch of other little things to truly make it functional. There might be some options for folding it all together, with idling, as that would still be important on rotating storage going forward.

This introduces a new blk_mq hw attribute time_slice_us which allows to specify a time slice in usecs.

Default value is 0 and implies no modification to blk-mq behaviour.

A positive value changes blk-mq to service only one software queue within this time slice until it expires or the software queue is empty. Then the next software queue with pending requests is selected.

Signed-off-by Andreas Herrmann <[email protected]>

---
 block/blk-mq-sysfs.c   |  27 +++++++
 block/blk-mq.c         | 208 +++++++++++++++++++++++++++++++++++++++++--------
 include/linux/blk-mq.h |   9 +++
 3 files changed, 211 insertions(+), 33 deletions(-)

Hi,

This update is long overdue (sorry for the delay).

Change to v1:
- time slice is now specified in usecs instead of msecs.
- time slice is extended (up to 3 times the initial value) when there was actually a request to be serviced for the software queue

Fio test results are sent in a separate mail to this.

Results for fio improved to some extent with this patch. But in reality the picture is quite mixed. Performance is highly dependend on task scheduling. There is no guarantee that the requests originated from one CPU belong to the same process.

I think for rotary devices CFQ is by far the best choice. A simple illustration is:

Copying two files (750MB in this case) in parallel on a rotary device. The elapsed wall clock time (seconds) for this is

                               mean    stdev
   cfq, slice_idle=8           16.18   4.95
   cfq, slice_idle=0           23.74   2.82
   blk-mq, time_slice_usec=0   24.37   2.05
   blk-mq, time_slice_usec=250 25.58   3.16

Regards,

Andreas

block/blk-mq-sysfs.c +27 -0

index 1cf1878..77c875c 100644
@@ -247,6 +247,26 @@ static ssize_t blk_mq_hw_sysfs_cpus_show(struct blk_mq_hw_ctx *hctx, char *page)
 	return ret;
 }
 
+static ssize_t blk_mq_hw_sysfs_tslice_show(struct blk_mq_hw_ctx *hctx,
+					  char *page)
+{
+	return sprintf(page, "%u\n", hctx->tslice_us);
+}
+
+static ssize_t blk_mq_hw_sysfs_tslice_store(struct blk_mq_hw_ctx *hctx,
+					const char *page, size_t length)
+{
+	unsigned long long store;
+	int err;
+
+	err = kstrtoull(page, 10, &store);
+	if (err)
+		return -EINVAL;
+
+	hctx->tslice_us = (unsigned)store;
+	return length;
+}
+
 static struct blk_mq_ctx_sysfs_entry blk_mq_sysfs_dispatched = {
 	.attr = {.name = "dispatched", .mode = S_IRUGO },
 	.show = blk_mq_sysfs_dispatched_show,

@@ -305,6 +325,12 @@ static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_poll = {
 	.show = blk_mq_hw_sysfs_poll_show,
 };
 
+static struct blk_mq_hw_ctx_sysfs_entry blk_mq_hw_sysfs_tslice = {
+	.attr = {.name = "time_slice_us", .mode = S_IRUGO | S_IWUSR },
+	.show = blk_mq_hw_sysfs_tslice_show,
+	.store = blk_mq_hw_sysfs_tslice_store,
+};
+
 static struct attribute *default_hw_ctx_attrs[] = {
 	&blk_mq_hw_sysfs_queued.attr,
 	&blk_mq_hw_sysfs_run.attr,

@@ -314,6 +340,7 @@ static struct attribute *default_hw_ctx_attrs[] = {
 	&blk_mq_hw_sysfs_cpus.attr,
 	&blk_mq_hw_sysfs_active.attr,
 	&blk_mq_hw_sysfs_poll.attr,
+	&blk_mq_hw_sysfs_tslice.attr,
 	NULL,
 };
 

block/blk-mq.c +175 -33

index 4c0622f..97d32f2 100644
@@ -68,6 +68,7 @@ static void blk_mq_hctx_mark_pending(struct blk_mq_hw_ctx *hctx,
 
 	if (!test_bit(CTX_TO_BIT(hctx, ctx), &bm->word))
 		set_bit(CTX_TO_BIT(hctx, ctx), &bm->word);
+	cpumask_set_cpu(ctx->cpu, hctx->cpu_pending_mask);
 }
 
 static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx,

@@ -76,6 +77,7 @@ static void blk_mq_hctx_clear_pending(struct blk_mq_hw_ctx *hctx,
 	struct blk_align_bitmap *bm = get_bm(hctx, ctx);
 
 	clear_bit(CTX_TO_BIT(hctx, ctx), &bm->word);
+	cpumask_clear_cpu(ctx->cpu, hctx->cpu_pending_mask);
 }
 
 void blk_mq_freeze_queue_start(struct request_queue *q)

@@ -682,15 +684,41 @@ static bool blk_mq_attempt_merge(struct request_queue *q,
 	return false;
 }
 
+static int tslice_flush_one_ctx(struct blk_mq_hw_ctx *hctx,
+				struct list_head *list,	int cpu)
+{
+	struct request_queue *q = hctx->queue;
+	struct blk_mq_ctx *ctx;
+
+	if (cpu == -1 || !hctx->tslice_us)
+		return 0;
+
+	ctx = __blk_mq_get_ctx(q, cpu);
+	spin_lock(&ctx->lock);
+	if (!list_empty(&ctx->rq_list)) {
+		list_splice_tail_init(&ctx->rq_list, list);
+		spin_lock(&hctx->lock);
+		hctx->tslice_inc = 1;
+		blk_mq_hctx_clear_pending(hctx, ctx);
+		spin_unlock(&hctx->lock);
+	}
+	spin_unlock(&ctx->lock);
+	return 1;
+}
+
 /*
  * Process software queues that have been marked busy, splicing them
  * to the for-dispatch
  */
-static void flush_busy_ctxs(struct blk_mq_hw_ctx *hctx, struct list_head *list)
+static void flush_busy_ctxs(struct blk_mq_hw_ctx *hctx,
+			struct list_head *list,	int cpu)
 {
 	struct blk_mq_ctx *ctx;
 	int i;
 
+	if (tslice_flush_one_ctx(hctx, list, cpu))
+		return;
+
 	for (i = 0; i < hctx->ctx_map.size; i++) {
 		struct blk_align_bitmap *bm = &hctx->ctx_map.map[i];
 		unsigned int off, bit;

@@ -706,9 +734,11 @@ static void flush_busy_ctxs(struct blk_mq_hw_ctx *hctx, struct list_head *list)
 				break;
 
 			ctx = hctx->ctxs[bit + off];
-			clear_bit(bit, &bm->word);
 			spin_lock(&ctx->lock);
 			list_splice_tail_init(&ctx->rq_list, list);
+			spin_lock(&hctx->lock);
+			blk_mq_hctx_clear_pending(hctx, ctx);
+			spin_unlock(&hctx->lock);
 			spin_unlock(&ctx->lock);
 
 			bit++;

@@ -717,6 +747,114 @@ static void flush_busy_ctxs(struct blk_mq_hw_ctx *hctx, struct list_head *list)
 }
 
 /*
+ * It'd be great if the workqueue API had a way to pass
+ * in a mask and had some smarts for more clever placement.
+ * For now we just round-robin here, switching for every
+ * BLK_MQ_CPU_WORK_BATCH queued items.
+ */
+static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
+{
+	if (hctx->queue->nr_hw_queues == 1)
+		return WORK_CPU_UNBOUND;
+
+	if (--hctx->next_cpu_batch <= 0) {
+		int cpu = hctx->next_cpu, next_cpu;
+
+		next_cpu = cpumask_next(hctx->next_cpu, hctx->cpumask);
+		if (next_cpu >= nr_cpu_ids)
+			next_cpu = cpumask_first(hctx->cpumask);
+
+		hctx->next_cpu = next_cpu;
+		hctx->next_cpu_batch = BLK_MQ_CPU_WORK_BATCH;
+
+		return cpu;
+	}
+
+	return hctx->next_cpu;
+}
+
+static int blk_mq_tslice_expired(struct blk_mq_hw_ctx *hctx)
+{
+	if (time_after_eq(jiffies, hctx->tslice_expiration))
+		return 1;
+
+	return 0;
+}
+
+static int blk_mq_tslice_next_cpu(struct blk_mq_hw_ctx *hctx)
+{
+	int c = hctx->tslice_cpu;
+
+	/* allow extension of time slice */
+	if (hctx->tslice_inc && hctx->tslice_inc_count < 4) {
+		hctx->tslice_inc = 0;
+		hctx->tslice_inc_count++;
+		goto out;
+	}
+	hctx->tslice_inc = 0;
+	hctx->tslice_inc_count = 0;
+
+	/* clear CPU for which tslice has expired */
+	if (c != -1)
+		cpumask_clear_cpu(c, hctx->cpu_service_mask);
+
+	/* calculate mask of remaining CPUs with pending work */
+	if (cpumask_and(hctx->cpu_next_mask, hctx->cpu_pending_mask,
+				hctx->cpu_service_mask)) {
+		c = cpumask_first(hctx->cpu_next_mask);
+	} else {
+		/*
+		 * no remaining CPUs with pending work, reset epoch,
+		 * start with first CPU that has requests pending
+		 */
+		hctx->tslice_expiration = 0;
+		cpumask_setall(hctx->cpu_service_mask);
+		c = cpumask_first(hctx->cpu_pending_mask);
+	}
+
+	/* no CPU with pending requests */
+	if (c >= nr_cpu_ids)
+		return -1;
+
+out:
+	hctx->tslice_expiration = jiffies + usecs_to_jiffies(hctx->tslice_us);
+	return c;
+}
+
+static int tslice_get_busy_ctx(struct blk_mq_hw_ctx *hctx)
+{
+	int cpu = -1;
+
+	if (hctx->tslice_us) {
+		spin_lock(&hctx->lock);
+		if (blk_mq_tslice_expired(hctx))
+			hctx->tslice_cpu = blk_mq_tslice_next_cpu(hctx);
+		cpu = hctx->tslice_cpu;
+		spin_unlock(&hctx->lock);
+	}
+
+	return cpu;
+}
+
+/*
+ * Ensure that hardware queue is run if there are pending
+ * requests in any software queue.
+ */
+static void tslice_schedule_pending_work(struct blk_mq_hw_ctx *hctx)
+{
+	long t;
+
+	if (!hctx->tslice_us || cpumask_empty(hctx->cpu_pending_mask))
+		return;
+
+	t = hctx->tslice_expiration - jiffies;
+	if (t < 0)
+		t = 0;
+	kblockd_schedule_delayed_work_on(blk_mq_hctx_next_cpu(hctx),
+					&hctx->run_work, (unsigned int)t);
+}
+
+/*
  * Run this hardware queue, pulling any software queues mapped to it in.
  * Note that this function currently has various problems around ordering
  * of IO. In particular, we'd like FIFO behaviour on handling existing

@@ -729,7 +867,7 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 	LIST_HEAD(rq_list);
 	LIST_HEAD(driver_list);
 	struct list_head *dptr;
-	int queued;
+	int queued, cpu;
 
 	WARN_ON(!cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask));
 

@@ -739,9 +877,11 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 	hctx->run++;
 
 	/*
-	 * Touch any software queue that has pending entries.
+	 * Touch dedicated software queue if time slice is set or any
+	 * software queue that has pending entries (cpu == -1).
 	 */
-	flush_busy_ctxs(hctx, &rq_list);
+	cpu = tslice_get_busy_ctx(hctx);
+	flush_busy_ctxs(hctx, &rq_list, cpu);
 
 	/*
 	 * If we have previous entries on our dispatch list, grab them

@@ -826,34 +966,8 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 		 * blk_mq_run_hw_queue() already checks the STOPPED bit
 		 **/
 		blk_mq_run_hw_queue(hctx, true);
-	}
-}
-
-/*
- * It'd be great if the workqueue API had a way to pass
- * in a mask and had some smarts for more clever placement.
- * For now we just round-robin here, switching for every
- * BLK_MQ_CPU_WORK_BATCH queued items.
- */
-static int blk_mq_hctx_next_cpu(struct blk_mq_hw_ctx *hctx)
-{
-	if (hctx->queue->nr_hw_queues == 1)
-		return WORK_CPU_UNBOUND;
-
-	if (--hctx->next_cpu_batch <= 0) {
-		int cpu = hctx->next_cpu, next_cpu;
-
-		next_cpu = cpumask_next(hctx->next_cpu, hctx->cpumask);
-		if (next_cpu >= nr_cpu_ids)
-			next_cpu = cpumask_first(hctx->cpumask);
-
-		hctx->next_cpu = next_cpu;
-		hctx->next_cpu_batch = BLK_MQ_CPU_WORK_BATCH;
-
-		return cpu;
-	}
-
-	return hctx->next_cpu;
+	} else
+		tslice_schedule_pending_work(hctx);
 }
 
 void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async)

@@ -992,7 +1106,10 @@ static void __blk_mq_insert_request(struct blk_mq_hw_ctx *hctx,
 	struct blk_mq_ctx *ctx = rq->mq_ctx;
 
 	__blk_mq_insert_req_list(hctx, ctx, rq, at_head);
+	spin_lock(&hctx->lock);
 	blk_mq_hctx_mark_pending(hctx, ctx);
+	spin_unlock(&hctx->lock);
+
 }
 
 void blk_mq_insert_request(struct request *rq, bool at_head, bool run_queue,

@@ -1583,7 +1700,9 @@ static int blk_mq_hctx_cpu_offline(struct blk_mq_hw_ctx *hctx, int cpu)
 	spin_lock(&ctx->lock);
 	if (!list_empty(&ctx->rq_list)) {
 		list_splice_init(&ctx->rq_list, &tmp);
+		spin_lock(&hctx->lock);
 		blk_mq_hctx_clear_pending(hctx, ctx);
+		spin_unlock(&hctx->lock);
 	}
 	spin_unlock(&ctx->lock);
 

@@ -1602,7 +1721,9 @@ static int blk_mq_hctx_cpu_offline(struct blk_mq_hw_ctx *hctx, int cpu)
 	}
 
 	hctx = q->mq_ops->map_queue(q, ctx->cpu);
+	spin_lock(&hctx->lock);
 	blk_mq_hctx_mark_pending(hctx, ctx);
+	spin_unlock(&hctx->lock);
 
 	spin_unlock(&ctx->lock);
 

@@ -1691,6 +1812,12 @@ static int blk_mq_init_hctx(struct request_queue *q,
 	hctx->queue_num = hctx_idx;
 	hctx->flags = set->flags & ~BLK_MQ_F_TAG_SHARED;
 
+	hctx->tslice_us = 0;
+	hctx->tslice_expiration = 0;
+	hctx->tslice_cpu = -1;
+	hctx->tslice_inc = 0;
+	hctx->tslice_inc_count = 0;
+
 	blk_mq_init_cpu_notifier(&hctx->cpu_notifier,
 					blk_mq_hctx_notify, hctx);
 	blk_mq_register_cpu_notifier(&hctx->cpu_notifier);

@@ -2006,6 +2133,18 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 						node))
 			goto err_hctxs;
 
+		if (!zalloc_cpumask_var_node(&hctxs[i]->cpu_pending_mask,
+						GFP_KERNEL, node))
+			goto err_hctxs;
+
+		if (!zalloc_cpumask_var_node(&hctxs[i]->cpu_service_mask,
+						GFP_KERNEL, node))
+			goto err_hctxs;
+
+		if (!zalloc_cpumask_var_node(&hctxs[i]->cpu_next_mask,
+						GFP_KERNEL, node))
+			goto err_hctxs;
+
 		atomic_set(&hctxs[i]->nr_active, 0);
 		hctxs[i]->numa_node = node;
 		hctxs[i]->queue_num = i;

@@ -2069,6 +2208,9 @@ err_hctxs:
 		if (!hctxs[i])
 			break;
 		free_cpumask_var(hctxs[i]->cpumask);
+		free_cpumask_var(hctxs[i]->cpu_pending_mask);
+		free_cpumask_var(hctxs[i]->cpu_service_mask);
+		free_cpumask_var(hctxs[i]->cpu_next_mask);
 		kfree(hctxs[i]);
 	}
 err_map:

include/linux/blk-mq.h +9 -0

index 7fc9296..a8ca685 100644
@@ -57,6 +57,15 @@ struct blk_mq_hw_ctx {
 
 	atomic_t		nr_active;
 
+	cpumask_var_t		cpu_service_mask; /* CPUs not yet serviced */
+	cpumask_var_t		cpu_pending_mask; /* CPUs with pending work */
+	cpumask_var_t		cpu_next_mask;	/* CPUs to be serviced */
+	int			tslice_us;	/* time slice in µs */
+	int			tslice_inc;	/* extend time slice */
+	int			tslice_inc_count;
+	unsigned long		tslice_expiration; /* in jiffies */
+	int			tslice_cpu;	/* cpu of time slice */
+
 	struct blk_mq_cpu_notifier	cpu_notifier;
 	struct kobject		kobj;
 

Signature · 1 line

1.9.1

[CC-ing linux-block and linux-scsi and adding some comments]

Quoted 4 lines · Andreas Herrmann

This introduces a new blk_mq hw attribute time_slice_us which allows to specify a time slice in usecs.

Fio test results are sent in a separate mail to this.

See http://marc.info/?l=linux-kernel&m=145436682607949&w=2

In short it shows significant performance gains in some tests, e.g. sequential read iops up by >40% with 8 jobs. But it's never on par with CFQ when more than 1 job was used during the test.

Quoted 8 lines

Results for fio improved to some extent with this patch. But in reality the picture is quite mixed. Performance is highly dependend on task scheduling. There is no guarantee that the requests originated from one CPU belong to the same process.

I think for rotary devices CFQ is by far the best choice. A simple illustration is:

Quoted 7 lines

Copying two files (750MB in this case) in parallel on a rotary device. The elapsed wall clock time (seconds) for this is

                               mean    stdev
   cfq, slice_idle=8           16.18   4.95
   cfq, slice_idle=0           23.74   2.82
   blk-mq, time_slice_usec=0   24.37   2.05
   blk-mq, time_slice_usec=250 25.58   3.16

This illustrates that although their was performance gain with fio tests, the patch can cause higher variance and lower performance in comparison to unmodified blk-mq with other tests. And it underscores superiority of CFQ for rotary disks.

Meanwhile my opinion is that it's not really worth to look further into introduction of I/O scheduling support in blk-mq. I don't see the need for scheduling support (deadline or something else) for fast storage devices. And rotary devices should really avoid usage of blk-mq and stick to CFQ.

Thus I think that introducing some coexistence of blk-mq and the legacy block with CFQ is the best option.

Recently Johannes sent a patch to enable scsi-mq per driver, see
http://marc.info/?l=linux-scsi&m=145347009631192&w=2

Probably that is a good solution (at least in the short term) to allow users to switch to blk-mq for some host adapters (with fast storage attached) but to stick to legacy stuff on other host adapters with rotary devices.

Quoted 47 lines · Andreas Herrmann

[CC-ing linux-block and linux-scsi and adding some comments]

Quoted 4 lines · Andreas Herrmann

This introduces a new blk_mq hw attribute time_slice_us which allows to specify a time slice in usecs.

Fio test results are sent in a separate mail to this.

See http://marc.info/?l=linux-kernel&m=145436682607949&w=2

In short it shows significant performance gains in some tests, e.g. sequential read iops up by >40% with 8 jobs. But it's never on par with CFQ when more than 1 job was used during the test.

Quoted 8 lines

Results for fio improved to some extent with this patch. But in reality the picture is quite mixed. Performance is highly dependend on task scheduling. There is no guarantee that the requests originated from one CPU belong to the same process.

I think for rotary devices CFQ is by far the best choice. A simple illustration is:

Quoted 7 lines

Copying two files (750MB in this case) in parallel on a rotary device. The elapsed wall clock time (seconds) for this is

                               mean    stdev
   cfq, slice_idle=8           16.18   4.95
   cfq, slice_idle=0           23.74   2.82
   blk-mq, time_slice_usec=0   24.37   2.05
   blk-mq, time_slice_usec=250 25.58   3.16

This illustrates that although their was performance gain with fio tests, the patch can cause higher variance and lower performance in comparison to unmodified blk-mq with other tests. And it underscores superiority of CFQ for rotary disks.

Meanwhile my opinion is that it's not really worth to look further into introduction of I/O scheduling support in blk-mq. I don't see the need for scheduling support (deadline or something else) for fast storage devices. And rotary devices should really avoid usage of blk-mq and stick to CFQ.

Thus I think that introducing some coexistence of blk-mq and the legacy block with CFQ is the best option.

Recently Johannes sent a patch to enable scsi-mq per driver, see
http://marc.info/?l=linux-scsi&m=145347009631192&w=2

Quoted 4 lines

Probably that is a good solution (at least in the short term) to allow users to switch to blk-mq for some host adapters (with fast storage attached) but to stick to legacy stuff on other host adapters with rotary devices.

I don't think that Johannes' patch is a good solution.

The best solution for the user would be if blk-mq could be toggled per drive (or even automatically enabled if queue/rotational == 0).

Is there a fundamental reason why this is not feasible?

Your solution is better than nothing, but it requires that the user
finds out the drive <=> host mapping by hand and then runs something
like:
echo "250" > /sys/devices/pci0000:00/0000:00:11.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb/mq/0/time_slice_us during boot for spinning rust drives...

Signature · 1 line

Markus

Quoted 3 lines · Markus Trippelsdorf

Quoted 3 lines

Recently Johannes sent a patch to enable scsi-mq per driver, see
http://marc.info/?l=linux-scsi&m=145347009631192&w=2

Quoted 6 lines

Quoted 4 lines

Probably that is a good solution (at least in the short term) to allow users to switch to blk-mq for some host adapters (with fast storage attached) but to stick to legacy stuff on other host adapters with rotary devices.

I don't think that Johannes' patch is a good solution.

Why? Because it's not per device?

Quoted 2 lines

The best solution for the user would be if blk-mq could be toggled per drive (or even automatically enabled if queue/rotational == 0).

Yes, I aggree, but ...

Quoted 1 line

Is there a fundamental reason why this is not feasible?

... it's not possible (*) with the current implementation.

Tag handling/command allocation differs. Respective functions are set per host.

(*) Or maybe it's possible but just hard to achieve and I didn't look long enough into relevant code to get an idea how to do it.

Quoted 5 lines

Your solution is better than nothing, but it requires that the user
finds out the drive <=> host mapping by hand and then runs something
like:
echo "250" > /sys/devices/pci0000:00/0000:00:11.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb/mq/0/time_slice_us during boot for spinning rust drives...

Or it could automatically be set in case of rotational device. (Once we know for sure that it doesn't cause performance degradation.)

Quoted 4 lines · Andreas Herrmann

Quoted 3 lines · Markus Trippelsdorf

Quoted 3 lines

Recently Johannes sent a patch to enable scsi-mq per driver, see
http://marc.info/?l=linux-scsi&m=145347009631192&w=2

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

Probably that is a good solution (at least in the short term) to allow users to switch to blk-mq for some host adapters (with fast storage attached) but to stick to legacy stuff on other host adapters with rotary devices.

I don't think that Johannes' patch is a good solution.

Why? Because it's not per device?

Yes. Like Christoph said in his reply to the patch: »The host is simply the wrong place to decide these things.«

Quoted 21 lines

Quoted 2 lines

The best solution for the user would be if blk-mq could be toggled per drive (or even automatically enabled if queue/rotational == 0).

Yes, I aggree, but ...

Quoted 1 line

Is there a fundamental reason why this is not feasible?

... it's not possible (*) with the current implementation.

Tag handling/command allocation differs. Respective functions are set per host.

(*) Or maybe it's possible but just hard to achieve and I didn't look long enough into relevant code to get an idea how to do it.

Quoted 5 lines

Your solution is better than nothing, but it requires that the user
finds out the drive <=> host mapping by hand and then runs something
like:
echo "250" > /sys/devices/pci0000:00/0000:00:11.0/ata2/host1/target1:0:0/1:0:0:0/block/sdb/mq/0/time_slice_us during boot for spinning rust drives...

Quoted 2 lines

Or it could automatically be set in case of rotational device. (Once we know for sure that it doesn't cause performance degradation.)

Yes, this sound like a good idea.

But, if I understand things correctly, your patch is only an interim solution until proper I/O scheduler support gets implemented for blk-mq, no?