Skip to contentSkip to conversation

cfq-iosched: don't idle if a deep seek queue is slow

PATCH In review Shaohua Li 3 participants 6 messages Nov 8, 2010 – Nov 9, 2010 Block layer

lore
Share & tools
lore

Shaohua Li

author
#0 lore raw

PATCH 3/3 cfq-iosched: don't idle if a deep seek queue is slow

If a deep seek queue slowly deliver requests but disk is much faster, idle for the queue just wastes disk throughput. If the queue delevers all requests before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept 128 requests at maxium and disk is fast. without the patch, the throughput is just around 30m/s, while with it, the speed is about 80m/s. The disk is a SSD, but is detected as a rotational disk. I can configure it as SSD, but I thought the deep seek queue logic should be fixed too, for example, considering a fast raid.

---
 block/cfq-iosched.c |   11 +++++++++++
 1 file changed, 11 insertions(+)
block/cfq-iosched.c +11 -0
Index: linux/block/cfq-iosched.c
===================================================================
@@ -2293,6 +2293,17 @@ static struct cfq_queue *cfq_select_queu
 		goto keep_queue;
 	}
 
+	/*
+	 * This is a deep seek queue, but the device is much faster than
+	 * the queue can deliver, don't idle
+	 **/
+	if (CFQQ_SEEKY(cfqq) && cfq_cfqq_idle_window(cfqq) &&
+	    (cfq_cfqq_slice_new(cfqq) ||
+	    (cfqq->slice_end - jiffies > jiffies - cfqq->slice_start))) {
+		cfq_clear_cfqq_deep(cfqq);
+		cfq_clear_cfqq_idle_window(cfqq);
+	}
+
 	if (cfqq->dispatched && cfq_should_idle(cfqd, cfqq)) {
 		cfqq = NULL;
 		goto keep_queue;
#1 lore raw
Quoted 10 lines · Shaohua Li

If a deep seek queue slowly deliver requests but disk is much faster, idle for the queue just wastes disk throughput. If the queue delevers all requests before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept 128 requests at maxium and disk is fast. without the patch, the throughput is just around 30m/s, while with it, the speed is about 80m/s. The disk is a SSD, but is detected as a rotational disk. I can configure it as SSD, but I thought the deep seek queue logic should be fixed too, for example, considering a fast raid.

Hi Shaohua,

So looks like you are trying to cut down queue idling in the case when device is fast and idling hurts. That's a noble goal, just that detetction of this condition only for deep queues does not seem to cover lots of cases. Manually one can set slice_idle=0 to handle this situation.

What about if you have lots of sequential queues (not deep) and they all will still idle.

Secondly, what if driver is just buffering lots of requests in its device queue and not necessarily device is processing the reuqests faster.

So I think it is a good idea to cut down on idling if we can find that underlying device is fast and idling on queue might hurt more. But discovering this only using deep queues does not sound very appleaing to me. This is help only a particular workload which is driving deep queues. So if there was a generic mechanism to tackle this, that would be much better.

Vivek

Quoted 29 lines
---
 block/cfq-iosched.c |   11 +++++++++++
 1 file changed, 11 insertions(+)

Index: linux/block/cfq-iosched.c
===================================================================

block/cfq-iosched.c +11 -0
@@ -2293,6 +2293,17 @@ static struct cfq_queue *cfq_select_queu
 		goto keep_queue;
 	}
 
+	/*
+	 * This is a deep seek queue, but the device is much faster than
+	 * the queue can deliver, don't idle
+	 **/
+	if (CFQQ_SEEKY(cfqq) && cfq_cfqq_idle_window(cfqq) &&
+	    (cfq_cfqq_slice_new(cfqq) ||
+	    (cfqq->slice_end - jiffies > jiffies - cfqq->slice_start))) {
+		cfq_clear_cfqq_deep(cfqq);
+		cfq_clear_cfqq_idle_window(cfqq);
+	}
+
 	if (cfqq->dispatched && cfq_should_idle(cfqd, cfqq)) {
 		cfqq = NULL;
 		goto keep_queue;
#2 lore raw
Quoted 19 lines · Vivek Goyal
Quoted 10 lines · Shaohua Li

If a deep seek queue slowly deliver requests but disk is much faster, idle for the queue just wastes disk throughput. If the queue delevers all requests before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept 128 requests at maxium and disk is fast. without the patch, the throughput is just around 30m/s, while with it, the speed is about 80m/s. The disk is a SSD, but is detected as a rotational disk. I can configure it as SSD, but I thought the deep seek queue logic should be fixed too, for example, considering a fast raid.

Hi Shaohua,

So looks like you are trying to cut down queue idling in the case when device is fast and idling hurts. That's a noble goal, just that detetction of this condition only for deep queues does not seem to cover lots of cases. Manually one can set slice_idle=0 to handle this situation.

Quoted 5 lines

What about if you have lots of sequential queues (not deep) and they all will still idle.

Secondly, what if driver is just buffering lots of requests in its device queue and not necessarily device is processing the reuqests faster.

That is not a valid concern, a driver should never extract more than it can process (pretty much) immediately.

Quoted 6 lines

So I think it is a good idea to cut down on idling if we can find that underlying device is fast and idling on queue might hurt more. But discovering this only using deep queues does not sound very appleaing to me. This is help only a particular workload which is driving deep queues. So if there was a generic mechanism to tackle this, that would be much better.

Agree, we could use better metrics for this.

Signature · 1 line
Jens Axboe
author
#3 lore raw
Quoted 29 lines · Jens Axboe
Quoted 19 lines · Vivek Goyal
Quoted 10 lines · Shaohua Li

If a deep seek queue slowly deliver requests but disk is much faster, idle for the queue just wastes disk throughput. If the queue delevers all requests before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept 128 requests at maxium and disk is fast. without the patch, the throughput is just around 30m/s, while with it, the speed is about 80m/s. The disk is a SSD, but is detected as a rotational disk. I can configure it as SSD, but I thought the deep seek queue logic should be fixed too, for example, considering a fast raid.

Hi Shaohua,

So looks like you are trying to cut down queue idling in the case when device is fast and idling hurts. That's a noble goal, just that detetction of this condition only for deep queues does not seem to cover lots of cases. Manually one can set slice_idle=0 to handle this situation.

Quoted 5 lines

What about if you have lots of sequential queues (not deep) and they all will still idle.

Secondly, what if driver is just buffering lots of requests in its device queue and not necessarily device is processing the reuqests faster.

That is not a valid concern, a driver should never extract more than it can process (pretty much) immediately.

Quoted 8 lines
Quoted 6 lines

So I think it is a good idea to cut down on idling if we can find that underlying device is fast and idling on queue might hurt more. But discovering this only using deep queues does not sound very appleaing to me. This is help only a particular workload which is driving deep queues. So if there was a generic mechanism to tackle this, that would be much better.

Agree, we could use better metrics for this.

Agree we'd better have a better method to measure device speed, but this seems not easy. Even in a fast device, a request might take long time to finish when NCQ is enabled. Before we have generic mechanism, we still need fix some particular cases.

Thanks,
Shaohua

#4 lore raw
Quoted 37 lines · Shaohua Li
Quoted 36 lines · Jens Axboe
Quoted 19 lines · Vivek Goyal
Quoted 10 lines · Shaohua Li

If a deep seek queue slowly deliver requests but disk is much faster, idle for the queue just wastes disk throughput. If the queue delevers all requests before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept 128 requests at maxium and disk is fast. without the patch, the throughput is just around 30m/s, while with it, the speed is about 80m/s. The disk is a SSD, but is detected as a rotational disk. I can configure it as SSD, but I thought the deep seek queue logic should be fixed too, for example, considering a fast raid.

Hi Shaohua,

So looks like you are trying to cut down queue idling in the case when device is fast and idling hurts. That's a noble goal, just that detetction of this condition only for deep queues does not seem to cover lots of cases. Manually one can set slice_idle=0 to handle this situation.

Quoted 5 lines

What about if you have lots of sequential queues (not deep) and they all will still idle.

Secondly, what if driver is just buffering lots of requests in its device queue and not necessarily device is processing the reuqests faster.

That is not a valid concern, a driver should never extract more than it can process (pretty much) immediately.

Quoted 6 lines

So I think it is a good idea to cut down on idling if we can find that underlying device is fast and idling on queue might hurt more. But discovering this only using deep queues does not sound very appleaing to me. This is help only a particular workload which is driving deep queues. So if there was a generic mechanism to tackle this, that would be much better.

Quoted 5 lines
Quoted 1 line

Agree, we could use better metrics for this.

Agree we'd better have a better method to measure device speed, but this seems not easy. Even in a fast device, a request might take long time to finish when NCQ is enabled. Before we have generic mechanism, we still need fix some particular cases.

Do you have a real workload for this case or it is just one of the synthetic workload simulated using fio?

Vivek

author
#5 lore raw
Quoted 38 lines · Vivek Goyal
Quoted 37 lines · Shaohua Li
Quoted 36 lines · Jens Axboe
Quoted 19 lines · Vivek Goyal
Quoted 10 lines · Shaohua Li

If a deep seek queue slowly deliver requests but disk is much faster, idle for the queue just wastes disk throughput. If the queue delevers all requests before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept 128 requests at maxium and disk is fast. without the patch, the throughput is just around 30m/s, while with it, the speed is about 80m/s. The disk is a SSD, but is detected as a rotational disk. I can configure it as SSD, but I thought the deep seek queue logic should be fixed too, for example, considering a fast raid.

Hi Shaohua,

So looks like you are trying to cut down queue idling in the case when device is fast and idling hurts. That's a noble goal, just that detetction of this condition only for deep queues does not seem to cover lots of cases. Manually one can set slice_idle=0 to handle this situation.

Quoted 5 lines

What about if you have lots of sequential queues (not deep) and they all will still idle.

Secondly, what if driver is just buffering lots of requests in its device queue and not necessarily device is processing the reuqests faster.

That is not a valid concern, a driver should never extract more than it can process (pretty much) immediately.

Quoted 6 lines

So I think it is a good idea to cut down on idling if we can find that underlying device is fast and idling on queue might hurt more. But discovering this only using deep queues does not sound very appleaing to me. This is help only a particular workload which is driving deep queues. So if there was a generic mechanism to tackle this, that would be much better.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Agree, we could use better metrics for this.

Agree we'd better have a better method to measure device speed, but this seems not easy. Even in a fast device, a request might take long time to finish when NCQ is enabled. Before we have generic mechanism, we still need fix some particular cases.

Do you have a real workload for this case or it is just one of the synthetic workload simulated using fio?

No, no real workload. We do a lot of fio tests with different scripts, this is from one of our tests.