cfq-iosched: Add some more documentation about idling
Subsystems (2)
PATCH 1/1 cfq-iosched: Add some more documentation about idling
There are always questions about why CFQ is idling on various conditions. Recent ones is Christoph asking again why to idle on REQ_NOIDLE. His assertion is that XFS is relying more and more on workqueues and is concerned that CFQ idling on IO from every workqueue will impact XFS badly.
So he suggested that I add some more documentation about CFQ idling and that can provide more clarity on the topic and also gives an opprotunity to poke a hole in theory and lead to improvements.
So here is my attempt at that. Any comments are welcome.
--- Documentation/block/cfq-iosched.txt | 70 +++++++++++++++++++++++++++++++++++ 1 files changed, 70 insertions(+), 0 deletions(-)
Documentation/block/cfq-iosched.txt +70 -0
@@ -43,3 +43,73 @@ If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches to IOPS mode and starts providing fairness in terms of number of requests dispatched. Note that this mode switching takes effect only for group scheduling. For non-cgroup users nothing should change. + +CFQ IO scheduler Idling Theory +============================== +Idling on a queue is primarily about waiting for next request to come on +same queue after completion of a request. In this process CFQ will not +dispatch requests from other cfq queues even if requests are pending +there. + +The rationale behind idling is that it can cut down on number of seeks +on rotational media. For example, if a process is doing dependent +sequential reads (next read will come on only after completion of previous +one), then not dispatching request from other queue sould help as we +did not move the disk head and kept on dispatching sequential IO from +one queue. + +CFQ does not do idling on all the queues. It primarily tries to do idling +on queues which are doing synchronous sequential IO. The synchronous +queues which are not doing sequential IO are put on a separate service +tree (called sync-noidle tree) where we do not idle on individual +cfq queue, but idle on the whole tree or IOW, idle on a group of cfq +queues. + +CFQ has following tree service trees and various queues are put on these +trees. + + sync-idle sync-noidle async + +All cfq queues doing synchronous sequential IO go on to sync-idle tree. +On this tree we idle on each queue individually. + +All synchronous non-sequential queues go on sync-noidle tree. Also any +request which are marked with REQ_NOIDLE go on this service tree. + +All async writes go on async service tree. There is no idling on async +queues. + +FAQ +=== +Q1. Why to idle at all on queues marked with REQ_NOIDLE. + +A1. We only do group idle on queues marked with REQ_NOIDLE. This helps in + providing isolation with all the sync-idle queues. Otherwise in presence + of many sequential readers, other synchronous IO might not get fair + share of disk. + + For example, if there are 10 sequential readers doing IO and they get + 100ms each. If a REQ_NOIDLE request comes in, it will be scheduled + roughly after 1 second. If after completion of REQ_NOIDLE request we + do not idle, and after a couple of mili seconds a another REQ_NOIDLE + request comes in, again it will be scheduled after 1second. Repeat it + and notice how a workload can lose its disk share and suffer due to + multiple sequnetial readers. + + fsync can generate dependent IO where bunch of data is written in the + context of fsync, and later some journaling data is written. Journaling + data comes in only after fsync has finished its IO (atleast for ext4 + that seemed to be the case). Now if one decides not to idle on fsync + thread due to REQ_NOIDLE, then next journaling write will not get + scheduled for another second. A process doing small fsync, will suffer + badly in presence of multiple sequntial readers. + + Hence doing group idling on threads using REQ_NOIDLE flag on requests + provides isolation from multiple sequntial readers and at the same + time we do not idle on individual threads. + +Q2. When to specify REQ_NOIDLE +A2. I would think whenever one is doing synchronous write and not expecting + more writes to be dispatched from same context soon, should be able + to specify REQ_NOIDLE on writes and that probably should work well for + most of the cases.
Signature · 1 line
1.7.4.4
Quoted 10 lines · Vivek Goyal
There are always questions about why CFQ is idling on various conditions. Recent ones is Christoph asking again why to idle on
REQ_NOIDLE. His assertion is that XFS is relying more and more on workqueues and is concerned that CFQ idling on IO from every workqueue will impact XFS badly.So he suggested that I add some more documentation about CFQ idling and that can provide more clarity on the topic and also gives an opprotunity to poke a hole in theory and lead to improvements.
Quoted 6 lines
So here is my attempt at that. Any comments are welcome.
Signed-off-by Vivek Goyal <[email protected]>--- Documentation/block/cfq-iosched.txt | 70 +++++++++++++++++++++++++++++++++++ 1 files changed, 70 insertions(+), 0 deletions(-)
The patch looks good and documents idling nicely. A few minor nits...
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 49
Quoted 12 lines
Documentation/block/cfq-iosched.txt +4
-0@@ -43,3 +43,73 @@ If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches to IOPS mode and starts providing fairness in terms of number of requests dispatched. Note that this mode switching takes effect only for group scheduling. For non-cgroup users nothing should change. + +CFQ IO scheduler Idling Theory +============================== +Idling on a queue is primarily about waiting for next request to come on
^^ the ?
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 57
Quoted 8 lines
Documentation/block/cfq-iosched.txt +8
-0+same queue after completion of a request. In this process CFQ will not +dispatch requests from other cfq queues even if requests are pending +there. + +The rationale behind idling is that it can cut down on number of seeks +on rotational media. For example, if a process is doing dependent +sequential reads (next read will come on only after completion of previous +one), then not dispatching request from other queue sould help as we
^^ should
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 68
Quoted 11 lines
Documentation/block/cfq-iosched.txt +11
-0+did not move the disk head and kept on dispatching sequential IO from +one queue. + +CFQ does not do idling on all the queues. It primarily tries to do idling +on queues which are doing synchronous sequential IO. The synchronous +queues which are not doing sequential IO are put on a separate service +tree (called sync-noidle tree) where we do not idle on individual +cfq queue, but idle on the whole tree or IOW, idle on a group of cfq +queues. + +CFQ has following tree service trees and various queues are put on these
^^ extraneous "tree" ?
There seems to be some redundant information between the paragraph above and below.. More room for brevity?
Quoted 47 lines
Documentation/block/cfq-iosched.txt +46
-0+trees. + + sync-idle sync-noidle async + +All cfq queues doing synchronous sequential IO go on to sync-idle tree. +On this tree we idle on each queue individually. + +All synchronous non-sequential queues go on sync-noidle tree. Also any +request which are marked with REQ_NOIDLE go on this service tree. + +All async writes go on async service tree. There is no idling on async +queues. +FAQ +=== +Q1. Why to idle at all on queues marked with REQ_NOIDLE. + +A1. We only do group idle on queues marked with REQ_NOIDLE. This helps in + providing isolation with all the sync-idle queues. Otherwise in presence + of many sequential readers, other synchronous IO might not get fair + share of disk. + + For example, if there are 10 sequential readers doing IO and they get + 100ms each. If a REQ_NOIDLE request comes in, it will be scheduled + roughly after 1 second. If after completion of REQ_NOIDLE request we + do not idle, and after a couple of mili seconds a another REQ_NOIDLE + request comes in, again it will be scheduled after 1second. Repeat it + and notice how a workload can lose its disk share and suffer due to + multiple sequnetial readers. + + fsync can generate dependent IO where bunch of data is written in the + context of fsync, and later some journaling data is written. Journaling + data comes in only after fsync has finished its IO (atleast for ext4 + that seemed to be the case). Now if one decides not to idle on fsync + thread due to REQ_NOIDLE, then next journaling write will not get + scheduled for another second. A process doing small fsync, will suffer + badly in presence of multiple sequntial readers. + + Hence doing group idling on threads using REQ_NOIDLE flag on requests + provides isolation from multiple sequntial readers and at the same + time we do not idle on individual threads. + +Q2. When to specify REQ_NOIDLE +A2. I would think whenever one is doing synchronous write and not expecting + more writes to be dispatched from same context soon, should be able + to specify REQ_NOIDLE on writes and that probably should work well for + most of the cases.
Signature · 1 line
Suresh Jayaraman
Quoted 49 lines · Vivek Goyal
There are always questions about why CFQ is idling on various conditions. Recent ones is Christoph asking again why to idle on
REQ_NOIDLE. His assertion is that XFS is relying more and more on workqueues and is concerned that CFQ idling on IO from every workqueue will impact XFS badly.So he suggested that I add some more documentation about CFQ idling and that can provide more clarity on the topic and also gives an opprotunity to poke a hole in theory and lead to improvements.
So here is my attempt at that. Any comments are welcome.
Signed-off-by Vivek Goyal <[email protected]>--- Documentation/block/cfq-iosched.txt | 70 +++++++++++++++++++++++++++++++++++ 1 files changed, 70 insertions(+), 0 deletions(-)Documentation/block/cfq-iosched.txt +25
-0@@ -43,3 +43,73 @@ If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches to IOPS mode and starts providing fairness in terms of number of requests dispatched. Note that this mode switching takes effect only for group scheduling. For non-cgroup users nothing should change. + +CFQ IO scheduler Idling Theory +============================== +Idling on a queue is primarily about waiting for next request to come on +same queue after completion of a request. In this process CFQ will not +dispatch requests from other cfq queues even if requests are pending +there. + +The rationale behind idling is that it can cut down on number of seeks +on rotational media. For example, if a process is doing dependent +sequential reads (next read will come on only after completion of previous +one), then not dispatching request from other queue sould help as we +did not move the disk head and kept on dispatching sequential IO from +one queue. + +CFQ does not do idling on all the queues. It primarily tries to do idling +on queues which are doing synchronous sequential IO. The synchronous +queues which are not doing sequential IO are put on a separate service +tree (called sync-noidle tree) where we do not idle on individual +cfq queue, but idle on the whole tree or IOW, idle on a group of cfq +queues. + +CFQ has following tree service trees and various queues are put on these +trees. +
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 80
Quoted 10 lines
Documentation/block/cfq-iosched.txt +10
-0+ sync-idle sync-noidle async + +All cfq queues doing synchronous sequential IO go on to sync-idle tree. +On this tree we idle on each queue individually. + +All synchronous non-sequential queues go on sync-noidle tree. Also any +request which are marked with REQ_NOIDLE go on this service tree. + +All async writes go on async service tree. There is no idling on async +queues.
Maybe mention CFQ don't do idle for SSD too.
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 86
Quoted 5 lines
Documentation/block/cfq-iosched.txt +5
-0+FAQ +=== +Q1. Why to idle at all on queues marked with REQ_NOIDLE. + +A1. We only do group idle on queues marked with REQ_NOIDLE. This helps in
^^^^^
tree or group? I suppose you are talking about tree, as below example doesn't mention group.
The sentence is a little confusing. we do tree/group idle for queue with random sync I/O too even without REQ_NOIDLE if the queue is the last one of the tree/group.
Quoted 11 lines
Documentation/block/cfq-iosched.txt +11
-0+ providing isolation with all the sync-idle queues. Otherwise in presence + of many sequential readers, other synchronous IO might not get fair + share of disk. + + For example, if there are 10 sequential readers doing IO and they get + 100ms each. If a REQ_NOIDLE request comes in, it will be scheduled + roughly after 1 second. If after completion of REQ_NOIDLE request we + do not idle, and after a couple of mili seconds a another REQ_NOIDLE + request comes in, again it will be scheduled after 1second. Repeat it + and notice how a workload can lose its disk share and suffer due to + multiple sequnetial readers.
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 107
Quoted 10 lines
Documentation/block/cfq-iosched.txt +10
-0+ + fsync can generate dependent IO where bunch of data is written in the + context of fsync, and later some journaling data is written. Journaling + data comes in only after fsync has finished its IO (atleast for ext4 + that seemed to be the case). Now if one decides not to idle on fsync + thread due to REQ_NOIDLE, then next journaling write will not get + scheduled for another second. A process doing small fsync, will suffer + badly in presence of multiple sequntial readers. + + Hence doing group idling on threads using REQ_NOIDLE flag on requests
^^^^^ same here.
Thanks,
Shaohua
Quoted 53 lines · Shaohua Li
Quoted 52 lines · Vivek Goyal
There are always questions about why CFQ is idling on various conditions. Recent ones is Christoph asking again why to idle on
REQ_NOIDLE. His assertion is that XFS is relying more and more on workqueues and is concerned that CFQ idling on IO from every workqueue will impact XFS badly.So he suggested that I add some more documentation about CFQ idling and that can provide more clarity on the topic and also gives an opprotunity to poke a hole in theory and lead to improvements.
So here is my attempt at that. Any comments are welcome.
Signed-off-by Vivek Goyal <[email protected]>--- Documentation/block/cfq-iosched.txt | 70 +++++++++++++++++++++++++++++++++++ 1 files changed, 70 insertions(+), 0 deletions(-)Documentation/block/cfq-iosched.txt +28
-0@@ -43,3 +43,73 @@ If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches to IOPS mode and starts providing fairness in terms of number of requests dispatched. Note that this mode switching takes effect only for group scheduling. For non-cgroup users nothing should change. + +CFQ IO scheduler Idling Theory +============================== +Idling on a queue is primarily about waiting for next request to come on +same queue after completion of a request. In this process CFQ will not +dispatch requests from other cfq queues even if requests are pending +there. + +The rationale behind idling is that it can cut down on number of seeks +on rotational media. For example, if a process is doing dependent +sequential reads (next read will come on only after completion of previous +one), then not dispatching request from other queue sould help as we +did not move the disk head and kept on dispatching sequential IO from +one queue. + +CFQ does not do idling on all the queues. It primarily tries to do idling +on queues which are doing synchronous sequential IO. The synchronous +queues which are not doing sequential IO are put on a separate service +tree (called sync-noidle tree) where we do not idle on individual +cfq queue, but idle on the whole tree or IOW, idle on a group of cfq +queues. + +CFQ has following tree service trees and various queues are put on these +trees. + + sync-idle sync-noidle async + +All cfq queues doing synchronous sequential IO go on to sync-idle tree.
Quoted 8 lines
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 80
Quoted 7 lines
Documentation/block/cfq-iosched.txt +7
-0+On this tree we idle on each queue individually. + +All synchronous non-sequential queues go on sync-noidle tree. Also any +request which are marked with REQ_NOIDLE go on this service tree. + +All async writes go on async service tree. There is no idling on async +queues.Maybe mention CFQ don't do idle for SSD too.
Ok. Will do that.
Quoted 9 lines
Documentation/block/cfq-iosched.txt If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches · line 86
Quoted 6 lines
Documentation/block/cfq-iosched.txt +5
-0+FAQ +=== +Q1. Why to idle at all on queues marked with REQ_NOIDLE. + +A1. We only do group idle on queues marked with REQ_NOIDLE. This helps in^^^^^tree or group? I suppose you are talking about tree, as below example doesn't mention group.
What I meant that the cfq queue group on this service tree. Ok, will change it to tree idle.
Quoted 3 lines
The sentence is a little confusing. we do tree/group idle for queue with random sync I/O too even without
REQ_NOIDLEif the queue is the last one of the tree/group.
We do. That has been covered above already that sync non-sequential queues go on sync-noidle tree. In this question I am only trying to answer the concern that when filesystem has marked a request as REQ_NOIDLE, then why do we still do tree idle on it.
Thanks
Vivek