block: make iolatency avg_lat exponentially decay
Subsystems (4)
PATCH v3 1/1 block: make iolatency avg_lat exponentially decay
From: "Dennis Zhou (Facebook)" <[email protected]>
Currently, avg_lat is calculated by accumulating the mean of every window in a long running cumulative average. As time goes on, the metric becomes less and less useful due to the accumulated history.
This patch reuses the same calculation done in load averages to make the avg_lat metric more lively. Unlike load averages, the avg only advances when a window elapses (due to an io). Idle periods extend the most recent window. Bucketing is used to limit the history of avg_lat by binding it to the window size. So, the window range for 1/exp (decay rate) is [1 min, 2.5 min) when windows elapse immediately.
The current sample window size is exposed in the debug info to enable calculation of the window range.
--- Documentation/admin-guide/cgroup-v2.rst | 21 +++++---- block/blk-iolatency.c | 60 ++++++++++++++++++------- 2 files changed, 57 insertions(+), 24 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +12 -9
@@ -1474,11 +1474,9 @@ So the ideal way to configure this is to set io.latency in groups A, B, and C. Generally you do not want to set a value lower than the latency your device supports. Experiment to find the value that works best for your workload. Start at higher than the expected latency for your device and watch the -total_lat_avg value in io.stat for your workload group to get an idea of the -latency you see during normal operation. Use this value as a basis for your -real setting, setting at 10-15% higher than the value in io.stat. -Experimentation is key here because total_lat_avg is a running total, so is the -"statistics" portion of "lies, damned lies, and statistics." +avg_lat value in io.stat for your workload group to get an idea of the +latency you see during normal operation. Use the avg_lat value as a basis for +your real setting, setting at 10-15% higher than the value in io.stat. How IO Latency Throttling Works ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1522,10 +1520,15 @@ IO Latency Interface Files This is the current queue depth for the group. avg_lat - The running average IO latency for this group in microseconds. - Running average is generally flawed, but will give an - administrator a general idea of the overall latency they can - expect for their workload on the given disk. + This is an exponential moving average with a decay rate of 1/exp + bound by the sampling interval. The decay rate interval can be + calculated by multiplying the win value in io.stat by the + corresponding number of samples based on the win value. + + win + The sampling window size in milliseconds. This is the minimum + duration of time between evaluation events. Windows only elapse + with IO activity. Idle periods extend the most recent window. PID ---
block/blk-iolatency.c +45 -15
105 lines not on this page · raw message
Signature · 1 line
2.17.1
Jens Axboe
maintainerQuoted 13 lines · Dennis Zhou
From: "Dennis Zhou (Facebook)" <[email protected]>
Currently,
avg_latis calculated by accumulating the mean of every window in a long running cumulative average. As time goes on, the metric becomes less and less useful due to the accumulated history.This patch reuses the same calculation done in load averages to make the
avg_latmetric more lively. Unlike load averages, the avg only advances when a window elapses (due to an io). Idle periods extend the most recent window. Bucketing is used to limit the history ofavg_latby binding it to the window size. So, the window range for 1/exp (decay rate) is [1 min, 2.5 min) when windows elapse immediately.
Quoted 2 lines
The current sample window size is exposed in the debug info to enable calculation of the window range.
Applied for 4.19, thanks Dennis.
Signature · 1 line
Jens Axboe
Tejun Heo
maintainerHello, Dennis.
Quoted 13 lines · Dennis Zhou
From: "Dennis Zhou (Facebook)" <[email protected]>
Currently,
avg_latis calculated by accumulating the mean of every window in a long running cumulative average. As time goes on, the metric becomes less and less useful due to the accumulated history.This patch reuses the same calculation done in load averages to make the
avg_latmetric more lively. Unlike load averages, the avg only advances when a window elapses (due to an io). Idle periods extend the most recent window. Bucketing is used to limit the history ofavg_latby binding it to the window size. So, the window range for 1/exp (decay rate) is [1 min, 2.5 min) when windows elapse immediately.
Quoted 7 lines
The current sample window size is exposed in the debug info to enable calculation of the window range.
Signed-off-by Dennis Zhou <[email protected]>Acked-by Tejun Heo <[email protected]>Acked-by Johannes Weiner <[email protected]>Acked-by Josef Bacik <[email protected]>
Heh, sorry about not thinking this up from the beginning but I think it'd be great to move this to blkcg core so that the stat is always available regardless of blk-iolatency. It's a really important metric and can be used for both monitoring and policy implementation.
Thanks!
Signature · 1 line
tejun
Jens Axboe
maintainerQuoted 24 lines · Tejun Heo
Hello, Dennis.
Quoted 13 lines · Dennis Zhou
From: "Dennis Zhou (Facebook)" <[email protected]>
Currently,
avg_latis calculated by accumulating the mean of every window in a long running cumulative average. As time goes on, the metric becomes less and less useful due to the accumulated history.This patch reuses the same calculation done in load averages to make the
avg_latmetric more lively. Unlike load averages, the avg only advances when a window elapses (due to an io). Idle periods extend the most recent window. Bucketing is used to limit the history ofavg_latby binding it to the window size. So, the window range for 1/exp (decay rate) is [1 min, 2.5 min) when windows elapse immediately.Quoted 7 lines
The current sample window size is exposed in the debug info to enable calculation of the window range.
Signed-off-by Dennis Zhou <[email protected]>Acked-by Tejun Heo <[email protected]>Acked-by Johannes Weiner <[email protected]>Acked-by Josef Bacik <[email protected]>
Quoted 4 lines
Heh, sorry about not thinking this up from the beginning but I think it'd be great to move this to blkcg core so that the stat is always available regardless of blk-iolatency. It's a really important metric and can be used for both monitoring and policy implementation.
Just make that change a separate patch, since v3 is already applied.
Signature · 1 line
Jens Axboe