fio: steady state for latency
cover fio: steady state for latency
I've been testing Claude Code for things lately, I figured one good task may be to try steady state latency support which we lack support for. It also fixed a small build bug.
Luis Chamberlain (2):
configure: libnfs + gnutls
fio: add latency steady state detection
HOWTO.rst | 12 +++++ client.c | 1 + configure | 4 +- example_latency_steadystate.fio | 47 ++++++++++++++++++++ options.c | 26 ++++++++++- stat.h | 10 +++++ steadystate.c | 78 ++++++++++++++++++++++++++++----- steadystate.h | 7 +++ 8 files changed, 170 insertions(+), 15 deletions(-) create mode 100644 example_latency_steadystate.fio
Signature · 1 line
2.47.2
PATCH 1/2 configure: libnfs + gnutls
The problem is that the configure script tries to get both libnfs and gnutls flags together, but gnutls is missing. Let me check the configure script's logic and fix it:
The configure script is trying to link both libnfs and gnutls together, but gnutls is not installed. Let me fix this by modifying the configure script to only require libnfs:
--- configure | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
configure +2 -2
@@ -2359,8 +2359,8 @@ print_config "DAOS File System (dfs) Engine" "$dfs" if test "$libnfs" != "no" ; then if $(pkg-config libnfs > /dev/null 2>&1); then libnfs="yes" - libnfs_cflags=$(pkg-config --cflags libnfs gnutls) - libnfs_libs=$(pkg-config --libs libnfs gnutls) + libnfs_cflags=$(pkg-config --cflags libnfs) + libnfs_libs=$(pkg-config --libs libnfs) else if test "$libnfs" = "yes" ; then feature_not_found "libnfs" "libnfs"
Signature · 1 line
2.47.2
Re: [PATCH 1/2] configure: libnfs + gnutls
Quoted 19 lines · Luis Chamberlain
The problem is that the configure script tries to get both libnfs and gnutls flags together, but gnutls is missing. Let me check the configure script's logic and fix it:
The configure script is trying to link both libnfs and gnutls together, but gnutls is not installed. Let me fix this by modifying the configure script to only require libnfs:
Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>--- configure | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)configure +0
-0
configure print_config "DAOS File System (dfs) Engine" "$dfs" · line 2366
Quoted 10 lines
configure +2
-2@@ -2359,8 +2359,8 @@ print_config "DAOS File System (dfs) Engine" "$dfs" if test "$libnfs" != "no" ; then if $(pkg-config libnfs > /dev/null 2>&1); then libnfs="yes" - libnfs_cflags=$(pkg-config --cflags libnfs gnutls) - libnfs_libs=$(pkg-config --libs libnfs gnutls) + libnfs_cflags=$(pkg-config --cflags libnfs) + libnfs_libs=$(pkg-config --libs libnfs) else if test "$libnfs" = "yes" ; then feature_not_found "libnfs" "libnfs"
The gnutls requirement was added via
https://lore.kernel.org/fio/[email protected]/T/#t
I can think of two ways to resolve this:
- Make sure gnutls is installed when building fio
- Modify the configure script to detect whether or not gnutls really is needed by libnfs and set up the flags accordingly
Vincent
PATCH 2/2 fio: add latency steady state detection
In reply to Luis ChamberlainAdd fio latency steady state support. This is based on the SNIA SSD Performance Test Specification requirements for latency steady state detection. The implementation calculates weighted average latency across all I/O directions and supports both maximum mean deviation and slope-based detection methods.
Tested successfully against NVMe device with debug output confirming proper latency calculation and steady state evaluation.
--- HOWTO.rst | 12 +++++ client.c | 1 + example_latency_steadystate.fio | 47 ++++++++++++++++++++ options.c | 26 ++++++++++- stat.h | 10 +++++ steadystate.c | 78 ++++++++++++++++++++++++++++----- steadystate.h | 7 +++ 7 files changed, 168 insertions(+), 13 deletions(-) create mode 100644 example_latency_steadystate.fio
HOWTO.rst +12 -0
@@ -4154,6 +4154,18 @@ Steady state Collect bandwidth data and calculate the least squares regression slope. Stop the job if the slope falls below the specified limit. + **lat** + Collect completion latency data and calculate the maximum mean + deviation. Stop the job if the deviation falls below the specified + limit. The latency values are weighted by the number of I/O samples + in each measurement interval. + + **lat_slope** + Collect completion latency data and calculate the least squares + regression slope. Stop the job if the slope falls below the + specified limit. The latency values are weighted by the number + of I/O samples in each measurement interval. + .. option:: steadystate_duration=time, ss_dur=time A rolling window of this duration will be used to judge whether steady
client.c +1 -0
@@ -1077,6 +1077,7 @@ static void convert_ts(struct thread_stat *dst, struct thread_stat *src)for (i =0 ; i < dst->ss_dur; i++ ) { dst->ss_iops_data[i] =le64_to_cpu (src->ss_iops_data[i]); dst->ss_bw_data[i] =le64_to_cpu (src->ss_bw_data[i]); + dst->ss_lat_data[i] =le64_to_cpu (src->ss_lat_data[i]); } }
example_latency_steadystate.fio new file +47 -0
47 lines not on this page · raw message
options.c +25 -1
@@ -1370,7 +1370,8 @@ static int str_steadystate_cb(void *data, const char *str)long long ll;if (td->o.ss_state !=FIO_SS_IOPS && td->o.ss_state !=FIO_SS_IOPS_SLOPE && - td->o.ss_state !=FIO_SS_BW && td->o.ss_state !=FIO_SS_BW_SLOPE ) { + td->o.ss_state !=FIO_SS_BW && td->o.ss_state !=FIO_SS_BW_SLOPE && + td->o.ss_state !=FIO_SS_LAT && td->o.ss_state !=FIO_SS_LAT_SLOPE ) {/* should be impossible to get here */ log_err ("fio: unknown steady state criterion\n" );return 1 ;
@@ -1414,6 +1415,21 @@ static int str_steadystate_cb(void *data, const char *str)return 0 ; td->o.ss_limit.u.f = val; + }else if (td->o.ss_state &FIO_SS_LAT ) { +long long tns; +if (check_str_time (nr, &tns,0 )) { +log_err ("fio: steadystate latency threshold parsing failed\n" ); +free (nr); +return 1 ; + } + +dprint (FD_PARSE ,"set steady state latency threshold to %lld nsec\n" , tns); +free (nr); +if (parse_dryrun ()) +return 0 ; + + td->o.ss_limit.u.f = (double ) tns; + }else {/* bandwidth criterion */ if (str_to_decimal (nr, &ll,1 , td,0 ,0 )) {log_err ("fio: steadystate BW threshold postfix parsing failed\n" );
@@ -5489,6 +5505,14 @@ struct fio_option fio_options[FIO_MAX_OPTS] = { .oval = FIO_SS_BW_SLOPE, .help = "slope calculated from bandwidth measurements", }, + { .ival = "lat", + .oval = FIO_SS_LAT, + .help = "maximum mean deviation of latency measurements", + }, + { .ival = "lat_slope", + .oval = FIO_SS_LAT_SLOPE, + .help = "slope calculated from latency measurements", + }, }, .category = FIO_OPT_C_GENERAL, .group = FIO_OPT_G_RUNTIME,
stat.h +10 -0
@@ -282,6 +282,16 @@ struct thread_stat {uint64_t pad5; }; +union { +uint64_t *ss_lat_data; +/* +* For FIO_NET_CMD_TS, the pointed to data will temporarily +* be stored at this offset from the start of the payload. +*/ +uint64_t ss_lat_data_offset; +uint64_t pad5b; + }; +union {struct clat_prio_stat *clat_prio[DDIR_RWDIR_CNT ];/*
steadystate.c +66 -12
210 lines not on this page · raw message
steadystate.h +7 -0
@@ -9,6 +9,7 @@ extern void steadystate_setup(void);extern int td_steadystate_init (struct thread_data *);extern uint64_t steadystate_bw_mean (struct thread_stat *);extern uint64_t steadystate_iops_mean (struct thread_stat *); +extern uint64_t steadystate_lat_mean (struct thread_stat *);extern bool steadystate_enabled;extern unsigned int ss_check_interval;
@@ -24,6 +25,7 @@ struct steadystate_data {unsigned int tail;uint64_t *iops_data;uint64_t *bw_data; +uint64_t *lat_data;double slope;double deviation;
@@ -38,6 +40,8 @@ struct steadystate_data {struct timespec prev_time;uint64_t prev_iops;uint64_t prev_bytes; +double prev_lat_sum; +uint64_t prev_lat_samples; };enum {
@@ -49,6 +53,7 @@ enum { __FIO_SS_DATA, __FIO_SS_PCT, __FIO_SS_BUFFER_FULL, + __FIO_SS_LAT, }; enum {
@@ -60,9 +65,11 @@ enum {FIO_SS_DATA =1 << __FIO_SS_DATA,FIO_SS_PCT =1 << __FIO_SS_PCT,FIO_SS_BUFFER_FULL =1 << __FIO_SS_BUFFER_FULL, +FIO_SS_LAT =1 << __FIO_SS_LAT,FIO_SS_IOPS_SLOPE =FIO_SS_IOPS |FIO_SS_SLOPE ,FIO_SS_BW_SLOPE =FIO_SS_BW |FIO_SS_SLOPE , + FIO_SS_LAT_SLOPE = FIO_SS_LAT | FIO_SS_SLOPE, };#endif
Signature · 1 line
2.47.2
Re: [PATCH 2/2] fio: add latency steady state detection
Quoted 11 lines · Luis Chamberlain
Add fio latency steady state support. This is based on the SNIA SSD Performance Test Specification requirements for latency steady state detection. The implementation calculates weighted average latency across all I/O directions and supports both maximum mean deviation and slope-based detection methods.
Tested successfully against NVMe device with debug output confirming proper latency calculation and steady state evaluation.
Generated-by Claude AISigned-off-by Luis Chamberlain <[email protected]>
Please add some more details about how closely this adheres to SNIA's definition of steady state. From
https://www.snia.org/sites/default/files/technical-work/pts/release/SNIA-SSS-PTS-2.0.2.pdf
Here is SNIA's definition of steady state:
2.1.24 Steady State: A device is said to be in Steady State when, for the dependent variable (y) being tracked:
a) Range(y) is less than 20% of Ave(y): Max(y)-Min(y) within the Measurement Window is no more than 20% of the Ave(y) within the Measurement Window; and
b) Slope(y) is less than 10%: Max(y)-Min(y), where Max(y) and Min(y) are the maximum and minimum values on the best linear curve fit of the y-values within the Measurement Window, is within 10% of Ave(y) value within the Measurement Window.
SNIA requires both slope *and* maximum deviation criteria to simultaneously be met whereas fio considers only one criterion. It might be worthwhile to add to fio an option to consider both criteria.
Also b) is unclear to me. "Slope(y) is less than 10%" of what? Perhaps this is meant to refer to 10% of Ave(y) but it would be better to spell this out explicitly.
Finally, fio does not consider the second part of the slope criterion concerning the range of values on the best linear curve fit.
Quoted 16 lines
--- HOWTO.rst | 12 +++++ client.c | 1 + example_latency_steadystate.fio | 47 ++++++++++++++++++++ options.c | 26 ++++++++++- stat.h | 10 +++++ steadystate.c | 78 ++++++++++++++++++++++++++++----- steadystate.h | 7 +++ 7 files changed, 168 insertions(+), 13 deletions(-)create mode 100644 example_latency_steadystate.fio
HOWTO.rst +0
-0
HOWTO.rst Steady state · line 4162
Quoted 9 lines
HOWTO.rst +6
-0@@ -4154,6 +4154,18 @@ Steady state Collect bandwidth data and calculate the least squares regression slope. Stop the job if the slope falls below the specified limit. + **lat** + Collect completion latency data and calculate the maximum mean + deviation. Stop the job if the deviation falls below the specified + limit. The latency values are weighted by the number of I/O samples + in each measurement interval. +
This is unstated but it should be made explicit that latency steady state is determined using mean latency values for each period with duration specified by ss_interval.
It's not obvious to me how the weigthing is accomplished or whether weighting is desirable. Having gone through the patch, I don't see any unexpected weighting happening. Each IO during an interval is given the same weight. I would just remove the statement about weighting.
HOWTO.rst Steady state · line 4168
Quoted 6 lines
HOWTO.rst +6
-0+ **lat_slope** + Collect completion latency data and calculate the least squares + regression slope. Stop the job if the slope falls below the + specified limit. The latency values are weighted by the number + of I/O samples in each measurement interval. +
Same as above.
Quoted 90 lines
HOWTO.rst +0
-0.. option:: steadystate_duration=time, ss_dur=time A rolling window of this duration will be used to judge whether steadyclient.c +1
-0@@ -1077,6 +1077,7 @@ static void convert_ts(struct thread_stat *dst, struct thread_stat *src)for (i =0 ; i < dst->ss_dur; i++ ) { dst->ss_iops_data[i] =le64_to_cpu (src->ss_iops_data[i]); dst->ss_bw_data[i] =le64_to_cpu (src->ss_bw_data[i]); + dst->ss_lat_data[i] =le64_to_cpu (src->ss_lat_data[i]); } }example_latency_steadystate.fio new file +47
-0@@ -0,0 +1,47 @@ +# Example FIO job file demonstrating latency steady state detection +# This example shows how to use FIO's latency steady state detection +# to automatically terminate workloads when latency stabilizes +# +# Based on SNIA SSD Performance Test Specification requirements: +# - Steady state is achieved when latency measurements don't change more than +# 20% for 5 measurement windows and remain within 5% of a line with 10% slope +# - This example uses more conservative 5% deviation threshold for demonstration + +[global] +# Basic I/O parameters +ioengine=libaio +iodepth=32 +bs=4k +direct=1 +rw=randread +numjobs=1 +time_based=1 +runtime=3600 # Max runtime: 1 hour (will terminate early if steady state reached) + +# Steady state detection parameters +steadystate=lat:5% # Stop when latency mean deviation < 5% of average +steadystate_duration=300 # Use 5-minute rolling window for measurements +steadystate_ramp_time=60 # Wait 1 minute before starting measurements +steadystate_check_interval=10 # Take measurements every 10 seconds + +# Output options +write_lat_log=lat_steadystate +log_avg_msec=10000 # Log average latency every 10 seconds + +[latency_steady_test] +filename=/dev/nvme3n1 +size=10G + +# Alternative steady state configurations (uncomment to try): + +# Use slope-based detection instead of deviation: +# steadystate=lat_slope:0.1% + +# More aggressive detection (faster convergence): +# steadystate=lat:2% +# steadystate_duration=120 # 2-minute window +# steadystate_check_interval=5 # Check every 5 seconds + +# More conservative detection (slower convergence): +# steadystate=lat:10% +# steadystate_duration=600 # 10-minute windowoptions.c +6
-1@@ -1370,7 +1370,8 @@ static int str_steadystate_cb(void *data, const char *str)long long ll;if (td->o.ss_state !=FIO_SS_IOPS && td->o.ss_state !=FIO_SS_IOPS_SLOPE && - td->o.ss_state !=FIO_SS_BW && td->o.ss_state !=FIO_SS_BW_SLOPE ) { + td->o.ss_state !=FIO_SS_BW && td->o.ss_state !=FIO_SS_BW_SLOPE && + td->o.ss_state !=FIO_SS_LAT && td->o.ss_state !=FIO_SS_LAT_SLOPE ) {/* should be impossible to get here */ log_err ("fio: unknown steady state criterion\n" );return 1 ;@@ -1414,6 +1415,21 @@ static int str_steadystate_cb(void *data, const char *str)return 0 ; td->o.ss_limit.u.f = val; + }else if (td->o.ss_state &FIO_SS_LAT ) { +long long tns; +if (check_str_time (nr, &tns,0 )) { +log_err ("fio: steadystate latency threshold parsing failed\n" );
options.c static int str_steadystate_cb(void *data, const char *str) · line 1431
Quoted 10 lines
options.c +10
-0+free (nr); +return 1 ; + } + +dprint (FD_PARSE ,"set steady state latency threshold to %lld nsec\n" , tns); +free (nr); +if (parse_dryrun ()) +return 0 ; + + td->o.ss_limit.u.f = (double ) tns;
Why make this a double instead of an integer?
Quoted 27 lines
options.c +9
-0+ }else {/* bandwidth criterion */ if (str_to_decimal (nr, &ll,1 , td,0 ,0 )) {log_err ("fio: steadystate BW threshold postfix parsing failed\n" );@@ -5489,6 +5505,14 @@ struct fio_option fio_options[FIO_MAX_OPTS] = { .oval = FIO_SS_BW_SLOPE, .help = "slope calculated from bandwidth measurements", }, + { .ival = "lat", + .oval = FIO_SS_LAT, + .help = "maximum mean deviation of latency measurements", + }, + { .ival = "lat_slope", + .oval = FIO_SS_LAT_SLOPE, + .help = "slope calculated from latency measurements", + }, }, .category = FIO_OPT_C_GENERAL, .group = FIO_OPT_G_RUNTIME,stat.h +0
-0@@ -282,6 +282,16 @@ struct thread_stat {uint64_t pad5; };
stat.h struct thread_stat { · line 294
Quoted 10 lines
stat.h +10
-0+union { +uint64_t *ss_lat_data; +/* +* For FIO_NET_CMD_TS, the pointed to data will temporarily +* be stored at this offset from the start of the payload. +*/ +uint64_t ss_lat_data_offset; +uint64_t pad5b; + }; +
I don't see any changes in client.c to process this data on the receiving end or in server.c to send this data from the server.
Quoted 27 lines
stat.h +0
-0union {struct clat_prio_stat *clat_prio[DDIR_RWDIR_CNT ];/* steadystate.c +3
-0@@ -10,8 +10,10 @@ void steadystate_free(struct thread_data *td) {free (td->ss.iops_data);free (td->ss.bw_data); +free (td->ss.lat_data); td->ss.iops_data =NULL ; td->ss.bw_data =NULL ; + td->ss.lat_data =NULL ; }static void steadystate_alloc (struct thread_data *td)@@ -20,6 +22,7 @@ static void steadystate_alloc(struct thread_data *td) td->ss.bw_data =calloc (intervals,sizeof (uint64_t )); td->ss.iops_data =calloc (intervals,sizeof (uint64_t )); + td->ss.lat_data =calloc (intervals,sizeof (uint64_t )); td->ss.state |=FIO_SS_DATA ; }
steadystate.c void steadystate_setup(void) · line 67
Quoted 6 lines
steadystate.c +1
-1@@ -60,7 +63,7 @@ void steadystate_setup(void)steadystate_alloc (prev_td); } -static bool steadystate_slope (uint64_t iops,uint64_t bw, +static bool steadystate_slope (uint64_t iops,uint64_t bw,double lat,struct thread_data *td)
I would make lat uint64_t instead of double to be consistent with iops and bw.
Quoted 51 lines
steadystate.c +13
-4{int i, j;@@ -71,11 +74,14 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw, ss->bw_data[ss->tail] = bw; ss->iops_data[ss->tail] = iops; + ss->lat_data[ss->tail] = (uint64_t )lat;if (ss->state &FIO_SS_IOPS ) new_val = iops; -else +else if (ss->state &FIO_SS_BW ) new_val = bw; +else + new_val = (uint64_t )lat;if (ss->state &FIO_SS_BUFFER_FULL || ss->tail - ss->head == intervals -1 ) {if (!(ss->state &FIO_SS_BUFFER_FULL )) {@@ -83,13 +89,17 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,for (i =0 , ss->sum_y =0 ; i < intervals; i++) {if (ss->state &FIO_SS_IOPS ) ss->sum_y += ss->iops_data[i]; -else +else if (ss->state &FIO_SS_BW ) ss->sum_y += ss->bw_data[i]; +else + ss->sum_y += ss->lat_data[i]; j = (ss->head + i) % intervals;if (ss->state &FIO_SS_IOPS ) ss->sum_xy += i * ss->iops_data[j]; -else +else if (ss->state &FIO_SS_BW ) ss->sum_xy += i * ss->bw_data[j]; +else + ss->sum_xy += i * ss->lat_data[j]; } ss->state |=FIO_SS_BUFFER_FULL ; }else {/* easy to update the sums */ @@ -100,8 +110,10 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,if (ss->state &FIO_SS_IOPS ) ss->oldest_y = ss->iops_data[ss->head]; -else +else if (ss->state &FIO_SS_BW ) ss->oldest_y = ss->bw_data[ss->head]; +else + ss->oldest_y = ss->lat_data[ss->head];/* * calculate slope as (sum_xy - sum_x * sum_y / n) / (sum_(x^2)
steadystate.c static bool steadystate_slope(uint64_t iops, uint64_t bw, · line 150
Quoted 6 lines
steadystate.c +1
-1@@ -134,7 +146,7 @@ static bool steadystate_slope(uint64_t iops, uint64_t bw,return false; } -static bool steadystate_deviation (uint64_t iops,uint64_t bw, +static bool steadystate_deviation (uint64_t iops,uint64_t bw,double lat,struct thread_data *td)
Same comment as above.
Quoted 108 lines
steadystate.c +27
-4{int i;@@ -146,6 +158,7 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw, ss->bw_data[ss->tail] = bw; ss->iops_data[ss->tail] = iops; + ss->lat_data[ss->tail] = (uint64_t )lat;if (ss->state &FIO_SS_BUFFER_FULL || ss->tail - ss->head == intervals -1 ) {if (!(ss->state &FIO_SS_BUFFER_FULL )) {@@ -153,22 +166,28 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,for (i =0 , ss->sum_y =0 ; i < intervals; i++) {if (ss->state &FIO_SS_IOPS ) ss->sum_y += ss->iops_data[i]; -else +else if (ss->state &FIO_SS_BW ) ss->sum_y += ss->bw_data[i]; +else + ss->sum_y += ss->lat_data[i]; } ss->state |=FIO_SS_BUFFER_FULL ; }else {/* easy to update the sum */ ss->sum_y -= ss->oldest_y;if (ss->state &FIO_SS_IOPS ) ss->sum_y += ss->iops_data[ss->tail]; -else +else if (ss->state &FIO_SS_BW ) ss->sum_y += ss->bw_data[ss->tail]; +else + ss->sum_y += ss->lat_data[ss->tail]; }if (ss->state &FIO_SS_IOPS ) ss->oldest_y = ss->iops_data[ss->head]; -else +else if (ss->state &FIO_SS_BW ) ss->oldest_y = ss->bw_data[ss->head]; +else + ss->oldest_y = ss->lat_data[ss->head]; mean = (double ) ss->sum_y / intervals; ss->deviation =0.0 ;@@ -176,8 +195,10 @@ static bool steadystate_deviation(uint64_t iops, uint64_t bw,for (i =0 ; i < intervals; i++) {if (ss->state &FIO_SS_IOPS ) diff = ss->iops_data[i] - mean; -else +else if (ss->state &FIO_SS_BW ) diff = ss->bw_data[i] - mean; +else + diff = ss->lat_data[i] - mean; ss->deviation =max (ss->deviation, diff * (diff <0.0 ?-1.0 :1.0 )); }@@ -209,13 +230,18 @@ int steadystate_check(void)unsigned long rate_time;struct timespec now;uint64_t group_bw =0 , group_iops =0 ; +double group_lat_sum =0.0 ; +uint64_t group_lat_samples =0 ;uint64_t td_iops, td_bytes; +double group_lat;bool ret; prev_groupid =-1 ;for_each_td (td) {const bool needs_lock =td_async_processing (td);struct steadystate_data *ss = &td->ss; +double td_lat_sum =0.0 ; +uint64_t td_lat_samples =0 ; if (!ss->dur || td->runstate <= TD_SETTING_UP || td->runstate >= TD_EXITED || !ss->state ||@@ -228,6 +254,8 @@ int steadystate_check(void) (td->o.group_reporting && td->groupid != prev_groupid)) { group_bw =0 ; group_iops =0 ; + group_lat_sum =0.0 ; + group_lat_samples =0 ; group_ramp_time_over =0 ; } prev_groupid = td->groupid;@@ -248,6 +276,9 @@ int steadystate_check(void)for (ddir =0 ; ddir <DDIR_RWDIR_CNT ; ddir++) { td_iops += td->io_blocks[ddir]; td_bytes += td->io_bytes[ddir]; + td_lat_sum += td->ts.clat_stat[ddir].mean.u.f * + td->ts.clat_stat[ddir].samples; + td_lat_samples += td->ts.clat_stat[ddir].samples; }if (needs_lock)@@ -261,10 +292,14 @@ int steadystate_check(void) (ss_check_interval * ss_check_interval /1000L ); group_iops += rate_time * (td_iops - ss->prev_iops) / (ss_check_interval * ss_check_interval /1000L ); + group_lat_sum += td_lat_sum - ss->prev_lat_sum; + group_lat_samples += td_lat_samples - ss->prev_lat_samples; ++group_ramp_time_over; } ss->prev_iops = td_iops; ss->prev_bytes = td_bytes; + ss->prev_lat_sum = td_lat_sum; + ss->prev_lat_samples = td_lat_samples;if (td->o.group_reporting && !(ss->state &FIO_SS_DATA ))continue ;
steadystate.c int steadystate_check(void) · line 325
Quoted 7 lines
steadystate.c +4
-0@@ -284,10 +319,14 @@ int steadystate_check(void) (unsigned long long ) group_bw, ss->head, ss->tail); + group_lat =0.0 ; +if (group_lat_samples) + group_lat = group_lat_sum / group_lat_samples; +
Are there any concerns about numerical precision here? Latency for an interval is calculated by taking differences in the sum of completion latency over all IOs at the beginning and the end of the period. The sums are calculated by multiplying the count by the average latency. For a long running workload count * (avg latency) can be very large and there may be few changes in avg latency. Might it be possible that these numerical issues would bias fio toward positive detection of steady state because of how avg latency within a period is calculated?
Quoted 18 lines
steadystate.c +7
-2if (ss->state &FIO_SS_SLOPE ) - ret =steadystate_slope (group_iops, group_bw, td); + ret =steadystate_slope (group_iops, group_bw, group_lat, td);else - ret =steadystate_deviation (group_iops, group_bw, td); + ret =steadystate_deviation (group_iops, group_bw, group_lat, td);if (ret) {if (td->o.group_reporting) {@@ -382,3 +421,18 @@ uint64_t steadystate_iops_mean(struct thread_stat *ts)return sum / intervals; } + +uint64_t steadystate_lat_mean (struct thread_stat *ts) +{ +int i; +uint64_t sum;
steadystate.c uint64_t steadystate_iops_mean(struct thread_stat *ts) · line 438
Quoted 10 lines
steadystate.c +10
-0+int intervals = ts->ss_dur / (ss_check_interval /1000L ); + +if (!ts->ss_dur) +return 0 ; + +for (i =0 , sum =0 ; i < intervals; i++) + sum += ts->ss_lat_data[i]; + +return sum / intervals; +}
There is an opportunity to eliminate duplicate code here with the steadystate_*_mean functions.
Quoted 30 lines
steadystate.h +4
-0@@ -9,6 +9,7 @@ extern void steadystate_setup(void);extern int td_steadystate_init (struct thread_data *);extern uint64_t steadystate_bw_mean (struct thread_stat *);extern uint64_t steadystate_iops_mean (struct thread_stat *); +extern uint64_t steadystate_lat_mean (struct thread_stat *);extern bool steadystate_enabled;extern unsigned int ss_check_interval;@@ -24,6 +25,7 @@ struct steadystate_data {unsigned int tail;uint64_t *iops_data;uint64_t *bw_data; +uint64_t *lat_data;double slope;double deviation;@@ -38,6 +40,8 @@ struct steadystate_data {struct timespec prev_time;uint64_t prev_iops;uint64_t prev_bytes; +double prev_lat_sum; +uint64_t prev_lat_samples; };enum {
steadystate.h enum { · line 56
Quoted 4 lines
steadystate.h +1
-0@@ -49,6 +53,7 @@ enum { __FIO_SS_DATA, __FIO_SS_PCT, __FIO_SS_BUFFER_FULL, + __FIO_SS_LAT,
Put this next to the labels for IOPS and BW since they belong together.
steadystate.h enum { · line 68
Quoted 8 lines
steadystate.h +1
-0};enum {@@ -60,9 +65,11 @@ enum {FIO_SS_DATA =1 << __FIO_SS_DATA,FIO_SS_PCT =1 << __FIO_SS_PCT,FIO_SS_BUFFER_FULL =1 << __FIO_SS_BUFFER_FULL, + FIO_SS_LAT = 1 << __FIO_SS_LAT,
Same comment as above.
Quoted 7 lines
steadystate.h +1
-0FIO_SS_IOPS_SLOPE =FIO_SS_IOPS |FIO_SS_SLOPE ,FIO_SS_BW_SLOPE =FIO_SS_BW |FIO_SS_SLOPE , + FIO_SS_LAT_SLOPE = FIO_SS_LAT | FIO_SS_SLOPE, };#endif
Re: [PATCH 2/2] fio: add latency steady state detection
In reply to Luis ChamberlainOn Fri, 25 Jul 2025 at 07:20, Luis Chamberlain <[email protected]> wrote:
<snip>
Quoted 10 lines
HOWTO.rst +2
-0@@ -4154,6 +4154,18 @@ Steady state Collect bandwidth data and calculate the least squares regression slope. Stop the job if the slope falls below the specified limit. + **lat** + Collect completion latency data and calculate the maximum mean
HOWTO.rst Steady state · line 4168
Quoted 10 lines
HOWTO.rst +10
-0+ deviation. Stop the job if the deviation falls below the specified + limit. The latency values are weighted by the number of I/O samples + in each measurement interval. + + **lat_slope** + Collect completion latency data and calculate the least squares + regression slope. Stop the job if the slope falls below the + specified limit. The latency values are weighted by the number + of I/O samples in each measurement interval. +
You'll need to update the man page (fio.1) too.
Signature · 1 line
Sitsofe