Introduce the end_syncfs option
PATCH v2 1/1 Introduce the end_syncfs option
When benchmarking a file system file writes with buffered I/Os, the options fsync, fsync_on_close and end_fsync allow for accounting for all performance with the file data safely written to media. All these options can be used in different scenarios to emulate applications use of the fsync() system call.
However, these different possibilities all involve the fsync() system call which implies that when called the written files are always open. Depending on the file system, this characteristic is very limiting. E.g. the XFS file system optimizes data placement of closed files to the same block group in order to generate a write command pattern that is very sequential and localized (tight packing). The use of fio existing sync option variants thus does not allow measuring the performance benefits of this optimization. Furthermore, the option end_fsync applies to all files of a job, causing a loop to open, fsync() and close each written file. For benchmarks with a very large number of files (e.g. a long run), this is very ineficient and slow, and often causes the performance results to much lower than expected.
Solve this by introducing the end_syncfs option. If enabled, this option results in a call to the syncfs() system call when a job completes. This allows syncing all written files with a single system call and also avoids the need to reopen all written files. This is thus much faster than using end_fsync, and also enables file system writeback optimizations that rely on the files being written back to be closed.
The syncfs() system call is supported by Linux only. This support is detected in the configure script and if detected, the CONFIG_SYNCFS configuration option defined. When not supported, the helpers.c file defines the syncfs() function to return an error.
Like other sync variants, end_syncfs causes issuing an io_u when a job completes so that the time taken to write back all written files is accounted for in the final performance statistics. The io_u data direction DDIR_SYNCFS is defined to control this and used in the fio_file_syncfs() function in backend.c
---
Changes from v1:
- Rebase on latest code to have the correct server version change.
HOWTO.rst | 7 +++++++ backend.c | 37 ++++++++++++++++++++++++++++++++----- cconv.c | 2 ++ configure | 21 +++++++++++++++++++++ fio.1 | 6 ++++++ helpers.c | 8 ++++++++ helpers.h | 3 +++ io_ddir.h | 1 + io_u.c | 11 +++++++++-- options.c | 10 ++++++++++ server.h | 2 +- thread_options.h | 5 ++++- zbd.c | 1 + 13 files changed, 105 insertions(+), 9 deletions(-)
HOWTO.rst +7 -0
@@ -1494,6 +1494,13 @@ I/O type If true, :manpage:`fsync(2)` file contents when a write stage has completed. Default: false. +.. option:: end_syncfs=bool + + Equivalent to :option:`end_fsync` but instead of executing + :manpage:`fsync(2)` for each file of a write stage, execute + :manpage:`syncfs(2)` to synchronize all written files with a single + system call when a write stage has completed. Default: false. + .. option:: fsync_on_close=bool If true, fio will :manpage:`fsync(2)` a dirty file on close. This differs
backend.c +32 -5
@@ -226,7 +226,8 @@ static bool check_min_rate(struct thread_data *td, struct timespec *now)* Helper to handle the final sync of a file. Works just like the normal * io path, just does everything sync. */ -static bool fio_io_sync (struct thread_data *td,struct fio_file *f) +static bool fio_io_sync (struct thread_data *td,struct fio_file *f, +enum fio_ddir ddir) {struct io_u *io_u =__get_io_u (td);enum fio_q_status ret;
@@ -234,7 +235,7 @@ static bool fio_io_sync(struct thread_data *td, struct fio_file *f)if (!io_u)return true; - io_u->ddir =DDIR_SYNC ; + io_u->ddir = ddir; io_u->file = f;io_u_set (td, io_u,IO_U_F_NO_FILE_PUT );
@@ -273,18 +274,34 @@ static int fio_file_fsync(struct thread_data *td, struct fio_file *f)int ret, ret2;if (fio_file_open (f)) -return fio_io_sync (td, f); +return fio_io_sync (td, f,DDIR_SYNC );if (td_io_open_file (td, f))return 1 ; - ret =fio_io_sync (td, f); + ret =fio_io_sync (td, f,DDIR_SYNC ); ret2 =0 ;if (fio_file_open (f)) ret2 =td_io_close_file (td, f);return (ret || ret2); } +static int fio_file_syncfs (struct thread_data *td,struct fio_file *f) +{ +int ret; + +if (fio_file_open (f)) +return fio_io_sync (td, f,DDIR_SYNCFS ); + +if (td_io_open_file (td, f)) +return 1 ; + + ret =fio_io_sync (td, f,DDIR_SYNCFS ); +td_io_close_file (td, f); + +return ret; +} +static inline void __update_ts_cache (struct thread_data *td) {fio_gettime (&td->ts_cache,NULL );
@@ -593,7 +610,7 @@ static void do_verify(struct thread_data *td, uint64_t verify_bytes)for_each_file (td, f, i) {if (!fio_file_open (f))continue ; -if (fio_io_sync (td, f)) +if (fio_io_sync (td, f,DDIR_SYNC ))break ;if (file_invalidate_cache (td, f))break ;
@@ -1280,6 +1297,16 @@ reap: f->file_name); } } + +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ; + } + } }else {if (td->o.io_submit_mode ==IO_MODE_OFFLOAD )workqueue_flush (&td->io_wq);
cconv.c +2 -0
@@ -174,6 +174,7 @@ int convert_thread_options_to_cpu(struct thread_options *o, o->create_only =le32_to_cpu (top->create_only); o->filetype =le32_to_cpu (top->filetype); o->end_fsync =le32_to_cpu (top->end_fsync); + o->end_syncfs =le32_to_cpu (top->end_syncfs); o->pre_read =le32_to_cpu (top->pre_read); o->sync_io =le32_to_cpu (top->sync_io); o->write_hint =le32_to_cpu (top->write_hint);
@@ -443,6 +444,7 @@ void convert_thread_options_to_net(struct thread_options_pack *top, top->create_only =cpu_to_le32 (o->create_only); top->filetype =cpu_to_le32 (o->filetype); top->end_fsync =cpu_to_le32 (o->end_fsync); + top->end_syncfs =cpu_to_le32 (o->end_syncfs); top->pre_read =cpu_to_le32 (o->pre_read); top->sync_io =cpu_to_le32 (o->sync_io); top->write_hint =cpu_to_le32 (o->write_hint);
configure +21 -0
@@ -1327,6 +1327,24 @@ if compile_prog "" "" "sync_file_range"; then fi print_config "sync_file_range" "$sync_file_range" +########################################## +# syncfs() probe +if test "$syncfs" != "yes" ; then + syncfs="no" +fi +cat > $TMPC << EOF +#include <stdio.h> +#include <unistd.h> +int main(int argc, char **argv) +{ + return syncfs(0); +} +EOF +if compile_prog "" "" "syncfs"; then + syncfs="yes" +fi +print_config "syncfs" "$syncfs" + ########################################## # ASharedMemory_create() probe if test "$ASharedMemory_create" != "yes" ; then
@@ -3129,6 +3147,9 @@ fi if test "$sync_file_range" = "yes" ; then output_sym "CONFIG_SYNC_FILE_RANGE" fi +if test "$syncfs" = "yes" ; then + output_sym "CONFIG_SYNCFS" +fi if test "$ASharedMemory_create" = "yes" ; then output_sym "CONFIG_ASHAREDMEMORY_CREATE" fi
fio.1 +6 -0
@@ -1284,6 +1284,12 @@ will be done. Default: false. If true, \fBfsync\fR\|(2) file contents when a write stage has completed. Default: false. .TP +.BI end_syncfs \fR=\fPbool +Equivalent to \fBend_fsync\fR but instead of executing \fBfsync\fR\|(2) for +each file of a write stage, execute \fBsyncfs\fR\|(2) to synchronize all +written files with a single system call when a write stage has completed. +Default: false. +.TP .BI fsync_on_close \fR=\fPbool If true, fio will \fBfsync\fR\|(2) a dirty file on close. This differs from \fBend_fsync\fR in that it will happen on every file close, not
helpers.c +8 -0
@@ -26,6 +26,14 @@ int sync_file_range(int fd, uint64_t offset, uint64_t nbytes, }#endif +#ifndef CONFIG_SYNCFS +int syncfs (int fd) +{ + errno =ENOSYS ; +return -1 ; +} +#endif +#ifndef CONFIG_POSIX_FADVISE int posix_fadvise (int fd,off_t offset,off_t len,int advice) {
helpers.h +3 -0
@@ -11,6 +11,9 @@ extern int posix_fallocate(int fd, off_t offset, off_t len);extern int sync_file_range (int fd,uint64_t offset,uint64_t nbytes,unsigned int flags);#endif +#ifndef CONFIG_SYNCFS +extern int syncfs (int fd); +#endif extern int posix_fadvise (int fd,off_t offset,off_t len,int advice);#endif /* FIO_HELPERS_H_ */
io_ddir.h +1 -0
@@ -8,6 +8,7 @@ enum fio_ddir {DDIR_SYNC =3 ,DDIR_DATASYNC ,DDIR_SYNC_FILE_RANGE , +DDIR_SYNCFS ,DDIR_WAIT ,DDIR_LAST , DDIR_INVAL = -1,
io_u.c +9 -2
@@ -2445,6 +2445,11 @@ void io_u_fill_buffer(struct thread_data *td, struct io_u *io_u,fill_io_buffer (td, io_u->buf, min_write, max_bs); } +static int do_syncfs (const struct thread_data *td,struct fio_file *f) +{ +return syncfs (f->fd); +} +static int do_sync_file_range (const struct thread_data *td,struct fio_file *f) {
@@ -2476,9 +2481,11 @@ int do_io_u_sync(const struct thread_data *td, struct io_u *io_u) ret = io_u->xfer_buflen; io_u->error =EINVAL ;#endif - }else if (io_u->ddir ==DDIR_SYNC_FILE_RANGE ) + }else if (io_u->ddir ==DDIR_SYNC_FILE_RANGE ) { ret =do_sync_file_range (td, io_u->file); -else { + }else if (io_u->ddir ==DDIR_SYNCFS ) { + ret =do_syncfs (td, io_u->file); + }else { ret = io_u->xfer_buflen; io_u->error =EINVAL ; }
options.c +10 -0
@@ -4610,6 +4610,16 @@ struct fio_option fio_options[FIO_MAX_OPTS] = { .category = FIO_OPT_C_FILE, .group = FIO_OPT_G_INVALID, }, + { + .name = "end_syncfs", + .lname = "End sync FS", + .type = FIO_OPT_BOOL, + .off1 = offsetof(struct thread_options, end_syncfs), + .help = "Include sync of FS at the end of job", + .def = "0", + .category = FIO_OPT_C_FILE, + .group = FIO_OPT_G_INVALID, + }, { .name = "unlink", .lname = "Unlink file",
server.h +1 -1
@@ -51,7 +51,7 @@ struct fio_net_cmd_reply { };enum { -FIO_SERVER_VER =116 , +FIO_SERVER_VER =117 ,FIO_SERVER_MAX_FRAGMENT_PDU =1024 ,FIO_SERVER_MAX_CMD_MB =2048 ,
thread_options.h +4 -1
@@ -138,6 +138,7 @@ struct thread_options {unsigned int create_on_open;unsigned int create_only;unsigned int end_fsync; +unsigned int end_syncfs;unsigned int pre_read;unsigned int sync_io;unsigned int write_hint;
@@ -525,6 +526,8 @@ struct thread_options_pack {uint32_t exitall_error;uint32_t sync_file_range; +uint32_t end_syncfs; +uint32_t pad;struct zone_split zone_split[DDIR_RWDIR_CNT ][ZONESPLIT_MAX ];uint32_t zone_split_nr[DDIR_RWDIR_CNT ];
@@ -623,7 +626,7 @@ struct thread_options_pack {uint32_t lat_percentiles;uint32_t slat_percentiles;uint32_t percentile_precision; -uint32_t pad; +uint32_t pad2;fio_fp64_t percentile_list[FIO_IO_U_LIST_MAX_LEN ];uint8_t read_iolog_file[FIO_TOP_STR_MAX ];
zbd.c +1 -0
@@ -2310,6 +2310,7 @@ retry:/* fall-through */ case DDIR_DATASYNC :case DDIR_SYNC_FILE_RANGE : +case DDIR_SYNCFS :case DDIR_WAIT :case DDIR_LAST :case DDIR_INVAL :
Signature · 1 line
2.52.0
The result of fio's continuous integration tests was: success
For more details see https://github.com/fiotestbot/fio/actions/runs/21349020694
Quoted 1 line · Damien Le Moal
<snip>
backend.c reap: · line 1307
Quoted 9 lines
}backend.c +8
-0+ +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ;
This seems like it will break out of the loop after syncfs-ing the first file. Is this intentional or should fio break only if there is an error?
Vincent
Quoted 15 lines · Vincent Fu
Quoted 1 line · Damien Le Moal
<snip>
backend.c reap: · line 1307
Quoted 9 lines
}backend.c +8
-0+ +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ;This seems like it will break out of the loop after syncfs-ing the first file. Is this intentional or should fio break only if there is an error?
It is intentional and it is the entire point of this patch: doing syncfs for the first file will sync the entire FS, so all the other files will be sync-ed as well. We thus avoid the entire loop on all files which is extremely slow when running with a large number of files. We only need to do syncfs() once for the first file. Ideally, we should do it for the directory specified for the files, but that is a little more involved as a change.
Signature · 2 lines
Damien Le Moal Western Digital Research
Quoted 18 lines · Damien Le Moal
Quoted 15 lines · Vincent Fu
Quoted 1 line · Damien Le Moal
<snip>
backend.c reap: · line 1307
Quoted 9 lines
}backend.c +8
-0+ +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ;This seems like it will break out of the loop after syncfs-ing the first file. Is this intentional or should fio break only if there is an error?
Quoted 6 lines
It is intentional and it is the entire point of this patch: doing syncfs for the first file will sync the entire FS, so all the other files will be sync-ed as well. We thus avoid the entire loop on all files which is extremely slow when running with a large number of files. We only need to do
syncfs()once for the first file. Ideally, we should do it for the directory specified for the files, but that is a little more involved as a change.
Ok thank you for clarifying.
Please emphasize in the documentation that this feature assumes that all files will reside on the same file system.
Vincent
Quoted 26 lines · Vincent Fu
Quoted 18 lines · Damien Le Moal
Quoted 15 lines · Vincent Fu
Quoted 1 line · Damien Le Moal
<snip>
backend.c reap: · line 1307
Quoted 9 lines
}backend.c +8
-0+ +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ;This seems like it will break out of the loop after syncfs-ing the first file. Is this intentional or should fio break only if there is an error?
Quoted 6 lines
It is intentional and it is the entire point of this patch: doing syncfs for the first file will sync the entire FS, so all the other files will be sync-ed as well. We thus avoid the entire loop on all files which is extremely slow when running with a large number of files. We only need to do
syncfs()once for the first file. Ideally, we should do it for the directory specified for the files, but that is a little more involved as a change.
Quoted 3 lines
Ok thank you for clarifying.
Please emphasize in the documentation that this feature assumes that all files will reside on the same file system.
OK. Will do in v3.
Quoted 2 lines
Vincent
Signature · 2 lines
Damien Le Moal Western Digital Research
Quoted 27 lines · Damien Le Moal
Quoted 26 lines · Vincent Fu
Quoted 24 lines · On Mon, Jan 26, 2026 at 9:05?PM Damien Le Moal <[email protected]> wrote:
Quoted 15 lines · Vincent Fu
Quoted 1 line · On Mon, Jan 26, 2026 at 1:22?AM Damien Le Moal <[email protected]> wrote:
<snip>
backend.c reap: · line 1307
Quoted 9 lines
}backend.c +8
-0+ +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ;This seems like it will break out of the loop after syncfs-ing the first file. Is this intentional or should fio break only if there is an error?
It is intentional and it is the entire point of this patch: doing syncfs for the first file will sync the entire FS, so all the other files will be sync-ed as well. We thus avoid the entire loop on all files which is extremely slow when running with a large number of files. We only need to do
syncfs()once for the first file. Ideally, we should do it for the directory specified for the files, but that is a little more involved as a change.
Quoted 5 lines
Quoted 3 lines
Ok thank you for clarifying.
Please emphasize in the documentation that this feature assumes that all files will reside on the same file system.OK. Will do in v3.
That seems like a very odd limitation that people will never know about, and hence it won't do what they think it does. We stat these files for getting the size, right? Then do a sync for each fs. Only doing the first is a cop-out imho, and somewhat of a lazy way of doing it.
We either sync any fs that has a file that was written, or let's not add this feature. That can be done in any number of ways. It's a terrible option otherwise.
Signature · 1 line
Jens Axbo
Quoted 32 lines · Jens Axboe
Quoted 31 lines · Damien Le Moal
Quoted 29 lines · Vincent Fu
Quoted 18 lines · On Mon, Jan 26, 2026 at 9:05?PM Damien Le Moal <[email protected]> wrote:
Quoted 15 lines · Vincent Fu
Quoted 1 line · On Mon, Jan 26, 2026 at 1:22?AM Damien Le Moal <[email protected]> wrote:
<snip>
backend.c reap: · line 1307
Quoted 9 lines
}backend.c +8
-0+ +if (td->o.end_syncfs) { +td_set_runstate (td,TD_FSYNCING ); + +for_each_file (td, f, i) { +if (fio_file_syncfs (td, f)) +log_err ("fio: end_syncfs failed\n" ); +break ;This seems like it will break out of the loop after syncfs-ing the first file. Is this intentional or should fio break only if there is an error?
Quoted 6 lines
It is intentional and it is the entire point of this patch: doing syncfs for the first file will sync the entire FS, so all the other files will be sync-ed as well. We thus avoid the entire loop on all files which is extremely slow when running with a large number of files. We only need to do
syncfs()once for the first file. Ideally, we should do it for the directory specified for the files, but that is a little more involved as a change.Ok thank you for clarifying.
Please emphasize in the documentation that this feature assumes that all files will reside on the same file system.
Quoted 6 lines
Quoted 1 line
OK. Will do in v3.
That seems like a very odd limitation that people will never know about, and hence it won't do what they think it does. We stat these files for getting the size, right? Then do a sync for each fs. Only doing the first is a cop-out imho, and somewhat of a lazy way of doing it.
OK.
Quoted 4 lines
We either sync any fs that has a file that was written, or let's not add this feature. That can be done in any number of ways. It's a terrible option otherwise.
Signature · 2 lines
Damien Le Moal Western Digital Research