fio: option to make engines load dynamically

Signed-off-by Yigal Korman <[email protected]>

---
 ioengines.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

ioengines.c +1 -0

index 78262bc1..20224e59 100644
@@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
 static void *dlopen_external(struct thread_data *td, const char *engine)
 {
 	char engine_path[PATH_MAX];
+	void *dlhandle;
 
 	sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
 

Hi,

The following patchset provides a configuration option that converts a select set of engines into external engines that are loaded dynamically on demand. The purpose of this option is to provide distribution packagers the ability to separate the core fio functionality from the addition optional engines. Some of the optional engines have multiple dependencies that increase the footprint of fio considerably even when the user requires only the basic 'sync' engine. See examples of the issue below.

I've tried to keep the changes as small and unobtrusive as possible. The only significant change is the way an engine is defined in the Makefile (variables are now prefixed with engine name). If the option is provided, the Makefile will build the engines as external libraries and install them in /usr/lib/fio by default. Fio will then search for the engines in that location if the engine is not compiled in. The last patch should help with the migration to the new scheme by suggesting that an additional package is required if the engine is not found.

Future work to consider:
* Move the engine help text from fio to the engines themselves. Create
* a whitelist of engines so that only engines that are part of the main
* codebase are searched and loaded automatically. Convert more engines
* to this scheme. The current set of engines was selected based on
* simplicity and external dependencies. Engines with multiple exported
* flavors (i.e. glusterfs) are not converted yet.

Branch is also available on github, at
https://github.com/ykorman/fio/tree/dynamic-libengines.

A sample of how to package fio with this new mode for RPM-based distributions is available here:
https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines

A sample of the packages for Fedora can be found here:
https://copr.fedorainfracloud.org/coprs/ykorman/fio/

See examples of how this improves our quality of life (tm) below the pull summary.

Yigal Korman (3):
configure/Makefile: engine LIBS consistency
configure: new --dynamic-libengines build option
fio: suggest installing engine package when missing

 .gitignore         |  1 +
 Makefile           | 86 +++++++++++++++++++++++++++++++++++++---------
 configure          | 20 +++++------
 engines/dev-dax.c  |  2 +-
 engines/guasi.c    |  2 +-
 engines/http.c     |  2 +-
 engines/libaio.c   |  2 +-
 engines/libhdfs.c  |  6 ++--
 engines/libiscsi.c |  6 ++--
 engines/libpmem.c  |  2 +-
 engines/libzbc.c   |  2 +-
 engines/nbd.c      |  2 +-
 engines/pmemblk.c  |  2 +-
 engines/rados.c    |  2 +-
 engines/rbd.c      |  2 +-
 engines/rdma.c     |  9 ++---
 ioengines.c        | 26 ++++++++++++--
 ioengines.h        |  6 ++++
 os/os-linux.h      |  2 ++
 19 files changed, 131 insertions(+), 51 deletions(-)

Quoted 10 lines · Yigal Korman

Hi,

The following patchset provides a configuration option that converts a select set of engines into external engines that are loaded dynamically on demand. The purpose of this option is to provide distribution packagers the ability to separate the core fio functionality from the addition optional engines. Some of the optional engines have multiple dependencies that increase the footprint of fio considerably even when the user requires only the basic 'sync' engine. See examples of the issue below.

Did you forget to include the examples?

I'm not sure I see a benefit to this change. fio isn't some core system service--it's a benchmarking tool. Maybe your examples shed more light on why this matters?

-Jeff

Quoted 18 lines

I've tried to keep the changes as small and unobtrusive as possible. The only significant change is the way an engine is defined in the Makefile (variables are now prefixed with engine name). If the option is provided, the Makefile will build the engines as external libraries and install them in /usr/lib/fio by default. Fio will then search for the engines in that location if the engine is not compiled in. The last patch should help with the migration to the new scheme by suggesting that an additional package is required if the engine is not found.

Future work to consider:
* Move the engine help text from fio to the engines themselves. Create
* a whitelist of engines so that only engines that are part of the main
* codebase are searched and loaded automatically. Convert more engines
* to this scheme. The current set of engines was selected based on
* simplicity and external dependencies. Engines with multiple exported
* flavors (i.e. glusterfs) are not converted yet.

Quoted 6 lines

Branch is also available on github, at
https://github.com/ykorman/fio/tree/dynamic-libengines.

A sample of how to package fio with this new mode for RPM-based distributions is available here:

https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines

Quoted 33 lines

A sample of the packages for Fedora can be found here:
https://copr.fedorainfracloud.org/coprs/ykorman/fio/

See examples of how this improves our quality of life (tm) below the pull summary.

Yigal Korman (3):
configure/Makefile: engine LIBS consistency
configure: new --dynamic-libengines build option
fio: suggest installing engine package when missing

 .gitignore         |  1 +
 Makefile           | 86 +++++++++++++++++++++++++++++++++++++---------
 configure          | 20 +++++------
 engines/dev-dax.c  |  2 +-
 engines/guasi.c    |  2 +-
 engines/http.c     |  2 +-
 engines/libaio.c   |  2 +-
 engines/libhdfs.c  |  6 ++--
 engines/libiscsi.c |  6 ++--
 engines/libpmem.c  |  2 +-
 engines/libzbc.c   |  2 +-
 engines/nbd.c      |  2 +-
 engines/pmemblk.c  |  2 +-
 engines/rados.c    |  2 +-
 engines/rbd.c      |  2 +-
 engines/rdma.c     |  9 ++---
 ioengines.c        | 26 ++++++++++++--
 ioengines.h        |  6 ++++
 os/os-linux.h      |  2 ++
 19 files changed, 131 insertions(+), 51 deletions(-)

Quoted 10 lines · Yigal Korman

Hi,

The following patchset provides a configuration option that converts a select set of engines into external engines that are loaded dynamically on demand. The purpose of this option is to provide distribution packagers the ability to separate the core fio functionality from the addition optional engines. Some of the optional engines have multiple dependencies that increase the footprint of fio considerably even when the user requires only the basic 'sync' engine. See examples of the issue below.

Did you forget to include the examples?

I'm not sure I see a benefit to this change. fio isn't some core system service--it's a benchmarking tool. Maybe your examples shed more light on why this matters?

-Jeff

Quoted 57 lines

I've tried to keep the changes as small and unobtrusive as possible. The only significant change is the way an engine is defined in the Makefile (variables are now prefixed with engine name). If the option is provided, the Makefile will build the engines as external libraries and install them in /usr/lib/fio by default. Fio will then search for the engines in that location if the engine is not compiled in. The last patch should help with the migration to the new scheme by suggesting that an additional package is required if the engine is not found.

Future work to consider:
* Move the engine help text from fio to the engines themselves. Create
* a whitelist of engines so that only engines that are part of the main
* codebase are searched and loaded automatically. Convert more engines
* to this scheme. The current set of engines was selected based on
* simplicity and external dependencies. Engines with multiple exported
* flavors (i.e. glusterfs) are not converted yet.

Branch is also available on github, at
https://github.com/ykorman/fio/tree/dynamic-libengines.

A sample of how to package fio with this new mode for RPM-based distributions is available here:
https://src.fedoraproject.org/fork/ykorman/rpms/fio/tree/dynamic-libengines

A sample of the packages for Fedora can be found here:
https://copr.fedorainfracloud.org/coprs/ykorman/fio/

See examples of how this improves our quality of life (tm) below the pull summary.

Yigal Korman (3):
configure/Makefile: engine LIBS consistency
configure: new --dynamic-libengines build option
fio: suggest installing engine package when missing

 .gitignore         |  1 +
 Makefile           | 86 +++++++++++++++++++++++++++++++++++++---------
 configure          | 20 +++++------
 engines/dev-dax.c  |  2 +-
 engines/guasi.c    |  2 +-
 engines/http.c     |  2 +-
 engines/libaio.c   |  2 +-
 engines/libhdfs.c  |  6 ++--
 engines/libiscsi.c |  6 ++--
 engines/libpmem.c  |  2 +-
 engines/libzbc.c   |  2 +-
 engines/nbd.c      |  2 +-
 engines/pmemblk.c  |  2 +-
 engines/rados.c    |  2 +-
 engines/rbd.c      |  2 +-
 engines/rdma.c     |  9 ++---
 ioengines.c        | 26 ++++++++++++--
 ioengines.h        |  6 ++++
 os/os-linux.h      |  2 ++
 19 files changed, 131 insertions(+), 51 deletions(-)

When enabled, some of the more dependency-heavy internal engines are converted to "plugin" engines, i.e. they are built into separate object files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not force a long list of package dependencies from the base fio package.

Quoted 5 lines · Yigal Korman

When enabled, some of the more dependency-heavy internal engines are converted to "plugin" engines, i.e. they are built into separate object files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not force a long list of package dependencies from the base fio package.

How does this impact the performance of the engine? It'd be interesting to run a test with something ala:

fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1

with the current build, then apply these patches (and turn the null engine into an externally loadable engine, of course), and re-run the test case.

For what it's worth, I like the change in general, as dependencies do pile on. But I'd like to ensure that we're not taking a performance hit for something like this.

--
Jens Axboe

Quoted 17 lines · Yigal Korman

Signed-off-by Yigal Korman <[email protected]>

---
 ioengines.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

ioengines.c +1 -0

index 78262bc1..20224e59 100644
@@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
 static void *dlopen_external(struct thread_data *td, const char *engine)
 {
      char engine_path[PATH_MAX];
+     void *dlhandle;
 
      sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
 

ioengines.c static struct ioengine_ops *find_ioengine(const char *name) · line 94

Quoted 10 lines

ioengines.c +6 -1

-     return dlopen(engine_path, RTLD_LAZY);
+     dlhandle = dlopen(engine_path, RTLD_LAZY);
+     if (!dlhandle)
+             log_info("Engine not available; Install the fio-engine-%s package\n",
+                      engine);
+
+     return dlhandle;
 }
 #else
 #define dlopen_external(td, engine) (NULL)

This assumes fio is part of a package, and that's not always (often?) the case. It may be that the engine simply wasn't built.

-Jeff

Quoted 11 lines · Jens Axboe

Quoted 5 lines · Yigal Korman

When enabled, some of the more dependency-heavy internal engines are converted to "plugin" engines, i.e. they are built into separate object files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not force a long list of package dependencies from the base fio package.

How does this impact the performance of the engine? It'd be interesting to run a test with something ala:

Quoted 8 lines

fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1

with the current build, then apply these patches (and turn the null engine into an externally loadable engine, of course), and re-run the test case.

For what it's worth, I like the change in general, as dependencies do pile on. But I'd like to ensure that we're not taking a performance hit for something like this.

Great to hear.

Here are the results of the run you suggested:

current build (statically linked) -

[root@host fio]# ./fio.static --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
stdev=38.59, samples=52
iops : min=969293, max=1015596, avg=1000951.42,
stdev=9879.80, samples=52

  cpu          : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s

(4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec

With patches applied[0] -

[root@host fio]# ./fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
stdev=45.93, samples=52
iops : min=960962, max=1021038, avg=1007067.92,
stdev=11758.92, samples=52

  cpu          : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s

(4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec

I wasn't expecting dynamic linking to have a performance impact and the results seem to agree.

Quoted 20 lines · Jeff Moyer

Quoted 17 lines · Yigal Korman

Signed-off-by Yigal Korman <[email protected]>

---
 ioengines.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

ioengines.c +1 -0

index 78262bc1..20224e59 100644
@@ -79,10 +79,16 @@ static struct ioengine_ops *find_ioengine(const char *name)
 static void *dlopen_external(struct thread_data *td, const char *engine)
 {
      char engine_path[PATH_MAX];
+     void *dlhandle;
 
      sprintf(engine_path, "%s/lib%s.so", FIO_EXT_ENG_DIR, engine);
 

Quoted 13 lines

ioengines.c static struct ioengine_ops *find_ioengine(const char *name) · line 94

Quoted 10 lines

ioengines.c +6 -1

-     return dlopen(engine_path, RTLD_LAZY);
+     dlhandle = dlopen(engine_path, RTLD_LAZY);
+     if (!dlhandle)
+             log_info("Engine not available; Install the fio-engine-%s package\n",
+                      engine);
+
+     return dlhandle;
 }
 #else
 #define dlopen_external(td, engine) (NULL)

This assumes fio is part of a package, and that's not always (often?) the case. It may be that the engine simply wasn't built.

I agree.
This patch is more of a showcase of how to help package maintainers transition users to the new packaging model.
I will drop it from the next patchset version.
This patch might have made more sense if I had implemented a whitelist of engines that were compiled dynamically.

Quoted 73 lines · Yigal Korman

Quoted 11 lines · Jens Axboe

Quoted 5 lines · Yigal Korman

When enabled, some of the more dependency-heavy internal engines are converted to "plugin" engines, i.e. they are built into separate object files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not force a long list of package dependencies from the base fio package.

How does this impact the performance of the engine? It'd be interesting to run a test with something ala:

Quoted 8 lines

fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1

with the current build, then apply these patches (and turn the null engine into an externally loadable engine, of course), and re-run the test case.

For what it's worth, I like the change in general, as dependencies do pile on. But I'd like to ensure that we're not taking a performance hit for something like this.

Great to hear.

Here are the results of the run you suggested:

current build (statically linked) -

[root@host fio]# ./fio.static --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
stdev=38.59, samples=52
iops : min=969293, max=1015596, avg=1000951.42,
stdev=9879.80, samples=52

  cpu          : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s

(4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec

With patches applied[0] -

[root@host fio]# ./fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
stdev=45.93, samples=52
iops : min=960962, max=1021038, avg=1007067.92,
stdev=11758.92, samples=52

  cpu          : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Quoted 6 lines

Run status group 0 (all jobs):

   READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s

(4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec

I wasn't expecting dynamic linking to have a performance impact and the results seem to agree.

That's nice to see. I'd probably add --cpus_allowed=0 to it for better locality, and then run 10 tests back-to-back with each just to be sure. There will be some fluctuations, but that should be enough for me to feel comfortable.

1000K IOPS is pretty slow though, what are you running this on? Pretty sure my laptop does 10x that.

Quoted 81 lines · Jens Axboe

Quoted 79 lines · Yigal Korman

Quoted 11 lines · Jens Axboe

Quoted 5 lines · Yigal Korman

When enabled, some of the more dependency-heavy internal engines are converted to "plugin" engines, i.e. they are built into separate object files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not force a long list of package dependencies from the base fio package.

How does this impact the performance of the engine? It'd be interesting to run a test with something ala:

Quoted 8 lines

fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1

with the current build, then apply these patches (and turn the null engine into an externally loadable engine, of course), and re-run the test case.

For what it's worth, I like the change in general, as dependencies do pile on. But I'd like to ensure that we're not taking a performance hit for something like this.

Great to hear.

Here are the results of the run you suggested:

current build (statically linked) -

[root@host fio]# ./fio.static --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
stdev=38.59, samples=52
iops : min=969293, max=1015596, avg=1000951.42,
stdev=9879.80, samples=52

  cpu          : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s

(4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec

With patches applied[0] -

[root@host fio]# ./fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
stdev=45.93, samples=52
iops : min=960962, max=1021038, avg=1007067.92,
stdev=11758.92, samples=52

  cpu          : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s

(4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec

I wasn't expecting dynamic linking to have a performance impact and the results seem to agree.

Quoted 7 lines

That's nice to see. I'd probably add --cpus_allowed=0 to it for better locality, and then run 10 tests back-to-back with each just to be sure. There will be some fluctuations, but that should be enough for me to feel comfortable.

1000K IOPS is pretty slow though, what are you running this on? Pretty sure my laptop does 10x that.

Ran the testing here, and nothing statistically significant.

--
Jens Axboe

Quoted 83 lines · Jens Axboe

Quoted 81 lines · Jens Axboe

Quoted 79 lines · Yigal Korman

Quoted 11 lines · Jens Axboe

Quoted 5 lines · Yigal Korman

When enabled, some of the more dependency-heavy internal engines are converted to "plugin" engines, i.e. they are built into separate object files and are loaded by fio on demand.
This helps downstream distros package these engines separately and not force a long list of package dependencies from the base fio package.

How does this impact the performance of the engine? It'd be interesting to run a test with something ala:

Quoted 8 lines

fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1

with the current build, then apply these patches (and turn the null engine into an externally loadable engine, of course), and re-run the test case.

For what it's worth, I like the change in general, as dependencies do pile on. But I'd like to ensure that we're not taking a performance hit for something like this.

Great to hear.

Here are the results of the run you suggested:

current build (statically linked) -

[root@host fio]# ./fio.static --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3910MiB/s][r=1001k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=155: Tue Jun 30 06:55:19 2020
read: IOPS=1000k, BW=3906MiB/s (4095MB/s)(100GiB/26219msec)
bw ( MiB/s): min= 3786, max= 3967, per=100.00%, avg=3909.96,
stdev=38.59, samples=52
iops : min=969293, max=1015596, avg=1000951.42,
stdev=9879.80, samples=52

  cpu          : usr=61.66%, sys=38.31%, ctx=103, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3906MiB/s (4095MB/s), 3906MiB/s-3906MiB/s

(4095MB/s-4095MB/s), io=100GiB (107GB), run=26219-26219msec

With patches applied[0] -

[root@host fio]# ./fio --name=test --ioengine=null --size=100g --rw=randread --norandommap --gtod_reduce=1
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.20-70-g9888
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=3905MiB/s][r=1000k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=158: Tue Jun 30 06:55:49 2020
read: IOPS=1006k, BW=3929MiB/s (4120MB/s)(100GiB/26060msec)
bw ( MiB/s): min= 3753, max= 3988, per=100.00%, avg=3933.86,
stdev=45.93, samples=52
iops : min=960962, max=1021038, avg=1007067.92,
stdev=11758.92, samples=52

  cpu          : usr=62.14%, sys=37.81%, ctx=117, majf=8, minf=5
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26214400,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   READ: bw=3929MiB/s (4120MB/s), 3929MiB/s-3929MiB/s

(4120MB/s-4120MB/s), io=100GiB (107GB), run=26060-26060msec

I wasn't expecting dynamic linking to have a performance impact and the results seem to agree.

Quoted 7 lines

Quoted 7 lines

That's nice to see. I'd probably add --cpus_allowed=0 to it for better locality, and then run 10 tests back-to-back with each just to be sure. There will be some fluctuations, but that should be enough for me to feel comfortable.

1000K IOPS is pretty slow though, what are you running this on? Pretty sure my laptop does 10x that.

Yeah, a tiny VM on my laptop for playing around with the package manager, not a good candidate for perf testing.
I re-ran as you suggested on the host (i5-6200U @ 2.30GHz), see results here[0].

Quoted 1 line

Quoted 1 line

Ran the testing here, and nothing statistically significant.

Yes, I got the same. Thanks!

Would you like a pull request for the patchset? (after I fix the last patch)