Recent changes (master)
The following changes since commit 942d66c85ee8f007ea5f1097d097cf9a44b662a0:
doc: update about size (2022-12-01 11:12:35 -0500)
are available in the Git repository at:
git://git.kernel.dk/fio.git master
for you to fetch changes up to 3afc2d8ac30c58372a1b7ccabaea0f3eae4ddaba:
engines/libblkio: Share a single blkio instance among threads in same process (2022-12-02 16:24:03 -0500)
----------------------------------------------------------------
Alberto Faria (10):
Add a libblkio engine
Add engine flag FIO_SKIPPABLE_IOMEM_ALLOC
engines/libblkio: Allow setting option mem/iomem engines/libblkio: Add support for poll queues
engines/libblkio: Add option libblkio_vectored
engines/libblkio: Add option libblkio_write_zeroes_on_trim engines/libblkio: Add option libblkio_wait_mode engines/libblkio: Add option libblkio_force_enable_completion_eventfd engines/libblkio: Add options for some driver-specific properties engines/libblkio: Share a single blkio instance among threads in same process
HOWTO.rst | 95 ++++ Makefile | 6 + configure | 25 + engines/libblkio.c | 914 ++++++++++++++++++++++++++++++ examples/libblkio-io_uring.fio | 29 + examples/libblkio-virtio-blk-vfio-pci.fio | 29 + fio.1 | 78 +++ ioengines.h | 2 + memory.c | 22 +- optgroup.h | 2 + 10 files changed, 1192 insertions(+), 10 deletions(-) create mode 100644 engines/libblkio.c create mode 100644 examples/libblkio-io_uring.fio create mode 100644 examples/libblkio-virtio-blk-vfio-pci.fio
---
Diff of recent changes:
HOWTO.rst +95 -0
@@ -2195,6 +2195,21 @@ I/O engine the SPDK NVMe driver, or your own custom NVMe driver. The xnvme engine includes engine specific options. (See https://xnvme.io). + **libblkio** + Use the libblkio library + (https://gitlab.com/libblkio/libblkio). The specific + *driver* to use must be set using + :option:`libblkio_driver`. If + :option:`mem`/:option:`iomem` is not specified, memory + allocation is delegated to libblkio (and so is + guaranteed to work with the selected *driver*). One + libblkio instance is used per process, so all jobs + setting option :option:`thread` will share a single + instance (with one queue per thread) and must specify + compatible options. Note that some drivers don't allow + several instances to access the same device or file + simultaneously, but allow it for threads. + I/O engine specific parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -2326,6 +2341,12 @@ with the caveat that when used on the command line, they must come after the by the application. The benefits are more efficient IO for high IOPS scenarios, and lower latencies for low queue depth IO. + [libblkio] + + Use poll queues. This is incompatible with + :option:`libblkio_wait_mode=eventfd <libblkio_wait_mode>` and + :option:`libblkio_force_enable_completion_eventfd`. + [pvsync2] Set RWF_HIPRI on I/O, indicating to the kernel that it's of higher priority
@@ -2847,6 +2868,80 @@ with the caveat that when used on the command line, they must come after the If this option is set. xnvme will use vectored read/write commands. +.. option:: libblkio_driver=str : [libblkio] + + The libblkio *driver* to use. Different drivers access devices through + different underlying interfaces. Available drivers depend on the + libblkio version in use and are listed at + https://libblkio.gitlab.io/libblkio/blkio.html#drivers + +.. option:: libblkio_path=str : [libblkio] + + Sets the value of the driver-specific "path" property before connecting + the libblkio instance, which identifies the target device or file on + which to perform I/O. Its exact semantics are driver-dependent and not + all drivers may support it; see + https://libblkio.gitlab.io/libblkio/blkio.html#drivers + +.. option:: libblkio_pre_connect_props=str : [libblkio] + + A colon-separated list of additional libblkio properties to be set after + creating but before connecting the libblkio instance. Each property must + have the format ``<name>=<value>``. Colons can be escaped as ``\:``. + These are set after the engine sets any other properties, so those can + be overriden. Available properties depend on the libblkio version in use + and are listed at + https://libblkio.gitlab.io/libblkio/blkio.html#properties + +.. option:: libblkio_num_entries=int : [libblkio] + + Sets the value of the driver-specific "num-entries" property before + starting the libblkio instance. Its exact semantics are driver-dependent + and not all drivers may support it; see + https://libblkio.gitlab.io/libblkio/blkio.html#drivers + +.. option:: libblkio_queue_size=int : [libblkio] + + Sets the value of the driver-specific "queue-size" property before + starting the libblkio instance. Its exact semantics are driver-dependent + and not all drivers may support it; see + https://libblkio.gitlab.io/libblkio/blkio.html#drivers + +.. option:: libblkio_pre_start_props=str : [libblkio] + + A colon-separated list of additional libblkio properties to be set after + connecting but before starting the libblkio instance. Each property must + have the format ``<name>=<value>``. Colons can be escaped as ``\:``. + These are set after the engine sets any other properties, so those can + be overriden. Available properties depend on the libblkio version in use + and are listed at + https://libblkio.gitlab.io/libblkio/blkio.html#properties + +.. option:: libblkio_vectored : [libblkio] + + Submit vectored read and write requests. + +.. option:: libblkio_write_zeroes_on_trim : [libblkio] + + Submit trims as "write zeroes" requests instead of discard requests. + +.. option:: libblkio_wait_mode=str : [libblkio] + + How to wait for completions: + + **block** (default) + Use a blocking call to ``blkioq_do_io()``. + **eventfd** + Use a blocking call to ``read()`` on the completion eventfd. + **loop** + Use a busy loop with a non-blocking call to ``blkioq_do_io()``. + +.. option:: libblkio_force_enable_completion_eventfd : [libblkio] + + Enable the queue's completion eventfd even when unused. This may impact + performance. The default is to enable it only if + :option:`libblkio_wait_mode=eventfd <libblkio_wait_mode>`. + I/O depth ~~~~~~~~~
Makefile +6 -0
@@ -237,6 +237,12 @@ ifdef CONFIG_LIBXNVME xnvme_CFLAGS = $(LIBXNVME_CFLAGS) ENGINES += xnvme endif +ifdef CONFIG_LIBBLKIO + libblkio_SRCS = engines/libblkio.c + libblkio_LIBS = $(LIBBLKIO_LIBS) + libblkio_CFLAGS = $(LIBBLKIO_CFLAGS) + ENGINES += libblkio +endif ifeq ($(CONFIG_TARGET_OS), Linux) SOURCE += diskutil.c fifo.c blktrace.c cgroup.c trim.c engines/sg.c \ oslib/linux-dev-lookup.c engines/io_uring.c engines/nvme.c
configure +25 -0
@@ -176,6 +176,7 @@ libiscsi="no" libnbd="no" libnfs="" xnvme="" +libblkio="" libzbc="" dfs="" seed_buckets=""
@@ -248,6 +249,8 @@ for opt do ;; --disable-xnvme) xnvme="no" ;; + --disable-libblkio) libblkio="no" + ;; --disable-tcmalloc) disable_tcmalloc="yes" ;; --disable-libnfs) libnfs="no"
@@ -304,6 +307,7 @@ if test "$show_help" = "yes" ; then echo "--enable-libiscsi Enable iscsi support" echo "--enable-libnbd Enable libnbd (NBD engine) support" echo "--disable-xnvme Disable xnvme support even if found" + echo "--disable-libblkio Disable libblkio support even if found" echo "--disable-libzbc Disable libzbc even if found" echo "--disable-tcmalloc Disable tcmalloc support" echo "--dynamic-libengines Lib-based ioengines as dynamic libraries"
@@ -2663,6 +2667,22 @@ if test "$xnvme" != "no" ; then fi print_config "xnvme engine" "$xnvme" +########################################## +# Check if we have libblkio +if test "$libblkio" != "no" ; then + if check_min_lib_version blkio 1.0.0; then + libblkio="yes" + libblkio_cflags=$(pkg-config --cflags blkio) + libblkio_libs=$(pkg-config --libs blkio) + else + if test "$libblkio" = "yes" ; then + feature_not_found "libblkio" "libblkio-dev or libblkio-devel" + fi + libblkio="no" + fi +fi +print_config "libblkio engine" "$libblkio" + ########################################## # check march=armv8-a+crc+crypto if test "$march_armv8_a_crc_crypto" != "yes" ; then
@@ -3276,6 +3296,11 @@ if test "$xnvme" = "yes" ; then echo "LIBXNVME_CFLAGS=$xnvme_cflags" >> $config_host_mak echo "LIBXNVME_LIBS=$xnvme_libs" >> $config_host_mak fi +if test "$libblkio" = "yes" ; then + output_sym "CONFIG_LIBBLKIO" + echo "LIBBLKIO_CFLAGS=$libblkio_cflags" >> $config_host_mak + echo "LIBBLKIO_LIBS=$libblkio_libs" >> $config_host_mak +fi if test "$dynamic_engines" = "yes" ; then output_sym "CONFIG_DYNAMIC_ENGINES" fi
engines/libblkio.c new file +914 -0
914 lines not on this page · raw message
examples/libblkio-io_uring.fio new file +29 -0
@@ -0,0 +1,29 @@ +; Benchmark accessing a regular file or block device using libblkio. +; +; Replace "/dev/nvme0n1" below with the path to your file or device, or override +; it by passing the '--libblkio_path=...' flag to fio. +; +; In the example below, the two subjobs of "job-B" *and* the single subjob of +; "job-C" will share a single libblkio instance, and "job-A" will use a separate +; libblkio instance. +; +; For information on libblkio, see: https://gitlab.com/libblkio/libblkio + +[global] +ioengine=libblkio +libblkio_driver=io_uring +libblkio_path=/dev/nvme0n1 ; REPLACE THIS WITH THE RIGHT PATH +rw=randread +blocksize=4k +direct=1 +time_based=1 +runtime=10s + +[job-A] + +[job-B] +numjobs=2 ; run two copies of this job simultaneously +thread=1 ; have each copy run as a separate thread in the *same* process + +[job-C] +thread=1 ; have the job run as a thread in the *same* process as "job-B"
examples/libblkio-virtio-blk-vfio-pci.fio new file +29 -0
@@ -0,0 +1,29 @@ +; Benchmark accessing a PCI virtio-blk device using libblkio. +; +; Replace "/sys/bus/pci/devices/0000:00:01.0" below with the path to your +; device's sysfs directory, or override it by passing the '--libblkio_path=...' +; flag to fio. +; +; In the example below, the two subjobs of "job-B" *and* the single subjob of +; "job-C" will share a single libblkio instance, and "job-A" will use a separate +; libblkio instance. +; +; For information on libblkio, see: https://gitlab.com/libblkio/libblkio + +[global] +ioengine=libblkio +libblkio_driver=virtio-blk-vfio-pci +libblkio_path=/sys/bus/pci/devices/0000:00:01.0 ; REPLACE THIS WITH THE RIGHT PATH +rw=randread +blocksize=4k +time_based=1 +runtime=10s + +[job-A] + +[job-B] +numjobs=2 ; run two copies of this job simultaneously +thread=1 ; have each copy run as a separate thread in the *same* process + +[job-C] +thread=1 ; have the job run as a thread in the *same* process as "job-B"
fio.1 +78 -0
90 lines not on this page · raw message
ioengines.h +2 -0
@@ -87,6 +87,8 @@ enum fio_ioengine_flags {FIO_NO_OFFLOAD =1 <<15 ,/* no async offload */ FIO_ASYNCIO_SETS_ISSUE_TIME =1 <<16 ,/* async ioengine with commit function that sets issue_time */ +FIO_SKIPPABLE_IOMEM_ALLOC + = 1 << 17, /* skip iomem_alloc & iomem_free if job sets mem/iomem */ };/*
memory.c +12 -10
@@ -305,16 +305,18 @@ int allocate_io_mem(struct thread_data *td)dprint (FD_MEM ,"Alloc %llu for buffers\n" , (unsigned long long ) total_mem);/* -* If the IO engine has hooks to allocate/free memory, use those. But -* error out if the user explicitly asked for something else. +* If the IO engine has hooks to allocate/free memory and the user +* doesn't explicitly ask for something else, use those. But fail if the +* user asks for something else with an engine that doesn't allow that. */ -if (td->io_ops->iomem_alloc) { -if (fio_option_is_set (&td->o, mem_type)) { -log_err ("fio: option 'mem/iomem' conflicts with specified IO engine\n" ); - ret =1 ; - }else - ret = td->io_ops->iomem_alloc (td, total_mem); - }else if (td->o.mem_type ==MEM_MALLOC ) +if (td->io_ops->iomem_alloc &&fio_option_is_set (&td->o, mem_type) && + !td_ioengine_flagged (td,FIO_SKIPPABLE_IOMEM_ALLOC )) { +log_err ("fio: option 'mem/iomem' conflicts with specified IO engine\n" ); + ret =1 ; + }else if (td->io_ops->iomem_alloc && + !fio_option_is_set (&td->o, mem_type)) + ret = td->io_ops->iomem_alloc (td, total_mem); +else if (td->o.mem_type ==MEM_MALLOC ) ret =alloc_mem_malloc (td, total_mem);else if (td->o.mem_type ==MEM_SHM || td->o.mem_type ==MEM_SHMHUGE ) ret =alloc_mem_shm (td, total_mem);
@@ -342,7 +344,7 @@ void free_io_mem(struct thread_data *td)if (td->o.odirect || td->o.oatomic) total_mem += page_mask; -if (td->io_ops->iomem_alloc) { +if (td->io_ops->iomem_alloc && !fio_option_is_set (&td->o, mem_type)) {if (td->io_ops->iomem_free) td->io_ops->iomem_free (td); }else if (td->o.mem_type ==MEM_MALLOC )
optgroup.h +2 -0
@@ -73,6 +73,7 @@ enum opt_category_group { __FIO_OPT_G_NFS, __FIO_OPT_G_WINDOWSAIO, __FIO_OPT_G_XNVME, + __FIO_OPT_G_LIBBLKIO,FIO_OPT_G_RATE = (1ULL << __FIO_OPT_G_RATE), FIO_OPT_G_ZONE = (1ULL << __FIO_OPT_G_ZONE),
@@ -120,6 +121,7 @@ enum opt_category_group {FIO_OPT_G_DFS = (1ULL << __FIO_OPT_G_DFS),FIO_OPT_G_WINDOWSAIO = (1ULL << __FIO_OPT_G_WINDOWSAIO),FIO_OPT_G_XNVME = (1ULL << __FIO_OPT_G_XNVME), + FIO_OPT_G_LIBBLKIO = (1ULL << __FIO_OPT_G_LIBBLKIO), };extern const struct opt_group *opt_group_from_mask (uint64_t *mask);