new options and fixes for xnvme ioengine
A message that is not in the archive (sent off-list or not yet received): CGME20221222044040epcas5p3abbde8c4a3775d39d237b0b6cf6dd04a@epcas5p3.samsung.com
cover new options and fixes for xnvme ioengine
This patch series adds
1. Few minor fixes and cleanups to the xnvme I/O engine.
2. xNVMe recently added a user-space VFIO based backend implemented using libvfn. This allow users to select that instead of SPDK.
3. Add subnqn option for fabrics target with multiple systems.
4. Add an option to select memory backend.
Ankit Kumar (4):
engines/xnvme: fixes for xnvme ioengine engines/xnvme: user space vfio based backend engines/xnvme: add subnqn to fio-options engines/xnvme: add support for picking mem backend
HOWTO.rst | 27 ++++++++++++++++++++++++++- engines/xnvme.c | 44 ++++++++++++++++++++++++++++++++++++++------ fio.1 | 32 +++++++++++++++++++++++++++++++- 3 files changed, 95 insertions(+), 8 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/4 engines/xnvme: fixes for xnvme ioengine
1. fix error-handling in xnvme_fioe_queue()
2. fix resource-leak in xnvme_fioe_init()
--- engines/xnvme.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
engines/xnvme.c +12 -3
@@ -322,12 +322,15 @@ static int xnvme_fioe_init(struct thread_data *td) xd->iocq =calloc (td->o.iodepth,sizeof (struct io_u *));if (!xd->iocq) { -log_err ("ioeng->init(): !calloc(), err(%d)\n" , errno); +free (xd); +log_err ("ioeng->init(): !calloc(xd->iocq), err(%d)\n" , errno);return 1 ; } xd->iovec =calloc (td->o.iodepth,sizeof (*xd->iovec));if (!xd->iovec) { +free (xd->iocq); +free (xd);log_err ("ioeng->init(): !calloc(xd->iovec), err(%d)\n" , errno);return 1 ; }
@@ -338,6 +341,10 @@ static int xnvme_fioe_init(struct thread_data *td)for_each_file (td, f, i) {if (_dev_open (td, f)) { +/* +* Note: We are not freeing xd, iocq and iovec. This +* will be done as part of cleanup routine. +*/ log_err ("ioeng->init(): failed; _dev_open(%s)\n" , f->file_name);return 1 ; }
@@ -506,9 +513,11 @@ static enum fio_q_status xnvme_fioe_queue(struct thread_data *td, struct io_u *idefault :log_err ("ioeng->queue(): ENOSYS: %u\n" , io_u->ddir); - err =-1 ; +xnvme_queue_put_cmd_ctx (ctx->async.queue, ctx); + + io_u->error =ENOSYS ;assert (false); -break ; +return FIO_Q_COMPLETED ; }if (vectored_io) {
Signature · 1 line
2.17.1
PATCH 2/4 engines/xnvme: user space vfio based backend
In reply to Ankit KumarAdd an option to use user-space VFIO-based backend, implemented using libvfn.
Update xnvme engine options for missing backends.
--- HOWTO.rst | 5 ++++- engines/xnvme.c | 7 ++++--- fio.1 | 6 +++++- 3 files changed, 13 insertions(+), 5 deletions(-)
HOWTO.rst +4 -1
@@ -2845,6 +2845,9 @@ with the caveat that when used on the command line, they must come after the **posix** Use the posix asynchronous I/O interface to perform one or more I/O operations asynchronously. + **vfio** + Use the user-space VFIO-based backend, implemented using + libvfn instead of SPDK. **nil** Do not transfer any data; just pretend to. This is mainly used for introspective performance evaluation.
@@ -2875,7 +2878,7 @@ with the caveat that when used on the command line, they must come after the .. option:: xnvme_dev_nsid=int : [xnvme] - xnvme namespace identifier for userspace NVMe driver, such as SPDK. + xnvme namespace identifier for userspace NVMe driver, SPDK or vfio. .. option:: xnvme_iovec=int : [xnvme]
engines/xnvme.c +4 -3
@@ -113,7 +113,8 @@ static struct fio_option options[] = { .lname = "xNVMe Asynchronous command-interface", .type = FIO_OPT_STR_STORE, .off1 = offsetof(struct xnvme_fioe_options, xnvme_async), - .help = "Select xNVMe async. interface: [emu,thrpool,io_uring,libaio,posix,nil]", + .help ="Select xNVMe async. interface: " + "[emu,thrpool,io_uring,io_uring_cmd,libaio,posix,vfio,nil]", .category = FIO_OPT_C_ENGINE, .group = FIO_OPT_G_XNVME, },
@@ -122,7 +123,7 @@ static struct fio_option options[] = { .lname = "xNVMe Synchronous. command-interface", .type = FIO_OPT_STR_STORE, .off1 = offsetof(struct xnvme_fioe_options, xnvme_sync), - .help = "Select xNVMe sync. interface: [nvme,psync]", + .help = "Select xNVMe sync. interface: [nvme,psync,block]", .category = FIO_OPT_C_ENGINE, .group = FIO_OPT_G_XNVME, },
@@ -131,7 +132,7 @@ static struct fio_option options[] = { .lname = "xNVMe Admin command-interface", .type = FIO_OPT_STR_STORE, .off1 = offsetof(struct xnvme_fioe_options, xnvme_admin), - .help = "Select xNVMe admin. cmd-interface: [nvme,block,file_as_ns]", + .help = "Select xNVMe admin. cmd-interface: [nvme,block]", .category = FIO_OPT_C_ENGINE, .group = FIO_OPT_G_XNVME, },
fio.1 +5 -1
@@ -2584,6 +2584,10 @@ Use Linux aio for Asynchronous I/O Use the posix asynchronous I/O interface to perform one or more I/O operations asynchronously. .TP +.BI vfio +Use the user-space VFIO-based backend, implemented using libvfn instead of +SPDK. +.TP .BI nil Do not transfer any data; just pretend to. This is mainly used for introspective performance evaluation.
@@ -2621,7 +2625,7 @@ Use Linux Block Layer ioctl() and sysfs for admin commands. .RE .TP .BI (xnvme)xnvme_dev_nsid\fR=\fPint -xnvme namespace identifier for userspace NVMe driver such as SPDK. +xnvme namespace identifier for userspace NVMe driver SPDK or vfio. .TP .BI (xnvme)xnvme_iovec If this option is set, xnvme will use vectored read/write commands.
Signature · 1 line
2.17.1
PATCH 3/4 engines/xnvme: add subnqn to fio-options
In reply to Ankit KumarFor fio to utilize a fabrics target with multiple systems, it needs a way for the user to specify which subsystem to use. This is done by providing 'subnqn' as fio-option.
--- HOWTO.rst | 5 +++++ engines/xnvme.c | 11 +++++++++++ fio.1 | 4 ++++ 3 files changed, 20 insertions(+)
HOWTO.rst +5 -0
@@ -2880,6 +2880,11 @@ with the caveat that when used on the command line, they must come after the xnvme namespace identifier for userspace NVMe driver, SPDK or vfio. +.. option:: xnvme_dev_subnqn=str : [xnvme] + + Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a + fabrics target with multiple systems. + .. option:: xnvme_iovec=int : [xnvme] If this option is set. xnvme will use vectored read/write commands.
engines/xnvme.c +11 -0
@@ -78,6 +78,7 @@ struct xnvme_fioe_options {char *xnvme_async;char *xnvme_sync;char *xnvme_admin; +char *xnvme_dev_subnqn; };static struct fio_option options[] = {
@@ -145,6 +146,15 @@ static struct fio_option options[] = { .category = FIO_OPT_C_ENGINE, .group = FIO_OPT_G_XNVME, }, + { + .name = "xnvme_dev_subnqn", + .lname = "Subsystem nqn for Fabrics", + .type = FIO_OPT_STR_STORE, + .off1 = offsetof(struct xnvme_fioe_options, xnvme_dev_subnqn), + .help = "Subsystem NQN for Fabrics", + .category = FIO_OPT_C_ENGINE, + .group = FIO_OPT_G_XNVME, + }, { .name = "xnvme_iovec", .lname = "Vectored IOs",
@@ -181,6 +191,7 @@ static struct xnvme_opts xnvme_opts_from_fioe(struct thread_data *td)struct xnvme_opts opts =xnvme_opts_default (); opts.nsid = o->xnvme_dev_nsid; + opts.subnqn = o->xnvme_dev_subnqn; opts.be = o->xnvme_be; opts.async = o->xnvme_async; opts.sync = o->xnvme_sync;
fio.1 +4 -0
@@ -2627,6 +2627,10 @@ Use Linux Block Layer ioctl() and sysfs for admin commands. .BI (xnvme)xnvme_dev_nsid\fR=\fPint xnvme namespace identifier for userspace NVMe driver SPDK or vfio. .TP +.BI (xnvme)xnvme_dev_subnqn\fR=\fPstr +Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics +target with multiple systems. +.TP .BI (xnvme)xnvme_iovec If this option is set, xnvme will use vectored read/write commands. .TP
Signature · 1 line
2.17.1
PATCH 4/4 engines/xnvme: add support for picking mem backend
In reply to Ankit KumarAdd option to the xnvme fio engine for picking a
memory backend. Update the fio document.
--- HOWTO.rst | 17 +++++++++++++++++ engines/xnvme.c | 11 +++++++++++ fio.1 | 22 ++++++++++++++++++++++ 3 files changed, 50 insertions(+)
HOWTO.rst +17 -0
@@ -2885,6 +2885,23 @@ with the caveat that when used on the command line, they must come after the Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics target with multiple systems. +.. option:: xnvme_mem=str : [xnvme] + + Select the xnvme memory backend. This can take these values. + + **posix** + This is the default posix memory backend for linux NVMe driver. + **hugepage** + Use hugepages, instead of existing posix memory backend. The + memory backend uses hugetlbfs. This require users to allocate + hugepages, mount hugetlbfs and set an enviornment variable for + XNVME_HUGETLB_PATH. + **spdk** + Uses SPDK's memory allocator. + **vfio** + Uses libvfn's memory allocator. This also specifies the use + of libvfn backend instead of SPDK. + .. option:: xnvme_iovec=int : [xnvme] If this option is set. xnvme will use vectored read/write commands.
engines/xnvme.c +11 -0
@@ -75,6 +75,7 @@ struct xnvme_fioe_options {unsigned int xnvme_dev_nsid;unsigned int xnvme_iovec;char *xnvme_be; +char *xnvme_mem;char *xnvme_async;char *xnvme_sync;char *xnvme_admin;
@@ -109,6 +110,15 @@ static struct fio_option options[] = { .category = FIO_OPT_C_ENGINE, .group = FIO_OPT_G_XNVME, }, + { + .name = "xnvme_mem", + .lname = "xNVMe Memory Backend", + .type = FIO_OPT_STR_STORE, + .off1 = offsetof(struct xnvme_fioe_options, xnvme_mem), + .help = "Select xNVMe memory backend", + .category = FIO_OPT_C_ENGINE, + .group = FIO_OPT_G_XNVME, + }, { .name = "xnvme_async", .lname = "xNVMe Asynchronous command-interface",
@@ -193,6 +203,7 @@ static struct xnvme_opts xnvme_opts_from_fioe(struct thread_data *td) opts.nsid = o->xnvme_dev_nsid; opts.subnqn = o->xnvme_dev_subnqn; opts.be = o->xnvme_be; + opts.mem = o->xnvme_mem; opts.async = o->xnvme_async; opts.sync = o->xnvme_sync; opts.admin = o->xnvme_admin;
fio.1 +22 -0
@@ -2631,6 +2631,28 @@ xnvme namespace identifier for userspace NVMe driver SPDK or vfio. Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics target with multiple systems. .TP +.BI (xnvme)xnvme_mem\fR=\fPstr +Select the xnvme memory backend. This can take these values. +.RS +.RS +.TP +.B posix +This is the default posix memory backend for linux NVMe driver. +.TP +.BI hugepage +Use hugepages, instead of existing posix memory backend. The memory backend +uses hugetlbfs. This require users to allocate hugepages, mount hugetlbfs and +set an enviornment variable for XNVME_HUGETLB_PATH. +.TP +.BI spdk +Uses SPDK's memory allocator. +.TP +.BI vfio +Uses libvfn's memory allocator. This also specifies the use of libvfn backend +instead of SPDK. +.RE +.RE +.TP .BI (xnvme)xnvme_iovec If this option is set, xnvme will use vectored read/write commands. .TP
Signature · 1 line
2.17.1
Vincent Fu
AppliedQuoted 13 lines · Ankit Kumar
This patch series adds
1. Few minor fixes and cleanups to the xnvme I/O engine.
2. xNVMe recently added a user-space VFIO based backend implemented using libvfn. This allow users to select that instead of SPDK.
3. Add subnqn option for fabrics target with multiple systems.
4. Add an option to select memory backend.Ankit Kumar (4):
engines/xnvme: fixes for xnvme ioengine engines/xnvme: user space vfio based backend engines/xnvme: add subnqn to fio-options engines/xnvme: add support for picking mem backend
Quoted 5 lines
HOWTO.rst | 27 ++++++++++++++++++++++++++- engines/xnvme.c | 44 ++++++++++++++++++++++++++++++++++++++------ fio.1 | 32 +++++++++++++++++++++++++++++++- 3 files changed, 95 insertions(+), 8 deletions(-)
Applied. Thanks.