Skip to contentSkip to conversation

new options and fixes for xnvme ioengine

PATCH Applied Ankit Kumar 2 participants 6 messages Dec 22, 2022 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CGME20221222044040epcas5p3abbde8c4a3775d39d237b0b6cf6dd04a@epcas5p3.samsung.com

#1 lore raw

cover new options and fixes for xnvme ioengine

This patch series adds
1. Few minor fixes and cleanups to the xnvme I/O engine.
2. xNVMe recently added a user-space VFIO based backend implemented using libvfn. This allow users to select that instead of SPDK.
3. Add subnqn option for fabrics target with multiple systems.
4. Add an option to select memory backend.

Ankit Kumar (4):
engines/xnvme: fixes for xnvme ioengine engines/xnvme: user space vfio based backend engines/xnvme: add subnqn to fio-options engines/xnvme: add support for picking mem backend

 HOWTO.rst       | 27 ++++++++++++++++++++++++++-
 engines/xnvme.c | 44 ++++++++++++++++++++++++++++++++++++++------
 fio.1           | 32 +++++++++++++++++++++++++++++++-
 3 files changed, 95 insertions(+), 8 deletions(-)
Signature · 1 line
2.17.1
#2 lore raw

PATCH 1/4 engines/xnvme: fixes for xnvme ioengine

1. fix error-handling in xnvme_fioe_queue()
2. fix resource-leak in xnvme_fioe_init()

---
 engines/xnvme.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
engines/xnvme.c +12 -3
index d8647481..dcc54998 100644
@@ -322,12 +322,15 @@ static int xnvme_fioe_init(struct thread_data *td)
 
 	xd->iocq = calloc(td->o.iodepth, sizeof(struct io_u *));
 	if (!xd->iocq) {
-		log_err("ioeng->init(): !calloc(), err(%d)\n", errno);
+		free(xd);
+		log_err("ioeng->init(): !calloc(xd->iocq), err(%d)\n", errno);
 		return 1;
 	}
 
 	xd->iovec = calloc(td->o.iodepth, sizeof(*xd->iovec));
 	if (!xd->iovec) {
+		free(xd->iocq);
+		free(xd);
 		log_err("ioeng->init(): !calloc(xd->iovec), err(%d)\n", errno);
 		return 1;
 	}
@@ -338,6 +341,10 @@ static int xnvme_fioe_init(struct thread_data *td)
 	for_each_file(td, f, i)
 	{
 		if (_dev_open(td, f)) {
+			/*
+			 * Note: We are not freeing xd, iocq and iovec. This
+			 * will be done as part of cleanup routine.
+			 */
 			log_err("ioeng->init(): failed; _dev_open(%s)\n", f->file_name);
 			return 1;
 		}
@@ -506,9 +513,11 @@ static enum fio_q_status xnvme_fioe_queue(struct thread_data *td, struct io_u *i
 
 	default:
 		log_err("ioeng->queue(): ENOSYS: %u\n", io_u->ddir);
-		err = -1;
+		xnvme_queue_put_cmd_ctx(ctx->async.queue, ctx);
+
+		io_u->error = ENOSYS;
 		assert(false);
-		break;
+		return FIO_Q_COMPLETED;
 	}
 
 	if (vectored_io) {
Signature · 1 line
2.17.1
#3 lore raw

PATCH 2/4 engines/xnvme: user space vfio based backend

In reply to Ankit Kumar

Add an option to use user-space VFIO-based backend, implemented using libvfn.
Update xnvme engine options for missing backends.

---
 HOWTO.rst       | 5 ++++-
 engines/xnvme.c | 7 ++++---
 fio.1           | 6 +++++-
 3 files changed, 13 insertions(+), 5 deletions(-)
HOWTO.rst +4 -1
index 97fe5350..aba6c9b3 100644
@@ -2845,6 +2845,9 @@ with the caveat that when used on the command line, they must come after the
 	**posix**
 		Use the posix asynchronous I/O interface to perform one or
 		more I/O operations asynchronously.
+	**vfio**
+		Use the user-space VFIO-based backend, implemented using
+		libvfn instead of SPDK.
 	**nil**
 		Do not transfer any data; just pretend to. This is mainly used
 		for introspective performance evaluation.
@@ -2875,7 +2878,7 @@ with the caveat that when used on the command line, they must come after the
 
 .. option:: xnvme_dev_nsid=int : [xnvme]
 
-	xnvme namespace identifier for userspace NVMe driver, such as SPDK.
+	xnvme namespace identifier for userspace NVMe driver, SPDK or vfio.
 
 .. option:: xnvme_iovec=int : [xnvme]
 
engines/xnvme.c +4 -3
index dcc54998..ee6b67c1 100644
@@ -113,7 +113,8 @@ static struct fio_option options[] = {
 		.lname = "xNVMe Asynchronous command-interface",
 		.type = FIO_OPT_STR_STORE,
 		.off1 = offsetof(struct xnvme_fioe_options, xnvme_async),
-		.help = "Select xNVMe async. interface: [emu,thrpool,io_uring,libaio,posix,nil]",
+		.help = "Select xNVMe async. interface: "
+			"[emu,thrpool,io_uring,io_uring_cmd,libaio,posix,vfio,nil]",
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
@@ -122,7 +123,7 @@ static struct fio_option options[] = {
 		.lname = "xNVMe Synchronous. command-interface",
 		.type = FIO_OPT_STR_STORE,
 		.off1 = offsetof(struct xnvme_fioe_options, xnvme_sync),
-		.help = "Select xNVMe sync. interface: [nvme,psync]",
+		.help = "Select xNVMe sync. interface: [nvme,psync,block]",
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
@@ -131,7 +132,7 @@ static struct fio_option options[] = {
 		.lname = "xNVMe Admin command-interface",
 		.type = FIO_OPT_STR_STORE,
 		.off1 = offsetof(struct xnvme_fioe_options, xnvme_admin),
-		.help = "Select xNVMe admin. cmd-interface: [nvme,block,file_as_ns]",
+		.help = "Select xNVMe admin. cmd-interface: [nvme,block]",
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
fio.1 +5 -1
index 1074b52a..004d3ba0 100644
@@ -2584,6 +2584,10 @@ Use Linux aio for Asynchronous I/O
 Use the posix asynchronous I/O interface to perform one or more I/O operations
 asynchronously.
 .TP
+.BI vfio
+Use the user-space VFIO-based backend, implemented using libvfn instead of
+SPDK.
+.TP
 .BI nil
 Do not transfer any data; just pretend to. This is mainly used for
 introspective performance evaluation.
@@ -2621,7 +2625,7 @@ Use Linux Block Layer ioctl() and sysfs for admin commands.
 .RE
 .TP
 .BI (xnvme)xnvme_dev_nsid\fR=\fPint
-xnvme namespace identifier for userspace NVMe driver such as SPDK.
+xnvme namespace identifier for userspace NVMe driver SPDK or vfio.
 .TP
 .BI (xnvme)xnvme_iovec
 If this option is set, xnvme will use vectored read/write commands.
Signature · 1 line
2.17.1
#4 lore raw

PATCH 3/4 engines/xnvme: add subnqn to fio-options

In reply to Ankit Kumar

For fio to utilize a fabrics target with multiple systems, it needs a way for the user to specify which subsystem to use. This is done by providing 'subnqn' as fio-option.

---
 HOWTO.rst       |  5 +++++
 engines/xnvme.c | 11 +++++++++++
 fio.1           |  4 ++++
 3 files changed, 20 insertions(+)
HOWTO.rst +5 -0
index aba6c9b3..0fec38a7 100644
@@ -2880,6 +2880,11 @@ with the caveat that when used on the command line, they must come after the
 
 	xnvme namespace identifier for userspace NVMe driver, SPDK or vfio.
 
+.. option:: xnvme_dev_subnqn=str : [xnvme]
+
+	Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a
+	fabrics target with multiple systems.
+
 .. option:: xnvme_iovec=int : [xnvme]
 
 	If this option is set. xnvme will use vectored read/write commands.
engines/xnvme.c +11 -0
index ee6b67c1..208d917d 100644
@@ -78,6 +78,7 @@ struct xnvme_fioe_options {
 	char *xnvme_async;
 	char *xnvme_sync;
 	char *xnvme_admin;
+	char *xnvme_dev_subnqn;
 };
 
 static struct fio_option options[] = {
@@ -145,6 +146,15 @@ static struct fio_option options[] = {
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
+	{
+		.name = "xnvme_dev_subnqn",
+		.lname = "Subsystem nqn for Fabrics",
+		.type = FIO_OPT_STR_STORE,
+		.off1 = offsetof(struct xnvme_fioe_options, xnvme_dev_subnqn),
+		.help = "Subsystem NQN for Fabrics",
+		.category = FIO_OPT_C_ENGINE,
+		.group = FIO_OPT_G_XNVME,
+	},
 	{
 		.name = "xnvme_iovec",
 		.lname = "Vectored IOs",
@@ -181,6 +191,7 @@ static struct xnvme_opts xnvme_opts_from_fioe(struct thread_data *td)
 	struct xnvme_opts opts = xnvme_opts_default();
 
 	opts.nsid = o->xnvme_dev_nsid;
+	opts.subnqn = o->xnvme_dev_subnqn;
 	opts.be = o->xnvme_be;
 	opts.async = o->xnvme_async;
 	opts.sync = o->xnvme_sync;
fio.1 +4 -0
index 004d3ba0..134aed54 100644
@@ -2627,6 +2627,10 @@ Use Linux Block Layer ioctl() and sysfs for admin commands.
 .BI (xnvme)xnvme_dev_nsid\fR=\fPint
 xnvme namespace identifier for userspace NVMe driver SPDK or vfio.
 .TP
+.BI (xnvme)xnvme_dev_subnqn\fR=\fPstr
+Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics
+target with multiple systems.
+.TP
 .BI (xnvme)xnvme_iovec
 If this option is set, xnvme will use vectored read/write commands.
 .TP
Signature · 1 line
2.17.1
#5 lore raw

PATCH 4/4 engines/xnvme: add support for picking mem backend

In reply to Ankit Kumar

Add option to the xnvme fio engine for picking a
memory backend. Update the fio document.

---
 HOWTO.rst       | 17 +++++++++++++++++
 engines/xnvme.c | 11 +++++++++++
 fio.1           | 22 ++++++++++++++++++++++
 3 files changed, 50 insertions(+)
HOWTO.rst +17 -0
index 0fec38a7..0a48a453 100644
@@ -2885,6 +2885,23 @@ with the caveat that when used on the command line, they must come after the
 	Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a
 	fabrics target with multiple systems.
 
+.. option:: xnvme_mem=str : [xnvme]
+
+	Select the xnvme memory backend. This can take these values.
+
+	**posix**
+		This is the default posix memory backend for linux NVMe driver.
+	**hugepage**
+		Use hugepages, instead of existing posix memory backend. The
+		memory backend uses hugetlbfs. This require users to allocate
+		hugepages, mount hugetlbfs and set an enviornment variable for
+		XNVME_HUGETLB_PATH.
+	**spdk**
+		Uses SPDK's memory allocator.
+	**vfio**
+		Uses libvfn's memory allocator. This also specifies the use
+		of libvfn backend instead of SPDK.
+
 .. option:: xnvme_iovec=int : [xnvme]
 
 	If this option is set. xnvme will use vectored read/write commands.
engines/xnvme.c +11 -0
index 208d917d..bb92a121 100644
@@ -75,6 +75,7 @@ struct xnvme_fioe_options {
 	unsigned int xnvme_dev_nsid;
 	unsigned int xnvme_iovec;
 	char *xnvme_be;
+	char *xnvme_mem;
 	char *xnvme_async;
 	char *xnvme_sync;
 	char *xnvme_admin;
@@ -109,6 +110,15 @@ static struct fio_option options[] = {
 		.category = FIO_OPT_C_ENGINE,
 		.group = FIO_OPT_G_XNVME,
 	},
+	{
+		.name = "xnvme_mem",
+		.lname = "xNVMe Memory Backend",
+		.type = FIO_OPT_STR_STORE,
+		.off1 = offsetof(struct xnvme_fioe_options, xnvme_mem),
+		.help = "Select xNVMe memory backend",
+		.category = FIO_OPT_C_ENGINE,
+		.group = FIO_OPT_G_XNVME,
+	},
 	{
 		.name = "xnvme_async",
 		.lname = "xNVMe Asynchronous command-interface",
@@ -193,6 +203,7 @@ static struct xnvme_opts xnvme_opts_from_fioe(struct thread_data *td)
 	opts.nsid = o->xnvme_dev_nsid;
 	opts.subnqn = o->xnvme_dev_subnqn;
 	opts.be = o->xnvme_be;
+	opts.mem = o->xnvme_mem;
 	opts.async = o->xnvme_async;
 	opts.sync = o->xnvme_sync;
 	opts.admin = o->xnvme_admin;
fio.1 +22 -0
index 134aed54..eb87533f 100644
@@ -2631,6 +2631,28 @@ xnvme namespace identifier for userspace NVMe driver SPDK or vfio.
 Sets the subsystem NQN for fabrics. This is for xNVMe to utilize a fabrics
 target with multiple systems.
 .TP
+.BI (xnvme)xnvme_mem\fR=\fPstr
+Select the xnvme memory backend. This can take these values.
+.RS
+.RS
+.TP
+.B posix
+This is the default posix memory backend for linux NVMe driver.
+.TP
+.BI hugepage
+Use hugepages, instead of existing posix memory backend. The memory backend
+uses hugetlbfs. This require users to allocate hugepages, mount hugetlbfs and
+set an enviornment variable for XNVME_HUGETLB_PATH.
+.TP
+.BI spdk
+Uses SPDK's memory allocator.
+.TP
+.BI vfio
+Uses libvfn's memory allocator. This also specifies the use of libvfn backend
+instead of SPDK.
+.RE
+.RE
+.TP
 .BI (xnvme)xnvme_iovec
 If this option is set, xnvme will use vectored read/write commands.
 .TP
Signature · 1 line
2.17.1
Applied
#6 lore raw
In reply to Ankit Kumar
Quoted 13 lines · Ankit Kumar

This patch series adds
1. Few minor fixes and cleanups to the xnvme I/O engine.
2. xNVMe recently added a user-space VFIO based backend implemented using libvfn. This allow users to select that instead of SPDK.
3. Add subnqn option for fabrics target with multiple systems.
4. Add an option to select memory backend.

Ankit Kumar (4):
engines/xnvme: fixes for xnvme ioengine engines/xnvme: user space vfio based backend engines/xnvme: add subnqn to fio-options engines/xnvme: add support for picking mem backend

Quoted 5 lines
  HOWTO.rst       | 27 ++++++++++++++++++++++++++-
  engines/xnvme.c | 44 ++++++++++++++++++++++++++++++++++++++------
  fio.1           | 32 +++++++++++++++++++++++++++++++-
  3 files changed, 95 insertions(+), 8 deletions(-)

Applied. Thanks.