Skip to contentSkip to conversation

fdp fixes, random placment id and xnvme support

PATCH Applied Ankit Kumar 2 participants 6 messages Jul 13, 2023 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CGME20230712050110epcas5p1282a4e40485cb37545ec7cb55a3dcff1@epcas5p1.samsung.com

#1 lore raw

v2 cover fdp fixes, random placment id and xnvme support

This series now enables us to either roundrobin or select a random placement ID from the available ones. In future a non-unifrom placement id selection such as zipf, pareto etc can be added.

The current boundary check for input placment ids was not correct. Added a fix for that.

This series also adds FDP support to the xnvme I/O engine.

Changes since v1:
- Split the macro change and placement id check fix into separate patches as suggested by Vincent

Ankit Kumar (4):
fdp: use macros
fdp: fix placement id check
fdp: support random placement id selection
engines/xnvme: add support for fdp

 HOWTO.rst              | 19 ++++++++--
 configure              |  2 +-
 engines/io_uring.c     |  2 +-
 engines/xnvme.c        | 78 +++++++++++++++++++++++++++++++++++++++++-
 examples/xnvme-fdp.fio | 36 +++++++++++++++++++
 fdp.c                  | 22 +++++++-----
 fdp.h                  | 13 +++++++
 fio.1                  | 20 +++++++++--
 fio.h                  |  2 ++
 init.c                 |  2 ++
 options.c              | 20 +++++++++++
 thread_options.h       |  2 ++
 12 files changed, 203 insertions(+), 15 deletions(-)
 create mode 100644 examples/xnvme-fdp.fio
Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 1/4 fdp: use macros

use macros for directive type and max ruhs.

---
 engines/io_uring.c | 2 +-
 fdp.c              | 8 ++++----
 fdp.h              | 3 +++
 3 files changed, 8 insertions(+), 5 deletions(-)
engines/io_uring.c +1 -1
index 5021239e..407d65ce 100644
@@ -1310,7 +1310,7 @@ static int fio_ioring_cmd_fetch_ruhs(struct thread_data *td, struct fio_file *f,
 	struct nvme_fdp_ruh_status *ruhs;
 	int bytes, ret, i;
 
-	bytes = sizeof(*ruhs) + 128 * sizeof(struct nvme_fdp_ruh_status_desc);
+	bytes = sizeof(*ruhs) + FDP_MAX_RUHS * sizeof(struct nvme_fdp_ruh_status_desc);
 	ruhs = scalloc(1, bytes);
 	if (!ruhs)
 		return -ENOMEM;
fdp.c +4 -4
index d92dbc67..b83fd660 100644
@@ -45,7 +45,7 @@ static int init_ruh_info(struct thread_data *td, struct fio_file *f)
 	struct fio_ruhs_info *ruhs, *tmp;
 	int i, ret;
 
-	ruhs = scalloc(1, sizeof(*ruhs) + 128 * sizeof(*ruhs->plis));
+	ruhs = scalloc(1, sizeof(*ruhs) + FDP_MAX_RUHS * sizeof(*ruhs->plis));
 	if (!ruhs)
 		return -ENOMEM;
 
@@ -56,8 +56,8 @@ static int init_ruh_info(struct thread_data *td, struct fio_file *f)
 		goto out;
 	}
 
-	if (ruhs->nr_ruhs > 128)
-		ruhs->nr_ruhs = 128;
+	if (ruhs->nr_ruhs > FDP_MAX_RUHS)
+		ruhs->nr_ruhs = FDP_MAX_RUHS;
 
 	if (td->o.fdp_nrpli == 0) {
 		f->ruhs_info = ruhs;
@@ -123,6 +123,6 @@ void fdp_fill_dspec_data(struct thread_data *td, struct io_u *io_u)
 		ruhs->pli_loc = 0;
 
 	dspec = ruhs->plis[ruhs->pli_loc++];
-	io_u->dtype = 2;
+	io_u->dtype = FDP_DIR_DTYPE;
 	io_u->dspec = dspec;
 }
fdp.h +3 -0
index 81691f62..6ecbfcbe 100644
@@ -3,6 +3,9 @@
 
 #include "io_u.h"
 
+#define FDP_DIR_DTYPE	2
+#define FDP_MAX_RUHS	128
+
 struct fio_ruhs_info {
 	uint32_t nr_ruhs;
 	uint32_t pli_loc;
Signature · 1 line
2.25.1
#3 lore raw

PATCH v2 2/4 fdp: fix placement id check

In reply to Ankit Kumar

Number of reclaim unit handle descriptors are 1 based, whereas the input placement id index / indices are 0 based. Add the correct check for that.

Fixes a7e8aae0 ("fio: add fdp support ..")
---
 fdp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fdp.c +1 -1
index b83fd660..6e124ce6 100644
@@ -65,7 +65,7 @@ static int init_ruh_info(struct thread_data *td, struct fio_file *f)
 	}
 
 	for (i = 0; i < td->o.fdp_nrpli; i++) {
-		if (td->o.fdp_plis[i] > ruhs->nr_ruhs) {
+		if (td->o.fdp_plis[i] >= ruhs->nr_ruhs) {
 			ret = -EINVAL;
 			goto out;
 		}
Signature · 1 line
2.25.1
#4 lore raw

PATCH v2 3/4 fdp: support random placement id selection

In reply to Ankit Kumar

Allow user to either roundrobin or select random placement ID from the available placement IDs.

---
 HOWTO.rst        | 15 +++++++++++++++
 fdp.c            | 12 +++++++++---
 fdp.h            | 10 ++++++++++
 fio.1            | 16 ++++++++++++++++
 fio.h            |  2 ++
 init.c           |  2 ++
 options.c        | 20 ++++++++++++++++++++
 thread_options.h |  2 ++
 8 files changed, 76 insertions(+), 3 deletions(-)
HOWTO.rst +15 -0
index 2e1e55c2..24789f41 100644
@@ -2435,6 +2435,21 @@ with the caveat that when used on the command line, they must come after the
 
 	Enable Flexible Data Placement mode for write commands.
 
+.. option:: fdp_pli_select=str : [io_uring_cmd]
+
+	Defines how fio decides which placement ID to use next. The following
+	types are defined:
+
+		**random**
+			Choose a placement ID at random (uniform).
+
+		**roundrobin**
+			Round robin over available placement IDs. This is the
+			default.
+
+	The available placement ID index/indices is defined by the option
+	:option:`fdp_pli`.
+
 .. option:: fdp_pli=str : [io_uring_cmd]
 
 	Select which Placement ID Index/Indicies this job is allowed to use for
fdp.c +9 -3
index 6e124ce6..49c80d2c 100644
@@ -119,10 +119,16 @@ void fdp_fill_dspec_data(struct thread_data *td, struct io_u *io_u)
 		return;
 	}
 
-	if (ruhs->pli_loc >= ruhs->nr_ruhs)
-		ruhs->pli_loc = 0;
+	if (td->o.fdp_pli_select == FIO_FDP_RR) {
+		if (ruhs->pli_loc >= ruhs->nr_ruhs)
+			ruhs->pli_loc = 0;
+
+		dspec = ruhs->plis[ruhs->pli_loc++];
+	} else {
+		ruhs->pli_loc = rand_between(&td->fdp_state, 0, ruhs->nr_ruhs - 1);
+		dspec = ruhs->plis[ruhs->pli_loc];
+	}
 
-	dspec = ruhs->plis[ruhs->pli_loc++];
 	io_u->dtype = FDP_DIR_DTYPE;
 	io_u->dspec = dspec;
 }
fdp.h +10 -0
index 6ecbfcbe..accbac38 100644
@@ -6,6 +6,16 @@
 #define FDP_DIR_DTYPE	2
 #define FDP_MAX_RUHS	128
 
+/*
+ * How fio chooses what placement identifier to use next. Choice of
+ * uniformly random, or roundrobin.
+ */
+
+enum {
+	FIO_FDP_RANDOM	= 0x1,
+	FIO_FDP_RR	= 0x2,
+};
+
 struct fio_ruhs_info {
 	uint32_t nr_ruhs;
 	uint32_t pli_loc;
fio.1 +16 -0
index 73b7e8c9..0257513b 100644
@@ -2195,6 +2195,22 @@ file blocks are fully allocated and the disk request could be issued immediately
 .BI (io_uring_cmd)fdp \fR=\fPbool
 Enable Flexible Data Placement mode for write commands.
 .TP
+.BI (io_uring_cmd)fdp_pli_select \fR=\fPstr
+Defines how fio decides which placement ID to use next. The following types
+are defined:
+.RS
+.RS
+.TP
+.B random
+Choose a placement ID at random (uniform).
+.TP
+.B roundrobin
+Round robin over available placement IDs. This is the default.
+.RE
+.P
+The available placement ID index/indices is defined by \fBfdp_pli\fR option.
+.RE
+.TP
 .BI (io_uring_cmd)fdp_pli \fR=\fPstr
 Select which Placement ID Index/Indicies this job is allowed to use for writes.
 By default, the job will cycle through all available Placement IDs, so use this
fio.h +2 -0
index c5453d13..a54f57c9 100644
@@ -144,6 +144,7 @@ enum {
 	FIO_RAND_POISSON3_OFF,
 	FIO_RAND_PRIO_CMDS,
 	FIO_RAND_DEDUPE_WORKING_SET_IX,
+	FIO_RAND_FDP_OFF,
 	FIO_RAND_NR_OFFS,
 };
 
@@ -262,6 +263,7 @@ struct thread_data {
 	struct frand_state verify_state_last_do_io;
 	struct frand_state trim_state;
 	struct frand_state delay_state;
+	struct frand_state fdp_state;
 
 	struct frand_state buf_state;
 	struct frand_state buf_state_prev;
init.c +2 -0
index 10e63cca..105339fa 100644
@@ -1082,6 +1082,8 @@ void td_fill_rand_seeds(struct thread_data *td)
 
 	init_rand_seed(&td->buf_state, td->rand_seeds[FIO_RAND_BUF_OFF], use64);
 	frand_copy(&td->buf_state_prev, &td->buf_state);
+
+	init_rand_seed(&td->fdp_state, td->rand_seeds[FIO_RAND_FDP_OFF], use64);
 }
 
 static int setup_random_seeds(struct thread_data *td)
options.c +20 -0
index a7c4ef6e..0f739317 100644
@@ -3679,6 +3679,26 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
 		.category = FIO_OPT_C_IO,
 		.group  = FIO_OPT_G_INVALID,
 	},
+	{
+		.name	= "fdp_pli_select",
+		.lname	= "FDP Placement ID select",
+		.type	= FIO_OPT_STR,
+		.off1	= offsetof(struct thread_options, fdp_pli_select),
+		.help	= "Select which FDP placement ID to use next",
+		.def	= "roundrobin",
+		.category = FIO_OPT_C_IO,
+		.group	= FIO_OPT_G_INVALID,
+		.posval	= {
+			  { .ival = "random",
+			    .oval = FIO_FDP_RANDOM,
+			    .help = "Choose a Placement ID at random (uniform)",
+			  },
+			  { .ival = "roundrobin",
+			    .oval = FIO_FDP_RR,
+			    .help = "Round robin select Placement IDs",
+			  },
+		},
+	},
 	{
 		.name	= "fdp_pli",
 		.lname	= "FDP Placement ID indicies",
thread_options.h +2 -0
index a24ebee6..1715b36c 100644
@@ -388,6 +388,7 @@ struct thread_options {
 
 #define FIO_MAX_PLIS 16
 	unsigned int fdp;
+	unsigned int fdp_pli_select;
 	unsigned int fdp_plis[FIO_MAX_PLIS];
 	unsigned int fdp_nrpli;
 
@@ -703,6 +704,7 @@ struct thread_options_pack {
 	uint32_t log_prio;
 
 	uint32_t fdp;
+	uint32_t fdp_pli_select;
 	uint32_t fdp_plis[FIO_MAX_PLIS];
 	uint32_t fdp_nrpli;
 
Signature · 1 line
2.25.1
#5 lore raw

PATCH v2 4/4 engines/xnvme: add support for fdp

In reply to Ankit Kumar

Add FDP support to xnvme I/O engine. This support can be used only with nvme-ns generic character device (/dev/ngXnY). The available backends are --xnvme_async=io_uring_cmd and --xnvme_sync=nvme.
Add a xnvme-fdp config example file.

Update the minimum required xnvme version to 0.7.0

---
 HOWTO.rst              |  6 ++--
 configure              |  2 +-
 engines/xnvme.c        | 78 +++++++++++++++++++++++++++++++++++++++++-
 examples/xnvme-fdp.fio | 36 +++++++++++++++++++
 fio.1                  |  6 ++--
 5 files changed, 120 insertions(+), 8 deletions(-)
 create mode 100644 examples/xnvme-fdp.fio
HOWTO.rst +3 -3
index 24789f41..b047877e 100644
@@ -2431,11 +2431,11 @@ with the caveat that when used on the command line, they must come after the
 	For direct I/O, requests will only succeed if cache invalidation isn't required,
 	file blocks are fully allocated and the disk request could be issued immediately.
 
-.. option:: fdp=bool : [io_uring_cmd]
+.. option:: fdp=bool : [io_uring_cmd] [xnvme]
 
 	Enable Flexible Data Placement mode for write commands.
 
-.. option:: fdp_pli_select=str : [io_uring_cmd]
+.. option:: fdp_pli_select=str : [io_uring_cmd] [xnvme]
 
 	Defines how fio decides which placement ID to use next. The following
 	types are defined:
@@ -2450,7 +2450,7 @@ with the caveat that when used on the command line, they must come after the
 	The available placement ID index/indices is defined by the option
 	:option:`fdp_pli`.
 
-.. option:: fdp_pli=str : [io_uring_cmd]
+.. option:: fdp_pli=str : [io_uring_cmd] [xnvme]
 
 	Select which Placement ID Index/Indicies this job is allowed to use for
 	writes. By default, the job will cycle through all available Placement
configure +1 -1
index 74416fd4..6c938251 100755
@@ -2651,7 +2651,7 @@ fi
 ##########################################
 # Check if we have xnvme
 if test "$xnvme" != "no" ; then
-  if check_min_lib_version xnvme 0.2.0; then
+  if check_min_lib_version xnvme 0.7.0; then
     xnvme="yes"
     xnvme_cflags=$(pkg-config --cflags xnvme)
     xnvme_libs=$(pkg-config --libs xnvme)
engines/xnvme.c +77 -1

114 lines not on this page · raw message

examples/xnvme-fdp.fio new file +36 -0
new file mode 100644
index 00000000..86fbe0d3
@@ -0,0 +1,36 @@
+; README
+;
+; This job-file is intended to be used either as:
+;
+; # Use the xNVMe io-engine engine io_uring_cmd async. impl.
+; fio examples/xnvme-fdp.fio \
+;   --section=default \
+;   --ioengine=xnvme \
+;   --xnvme_async=io_uring_cmd \
+;   --filename=/dev/ng0n1
+;
+; # Use the xNVMe io-engine engine with nvme sync. impl.
+; fio examples/xnvme-fdp.fio \
+;   --section=default \
+;   --ioengine=xnvme \
+;   --xnvme_sync=nvme \
+;   --filename=/dev/ng0n1
+;
+; FIO_BS="512" FIO_RW="read" FIO_IODEPTH=16 fio examples/xnvme-fdp.fio \
+;   --section=override --ioengine=xnvme --xnvme_sync=nvme --filename=/dev/ng0n1
+;
+[global]
+rw=randwrite
+size=2M
+iodepth=1
+bs=4K
+thread=1
+fdp=1
+fdp_pli=4,5
+
+[default]
+
+[override]
+rw=${FIO_RW}
+iodepth=${FIO_IODEPTH}
+bs=${FIO_BS}
fio.1 +3 -3
index 0257513b..86cb2af6 100644
@@ -2192,10 +2192,10 @@ cached data. Currently the RWF_NOWAIT flag does not supported for cached write.
 For direct I/O, requests will only succeed if cache invalidation isn't required,
 file blocks are fully allocated and the disk request could be issued immediately.
 .TP
-.BI (io_uring_cmd)fdp \fR=\fPbool
+.BI (io_uring_cmd,xnvme)fdp \fR=\fPbool
 Enable Flexible Data Placement mode for write commands.
 .TP
-.BI (io_uring_cmd)fdp_pli_select \fR=\fPstr
+.BI (io_uring_cmd,xnvme)fdp_pli_select \fR=\fPstr
 Defines how fio decides which placement ID to use next. The following types
 are defined:
 .RS
@@ -2211,7 +2211,7 @@ Round robin over available placement IDs. This is the default.
 The available placement ID index/indices is defined by \fBfdp_pli\fR option.
 .RE
 .TP
-.BI (io_uring_cmd)fdp_pli \fR=\fPstr
+.BI (io_uring_cmd,xnvme)fdp_pli \fR=\fPstr
 Select which Placement ID Index/Indicies this job is allowed to use for writes.
 By default, the job will cycle through all available Placement IDs, so use this
 to isolate these identifiers to specific jobs. If you want fio to use placement
Signature · 1 line
2.25.1
Applied
#6 lore raw
In reply to Ankit Kumar
Quoted 32 lines · Ankit Kumar

This series now enables us to either roundrobin or select a random placement ID from the available ones. In future a non-unifrom placement id selection such as zipf, pareto etc can be added.

The current boundary check for input placment ids was not correct. Added a fix for that.

This series also adds FDP support to the xnvme I/O engine.

Changes since v1:
- Split the macro change and placement id check fix into separate patches as suggested by Vincent

Ankit Kumar (4):
fdp: use macros
fdp: fix placement id check
fdp: support random placement id selection
engines/xnvme: add support for fdp

  HOWTO.rst              | 19 ++++++++--
  configure              |  2 +-
  engines/io_uring.c     |  2 +-
  engines/xnvme.c        | 78 +++++++++++++++++++++++++++++++++++++++++-
  examples/xnvme-fdp.fio | 36 +++++++++++++++++++
  fdp.c                  | 22 +++++++-----
  fdp.h                  | 13 +++++++
  fio.1                  | 20 +++++++++--
  fio.h                  |  2 ++
  init.c                 |  2 ++
  options.c              | 20 +++++++++++
  thread_options.h       |  2 ++
  12 files changed, 203 insertions(+), 15 deletions(-)
Quoted 2 lines

create mode 100644 examples/xnvme-fdp.fio

Applied, thanks.

Vincent