Skip to contentSkip to conversation

Introduce a page_fault ioengine for MM workflows

RFC Superseded Nico Pache 4 participants 8 messages Jan 29 – Feb 2 Kbuild

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Nico Pache

author
#0 lore raw

cover Introduce a page_fault ioengine for MM workflows

This series introduces a new page_fault ioengine and a follow‑up option to make khugepaged behavior more deterministic for page fault testing.

The first patch adds an mmap‑backed ioengine that allocates anonymous memory and copies data on read/write to intentionally trigger faults. This enables using fio’s existing framework and job files for memory management style workloads without relying on a filesystem. An example job file is included to demonstrate usage.

The second patch adds a hugepage_delay option to the new engine. It initially maps memory with MADV_NOHUGEPAGE, then (after a configurable delay) switches to MADV_HUGEPAGE via a helper thread. This makes khugepaged candidates reproducible, allows for easier khugepaged testing and improves repeatability for page_fault tests.

Nico Pache (2):
page_fault: add mmap-backed ioengine for anonymous faults
page_fault: add hugepage_delay option for delayed MADV_HUGEPAGE

 Makefile                |   2 +-
 cconv.c                 |   3 +
 engines/page_fault.c    | 185 ++++++++++++++++++++++++++++++++++++++++
 examples/page_fault.fio |   9 ++
 options.c               |  10 +++
 thread_options.h        |   3 +
 6 files changed, 211 insertions(+), 1 deletion(-)
 create mode 100644 engines/page_fault.c
 create mode 100644 examples/page_fault.fio
Signature · 1 line
2.52.0
author
#1 lore raw

RFC 1/2 page_fault: add mmap-backed ioengine for anonymous faults

Introduce a new ioengine that mmaps anonymous memory and copies data on read/write to trigger page faults. This allows us to leverage FIOs powerful framework for MM related testing, and will ideally allow us to quickly expand testing, by leveraging previously FS related fio scripts.

---
 Makefile                |   2 +-
 engines/page_fault.c    | 105 ++++++++++++++++++++++++++++++++++++++++
 examples/page_fault.fio |   9 ++++
 3 files changed, 115 insertions(+), 1 deletion(-)
 create mode 100644 engines/page_fault.c
 create mode 100644 examples/page_fault.fio
Makefile +1 -1
index 0337e8fe..099e2f94 100644
@@ -57,7 +57,7 @@ SOURCE :=	$(sort $(patsubst $(SRCDIR)/%,%,$(wildcard $(SRCDIR)/crc/*.c)) \
 		smalloc.c filehash.c profile.c debug.c engines/cpu.c \
 		engines/mmap.c engines/sync.c engines/null.c engines/net.c \
 		engines/ftruncate.c engines/fileoperations.c \
-		engines/exec.c \
+		engines/exec.c engines/page_fault.c \
 		server.c client.c iolog.c backend.c libfio.c flow.c cconv.c \
 		gettime-thread.c helpers.c json.c idletime.c td_error.c \
 		profiles/tiobench.c profiles/act.c io_u_queue.c filelock.c \
engines/page_fault.c new file +105 -0

106 lines not on this page · raw message

examples/page_fault.fio new file +9 -0
new file mode 100644
index 00000000..9001f570
@@ -0,0 +1,9 @@
+[global]
+bs=4k
+
+[page_fault]
+ioengine=page_fault
+size=2g
+rw=randrw
+rwmixread=50
+verify=crc32c
\ No newline at end of file
Signature · 1 line
2.52.0
author
#2 lore raw

RFC 2/2 page_fault: add hugepage_delay option for delayed MADV_HUGEPAGE

In reply to Nico Pache

Introduce hugepage_delay to map with MADV_NOHUGEPAGE first, then madvise MADV_HUGEPAGE after a configurable delay via a helper thread. This makes khugepaged candidates reproducible for page_fault tests.

---
 cconv.c              |  3 ++
 engines/page_fault.c | 94 ++++++++++++++++++++++++++++++++++++++++----
 options.c            | 10 +++++
 thread_options.h     |  3 ++
 4 files changed, 103 insertions(+), 7 deletions(-)
cconv.c +3 -0
index 0c4a3f2d..4fafbf83 100644
@@ -242,6 +242,7 @@ int convert_thread_options_to_cpu(struct thread_options *o,
 	o->random_center.u.f = fio_uint64_to_double(le64_to_cpu(top->random_center.u.i));
 	o->random_generator = le32_to_cpu(top->random_generator);
 	o->hugepage_size = le32_to_cpu(top->hugepage_size);
+	o->hugepage_delay = le32_to_cpu(top->hugepage_delay);
 	o->rw_min_bs = le64_to_cpu(top->rw_min_bs);
 	o->thinkcycles = le32_to_cpu(top->thinkcycles);
 	o->thinktime = le32_to_cpu(top->thinktime);
@@ -494,6 +495,8 @@ void convert_thread_options_to_net(struct thread_options_pack *top,
 	top->random_center.u.i = __cpu_to_le64(fio_double_to_uint64(o->random_center.u.f));
 	top->random_generator = cpu_to_le32(o->random_generator);
 	top->hugepage_size = cpu_to_le32(o->hugepage_size);
+	top->hugepage_delay = cpu_to_le32(o->hugepage_delay);
+	top->hugepage_delay_pad = 0;
 	top->rw_min_bs = __cpu_to_le64(o->rw_min_bs);
 	top->thinkcycles = cpu_to_le32(o->thinkcycles);
 	top->thinktime = cpu_to_le32(o->thinktime);
engines/page_fault.c +87 -7

125 lines not on this page · raw message

options.c +10 -0
index f526f5eb..5f8c53cd 100644
@@ -5490,6 +5490,16 @@ struct fio_option fio_options[FIO_MAX_OPTS] = {
 		.category = FIO_OPT_C_GENERAL,
 		.group	= FIO_OPT_G_INVALID,
 	},
+	{
+		.name	= "hugepage_delay",
+		.lname	= "Hugepage delay",
+		.type	= FIO_OPT_INT,
+		.off1	= offsetof(struct thread_options, hugepage_delay),
+		.help	= "For mmap, map with MADV_NOHUGEPAGE then MADV_HUGEPAGE after delay (in ms)",
+		.def	= "0",
+		.category = FIO_OPT_C_GENERAL,
+		.group	= FIO_OPT_G_INVALID,
+	},
 	{
 		.name	= "flow_id",
 		.lname	= "I/O flow ID",
thread_options.h +3 -0
index b4dd8d7a..f288664f 100644
@@ -203,6 +203,7 @@ struct thread_options {
 	unsigned int perc_rand[DDIR_RWDIR_CNT];
 
 	unsigned int hugepage_size;
+	unsigned int hugepage_delay;
 	unsigned long long rw_min_bs;
 	unsigned int fsync_blocks;
 	unsigned int fdatasync_blocks;
@@ -539,6 +540,8 @@ struct thread_options_pack {
 	uint32_t perc_rand[DDIR_RWDIR_CNT];
 
 	uint32_t hugepage_size;
+	uint32_t hugepage_delay;
+	uint32_t hugepage_delay_pad;
 	uint64_t rw_min_bs;
 	uint32_t fsync_blocks;
 	uint32_t fdatasync_blocks;
Signature · 1 line
2.52.0
#3 lore raw

Re: [RFC 2/2] page_fault: add hugepage_delay option for delayed MADV_HUGEPAGE

Quoted 7 lines · Nico Pache

Introduce hugepage_delay to map with MADV_NOHUGEPAGE first, then madvise MADV_HUGEPAGE after a configurable delay via a helper thread. This makes khugepaged candidates reproducible for page_fault tests.

---

I would add this option as an *ioengine* option instead of a general FIo option. There are examples in sync.c, io_uring.c, etc.

This option also needs to be added to the documentation.

Vincent

author
#4 lore raw

Re: [RFC 2/2] page_fault: add hugepage_delay option for delayed MADV_HUGEPAGE

Quoted 7 lines · Vincent Fu
Quoted 5 lines · Nico Pache

Introduce hugepage_delay to map with MADV_NOHUGEPAGE first, then madvise MADV_HUGEPAGE after a configurable delay via a helper thread. This makes khugepaged candidates reproducible for page_fault tests.

Quoted 7 lines
Quoted 2 lines

---

I would add this option as an *ioengine* option instead of a general FIo option. There are examples in sync.c, io_uring.c, etc.

This option also needs to be added to the documentation.

Hey Vincent,

Thanks for the review. I will switch over to ioengine options and add some documentation!

Cheers,
-- Nico

Quoted 3 lines

Vincent

#6 lore raw
In reply to Nico Pache
Quoted 9 lines · Nico Pache

This series introduces a new page_fault ioengine and a follow?up option to make khugepaged behavior more deterministic for page fault testing.

The first patch adds an mmap?backed ioengine that allocates anonymous memory and copies data on read/write to intentionally trigger faults. This enables using fio?s existing framework and job files for memory management style workloads without relying on a filesystem. An example job file is included to demonstrate usage.

Quoted 5 lines

The second patch adds a hugepage_delay option to the new engine. It initially maps memory with MADV_NOHUGEPAGE, then (after a configurable delay) switches to MADV_HUGEPAGE via a helper thread. This makes khugepaged candidates reproducible, allows for easier khugepaged testing and improves repeatability for page_fault tests.

Looks pretty clean - echo what Vincent added on the option that should be an engine specific option, but outside of that, the formatting of the code is all wrong. Please follow the style of the code that's in the repo already, which means:

1) Placement of braces
2) Tabs are... tabs, not 4 spaces. The code is a mix and match of this.
3) Appropriate empty lines after variable declarations
4) Style of switch case (indentation and braces)

etc. It's a good start to just use the clang format from the kernel, that should get you basically to where you need to be.

Signature · 1 line
Jens Axboe
author
#7 lore raw
Quoted 17 lines · Jens Axboe
Quoted 9 lines · Nico Pache

This series introduces a new page_fault ioengine and a follow?up option to make khugepaged behavior more deterministic for page fault testing.

The first patch adds an mmap?backed ioengine that allocates anonymous memory and copies data on read/write to intentionally trigger faults. This enables using fio?s existing framework and job files for memory management style workloads without relying on a filesystem. An example job file is included to demonstrate usage.

Quoted 5 lines

The second patch adds a hugepage_delay option to the new engine. It initially maps memory with MADV_NOHUGEPAGE, then (after a configurable delay) switches to MADV_HUGEPAGE via a helper thread. This makes khugepaged candidates reproducible, allows for easier khugepaged testing and improves repeatability for page_fault tests.

Quoted 4 lines

Looks pretty clean - echo what Vincent added on the option that should be an engine specific option, but outside of that, the formatting of the code is all wrong. Please follow the style of the code that's in the repo already, which means:

Thanks! Yeah the engine specific option makes total sense, I will fix that. That should also allow me to add some other functionality to this engine :)

Quoted 8 lines

1) Placement of braces
2) Tabs are... tabs, not 4 spaces. The code is a mix and match of this.
3) Appropriate empty lines after variable declarations
4) Style of switch case (indentation and braces)

etc. It's a good start to just use the clang format from the kernel, that should get you basically to where you need to be.

Sounds good! yeah sorry, didn't really consider the formatting at all for the RFC, was just trying to get the idea out, and make sure there were no glaring issues!

Thanks for the review!
-- Nico

Quoted 4 lines

--
Jens Axboe