Skip to contentSkip to conversation

linux/mem: atomically prefault hugepages in alloc_seg

PATCH In review Michal Sieron 3 participants 8 messages May 20 – Jun 3

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 linux/mem: atomically prefault hugepages in alloc_seg

In rare cases, when a secondary process calls rte_eal_init() it can cause a data race during page prefaulting in alloc_seg().

An atomic compare-exchange in a loop should eliminate the data race.

---
 lib/eal/linux/eal_memalloc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
lib/eal/linux/eal_memalloc.c +4 -1
index a39bc31c7b..cb92fda2e8 100644
@@ -30,6 +30,7 @@
 #include <rte_eal.h>
 #include <rte_memory.h>
 #include <rte_cycles.h>
+#include <rte_atomic.h>
 
 #include "eal_filesystem.h"
 #include "eal_internal_cfg.h"
@@ -600,7 +601,9 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 	 * that is already there, so read the old value, and write itback.
 	 * kernel populates the page with zeroes initially.
 	 */
-	*(volatile int *)addr = *(volatile int *)addr;
+	int snapshot = *(volatile int *)addr;
+	while (!rte_atomic_compare_exchange_strong((volatile int *)addr, &snapshot, snapshot))
+		;
 
 	iova = rte_mem_virt2iova(addr);
 	if (iova == RTE_BAD_PHYS_ADDR) {
Signature · 1 line
2.43.0
#1 lore raw
Quoted 23 lines · On Wed, 20 May 2026 14:57:56 +0200 Michal Sieron <[email protected]>…

In rare cases, when a secondary process calls rte_eal_init() it can cause a data race during page prefaulting in alloc_seg().

An atomic compare-exchange in a loop should eliminate the data race.

---
 lib/eal/linux/eal_memalloc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
lib/eal/linux/eal_memalloc.c +1 -0
index a39bc31c7b..cb92fda2e8 100644
@@ -30,6 +30,7 @@
 #include <rte_eal.h>
 #include <rte_memory.h>
 #include <rte_cycles.h>
+#include <rte_atomic.h>
 
 #include "eal_filesystem.h"
 #include "eal_internal_cfg.h"

lib/eal/linux/eal_memalloc.c alloc_seg(struct rte_memseg *ms, void *addr, int socket_id, · line 609

Quoted 10 lines
lib/eal/linux/eal_memalloc.c +3 -1
@@ -600,7 +601,9 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 	 * that is already there, so read the old value, and write itback.
 	 * kernel populates the page with zeroes initially.
 	 */
-	*(volatile int *)addr = *(volatile int *)addr;
+	int snapshot = *(volatile int *)addr;
+	while (!rte_atomic_compare_exchange_strong((volatile int *)addr, &snapshot, snapshot))
+		;
 
 	iova = rte_mem_virt2iova(addr);
 	if (iova == RTE_BAD_PHYS_ADDR) {

No don't use a loop with compare_exchange_strong here. It could get stuck.
Should just a an relaxed load be enough to get the page in?

#2 lore raw
In reply to Michal Sieron
Quoted 7 lines · On Wed, 20 May 2026 14:57:56 +0200 Michal Sieron <[email protected]>…

In rare cases, when a secondary process calls rte_eal_init() it can cause a data race during page prefaulting in alloc_seg().

An atomic compare-exchange in a loop should eliminate the data race.

---

Build fails. Fix and resubmit.
Looks like you did this against older version of DPDK before stdatomic.

FAILED: [code=1] lib/librte_eal.a.p/eal_linux_eal_memalloc.c.o
ccache clang -Ilib/librte_eal.a.p -Ilib -I../lib -Ilib/eal/common -I../lib/eal/common -I. -I.. -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -I../kernel/linux -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry -I../lib/telemetry -Ilib/argparse -I../lib/argparse -Xclang -fcolor-diagnostics -fsanitize=undefined -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -include rte_config.h -Wvla -Wcast-qual -Wcomma -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wshadow -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=corei7 -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API '-DABI_VERSION="26.2"' -DRTE_EAL_PTHREAD_ATTR_SETAFFINITY_NP -DRTE_LOG_DEFAULT_LOGTYPE=lib.eal -DRTE_ANNOTATE_LOCKS -Wthread-safety -MD -MQ lib/librte_eal.a.p/eal_linux_eal_memalloc.c.o -MF lib/librte_eal.a.p/eal_linux_eal_memalloc.c.o.d -o lib/librte_eal.a.p/eal_linux_eal_memalloc.c.o -c ../lib/eal/linux/eal_memalloc.c ../lib/eal/linux/eal_memalloc.c:605:10: error: implicit declaration of function 'rte_atomic_compare_exchange_strong' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
while (!rte_atomic_compare_exchange_strong((volatile int *)addr, &snapshot, snapshot))
^
../lib/eal/linux/eal_memalloc.c:605:10: note: did you mean '__atomic_compare_exchange_n'?
../lib/eal/include/generic/rte_rwlock.h:189:6: note: '__atomic_compare_exchange_n' declared here

            rte_atomic_compare_exchange_weak_explicit(&rwl->cnt, &x, x + RTE_RWLOCK_WRITE,
            ^
../lib/eal/include/rte_stdatomic.h:151:2: note: expanded from macro 'rte_atomic_compare_exchange_weak_explicit'
        __atomic_compare_exchange_n(ptr, expected, desired, 1, \
        ^

1 error generated.

#3 lore raw
In reply to Michal Sieron
Quoted 7 lines · On Wed, 20 May 2026 14:57:56 +0200 Michal Sieron <[email protected]>…

In rare cases, when a secondary process calls rte_eal_init() it can cause a data race during page prefaulting in alloc_seg().

An atomic compare-exchange in a loop should eliminate the data race.

---

AI had good suggestion when reviewing this.
Your version is still racy (on the read side).

A simple non-racy, and no loop version would be:

	rte_atomic_fetch_or_explicit((int *)addr, 0, rte_memory_order_relaxed);
#4 lore raw

PATCH 1/1 eal: fix data race in hugepage prefault

In reply to Michal Sieron

The prefault step in alloc_seg() reads a value from the hugepage and writes it back unchanged to force the kernel to commit the backing page. The read and write were not atomic, which races with concurrent access to the same physical page from a secondary process attaching to the hugetlbfs-backed mapping during rte_eal_init().

Replace the non-atomic load+store with a single atomic fetch-or of zero. This touches the page with an atomic read-modify-write without changing its contents, eliminating the race while preserving the original intent of forcing a write fault.

Fixes 0f1631be24bd ("mem: fix page fault trigger")
---
 .mailmap                     | 1 +
 lib/eal/linux/eal_memalloc.c | 7 ++++---
 2 files changed, 5 insertions(+), 3 deletions(-)
.mailmap +1 -0
index 4d26d9c286..07c49eb32f 100644
@@ -1086,6 +1086,7 @@ Michal Mazurek <[email protected]>
 Michal Michalik <[email protected]>
 Michal Nowak <[email protected]>
 Michal Schmidt <[email protected]>
+Michal Sieron <[email protected]>
 Michal Swiatkowski <[email protected]>
 Michal Wilczynski <[email protected]>
 Michał Mirosław <[email protected]> <[email protected]>
lib/eal/linux/eal_memalloc.c +4 -3
index a39bc31c7b..e73a0c11a6 100644
@@ -25,6 +25,7 @@
 #include <linux/falloc.h>
 #include <linux/mman.h> /* for hugetlb-related mmap flags */
 
+#include <rte_atomic.h>
 #include <rte_common.h>
 #include <rte_log.h>
 #include <rte_eal.h>
@@ -597,10 +598,10 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 
 	/* we need to trigger a write to the page to enforce page fault and
 	 * ensure that page is accessible to us, but we can't overwrite value
-	 * that is already there, so read the old value, and write itback.
-	 * kernel populates the page with zeroes initially.
+	 * that is already there.
+	 * Use an atomic OR with zero to touch the page without changing its contents.
 	 */
-	*(volatile int *)addr = *(volatile int *)addr;
+	(void)rte_atomic_fetch_or_explicit((int *)addr, 0, rte_memory_order_relaxed);
 
 	iova = rte_mem_virt2iova(addr);
 	if (iova == RTE_BAD_PHYS_ADDR) {
Signature · 1 line
2.53.0
#5 lore raw

Re: [PATCH] eal: fix data race in hugepage prefault

20/05/2026 19:08, Stephen Hemminger:

Quoted 19 lines

The prefault step in alloc_seg() reads a value from the hugepage and writes it back unchanged to force the kernel to commit the backing page. The read and write were not atomic, which races with concurrent access to the same physical page from a secondary process attaching to the hugetlbfs-backed mapping during rte_eal_init().

Replace the non-atomic load+store with a single atomic fetch-or of zero. This touches the page with an atomic read-modify-write without changing its contents, eliminating the race while preserving the original intent of forcing a write fault.

Fixes 0f1631be24bd ("mem: fix page fault trigger")

---

lib/eal/linux/eal_memalloc.c alloc_seg(struct rte_memseg *ms, void *addr, int socket_id, · line 604

Quoted 2 lines
lib/eal/linux/eal_memalloc.c +1 -1
-	*(volatile int *)addr = *(volatile int *)addr;
+	(void)rte_atomic_fetch_or_explicit((int *)addr, 0, rte_memory_order_relaxed);

There is a compilation failure:

lib/eal/linux/eal_memalloc.c:604:8: error: address argument to atomic operation must be a pointer to _Atomic type ('int *' invalid)

        (void)rte_atomic_fetch_or_explicit((int *)addr, 0, rte_memory_order_relaxed);
              ^                            ~~~~~~~~~~~
#6 lore raw

PATCH v2 1/1 eal: fix data race in hugepage prefault

In reply to Stephen Hemminger

The prefault step in alloc_seg() reads a value from the hugepage and writes it back unchanged to force the kernel to commit the backing page. The read and write were not atomic, which races with concurrent access to the same physical page from a secondary process attaching to the hugetlbfs-backed mapping during rte_eal_init().

Replace the non-atomic load+store with a single atomic fetch-or of zero. This touches the page with an atomic read-modify-write without changing its contents, eliminating the race while preserving the original intent of forcing a write fault.

Fixes 0f1631be24bd ("mem: fix page fault trigger")
---
 .mailmap                     | 1 +
 lib/eal/linux/eal_memalloc.c | 8 +++++---
 2 files changed, 6 insertions(+), 3 deletions(-)
.mailmap +1 -0
index 43febb9030..3c45e365d3 100644
@@ -1094,6 +1094,7 @@ Michal Mazurek <[email protected]>
 Michal Michalik <[email protected]>
 Michal Nowak <[email protected]>
 Michal Schmidt <[email protected]>
+Michal Sieron <[email protected]>
 Michal Swiatkowski <[email protected]>
 Michal Wilczynski <[email protected]>
 Michał Mirosław <[email protected]> <[email protected]>
lib/eal/linux/eal_memalloc.c +5 -3
index a39bc31c7b..7359a41d3f 100644
@@ -25,6 +25,7 @@
 #include <linux/falloc.h>
 #include <linux/mman.h> /* for hugetlb-related mmap flags */
 
+#include <rte_atomic.h>
 #include <rte_common.h>
 #include <rte_log.h>
 #include <rte_eal.h>
@@ -597,10 +598,11 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 
 	/* we need to trigger a write to the page to enforce page fault and
 	 * ensure that page is accessible to us, but we can't overwrite value
-	 * that is already there, so read the old value, and write itback.
-	 * kernel populates the page with zeroes initially.
+	 * that is already there.
+	 * Use an atomic OR with zero to touch the page without changing its contents.
 	 */
-	*(volatile int *)addr = *(volatile int *)addr;
+	(void)rte_atomic_fetch_or_explicit((__rte_atomic uint64_t *)addr, 0,
+					   rte_memory_order_relaxed);
 
 	iova = rte_mem_virt2iova(addr);
 	if (iova == RTE_BAD_PHYS_ADDR) {
Signature · 1 line
2.53.0
#7 lore raw

Re: [PATCH v2] eal: fix data race in hugepage prefault

01/06/2026 18:00, Stephen Hemminger:

Quoted 11 lines

The prefault step in alloc_seg() reads a value from the hugepage and writes it back unchanged to force the kernel to commit the backing page. The read and write were not atomic, which races with concurrent access to the same physical page from a secondary process attaching to the hugetlbfs-backed mapping during rte_eal_init().

Replace the non-atomic load+store with a single atomic fetch-or of zero. This touches the page with an atomic read-modify-write without changing its contents, eliminating the race while preserving the original intent of forcing a write fault.

Quoted 5 lines
Fixes 0f1631be24bd ("mem: fix page fault trigger")

Applied, thanks.