Skip to contentSkip to conversation

Windows basic memory management

RFC PATCH Superseded Dmitry Kozlyuk 13 participants 218 messages Mar 30, 2020 – Jun 16, 2020 Page 3 of 3

Subsystems (2)
lore
Share & tools
lore
#160 lore raw

PATCH v7 4/11 eal/mem: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c |  97 ++++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  62 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       |  94 ++++--------------
 lib/librte_eal/linux/eal_memory.c         | 115 +++++-----------------
 4 files changed, 200 insertions(+), 168 deletions(-)
lib/librte_eal/common/eal_common_memory.c +97 -0
index f9fbd3e4e..3325d8c35 100644
@@ -25,6 +25,7 @@
 #include "eal_private.h"
 #include "eal_internal_cfg.h"
 #include "eal_memcfg.h"
+#include "eal_options.h"
 #include "malloc_heap.h"
 
 /*
@@ -182,6 +183,102 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	return aligned_addr;
 }
 
+int
+eal_memseg_list_init_named(struct rte_memseg_list *msl, const char *name,
+		uint64_t page_sz, int n_segs, int socket_id, bool heap)
+{
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL,
+		"Memseg list allocated at socket %i, page size 0x%zxkB\n",
+		socket_id, (size_t)page_sz >> 10);
+
+	return 0;
+}
+
+int
+eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+
+	return eal_memseg_list_init_named(
+		msl, name, page_sz, n_segs, socket_id, heap);
+}
+
+int
+eal_memseg_list_alloc(struct rte_memseg_list *msl, int reserve_flags)
+{
+	size_t page_sz, mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(
+		msl->base_va, &mem_sz, page_sz, 0, reserve_flags);
+	if (addr == NULL) {
+#ifndef RTE_EXEC_ENV_WINDOWS
+		/* The hint would be misleading on Windows, but this function
+		 * is called from many places, including common code,
+		 * so don't duplicate the message.
+		 */
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+		else
+			RTE_LOG(ERR, EAL, "Cannot reserve memory\n");
+#endif
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	RTE_LOG(DEBUG, EAL, "VA reserved for memseg list at %p, size %zx\n",
+			addr, mem_sz);
+
+	return 0;
+}
+
+void
+eal_memseg_list_populate(struct rte_memseg_list *msl, void *addr, int n_segs)
+{
+	size_t page_sz = msl->page_sz;
+	int i;
+
+	for (i = 0; i < n_segs; i++) {
+		struct rte_fbarray *arr = &msl->memseg_arr;
+		struct rte_memseg *ms = rte_fbarray_get(arr, i);
+
+		if (rte_eal_iova_mode() == RTE_IOVA_VA)
+			ms->iova = (uintptr_t)addr;
+		else
+			ms->iova = RTE_BAD_IOVA;
+		ms->addr = addr;
+		ms->hugepage_sz = page_sz;
+		ms->socket_id = 0;
+		ms->len = page_sz;
+
+		rte_fbarray_set_used(arr, i);
+
+		addr = RTE_PTR_ADD(addr, page_sz);
+	}
+}
+
 static struct rte_memseg *
 virt2memseg(const void *addr, const struct rte_memseg_list *msl)
 {
lib/librte_eal/common/eal_private.h +62 -0

68 lines not on this page · raw message

lib/librte_eal/freebsd/eal_memory.c +18 -76

156 lines not on this page · raw message

lib/librte_eal/linux/eal_memory.c +23 -92

216 lines not on this page · raw message

Signature · 1 line
2.25.4
#161 lore raw

Re: [dpdk-dev] [PATCH v7 04/11] eal/mem: extract common code for memseg list initialization

Quoted 21 lines

Subject: [PATCH v7 04/11] eal/mem: extract common code for memseg list initialization

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c |  97
++++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  62 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       |  94 ++++--------------
 lib/librte_eal/linux/eal_memory.c         | 115 +++++-----------------
 4 files changed, 200 insertions(+), 168 deletions(-)
lib/librte_eal/common/eal_common_memory.c +0 -0
index f9fbd3e4e..3325d8c35 100644

[snip]

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 265

Quoted 9 lines
lib/librte_eal/common/eal_common_memory.c +9 -0
+void
+eal_memseg_list_populate(struct rte_memseg_list *msl, void *addr, int
+n_segs) {
+	size_t page_sz = msl->page_sz;
+	int i;
+
+	for (i = 0; i < n_segs; i++) {
+		struct rte_fbarray *arr = &msl->memseg_arr;
+		struct rte_memseg *ms = rte_fbarray_get(arr, i);

Since rte_fbarray_get() can return NULL you should verify *ms isn't dereferenced.

Quoted 15 lines
lib/librte_eal/common/eal_common_memory.c +15 -0
+
+		if (rte_eal_iova_mode() == RTE_IOVA_VA)
+			ms->iova = (uintptr_t)addr;
+		else
+			ms->iova = RTE_BAD_IOVA;
+		ms->addr = addr;
+		ms->hugepage_sz = page_sz;
+		ms->socket_id = 0;
+		ms->len = page_sz;
+
+		rte_fbarray_set_used(arr, i);
+
+		addr = RTE_PTR_ADD(addr, page_sz);
+	}
+}

[snip]

Quoted 2 lines

--
2.25.4

#162 lore raw

Re: [dpdk-dev] [PATCH v7 04/11] eal/mem: extract common code for memseg list initialization

Quoted 24 lines · Tal Shnaiderman
Quoted 21 lines

Subject: [PATCH v7 04/11] eal/mem: extract common code for memseg list initialization

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
  lib/librte_eal/common/eal_common_memory.c |  97
++++++++++++++++++
  lib/librte_eal/common/eal_private.h       |  62 ++++++++++++
  lib/librte_eal/freebsd/eal_memory.c       |  94 ++++--------------
  lib/librte_eal/linux/eal_memory.c         | 115 +++++-----------------
  4 files changed, 200 insertions(+), 168 deletions(-)
lib/librte_eal/common/eal_common_memory.c +0 -0
index f9fbd3e4e..3325d8c35 100644

[snip]

Quoted 11 lines

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 265

Quoted 9 lines
lib/librte_eal/common/eal_common_memory.c +9 -0
+void
+eal_memseg_list_populate(struct rte_memseg_list *msl, void *addr, int
+n_segs) {
+	size_t page_sz = msl->page_sz;
+	int i;
+
+	for (i = 0; i < n_segs; i++) {
+		struct rte_fbarray *arr = &msl->memseg_arr;
+		struct rte_memseg *ms = rte_fbarray_get(arr, i);

Since rte_fbarray_get() can return NULL you should verify *ms isn't dereferenced.

I don't think it's necessary in this case, since this fbarray was just initialized with the n_segs value in the calling code.

Quoted 22 lines
Quoted 16 lines
lib/librte_eal/common/eal_common_memory.c +15 -0
+
+		if (rte_eal_iova_mode() == RTE_IOVA_VA)
+			ms->iova = (uintptr_t)addr;
+		else
+			ms->iova = RTE_BAD_IOVA;
+		ms->addr = addr;
+		ms->hugepage_sz = page_sz;
+		ms->socket_id = 0;
+		ms->len = page_sz;
+
+		rte_fbarray_set_used(arr, i);
+
+		addr = RTE_PTR_ADD(addr, page_sz);
+	}
+}

[snip]

Quoted 2 lines

--
2.25.4

Signature · 2 lines
Thanks,
Anatoly
#163 lore raw

PATCH v7 5/11 eal/mem: extract common code for dynamic memory allocation

In reply to Dmitry Kozlyuk

Code in Linux EAL that supports dynamic memory allocation (as opposed to static allocation used by FreeBSD) is not OS-dependent and can be reused by Windows EAL. Move such code to a file compiled only for the OS that require it. Keep Anatoly Burakov maintainer of extracted code.

---
 MAINTAINERS                               |   1 +
 lib/librte_eal/common/eal_common_dynmem.c | 521 +++++++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  43 +-
 lib/librte_eal/common/meson.build         |   4 +
 lib/librte_eal/freebsd/eal_memory.c       |  12 +-
 lib/librte_eal/linux/Makefile             |   1 +
 lib/librte_eal/linux/eal_memory.c         | 523 +---------------------
 7 files changed, 582 insertions(+), 523 deletions(-)
 create mode 100644 lib/librte_eal/common/eal_common_dynmem.c
MAINTAINERS +1 -0
index 1d9aff26d..a1722ca73 100644
@@ -208,6 +208,7 @@ F: lib/librte_eal/include/rte_fbarray.h
 F: lib/librte_eal/include/rte_mem*
 F: lib/librte_eal/include/rte_malloc.h
 F: lib/librte_eal/common/*malloc*
+F: lib/librte_eal/common/eal_common_dynmem.c
 F: lib/librte_eal/common/eal_common_fbarray.c
 F: lib/librte_eal/common/eal_common_mem*
 F: lib/librte_eal/common/eal_hugepages.h
lib/librte_eal/common/eal_common_dynmem.c new file +521 -0

521 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +42 -1

61 lines not on this page · raw message

lib/librte_eal/common/meson.build +4 -0
index 55aaeb18e..d91c22220 100644
@@ -56,3 +56,7 @@ sources += files(
 	'rte_reciprocal.c',
 	'rte_service.c',
 )
+
+if is_linux
+	sources += files('eal_common_dynmem.c')
+endif
lib/librte_eal/freebsd/eal_memory.c +2 -10
index 29c3ed5a9..7106b8b84 100644
@@ -317,14 +317,6 @@ get_mem_amount(uint64_t page_sz, uint64_t max_mem)
 	return RTE_ALIGN(area_sz, page_sz);
 }
 
-static int
-memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
-		int n_segs, int socket_id, int type_msl_idx)
-{
-	return eal_memseg_list_init(
-		msl, page_sz, n_segs, socket_id, type_msl_idx, false);
-}
-
 static int
 memseg_list_alloc(struct rte_memseg_list *msl)
 {
@@ -421,8 +413,8 @@ memseg_primary_init(void)
 					cur_max_mem);
 			n_segs = cur_mem / hugepage_sz;
 
-			if (memseg_list_init(msl, hugepage_sz, n_segs,
-					0, type_msl_idx))
+			if (eal_memseg_list_init(msl, hugepage_sz, n_segs,
+					0, type_msl_idx, false))
 				return -1;
 
 			total_segs += msl->memseg_arr.len;
lib/librte_eal/linux/Makefile +1 -0
index 8febf2212..07ce643ba 100644
@@ -50,6 +50,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_timer.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memzone.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_dynmem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_mcfg.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memory.c
lib/librte_eal/linux/eal_memory.c +11 -512

586 lines not on this page · raw message

Signature · 1 line
2.25.4
#164 lore raw

PATCH v7 6/11 trace: add size_t field emitter

In reply to Dmitry Kozlyuk

It is not guaranteed that sizeof(long) == sizeof(size_t). On Windows, sizeof(long) == 4 and sizeof(size_t) == 8 for 64-bit programs. Tracepoints using "long" field emitter are therefore invalid there. Add dedicated field emitter for size_t and use it to store size_t values in all existing tracepoints.

---
 lib/librte_eal/include/rte_eal_trace.h   |  8 ++++----
 lib/librte_eal/include/rte_trace_point.h |  3 +++
 lib/librte_mempool/rte_mempool_trace.h   | 10 +++++-----
 3 files changed, 12 insertions(+), 9 deletions(-)
lib/librte_eal/include/rte_eal_trace.h +4 -4
index 1ebb2905a..bcfef0cfa 100644
@@ -143,7 +143,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(const char *type, size_t size, unsigned int align,
 		int socket, void *ptr),
 	rte_trace_point_emit_string(type);
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -154,7 +154,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(const char *type, size_t size, unsigned int align,
 		int socket, void *ptr),
 	rte_trace_point_emit_string(type);
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -164,7 +164,7 @@ RTE_TRACE_POINT(
 	rte_eal_trace_mem_realloc,
 	RTE_TRACE_POINT_ARGS(size_t size, unsigned int align, int socket,
 		void *ptr),
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -183,7 +183,7 @@ RTE_TRACE_POINT(
 		unsigned int flags, unsigned int align, unsigned int bound,
 		const void *mz),
 	rte_trace_point_emit_string(name);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_int(socket_id);
 	rte_trace_point_emit_u32(flags);
 	rte_trace_point_emit_u32(align);
lib/librte_eal/include/rte_trace_point.h +3 -0
index b45171275..377c2414a 100644
@@ -138,6 +138,8 @@ _tp _args \
 #define rte_trace_point_emit_int(val)
 /** Tracepoint function payload for long datatype */
 #define rte_trace_point_emit_long(val)
+/** Tracepoint function payload for size_t datatype */
+#define rte_trace_point_emit_size_t(val)
 /** Tracepoint function payload for float datatype */
 #define rte_trace_point_emit_float(val)
 /** Tracepoint function payload for double datatype */
@@ -395,6 +397,7 @@ do { \
 #define rte_trace_point_emit_i8(in) __rte_trace_point_emit(in, int8_t)
 #define rte_trace_point_emit_int(in) __rte_trace_point_emit(in, int32_t)
 #define rte_trace_point_emit_long(in) __rte_trace_point_emit(in, long)
+#define rte_trace_point_emit_size_t(in) __rte_trace_point_emit(in, size_t)
 #define rte_trace_point_emit_float(in) __rte_trace_point_emit(in, float)
 #define rte_trace_point_emit_double(in) __rte_trace_point_emit(in, double)
 #define rte_trace_point_emit_ptr(in) __rte_trace_point_emit(in, uintptr_t)
lib/librte_mempool/rte_mempool_trace.h +5 -5
index e776df0a6..087c913c8 100644
@@ -72,7 +72,7 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(mempool->name);
 	rte_trace_point_emit_ptr(vaddr);
 	rte_trace_point_emit_u64(iova);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_ptr(free_cb);
 	rte_trace_point_emit_ptr(opaque);
 )
@@ -84,8 +84,8 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_ptr(mempool);
 	rte_trace_point_emit_string(mempool->name);
 	rte_trace_point_emit_ptr(addr);
-	rte_trace_point_emit_long(len);
-	rte_trace_point_emit_long(pg_sz);
+	rte_trace_point_emit_size_t(len);
+	rte_trace_point_emit_size_t(pg_sz);
 	rte_trace_point_emit_ptr(free_cb);
 	rte_trace_point_emit_ptr(opaque);
 )
@@ -126,7 +126,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(struct rte_mempool *mempool, size_t pg_sz),
 	rte_trace_point_emit_ptr(mempool);
 	rte_trace_point_emit_string(mempool->name);
-	rte_trace_point_emit_long(pg_sz);
+	rte_trace_point_emit_size_t(pg_sz);
 )
 
 RTE_TRACE_POINT(
@@ -139,7 +139,7 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_u32(max_objs);
 	rte_trace_point_emit_ptr(vaddr);
 	rte_trace_point_emit_u64(iova);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_ptr(obj_cb);
 	rte_trace_point_emit_ptr(obj_cb_arg);
 )
Signature · 1 line
2.25.4
#165 lore raw

PATCH v7 7/11 eal/windows: add tracing support stubs

In reply to Dmitry Kozlyuk

EAL common code depends on tracepoint calls, but generic implementation cannot be enabled on Windows due to missing standard library facilities. Add stub functions to support tracepoint compilation, so that common code does not have to conditionally include tracepoints until proper support is added.

---
 lib/librte_eal/common/eal_common_thread.c |  5 +---
 lib/librte_eal/common/meson.build         |  1 +
 lib/librte_eal/windows/eal.c              | 34 ++++++++++++++++++++++-
 3 files changed, 35 insertions(+), 5 deletions(-)
lib/librte_eal/common/eal_common_thread.c +1 -4
index f9f588c17..370bb1b63 100644
@@ -15,9 +15,7 @@
 #include <rte_lcore.h>
 #include <rte_memory.h>
 #include <rte_log.h>
-#ifndef RTE_EXEC_ENV_WINDOWS
 #include <rte_trace_point.h>
-#endif
 
 #include "eal_internal_cfg.h"
 #include "eal_private.h"
@@ -169,9 +167,8 @@ static void *rte_thread_init(void *arg)
 		free(params);
 	}
 
-#ifndef RTE_EXEC_ENV_WINDOWS
 	__rte_trace_mem_per_thread_alloc();
-#endif
+
 	return start_routine(routine_arg);
 }
 
lib/librte_eal/common/meson.build +1 -0
index d91c22220..4e9208129 100644
@@ -14,6 +14,7 @@ if is_windows
 		'eal_common_log.c',
 		'eal_common_options.c',
 		'eal_common_thread.c',
+		'eal_common_trace_points.c',
 	)
 	subdir_done()
 endif
lib/librte_eal/windows/eal.c +33 -1
index d084606a6..e7461f731 100644
@@ -17,6 +17,7 @@
 #include <eal_filesystem.h>
 #include <eal_options.h>
 #include <eal_private.h>
+#include <rte_trace_point.h>
 
 #include "eal_windows.h"
 
@@ -221,7 +222,38 @@ rte_eal_init_alert(const char *msg)
 	RTE_LOG(ERR, EAL, "%s\n", msg);
 }
 
- /* Launch threads, called at application init(). */
+/* Stubs to enable EAL trace point compilation
+ * until eal_common_trace.c can be compiled.
+ */
+
+RTE_DEFINE_PER_LCORE(volatile int, trace_point_sz);
+RTE_DEFINE_PER_LCORE(void *, trace_mem);
+
+void
+__rte_trace_mem_per_thread_alloc(void)
+{
+}
+
+void
+__rte_trace_point_emit_field(size_t sz, const char *field,
+	const char *type)
+{
+	RTE_SET_USED(sz);
+	RTE_SET_USED(field);
+	RTE_SET_USED(type);
+}
+
+int
+__rte_trace_point_register(rte_trace_point_t *trace, const char *name,
+	void (*register_fn)(void))
+{
+	RTE_SET_USED(trace);
+	RTE_SET_USED(name);
+	RTE_SET_USED(register_fn);
+	return -ENOTSUP;
+}
+
+/* Launch threads, called at application init(). */
 int
 rte_eal_init(int argc, char **argv)
 {
Signature · 1 line
2.25.4
#166 lore raw

PATCH v7 8/11 eal/windows: replace sys/queue.h with a complete one from FreeBSD

In reply to Dmitry Kozlyuk

Limited version imported previously lacks at least SLIST macros. Import a complete file from FreeBSD, since its license exception is already approved by Technical Board.

---
 lib/librte_eal/windows/include/sys/queue.h | 663 +++++++++++++++++++--
 1 file changed, 601 insertions(+), 62 deletions(-)
lib/librte_eal/windows/include/sys/queue.h +601 -62
index a65949a78..9756bee6f 100644
@@ -8,7 +8,36 @@
 #define	_SYS_QUEUE_H_
 
 /*
- * This file defines tail queues.
+ * This file defines four types of data structures: singly-linked lists,
+ * singly-linked tail queues, lists and tail queues.
+ *
+ * A singly-linked list is headed by a single forward pointer. The elements
+ * are singly linked for minimum space and pointer manipulation overhead at
+ * the expense of O(n) removal for arbitrary elements. New elements can be
+ * added to the list after an existing element or at the head of the list.
+ * Elements being removed from the head of the list should use the explicit
+ * macro for this purpose for optimum efficiency. A singly-linked list may
+ * only be traversed in the forward direction.  Singly-linked lists are ideal
+ * for applications with large datasets and few or no removals or for
+ * implementing a LIFO queue.
+ *
+ * A singly-linked tail queue is headed by a pair of pointers, one to the
+ * head of the list and the other to the tail of the list. The elements are
+ * singly linked for minimum space and pointer manipulation overhead at the
+ * expense of O(n) removal for arbitrary elements. New elements can be added
+ * to the list after an existing element, at the head of the list, or at the
+ * end of the list. Elements being removed from the head of the tail queue
+ * should use the explicit macro for this purpose for optimum efficiency.
+ * A singly-linked tail queue may only be traversed in the forward direction.
+ * Singly-linked tail queues are ideal for applications with large datasets
+ * and few or no removals or for implementing a FIFO queue.
+ *
+ * A list is headed by a single forward pointer (or an array of forward
+ * pointers for a hash table header). The elements are doubly linked
+ * so that an arbitrary element can be removed without a need to
+ * traverse the list. New elements can be added to the list before
+ * or after an existing element or at the head of the list. A list
+ * may be traversed in either direction.
  *
  * A tail queue is headed by a pair of pointers, one to the head of the
  * list and the other to the tail of the list. The elements are doubly
@@ -17,65 +46,93 @@
  * after an existing element, at the head of the list, or at the end of
  * the list. A tail queue may be traversed in either direction.
  *
+ * For details on the use of these macros, see the queue(3) manual page.
+ *
  * Below is a summary of implemented functions where:
  *  +  means the macro is available
  *  -  means the macro is not available
  *  s  means the macro is available but is slow (runs in O(n) time)
  *
- *				TAILQ
- * _HEAD			+
- * _CLASS_HEAD			+
- * _HEAD_INITIALIZER		+
- * _ENTRY			+
- * _CLASS_ENTRY			+
- * _INIT			+
- * _EMPTY			+
- * _FIRST			+
- * _NEXT			+
- * _PREV			+
- * _LAST			+
- * _LAST_FAST			+
- * _FOREACH			+
- * _FOREACH_FROM		+
- * _FOREACH_SAFE		+
- * _FOREACH_FROM_SAFE		+
- * _FOREACH_REVERSE		+
- * _FOREACH_REVERSE_FROM	+
- * _FOREACH_REVERSE_SAFE	+
- * _FOREACH_REVERSE_FROM_SAFE	+
- * _INSERT_HEAD			+
- * _INSERT_BEFORE		+
- * _INSERT_AFTER		+
- * _INSERT_TAIL			+
- * _CONCAT			+
- * _REMOVE_AFTER		-
- * _REMOVE_HEAD			-
- * _REMOVE			+
- * _SWAP			+
+ *				SLIST	LIST	STAILQ	TAILQ
+ * _HEAD			+	+	+	+
+ * _CLASS_HEAD			+	+	+	+
+ * _HEAD_INITIALIZER		+	+	+	+
+ * _ENTRY			+	+	+	+
+ * _CLASS_ENTRY			+	+	+	+
+ * _INIT			+	+	+	+
+ * _EMPTY			+	+	+	+
+ * _FIRST			+	+	+	+
+ * _NEXT			+	+	+	+
+ * _PREV			-	+	-	+
+ * _LAST			-	-	+	+
+ * _LAST_FAST			-	-	-	+
+ * _FOREACH			+	+	+	+
+ * _FOREACH_FROM		+	+	+	+
+ * _FOREACH_SAFE		+	+	+	+
+ * _FOREACH_FROM_SAFE		+	+	+	+
+ * _FOREACH_REVERSE		-	-	-	+
+ * _FOREACH_REVERSE_FROM	-	-	-	+
+ * _FOREACH_REVERSE_SAFE	-	-	-	+
+ * _FOREACH_REVERSE_FROM_SAFE	-	-	-	+
+ * _INSERT_HEAD			+	+	+	+
+ * _INSERT_BEFORE		-	+	-	+
+ * _INSERT_AFTER		+	+	+	+
+ * _INSERT_TAIL			-	-	+	+
+ * _CONCAT			s	s	+	+
+ * _REMOVE_AFTER		+	-	+	-
+ * _REMOVE_HEAD			+	-	+	-
+ * _REMOVE			s	+	s	+
+ * _SWAP			+	+	+	+
  *
  */
-
-#ifdef __cplusplus
-extern "C" {
+#ifdef QUEUE_MACRO_DEBUG
+#warn Use QUEUE_MACRO_DEBUG_TRACE and/or QUEUE_MACRO_DEBUG_TRASH
+#define	QUEUE_MACRO_DEBUG_TRACE
+#define	QUEUE_MACRO_DEBUG_TRASH
 #endif
 
-/*
- * List definitions.
- */
-#define	LIST_HEAD(name, type)						\
-struct name {								\
-	struct type *lh_first;	/* first element */			\
-}
+#ifdef QUEUE_MACRO_DEBUG_TRACE
+/* Store the last 2 places the queue element or head was altered */
+struct qm_trace {
+	unsigned long	 lastline;
+	unsigned long	 prevline;
+	const char	*lastfile;
+	const char	*prevfile;
+};
+
+#define	TRACEBUF	struct qm_trace trace;
+#define	TRACEBUF_INITIALIZER	{ __LINE__, 0, __FILE__, NULL } ,
+
+#define	QMD_TRACE_HEAD(head) do {					\
+	(head)->trace.prevline = (head)->trace.lastline;		\
+	(head)->trace.prevfile = (head)->trace.lastfile;		\
+	(head)->trace.lastline = __LINE__;				\
+	(head)->trace.lastfile = __FILE__;				\
+} while (0)
 
+#define	QMD_TRACE_ELEM(elem) do {					\
+	(elem)->trace.prevline = (elem)->trace.lastline;		\
+	(elem)->trace.prevfile = (elem)->trace.lastfile;		\
+	(elem)->trace.lastline = __LINE__;				\
+	(elem)->trace.lastfile = __FILE__;				\
+} while (0)
+
+#else	/* !QUEUE_MACRO_DEBUG_TRACE */
 #define	QMD_TRACE_ELEM(elem)
 #define	QMD_TRACE_HEAD(head)
 #define	TRACEBUF
 #define	TRACEBUF_INITIALIZER
+#endif	/* QUEUE_MACRO_DEBUG_TRACE */
 
+#ifdef QUEUE_MACRO_DEBUG_TRASH
+#define	QMD_SAVELINK(name, link)	void **name = (void *)&(link)
+#define	TRASHIT(x)		do {(x) = (void *)-1;} while (0)
+#define	QMD_IS_TRASHED(x)	((x) == (void *)(intptr_t)-1)
+#else	/* !QUEUE_MACRO_DEBUG_TRASH */
+#define	QMD_SAVELINK(name, link)
 #define	TRASHIT(x)
 #define	QMD_IS_TRASHED(x)	0
-
-#define	QMD_SAVELINK(name, link)
+#endif	/* QUEUE_MACRO_DEBUG_TRASH */
 
 #ifdef __cplusplus
 /*
@@ -86,6 +143,445 @@ struct name {								\
 #define	QUEUE_TYPEOF(type) struct type
 #endif
 
+/*
+ * Singly-linked List declarations.
+ */
+#define	SLIST_HEAD(name, type)						\
+struct name {								\
+	struct type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	SLIST_ENTRY(type)						\
+struct {								\
+	struct type *sle_next;	/* next element */			\
+}
+
+#define	SLIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *sle_next;		/* next element */		\
+}
+
+/*
+ * Singly-linked List functions.
+ */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm) do {			\
+	if (*(prevp) != (elm))						\
+		panic("Bad prevptr *(%p) == %p != %p",			\
+		    (prevp), *(prevp), (elm));				\
+} while (0)
+#else
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm)
+#endif
+
+#define SLIST_CONCAT(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head1);		\
+	if (curelm == NULL) {						\
+		if ((SLIST_FIRST(head1) = SLIST_FIRST(head2)) != NULL)	\
+			SLIST_INIT(head2);				\
+	} else if (SLIST_FIRST(head2) != NULL) {			\
+		while (SLIST_NEXT(curelm, field) != NULL)		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_NEXT(curelm, field) = SLIST_FIRST(head2);		\
+		SLIST_INIT(head2);					\
+	}								\
+} while (0)
+
+#define	SLIST_EMPTY(head)	((head)->slh_first == NULL)
+
+#define	SLIST_FIRST(head)	((head)->slh_first)
+
+#define	SLIST_FOREACH(var, head, field)					\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			\
+	for ((varp) = &SLIST_FIRST((head));				\
+	    ((var) = *(varp)) != NULL;					\
+	    (varp) = &SLIST_NEXT((var), field))
+
+#define	SLIST_INIT(head) do {						\
+	SLIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	SLIST_INSERT_AFTER(slistelm, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_NEXT((slistelm), field);	\
+	SLIST_NEXT((slistelm), field) = (elm);				\
+} while (0)
+
+#define	SLIST_INSERT_HEAD(head, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_FIRST((head));			\
+	SLIST_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)
+
+#define	SLIST_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.sle_next);			\
+	if (SLIST_FIRST((head)) == (elm)) {				\
+		SLIST_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head);		\
+		while (SLIST_NEXT(curelm, field) != (elm))		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_REMOVE_AFTER(curelm, field);			\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define SLIST_REMOVE_AFTER(elm, field) do {				\
+	SLIST_NEXT(elm, field) =					\
+	    SLIST_NEXT(SLIST_NEXT(elm, field), field);			\
+} while (0)
+
+#define	SLIST_REMOVE_HEAD(head, field) do {				\
+	SLIST_FIRST((head)) = SLIST_NEXT(SLIST_FIRST((head)), field);	\
+} while (0)
+
+#define	SLIST_REMOVE_PREVPTR(prevp, elm, field) do {			\
+	QMD_SLIST_CHECK_PREVPTR(prevp, elm);				\
+	*(prevp) = SLIST_NEXT(elm, field);				\
+	TRASHIT((elm)->field.sle_next);					\
+} while (0)
+
+#define SLIST_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = SLIST_FIRST(head1);		\
+	SLIST_FIRST(head1) = SLIST_FIRST(head2);			\
+	SLIST_FIRST(head2) = swap_first;				\
+} while (0)
+
+/*
+ * Singly-linked Tail queue declarations.
+ */
+#define	STAILQ_HEAD(name, type)						\
+struct name {								\
+	struct type *stqh_first;/* first element */			\
+	struct type **stqh_last;/* addr of last next element */		\
+}
+
+#define	STAILQ_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *stqh_first;	/* first element */			\
+	class type **stqh_last;	/* addr of last next element */		\
+}
+
+#define	STAILQ_HEAD_INITIALIZER(head)					\
+	{ NULL, &(head).stqh_first }
+
+#define	STAILQ_ENTRY(type)						\
+struct {								\
+	struct type *stqe_next;	/* next element */			\
+}
+
+#define	STAILQ_CLASS_ENTRY(type)					\
+struct {								\
+	class type *stqe_next;	/* next element */			\
+}
+
+/*
+ * Singly-linked Tail queue functions.
+ */
+#define	STAILQ_CONCAT(head1, head2) do {				\
+	if (!STAILQ_EMPTY((head2))) {					\
+		*(head1)->stqh_last = (head2)->stqh_first;		\
+		(head1)->stqh_last = (head2)->stqh_last;		\
+		STAILQ_INIT((head2));					\
+	}								\
+} while (0)
+
+#define	STAILQ_EMPTY(head)	((head)->stqh_first == NULL)
+
+#define	STAILQ_FIRST(head)	((head)->stqh_first)
+
+#define	STAILQ_FOREACH(var, head, field)				\
+	for((var) = STAILQ_FIRST((head));				\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = STAILQ_FIRST((head));				\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_FOREACH_FROM_SAFE(var, head, field, tvar)		\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_INIT(head) do {						\
+	STAILQ_FIRST((head)) = NULL;					\
+	(head)->stqh_last = &STAILQ_FIRST((head));			\
+} while (0)
+
+#define	STAILQ_INSERT_AFTER(head, tqelm, elm, field) do {		\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_NEXT((tqelm), field)) == NULL)\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_NEXT((tqelm), field) = (elm);				\
+} while (0)
+
+#define	STAILQ_INSERT_HEAD(head, elm, field) do {			\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_FIRST((head))) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	STAILQ_INSERT_TAIL(head, elm, field) do {			\
+	STAILQ_NEXT((elm), field) = NULL;				\
+	*(head)->stqh_last = (elm);					\
+	(head)->stqh_last = &STAILQ_NEXT((elm), field);			\
+} while (0)
+
+#define	STAILQ_LAST(head, type, field)				\
+	(STAILQ_EMPTY((head)) ? NULL :				\
+	    __containerof((head)->stqh_last,			\
+	    QUEUE_TYPEOF(type), field.stqe_next))
+
+#define	STAILQ_NEXT(elm, field)	((elm)->field.stqe_next)
+
+#define	STAILQ_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.stqe_next);			\
+	if (STAILQ_FIRST((head)) == (elm)) {				\
+		STAILQ_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = STAILQ_FIRST(head);	\
+		while (STAILQ_NEXT(curelm, field) != (elm))		\
+			curelm = STAILQ_NEXT(curelm, field);		\
+		STAILQ_REMOVE_AFTER(head, curelm, field);		\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define STAILQ_REMOVE_AFTER(head, elm, field) do {			\
+	if ((STAILQ_NEXT(elm, field) =					\
+	     STAILQ_NEXT(STAILQ_NEXT(elm, field), field)) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+} while (0)
+
+#define	STAILQ_REMOVE_HEAD(head, field) do {				\
+	if ((STAILQ_FIRST((head)) =					\
+	     STAILQ_NEXT(STAILQ_FIRST((head)), field)) == NULL)		\
+		(head)->stqh_last = &STAILQ_FIRST((head));		\
+} while (0)
+
+#define STAILQ_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = STAILQ_FIRST(head1);		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->stqh_last;		\
+	STAILQ_FIRST(head1) = STAILQ_FIRST(head2);			\
+	(head1)->stqh_last = (head2)->stqh_last;			\
+	STAILQ_FIRST(head2) = swap_first;				\
+	(head2)->stqh_last = swap_last;					\
+	if (STAILQ_EMPTY(head1))					\
+		(head1)->stqh_last = &STAILQ_FIRST(head1);		\
+	if (STAILQ_EMPTY(head2))					\
+		(head2)->stqh_last = &STAILQ_FIRST(head2);		\
+} while (0)
+
+
+/*
+ * List declarations.
+ */
+#define	LIST_HEAD(name, type)						\
+struct name {								\
+	struct type *lh_first;	/* first element */			\
+}
+
+#define	LIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *lh_first;	/* first element */			\
+}
+
+#define	LIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	LIST_ENTRY(type)						\
+struct {								\
+	struct type *le_next;	/* next element */			\
+	struct type **le_prev;	/* address of previous next element */	\
+}
+
+#define	LIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *le_next;	/* next element */			\
+	class type **le_prev;	/* address of previous next element */	\
+}
+
+/*
+ * List functions.
+ */
+
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_LIST_CHECK_HEAD(LIST_HEAD *head, LIST_ENTRY NAME)
+ *
+ * If the list is non-empty, validates that the first element of the list
+ * points back at 'head.'
+ */
+#define	QMD_LIST_CHECK_HEAD(head, field) do {				\
+	if (LIST_FIRST((head)) != NULL &&				\
+	    LIST_FIRST((head))->field.le_prev !=			\
+	     &LIST_FIRST((head)))					\
+		panic("Bad list head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_NEXT(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the list, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_LIST_CHECK_NEXT(elm, field) do {				\
+	if (LIST_NEXT((elm), field) != NULL &&				\
+	    LIST_NEXT((elm), field)->field.le_prev !=			\
+	     &((elm)->field.le_next))					\
+	     	panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_PREV(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the list) points to 'elm.'
+ */
+#define	QMD_LIST_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.le_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
+#define	QMD_LIST_CHECK_HEAD(head, field)
+#define	QMD_LIST_CHECK_NEXT(elm, field)
+#define	QMD_LIST_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
+
+#define LIST_CONCAT(head1, head2, type, field) do {			      \
+	QUEUE_TYPEOF(type) *curelm = LIST_FIRST(head1);			      \
+	if (curelm == NULL) {						      \
+		if ((LIST_FIRST(head1) = LIST_FIRST(head2)) != NULL) {	      \
+			LIST_FIRST(head2)->field.le_prev =		      \
+			    &LIST_FIRST((head1));			      \
+			LIST_INIT(head2);				      \
+		}							      \
+	} else if (LIST_FIRST(head2) != NULL) {				      \
+		while (LIST_NEXT(curelm, field) != NULL)		      \
+			curelm = LIST_NEXT(curelm, field);		      \
+		LIST_NEXT(curelm, field) = LIST_FIRST(head2);		      \
+		LIST_FIRST(head2)->field.le_prev = &LIST_NEXT(curelm, field); \
+		LIST_INIT(head2);					      \
+	}								      \
+} while (0)
+
+#define	LIST_EMPTY(head)	((head)->lh_first == NULL)
+
+#define	LIST_FIRST(head)	((head)->lh_first)
+
+#define	LIST_FOREACH(var, head, field)					\
+	for ((var) = LIST_FIRST((head));				\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = LIST_FIRST((head));				\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_INIT(head) do {						\
+	LIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	LIST_INSERT_AFTER(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_NEXT(listelm, field);				\
+	if ((LIST_NEXT((elm), field) = LIST_NEXT((listelm), field)) != NULL)\
+		LIST_NEXT((listelm), field)->field.le_prev =		\
+		    &LIST_NEXT((elm), field);				\
+	LIST_NEXT((listelm), field) = (elm);				\
+	(elm)->field.le_prev = &LIST_NEXT((listelm), field);		\
+} while (0)
+
+#define	LIST_INSERT_BEFORE(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_PREV(listelm, field);				\
+	(elm)->field.le_prev = (listelm)->field.le_prev;		\
+	LIST_NEXT((elm), field) = (listelm);				\
+	*(listelm)->field.le_prev = (elm);				\
+	(listelm)->field.le_prev = &LIST_NEXT((elm), field);		\
+} while (0)
+
+#define	LIST_INSERT_HEAD(head, elm, field) do {				\
+	QMD_LIST_CHECK_HEAD((head), field);				\
+	if ((LIST_NEXT((elm), field) = LIST_FIRST((head))) != NULL)	\
+		LIST_FIRST((head))->field.le_prev = &LIST_NEXT((elm), field);\
+	LIST_FIRST((head)) = (elm);					\
+	(elm)->field.le_prev = &LIST_FIRST((head));			\
+} while (0)
+
+#define	LIST_NEXT(elm, field)	((elm)->field.le_next)
+
+#define	LIST_PREV(elm, head, type, field)			\
+	((elm)->field.le_prev == &LIST_FIRST((head)) ? NULL :	\
+	    __containerof((elm)->field.le_prev,			\
+	    QUEUE_TYPEOF(type), field.le_next))
+
+#define	LIST_REMOVE(elm, field) do {					\
+	QMD_SAVELINK(oldnext, (elm)->field.le_next);			\
+	QMD_SAVELINK(oldprev, (elm)->field.le_prev);			\
+	QMD_LIST_CHECK_NEXT(elm, field);				\
+	QMD_LIST_CHECK_PREV(elm, field);				\
+	if (LIST_NEXT((elm), field) != NULL)				\
+		LIST_NEXT((elm), field)->field.le_prev = 		\
+		    (elm)->field.le_prev;				\
+	*(elm)->field.le_prev = LIST_NEXT((elm), field);		\
+	TRASHIT(*oldnext);						\
+	TRASHIT(*oldprev);						\
+} while (0)
+
+#define LIST_SWAP(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *swap_tmp = LIST_FIRST(head1);		\
+	LIST_FIRST((head1)) = LIST_FIRST((head2));			\
+	LIST_FIRST((head2)) = swap_tmp;					\
+	if ((swap_tmp = LIST_FIRST((head1))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head1));		\
+	if ((swap_tmp = LIST_FIRST((head2))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head2));		\
+} while (0)
+
 /*
  * Tail queue declarations.
  */
@@ -123,10 +619,58 @@ struct {								\
 /*
  * Tail queue functions.
  */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_TAILQ_CHECK_HEAD(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * If the tailq is non-empty, validates that the first element of the tailq
+ * points back at 'head.'
+ */
+#define	QMD_TAILQ_CHECK_HEAD(head, field) do {				\
+	if (!TAILQ_EMPTY(head) &&					\
+	    TAILQ_FIRST((head))->field.tqe_prev !=			\
+	     &TAILQ_FIRST((head)))					\
+		panic("Bad tailq head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_TAIL(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * Validates that the tail of the tailq is a pointer to pointer to NULL.
+ */
+#define	QMD_TAILQ_CHECK_TAIL(head, field) do {				\
+	if (*(head)->tqh_last != NULL)					\
+	    	panic("Bad tailq NEXT(%p->tqh_last) != NULL", (head)); 	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_NEXT(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the tailq, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_NEXT(elm, field) do {				\
+	if (TAILQ_NEXT((elm), field) != NULL &&				\
+	    TAILQ_NEXT((elm), field)->field.tqe_prev !=			\
+	     &((elm)->field.tqe_next))					\
+		panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_PREV(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the tailq) points to 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.tqe_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
 #define	QMD_TAILQ_CHECK_HEAD(head, field)
 #define	QMD_TAILQ_CHECK_TAIL(head, headname)
 #define	QMD_TAILQ_CHECK_NEXT(elm, field)
 #define	QMD_TAILQ_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
 
 #define	TAILQ_CONCAT(head1, head2, field) do {				\
 	if (!TAILQ_EMPTY(head2)) {					\
@@ -191,9 +735,8 @@ struct {								\
 
 #define	TAILQ_INSERT_AFTER(head, listelm, elm, field) do {		\
 	QMD_TAILQ_CHECK_NEXT(listelm, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field);	\
-	if (TAILQ_NEXT((listelm), field) != NULL)			\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    &TAILQ_NEXT((elm), field);				\
 	else {								\
 		(head)->tqh_last = &TAILQ_NEXT((elm), field);		\
@@ -217,8 +760,7 @@ struct {								\
 
 #define	TAILQ_INSERT_HEAD(head, elm, field) do {			\
 	QMD_TAILQ_CHECK_HEAD(head, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_FIRST((head));			\
-	if (TAILQ_FIRST((head)) != NULL)				\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL)	\
 		TAILQ_FIRST((head))->field.tqe_prev =			\
 		    &TAILQ_NEXT((elm), field);				\
 	else								\
@@ -250,21 +792,24 @@ struct {								\
  * you may want to prefetch the last data element.
  */
 #define	TAILQ_LAST_FAST(head, type, field)			\
-	(TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last,	\
-	QUEUE_TYPEOF(type), field.tqe_next))
+    (TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last, QUEUE_TYPEOF(type), field.tqe_next))
 
 #define	TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
 
 #define	TAILQ_PREV(elm, headname, field)				\
 	(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+#define	TAILQ_PREV_FAST(elm, head, type, field)				\
+    ((elm)->field.tqe_prev == &(head)->tqh_first ? NULL :		\
+     __containerof((elm)->field.tqe_prev, QUEUE_TYPEOF(type), field.tqe_next))
+
 #define	TAILQ_REMOVE(head, elm, field) do {				\
 	QMD_SAVELINK(oldnext, (elm)->field.tqe_next);			\
 	QMD_SAVELINK(oldprev, (elm)->field.tqe_prev);			\
 	QMD_TAILQ_CHECK_NEXT(elm, field);				\
 	QMD_TAILQ_CHECK_PREV(elm, field);				\
 	if ((TAILQ_NEXT((elm), field)) != NULL)				\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    (elm)->field.tqe_prev;				\
 	else {								\
 		(head)->tqh_last = (elm)->field.tqe_prev;		\
@@ -277,26 +822,20 @@ struct {								\
 } while (0)
 
 #define TAILQ_SWAP(head1, head2, type, field) do {			\
-	QUEUE_TYPEOF(type) * swap_first = (head1)->tqh_first;		\
-	QUEUE_TYPEOF(type) * *swap_last = (head1)->tqh_last;		\
+	QUEUE_TYPEOF(type) *swap_first = (head1)->tqh_first;		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->tqh_last;		\
 	(head1)->tqh_first = (head2)->tqh_first;			\
 	(head1)->tqh_last = (head2)->tqh_last;				\
 	(head2)->tqh_first = swap_first;				\
 	(head2)->tqh_last = swap_last;					\
-	swap_first = (head1)->tqh_first;				\
-	if (swap_first != NULL)						\
+	if ((swap_first = (head1)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head1)->tqh_first;	\
 	else								\
 		(head1)->tqh_last = &(head1)->tqh_first;		\
-	swap_first = (head2)->tqh_first;				\
-	if (swap_first != NULL)			\
+	if ((swap_first = (head2)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head2)->tqh_first;	\
 	else								\
 		(head2)->tqh_last = &(head2)->tqh_first;		\
 } while (0)
 
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _SYS_QUEUE_H_ */
+#endif /* !_SYS_QUEUE_H_ */
Signature · 1 line
2.25.4
#167 lore raw

PATCH v7 9/11 eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")
---
 lib/librte_eal/windows/eal_lcore.c   | 185 +++++++++++++++++----------
 lib/librte_eal/windows/eal_windows.h |  10 ++
 2 files changed, 124 insertions(+), 71 deletions(-)
lib/librte_eal/windows/eal_lcore.c +114 -71
index 82ee45413..9d931d50a 100644
@@ -3,103 +3,146 @@
  */
 
 #include <pthread.h>
+#include <stdbool.h>
 #include <stdint.h>
 
 #include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_lcore.h>
+#include <rte_os.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
 #include "eal_windows.h"
 
-/* global data structure that contains the CPU map */
-static struct _wcpu_map {
-	unsigned int total_procs;
-	unsigned int proc_sockets;
-	unsigned int proc_cores;
-	unsigned int reserved;
-	struct _win_lcore_map {
-		uint8_t socket_id;
-		uint8_t core_id;
-	} wlcore_map[RTE_MAX_LCORE];
-} wcpu_map = { 0 };
-
-/*
- * Create a map of all processors and associated cores on the system
- */
+/** Number of logical processors (cores) in a processor group (32 or 64). */
+#define EAL_PROCESSOR_GROUP_SIZE (sizeof(KAFFINITY) * CHAR_BIT)
+
+struct lcore_map {
+	uint8_t socket_id;
+	uint8_t core_id;
+};
+
+struct socket_map {
+	uint16_t node_id;
+};
+
+struct cpu_map {
+	unsigned int socket_count;
+	unsigned int lcore_count;
+	struct lcore_map lcores[RTE_MAX_LCORE];
+	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct cpu_map cpu_map = { 0 };
+
 void
-eal_create_cpu_map()
+eal_create_cpu_map(void)
 {
-	wcpu_map.total_procs =
-		GetActiveProcessorCount(ALL_PROCESSOR_GROUPS);
-
-	LOGICAL_PROCESSOR_RELATIONSHIP lprocRel;
-	DWORD lprocInfoSize = 0;
-	BOOL ht_enabled = FALSE;
-
-	/* First get the processor package information */
-	lprocRel = RelationProcessorPackage;
-	/* Determine the size of buffer we need (pass NULL) */
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_sockets = lprocInfoSize / 48;
-
-	lprocInfoSize = 0;
-	/* Next get the processor core information */
-	lprocRel = RelationProcessorCore;
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_cores = lprocInfoSize / 48;
-
-	if (wcpu_map.total_procs > wcpu_map.proc_cores)
-		ht_enabled = TRUE;
-
-	/* Distribute the socket and core ids appropriately
-	 * across the logical cores. For now, split the cores
-	 * equally across the sockets.
-	 */
-	unsigned int lcore = 0;
-	for (unsigned int socket = 0; socket <
-			wcpu_map.proc_sockets; ++socket) {
-		for (unsigned int core = 0;
-			core < (wcpu_map.proc_cores / wcpu_map.proc_sockets);
-			++core) {
-			wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-			wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-			lcore++;
-			if (ht_enabled) {
-				wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-				wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-				lcore++;
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
+	DWORD infos_size;
+	bool full = false;
+
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			rte_panic("cannot get NUMA node info size, error %lu",
+				GetLastError());
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		rte_panic("cannot allocate memory for NUMA node information");
+		return;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		rte_panic("cannot get NUMA node information, error %lu",
+			GetLastError());
+	}
+
+	info = infos;
+	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
+		unsigned int node_id = info->NumaNode.NodeNumber;
+		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+		struct lcore_map *lcore;
+		unsigned int i, socket_id;
+
+		/* NUMA node may be reported multiple times if it includes
+		 * cores from different processor groups, e. g. 80 cores
+		 * of a physical processor comprise one NUMA node, but two
+		 * processor groups, because group size is limited by 32/64.
+		 */
+		for (socket_id = 0; socket_id < cpu_map.socket_count;
+		    socket_id++) {
+			if (cpu_map.sockets[socket_id].node_id == node_id)
+				break;
+		}
+
+		if (socket_id == cpu_map.socket_count) {
+			if (socket_id == RTE_DIM(cpu_map.sockets)) {
+				full = true;
+				goto exit;
 			}
+
+			cpu_map.sockets[socket_id].node_id = node_id;
+			cpu_map.socket_count++;
+		}
+
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+				continue;
+
+			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
+				full = true;
+				goto exit;
+			}
+
+			lcore = &cpu_map.lcores[cpu_map.lcore_count];
+			lcore->socket_id = socket_id;
+			lcore->core_id =
+				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+			cpu_map.lcore_count++;
 		}
+
+		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
+			(uint8_t *)info + info->Size);
+	}
+
+exit:
+	if (full) {
+		/* RTE_LOG() may not be available, but this is important. */
+		fprintf(stderr, "Enumerated maximum of %u NUMA nodes and %u cores\n",
+			cpu_map.socket_count, cpu_map.lcore_count);
 	}
+
+	free(infos);
 }
 
-/*
- * Check if a cpu is present by the presence of the cpu information for it
- */
 int
 eal_cpu_detected(unsigned int lcore_id)
 {
-	return (lcore_id < wcpu_map.total_procs);
+	return lcore_id < cpu_map.lcore_count;
 }
 
-/*
- * Get CPU socket id for a logical core
- */
 unsigned
 eal_cpu_socket_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].socket_id;
+	return cpu_map.lcores[lcore_id].socket_id;
 }
 
-/*
- * Get CPU socket id (NUMA node) for a logical core
- */
 unsigned
 eal_cpu_core_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].core_id;
+	return cpu_map.lcores[lcore_id].core_id;
+}
+
+unsigned int
+eal_socket_numa_node(unsigned int socket_id)
+{
+	return cpu_map.sockets[socket_id].node_id;
 }
lib/librte_eal/windows/eal_windows.h +10 -0
index fadd676b2..390d2fd66 100644
@@ -26,4 +26,14 @@ void eal_create_cpu_map(void);
  */
 int eal_thread_create(pthread_t *thread);
 
+/**
+ * Get system NUMA node number for a socket ID.
+ *
+ * @param socket_id
+ *  Valid EAL socket ID.
+ * @return
+ *  NUMA node number to use with Win32 API.
+ */
+unsigned int eal_socket_numa_node(unsigned int socket_id);
+
 #endif /* _EAL_WINDOWS_H_ */
Signature · 1 line
2.25.4
#168 lore raw

PATCH v7 10/11 eal/windows: initialize hugepage info

In reply to Dmitry Kozlyuk

Add hugepages discovery ("large pages" in Windows terminology) and update documentation for required privilege setup. Only 2MB hugepages are supported and their number is estimated roughly due to the lack or unstable status of suitable OS APIs. Assign myself as maintainer for the implementation file.

---
 MAINTAINERS                            |   4 +
 config/meson.build                     |   2 +
 doc/guides/windows_gsg/build_dpdk.rst  |  20 -----
 doc/guides/windows_gsg/index.rst       |   1 +
 doc/guides/windows_gsg/run_apps.rst    |  47 +++++++++++
 lib/librte_eal/windows/eal.c           |  14 ++++
 lib/librte_eal/windows/eal_hugepages.c | 108 +++++++++++++++++++++++++
 lib/librte_eal/windows/meson.build     |   1 +
 8 files changed, 177 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
MAINTAINERS +4 -0
index a1722ca73..19b818f69 100644
@@ -336,6 +336,10 @@ F: lib/librte_eal/windows/
 F: lib/librte_eal/rte_eal_exports.def
 F: doc/guides/windows_gsg/
 
+Windows memory allocation
+M: Dmitry Kozlyuk <[email protected]>
+F: lib/librte_eal/eal_hugepages.c
+
 
 Core Libraries
 --------------
config/meson.build +2 -0
index 43ab11310..c1e80de4b 100644
@@ -268,6 +268,8 @@ if is_windows
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
+
+	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/build_dpdk.rst +0 -20
index d46e84e3f..650483e3b 100644
@@ -111,23 +111,3 @@ Depending on the distribution, paths in this file may need adjustments.
 
     meson --cross-file config/x86/meson_mingw.txt -Dexamples=helloworld build
     ninja -C build
-
-
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).
doc/guides/windows_gsg/index.rst +1 -0
index d9b7990a8..e94593572 100644
@@ -12,3 +12,4 @@ Getting Started Guide for Windows
 
     intro
     build_dpdk
+    run_apps
doc/guides/windows_gsg/run_apps.rst new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/windows/eal.c +14 -0
index e7461f731..7c2fcc860 100644
@@ -19,8 +19,11 @@
 #include <eal_private.h>
 #include <rte_trace_point.h>
 
+#include "eal_hugepages.h"
 #include "eal_windows.h"
 
+#define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
@@ -276,6 +279,17 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
+		rte_eal_init_alert("Cannot get hugepage information");
+		rte_errno = EACCES;
+		return -1;
+	}
+
+	if (internal_config.memory == 0 && !internal_config.force_sockets) {
+		if (internal_config.no_hugetlbfs)
+			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_hugepages.c new file +108 -0

108 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index adfc8b9b7..52978e9d7 100644
@@ -6,6 +6,7 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_log.c',
 	'eal_thread.c',
Signature · 1 line
2.25.4
#169 lore raw

PATCH v7 11/11 eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode. Multi-process mode is not implemented and is forcefully disabled at startup. Assign myself as a maintainer for Windows file and memory management implementation.

---
 MAINTAINERS                                   |   1 +
 config/meson.build                            |  12 +-
 doc/guides/windows_gsg/run_apps.rst           |  54 +-
 lib/librte_eal/common/meson.build             |  11 +
 lib/librte_eal/common/rte_malloc.c            |   1 +
 lib/librte_eal/rte_eal_exports.def            | 119 +++
 lib/librte_eal/windows/eal.c                  |  63 +-
 lib/librte_eal/windows/eal_file.c             | 125 +++
 lib/librte_eal/windows/eal_memalloc.c         | 441 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 710 ++++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  75 ++
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |  17 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   6 +
 18 files changed, 1771 insertions(+), 7 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_file.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
MAINTAINERS +1 -0
index 19b818f69..5140756b3 100644
@@ -339,6 +339,7 @@ F: doc/guides/windows_gsg/
 Windows memory allocation
 M: Dmitry Kozlyuk <[email protected]>
 F: lib/librte_eal/eal_hugepages.c
+F: lib/librte_eal/eal_mem*
 
 
 Core Libraries
config/meson.build +9 -3
index c1e80de4b..d3f05f878 100644
@@ -261,15 +261,21 @@ if is_freebsd
 endif
 
 if is_windows
-	# Minimum supported API is Windows 7.
-	add_project_arguments('-D_WIN32_WINNT=0x0601', language: 'c')
+	# VirtualAlloc2() is available since Windows 10 / Server 2016.
+	add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'c')
 
 	# Use MinGW-w64 stdio, because DPDK assumes ANSI-compliant formatting.
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	# Contrary to docs, VirtualAlloc2() is exported by mincore.lib
+	# in Windows SDK, while MinGW exports it by advapi32.a.
+	if is_ms_linker
+		add_project_link_arguments('-lmincore', language: 'c')
+	endif
+
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +51 -3

68 lines not on this page · raw message

lib/librte_eal/common/meson.build +11 -0
index 4e9208129..310844269 100644
@@ -8,13 +8,24 @@ if is_windows
 		'eal_common_bus.c',
 		'eal_common_class.c',
 		'eal_common_devargs.c',
+		'eal_common_dynmem.c',
 		'eal_common_errno.c',
+		'eal_common_fbarray.c',
 		'eal_common_launch.c',
 		'eal_common_lcore.c',
 		'eal_common_log.c',
+		'eal_common_mcfg.c',
+		'eal_common_memalloc.c',
+		'eal_common_memory.c',
+		'eal_common_memzone.c',
 		'eal_common_options.c',
+		'eal_common_string_fns.c',
+		'eal_common_tailqs.c',
 		'eal_common_thread.c',
 		'eal_common_trace_points.c',
+		'malloc_elem.c',
+		'malloc_heap.c',
+		'rte_malloc.c',
 	)
 	subdir_done()
 endif
lib/librte_eal/common/rte_malloc.c +1 -0
index f1b73168b..9d39e58c0 100644
@@ -20,6 +20,7 @@
 #include <rte_lcore.h>
 #include <rte_common.h>
 #include <rte_spinlock.h>
+
 #include <rte_eal_trace.h>
 
 #include <rte_malloc.h>
lib/librte_eal/rte_eal_exports.def +119 -0

128 lines not on this page · raw message

lib/librte_eal/windows/eal.c +62 -1

87 lines not on this page · raw message

lib/librte_eal/windows/eal_file.c new file +125 -0

125 lines not on this page · raw message

lib/librte_eal/windows/eal_memalloc.c new file +441 -0

441 lines not on this page · raw message

lib/librte_eal/windows/eal_memory.c new file +710 -0

710 lines not on this page · raw message

lib/librte_eal/windows/eal_mp.c new file +103 -0

103 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +75 -0

87 lines not on this page · raw message

lib/librte_eal/windows/include/meson.build +1 -0
index 5fb1962ac..b3534b025 100644
@@ -5,5 +5,6 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_virt2phys.h',
         'rte_windows.h',
 )
lib/librte_eal/windows/include/rte_os.h +17 -0
index 510e39e03..cb10d6494 100644
@@ -14,6 +14,7 @@
 #include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
+#include <string.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -36,6 +37,9 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
@@ -46,6 +50,7 @@ extern "C" {
 typedef long long ssize_t;
 
 #ifndef RTE_TOOLCHAIN_GCC
+
 static inline int
 asprintf(char **buffer, const char *format, ...)
 {
@@ -72,6 +77,18 @@ asprintf(char **buffer, const char *format, ...)
 	}
 	return ret;
 }
+
+static inline const char *
+eal_strerror(int code)
+{
+	static char buffer[128];
+
+	strerror_s(buffer, sizeof(buffer), code);
+	return buffer;
+}
+
+#define strerror eal_strerror
+
 #endif /* RTE_TOOLCHAIN_GCC */
 
 #ifdef __cplusplus
lib/librte_eal/windows/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/meson.build +6 -0
index 52978e9d7..ded5a2b80 100644
@@ -6,10 +6,16 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_file.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_log.c',
+	'eal_memalloc.c',
+	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'fnmatch.c',
 	'getopt.c',
 )
+
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
Signature · 1 line
2.25.4
#170 lore raw

v8 cover Windows basic memory management

In reply to Dmitry Kozlyuk

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing without IOVA is available.

Testing revealed Windows Server 2019 does not allow allocating hugepage memory at a reserved address, despite advertised API. So allocator has to temporary free the region to be allocated. This creates in inherent race condition. This issue is being discussed with Microsoft privately.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications: rte_mem_map, rte_mem_unmap, rte_mem_lock, rte_mem_page_size.

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

To avoid code duplication between Linux and Windows EAL, common code for EALs supporting dynamic memory allocation is extracted (discussed with Anatoly Burakov in v4 thread). This is a separate patch to ease the review, but it can be merged with the previous one.

EAL tracepoints save size_t values as long, which is invalid on Windows. New size_t emitter for tracepoints is introduced (suggested by Jerin Jacob to Fady Bader, see [1]). Also, to avoid workaround in every file using the tracepoints, stubs are added to Windows EAL.

Entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file. The file is imported as-is, so it causes a bunch of checkpatch warnings.

[1]: http://mails.dpdk.org/archives/dev/2020-May/168076.html

---

v8:
* Log eal_memseg_list_alloc() failure at caller sites (Anatoly Burakov).

v7:
* Change EAL internal file management API (Neil Horman).

v6:
* Fix 32-bit build on x86 (CI).
* Fix Makefile build (Anatoly Burakov, Thomas Monjalon).
* Restore 32-bit common code (Anatoly Burakov).
* Fix error reporting in memory management (Anatoly Burakov).
* Add Doxygen comment for size_t tracepoint emitter (Jerin Jacob).
* Update MAINTAINERS for new files and new code (Thomas Monjalon).
* Rename rte_get_page_size to rte_mem_page_size.
* Mark DPDK-only wrappers internal, move them to separate file.
* Get rid of warnings in enabled common code with Clang on Windows.

v5:
* Fix allocation and deallocation on Windows Server (Fady Bader).
* Replace remaining VirtualFree with VirtualFreeEx (Ranjit Menon).
* Fix errors in eal_get_virtual_area (Anatoly Burakov).
* Fix error handling and documentation for rte_mem_lock (Anatoly Burakov).
* Extract common code for EALs w/dynamic allocation (Anatoly Burakov).
* Use POSIX value for rte_errno after rte_mem_unmap() on Windows.
* Add stubs to use tracing functions without workarounds.

v4:
* Rebase on ToT, drop patches merged into master.
* Rearrange patches to split Windows code (Jerin).
* Fix Linux and FreeBSD build with make (Ophir).
* Use int instead of enum to hold a set of flags (Anatoly).
* Rename eal_mem_reserve items and fix their description (Anatoly).
* Add eal_mem_set_dump() wrapper around madvise (Anatoly).
* Don't claim Windows Server 2016 support due to lack of API (Tal).
* Replace enum rte_page_sizes with a set of #defines (Jerin).
* Fix documentation, SPDX tags, logging (Thomas).

v3:
* Fix Linux build on and aarch64 and 32-bit x86 (reported by CI).
* Fix logic and error handling while allocating segments.
* Fix Unix rte_mem_map(): return NULL on failure.
* Fix some checkpatch.sh issues:
* Do not return positive errno, use DWORD for GetLastError().
* Make dpdk-kmods source files non-executable.
* Improve GSG for Windows Server (suggested by Ranjit Menon).

v2:
* Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there.
* Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler.
* Use EAL prefix for private facilities instead if RTE.
* Improve documentation comments for new functions.
* Remove co-installer for virt2phys. Add a typecast for clarity.
* Document virt2phys in user guide, improve its own README.
* Explicitly and forcefully disable multi-process.

*** BLURB HERE ***

Dmitry Kozlyuk (11):
eal: replace rte_page_sizes with a set of constants
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal/mem: extract common code for memseg list initialization eal/mem: extract common code for dynamic memory allocation
trace: add size_t field emitter
eal/windows: add tracing support stubs
eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: improve CPU and NUMA node detection eal/windows: initialize hugepage info
eal/windows: implement basic memory management

 MAINTAINERS                                   |   9 +
 config/meson.build                            |  12 +-
 doc/guides/rel_notes/release_20_08.rst        |   2 +
 doc/guides/windows_gsg/build_dpdk.rst         |  20 -
 doc/guides/windows_gsg/index.rst              |   1 +
 doc/guides/windows_gsg/run_apps.rst           |  95 +++
 lib/librte_eal/common/eal_common_dynmem.c     | 521 +++++++++++++
 lib/librte_eal/common/eal_common_fbarray.c    |  71 +-
 lib/librte_eal/common/eal_common_memory.c     | 156 +++-
 lib/librte_eal/common/eal_common_thread.c     |   5 +-
 lib/librte_eal/common/eal_private.h           | 254 ++++++-
 lib/librte_eal/common/meson.build             |  16 +
 lib/librte_eal/common/rte_malloc.c            |   1 +
 lib/librte_eal/freebsd/Makefile               |   5 +
 lib/librte_eal/freebsd/eal_memory.c           |  98 +--
 lib/librte_eal/include/rte_eal_memory.h       |  93 +++
 lib/librte_eal/include/rte_eal_trace.h        |   8 +-
 lib/librte_eal/include/rte_memory.h           |  23 +-
 lib/librte_eal/include/rte_trace_point.h      |   3 +
 lib/librte_eal/linux/Makefile                 |   6 +
 lib/librte_eal/linux/eal_memalloc.c           |   5 +-
 lib/librte_eal/linux/eal_memory.c             | 618 +--------------
 lib/librte_eal/meson.build                    |   4 +
 lib/librte_eal/rte_eal_exports.def            | 119 +++
 lib/librte_eal/rte_eal_version.map            |   9 +
 lib/librte_eal/unix/eal_file.c                |  80 ++
 lib/librte_eal/unix/eal_unix_memory.c         | 152 ++++
 lib/librte_eal/unix/meson.build               |   7 +
 lib/librte_eal/windows/eal.c                  | 107 +++
 lib/librte_eal/windows/eal_file.c             | 125 +++
 lib/librte_eal/windows/eal_hugepages.c        | 108 +++
 lib/librte_eal/windows/eal_lcore.c            | 185 +++--
 lib/librte_eal/windows/eal_memalloc.c         | 441 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 710 ++++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  85 +++
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |  17 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/sys/queue.h    | 663 ++++++++++++++--
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   7 +
 lib/librte_mempool/rte_mempool_trace.h        |  10 +-
 44 files changed, 4056 insertions(+), 938 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/common/eal_common_dynmem.c
 create mode 100644 lib/librte_eal/include/rte_eal_memory.h
 create mode 100644 lib/librte_eal/unix/eal_file.c
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_file.c
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
Signature · 1 line
2.25.4
#171 lore raw

PATCH v8 1/11 eal: replace rte_page_sizes with a set of constants

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_sizes has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Remove rte_page_sizes and replace its values with #define's. This enumeration is not used in public API, so there's no ABI breakage. Announce API changes for 20.08 in documentation.

---
 doc/guides/rel_notes/release_20_08.rst |  2 ++
 lib/librte_eal/include/rte_memory.h    | 23 ++++++++++-------------
 2 files changed, 12 insertions(+), 13 deletions(-)
doc/guides/rel_notes/release_20_08.rst +2 -0
index 39064afbe..2041a29b9 100644
@@ -85,6 +85,8 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ``rte_page_sizes`` enumeration is replaced with ``RTE_PGSIZE_xxx`` defines.
+
 
 ABI Changes
 -----------
lib/librte_eal/include/rte_memory.h +10 -13
index 3d8d0bd69..65374d53a 100644
@@ -24,19 +24,16 @@ extern "C" {
 #include <rte_config.h>
 #include <rte_fbarray.h>
 
-__extension__
-enum rte_page_sizes {
-	RTE_PGSIZE_4K    = 1ULL << 12,
-	RTE_PGSIZE_64K   = 1ULL << 16,
-	RTE_PGSIZE_256K  = 1ULL << 18,
-	RTE_PGSIZE_2M    = 1ULL << 21,
-	RTE_PGSIZE_16M   = 1ULL << 24,
-	RTE_PGSIZE_256M  = 1ULL << 28,
-	RTE_PGSIZE_512M  = 1ULL << 29,
-	RTE_PGSIZE_1G    = 1ULL << 30,
-	RTE_PGSIZE_4G    = 1ULL << 32,
-	RTE_PGSIZE_16G   = 1ULL << 34,
-};
+#define RTE_PGSIZE_4K   (1ULL << 12)
+#define RTE_PGSIZE_64K  (1ULL << 16)
+#define RTE_PGSIZE_256K (1ULL << 18)
+#define RTE_PGSIZE_2M   (1ULL << 21)
+#define RTE_PGSIZE_16M  (1ULL << 24)
+#define RTE_PGSIZE_256M (1ULL << 28)
+#define RTE_PGSIZE_512M (1ULL << 29)
+#define RTE_PGSIZE_1G   (1ULL << 30)
+#define RTE_PGSIZE_4G   (1ULL << 32)
+#define RTE_PGSIZE_16G  (1ULL << 34)
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
 
Signature · 1 line
2.25.4
#172 lore raw

PATCH v8 2/11 eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk

Introduce OS-independent wrappers in order to support common EAL code on Unix and Windows:

* eal_file_open: open or create a file.
* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. These thin wrappers require no special maintenance.

Common code supporting multi-process doesn't use the new wrappers, because it is inherently Unix-specific and would impose excessive requirements on the wrappers.

---
 MAINTAINERS                                |  3 +
 lib/librte_eal/common/eal_common_fbarray.c | 31 ++++-----
 lib/librte_eal/common/eal_private.h        | 73 ++++++++++++++++++++
 lib/librte_eal/freebsd/Makefile            |  4 ++
 lib/librte_eal/linux/Makefile              |  4 ++
 lib/librte_eal/meson.build                 |  4 ++
 lib/librte_eal/unix/eal_file.c             | 80 ++++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |  6 ++
 8 files changed, 186 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_eal/unix/eal_file.c
 create mode 100644 lib/librte_eal/unix/meson.build
MAINTAINERS +3 -0
index d2b286701..1d9aff26d 100644
@@ -323,6 +323,9 @@ FreeBSD UIO
 M: Bruce Richardson <[email protected]>
 F: kernel/freebsd/nic_uio/
 
+Unix shared files
+F: lib/librte_eal/unix/
+
 Windows support
 M: Harini Ramakrishnan <[email protected]>
 M: Omar Cardona <[email protected]>
lib/librte_eal/common/eal_common_fbarray.c +12 -19
index 4f8f1af73..c52ddb967 100644
@@ -8,8 +8,8 @@
 #include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
-#include <sys/file.h>
 #include <string.h>
+#include <unistd.h>
 
 #include <rte_common.h>
 #include <rte_log.h>
@@ -85,10 +85,8 @@ resize_and_map(int fd, void *addr, size_t len)
 	char path[PATH_MAX];
 	void *map_addr;
 
-	if (ftruncate(fd, len)) {
+	if (eal_file_truncate(fd, len)) {
 		RTE_LOG(ERR, EAL, "Cannot truncate %s\n", path);
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 
@@ -772,15 +770,15 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		 * and see if we succeed. If we don't, someone else is using it
 		 * already.
 		 */
-		fd = open(path, O_CREAT | O_RDWR, 0600);
+		fd = eal_file_open(path, EAL_OPEN_CREATE | EAL_OPEN_READWRITE);
 		if (fd < 0) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't open %s: %s\n",
-					__func__, path, strerror(errno));
-			rte_errno = errno;
+				__func__, path, rte_strerror(rte_errno));
 			goto fail;
-		} else if (flock(fd, LOCK_EX | LOCK_NB)) {
+		} else if (eal_file_lock(
+				fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't lock %s: %s\n",
-					__func__, path, strerror(errno));
+				__func__, path, rte_strerror(rte_errno));
 			rte_errno = EBUSY;
 			goto fail;
 		}
@@ -789,10 +787,8 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		 * still attach to it, but no other process could reinitialize
 		 * it.
 		 */
-		if (flock(fd, LOCK_SH | LOCK_NB)) {
-			rte_errno = errno;
+		if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 			goto fail;
-		}
 
 		if (resize_and_map(fd, data, mmap_len))
 			goto fail;
@@ -888,17 +884,14 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 
 	eal_get_fbarray_path(path, sizeof(path), arr->name);
 
-	fd = open(path, O_RDWR);
+	fd = eal_file_open(path, EAL_OPEN_READWRITE);
 	if (fd < 0) {
-		rte_errno = errno;
 		goto fail;
 	}
 
 	/* lock the file, to let others know we're using it */
-	if (flock(fd, LOCK_SH | LOCK_NB)) {
-		rte_errno = errno;
+	if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 		goto fail;
-	}
 
 	if (resize_and_map(fd, data, mmap_len))
 		goto fail;
@@ -1025,7 +1018,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		 * has been detached by all other processes
 		 */
 		fd = tmp->fd;
-		if (flock(fd, LOCK_EX | LOCK_NB)) {
+		if (eal_file_lock(fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "Cannot destroy fbarray - another process is using it\n");
 			rte_errno = EBUSY;
 			ret = -1;
@@ -1042,7 +1035,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 			 * we're still holding an exclusive lock, so drop it to
 			 * shared.
 			 */
-			flock(fd, LOCK_SH | LOCK_NB);
+			eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN);
 
 			ret = -1;
 			goto out;
lib/librte_eal/common/eal_private.h +73 -0

77 lines not on this page · raw message

lib/librte_eal/freebsd/Makefile +4 -0
index af95386d4..0f8741d96 100644
@@ -7,6 +7,7 @@ LIB = librte_eal.a
 
 ARCH_DIR ?= $(RTE_ARCH)
 VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/unix
 VPATH += $(RTE_SDK)/lib/librte_eal/common
 
 CFLAGS += -I$(SRCDIR)/include
@@ -74,6 +75,9 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_service.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_random.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
+# from unix dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_file.c
+
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_hypervisor.c
lib/librte_eal/linux/Makefile +4 -0
index 48cc34844..331489f99 100644
@@ -7,6 +7,7 @@ LIB = librte_eal.a
 
 ARCH_DIR ?= $(RTE_ARCH)
 VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/unix
 VPATH += $(RTE_SDK)/lib/librte_eal/common
 
 CFLAGS += -I$(SRCDIR)/include
@@ -81,6 +82,9 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_service.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_random.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
+# from unix dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_file.c
+
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_hypervisor.c
lib/librte_eal/meson.build +4 -0
index e301f4558..8d492897d 100644
@@ -6,6 +6,10 @@ subdir('include')
 
 subdir('common')
 
+if not is_windows
+	subdir('unix')
+endif
+
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
 subdir(exec_env)
 
lib/librte_eal/unix/eal_file.c new file +80 -0

80 lines not on this page · raw message

lib/librte_eal/unix/meson.build new file +6 -0
new file mode 100644
index 000000000..21029ba1a
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Dmitry Kozlyuk
+
+sources += files(
+	'eal_file.c',
+)
Signature · 1 line
2.25.4
#173 lore raw

Re: [dpdk-dev] [PATCH v8 02/11] eal: introduce internal wrappers for file operations

10/06/2020 16:27, Dmitry Kozlyuk:

Quoted 10 lines

Introduce OS-independent wrappers in order to support common EAL code on Unix and Windows:

* eal_file_open: open or create a file.
* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. These thin wrappers require no special maintenance.

[...]

MAINTAINERS FreeBSD UIO · line 327

Quoted 4 lines
MAINTAINERS +2 -0
+Unix shared files
+F: lib/librte_eal/unix/

Can be moved in "EAL API and common code".

#174 lore raw

PATCH v8 3/11 eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

Introduce OS-independent wrappers for memory management operations used across DPDK and specifically in common code of EAL:

* rte_mem_map()
* rte_mem_unmap()
* rte_mem_page_size()
* rte_mem_lock()

Windows uses different APIs for memory mapping and reservation, while Unices reserve memory by mapping it. Introduce EAL private functions to support memory reservation in common code:

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive. New symbols are internal. Being thin wrappers, they require no special maintenance.

---
 lib/librte_eal/common/eal_common_fbarray.c |  40 +++---
 lib/librte_eal/common/eal_common_memory.c  |  61 ++++-----
 lib/librte_eal/common/eal_private.h        |  78 ++++++++++-
 lib/librte_eal/freebsd/Makefile            |   1 +
 lib/librte_eal/include/rte_eal_memory.h    |  93 +++++++++++++
 lib/librte_eal/linux/Makefile              |   1 +
 lib/librte_eal/linux/eal_memalloc.c        |   5 +-
 lib/librte_eal/rte_eal_version.map         |   9 ++
 lib/librte_eal/unix/eal_unix_memory.c      | 152 +++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |   1 +
 10 files changed, 376 insertions(+), 65 deletions(-)
 create mode 100644 lib/librte_eal/include/rte_eal_memory.h
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
lib/librte_eal/common/eal_common_fbarray.c +19 -21
index c52ddb967..98fd6e1f6 100644
@@ -5,15 +5,16 @@
 #include <fcntl.h>
 #include <inttypes.h>
 #include <limits.h>
-#include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
 #include <string.h>
 #include <unistd.h>
 
 #include <rte_common.h>
-#include <rte_log.h>
+#include <rte_eal_memory.h>
 #include <rte_errno.h>
+#include <rte_log.h>
+#include <rte_memory.h>
 #include <rte_spinlock.h>
 #include <rte_tailq.h>
 
@@ -90,12 +91,9 @@ resize_and_map(int fd, void *addr, size_t len)
 		return -1;
 	}
 
-	map_addr = mmap(addr, len, PROT_READ | PROT_WRITE,
-			MAP_SHARED | MAP_FIXED, fd, 0);
+	map_addr = rte_mem_map(addr, len, RTE_PROT_READ | RTE_PROT_WRITE,
+			RTE_MAP_SHARED | RTE_MAP_FORCE_ADDRESS, fd, 0);
 	if (map_addr != addr) {
-		RTE_LOG(ERR, EAL, "mmap() failed: %s\n", strerror(errno));
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 	return 0;
@@ -733,7 +731,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -754,11 +752,13 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 
 	if (internal_config.no_shconf) {
 		/* remap virtual area as writable */
-		void *new_data = mmap(data, mmap_len, PROT_READ | PROT_WRITE,
-				MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, fd, 0);
-		if (new_data == MAP_FAILED) {
+		static const int flags = RTE_MAP_FORCE_ADDRESS |
+			RTE_MAP_PRIVATE | RTE_MAP_ANONYMOUS;
+		void *new_data = rte_mem_map(data, mmap_len,
+			RTE_PROT_READ | RTE_PROT_WRITE, flags, fd, 0);
+		if (new_data == NULL) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't remap anonymous memory: %s\n",
-					__func__, strerror(errno));
+					__func__, rte_strerror(rte_errno));
 			goto fail;
 		}
 	} else {
@@ -820,7 +820,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -858,7 +858,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -909,7 +909,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -937,8 +937,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -957,7 +956,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 		goto out;
 	}
 
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, close fd and remove the tailq entry */
 	if (tmp->fd >= 0)
@@ -992,8 +991,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -1042,7 +1040,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		}
 		close(fd);
 	}
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, remove the tailq entry */
 	TAILQ_REMOVE(&mem_area_tailq, tmp, next);
lib/librte_eal/common/eal_common_memory.c +23 -38

143 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +75 -3

106 lines not on this page · raw message

lib/librte_eal/freebsd/Makefile +1 -0
index 0f8741d96..2374ba0b7 100644
@@ -77,6 +77,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_file.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
lib/librte_eal/include/rte_eal_memory.h new file +93 -0

93 lines not on this page · raw message

lib/librte_eal/linux/Makefile +1 -0
index 331489f99..8febf2212 100644
@@ -84,6 +84,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_file.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
lib/librte_eal/linux/eal_memalloc.c +2 -3
index 2c717f8bd..bf29b83c6 100644
@@ -630,7 +630,7 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 mapped:
 	munmap(addr, alloc_sz);
 unmapped:
-	flags = MAP_FIXED;
+	flags = EAL_RESERVE_FORCE_ADDRESS;
 	new_addr = eal_get_virtual_area(addr, &alloc_sz, alloc_sz, 0, flags);
 	if (new_addr != addr) {
 		if (new_addr != NULL)
@@ -687,8 +687,7 @@ free_seg(struct rte_memseg *ms, struct hugepage_info *hi,
 		return -1;
 	}
 
-	if (madvise(ms->addr, ms->len, MADV_DONTDUMP) != 0)
-		RTE_LOG(DEBUG, EAL, "madvise failed: %s\n", strerror(errno));
+	eal_mem_set_dump(ms->addr, ms->len, false);
 
 	exit_early = false;
 
lib/librte_eal/rte_eal_version.map +9 -0
index d8038749a..196eef5af 100644
@@ -387,3 +387,12 @@ EXPERIMENTAL {
 	rte_trace_regexp;
 	rte_trace_save;
 };
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};
lib/librte_eal/unix/eal_unix_memory.c new file +152 -0

152 lines not on this page · raw message

lib/librte_eal/unix/meson.build +1 -0
index 21029ba1a..e733910a1 100644
@@ -3,4 +3,5 @@
 
 sources += files(
 	'eal_file.c',
+	'eal_unix_memory.c',
 )
Signature · 1 line
2.25.4
#175 lore raw

Re: [dpdk-dev] [PATCH v8 03/11] eal: introduce memory management wrappers

10/06/2020 16:27, Dmitry Kozlyuk:

Quoted 20 lines

Introduce OS-independent wrappers for memory management operations used across DPDK and specifically in common code of EAL:

* rte_mem_map()
* rte_mem_unmap()
* rte_mem_page_size()
* rte_mem_lock()

Windows uses different APIs for memory mapping and reservation, while Unices reserve memory by mapping it. Introduce EAL private functions to support memory reservation in common code:

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive. New symbols are internal. Being thin wrappers, they require no special maintenance.

[...]

Quoted 7 lines

+/**
+ * Reserve a region of virtual memory.
+ *
+ * Use eal_mem_free() to free reserved memory.
+ *
+ * @param requested_addr
+ * A desired reservation addressm which must be page-aligned.

Typo: addressm

Quoted 12 lines

+ * The system might not respect it.
+ * NULL means the address will be chosen by the system.
+ * @param size
+ * Reservation size. Must be a multiple of system page size.
+ * @param flags
+ * Reservation options, a combination of eal_mem_reserve_flags.
+ * @returns
+ * Starting address of the reserved area on success, NULL on failure.
+ * Callers must not access this memory until remapping it.
+ */

lib/librte_eal/common/eal_private.h +2 -0
+void *
+eal_mem_reserve(void *requested_addr, size_t size, int flags);

[...]

Quoted 2 lines

+/**
+ * Configure memory region inclusion into core dumps.

Not sure about the word "core" here.

Quoted 12 lines

+ *
+ * @param virt
+ * Starting address of the region.
+ * @param size
+ * Size of the region.
+ * @param dump
+ * True to include memory into core dumps, false to exclude.
+ * @return
+ * 0 on success, (-1) on failure and rte_errno is set.
+ */

lib/librte_eal/common/eal_private.h +2 -0
+int
+eal_mem_set_dump(void *virt, size_t size, bool dump);

[...]

lib/librte_eal/include/rte_eal_memory.h line 9

Quoted 12 lines
lib/librte_eal/include/rte_eal_memory.h new file +9 -0
@@ -0,0 +1,93 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Dmitry Kozlyuk
+ */
+
+#include <stdint.h>
+
+#include <rte_compat.h>
+
+/** @file Mamory management wrappers used across DPDK. */

typo on "Mamory"
+
@file must be on a separate line:

/** @file
*
* Memory management wrappers used across DPDK.
*/

lib/librte_eal/include/rte_eal_memory.h line 18

Quoted 9 lines
lib/librte_eal/include/rte_eal_memory.h new file +9 -0
+
+/** Memory protection flags. */
+enum rte_mem_prot {
+	RTE_PROT_READ = 1 << 0,   /**< Read access. */
+	RTE_PROT_WRITE = 1 << 1,  /**< Write access. */
+	RTE_PROT_EXECUTE = 1 << 2 /**< Code execution. */
+};
+
+/** Additional flags for memory mapping. */

Typo on "Addtional"

Quoted 15 lines
lib/librte_eal/include/rte_eal_memory.h new file +15 -0
+enum rte_map_flags {
+	/** Changes to the mapped memory are visible to other processes. */
+	RTE_MAP_SHARED = 1 << 0,
+	/** Mapping is not backed by a regular file. */
+	RTE_MAP_ANONYMOUS = 1 << 1,
+	/** Copy-on-write mapping, changes are invisible to other processes. */
+	RTE_MAP_PRIVATE = 1 << 2,
+	/**
+	 * Force mapping to the requested address. This flag should be used
+	 * with caution, because to fulfill the request implementation
+	 * may remove all other mappings in the requested region. However,
+	 * it is not required to do so, thus mapping with this flag may fail.
+	 */
+	RTE_MAP_FORCE_ADDRESS = 1 << 3
+};

[...]

lib/librte_eal/include/rte_eal_memory.h

Quoted 8 lines
lib/librte_eal/include/rte_eal_memory.h new file +8 -0
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};

Not sure why these functions are internal.
They may be useful for DPDK applications.
We would need to add the file in doxygen index.

If we want to keep them internal, we should add a doxygen marker @internal.

Quoted 1 line

+#include <rte_eal_memory.h>

I think we should find a better file name for these wrappers. "EAL memory" means DPDK memory allocator in my mind. We need a file name which is about OS-independent wrappers, or libc wrappers.
What about rte_libc_mem.h? rte_mem_os.h? something else?

#176 lore raw

Re: [dpdk-dev] [PATCH v8 03/11] eal: introduce memory management wrappers

Quoted 14 lines

[...]

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 8 lines
lib/librte_eal/rte_eal_version.map +8 -0
+INTERNAL {
+     global:
+
+     rte_mem_lock;
+     rte_mem_map;
+     rte_mem_page_size;
+     rte_mem_unmap;
+};

Not sure why these functions are internal.
They may be useful for DPDK applications.
We would need to add the file in doxygen index.

Not sure if they are in DPDK scope, apart from rte_mem_lock, which generalizes rte_mem_lock_page already in rte_memory.h. What may be typical use cases for data-plane apps? I can see testpmd using mmap for allocating external memory (because of possible use of hugepages), does it need these functions exposed?

Quoted 3 lines

If we want to keep them internal, we should add a doxygen marker @internal.

IIRC, it were you who proposed making them internal instead of experimental. And internal symbols can always be exposed later.

Quoted 7 lines
Quoted 1 line

+#include <rte_eal_memory.h>

I think we should find a better file name for these wrappers. "EAL memory" means DPDK memory allocator in my mind. We need a file name which is about OS-independent wrappers, or libc wrappers.
What about rte_libc_mem.h? rte_mem_os.h? something else?

See above, but anyway, "libc" is non-generic.

#177 lore raw

Re: [dpdk-dev] [PATCH v8 03/11] eal: introduce memory management wrappers

12/06/2020 15:44, Dmitry Kozliuk:

Quoted 14 lines
Quoted 13 lines

[...]

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 8 lines
lib/librte_eal/rte_eal_version.map +8 -0
+INTERNAL {
+     global:
+
+     rte_mem_lock;
+     rte_mem_map;
+     rte_mem_page_size;
+     rte_mem_unmap;
+};

Not sure why these functions are internal.
They may be useful for DPDK applications.
We would need to add the file in doxygen index.

Quoted 5 lines

Not sure if they are in DPDK scope, apart from rte_mem_lock, which generalizes rte_mem_lock_page already in rte_memory.h. What may be typical use cases for data-plane apps? I can see testpmd using mmap for allocating external memory (because of possible use of hugepages), does it need these functions exposed?

There is a chance the application needs such functions for another part of its dataplane.

Quoted 5 lines
Quoted 2 lines

If we want to keep them internal, we should add a doxygen marker @internal.

IIRC, it were you who proposed making them internal instead of experimental. And internal symbols can always be exposed later.

They they can be exposed later.
I think it's good to start internal.
Please add the @internal tag in doxygen to make the status clear.

Quoted 9 lines
Quoted 7 lines
Quoted 1 line

+#include <rte_eal_memory.h>

I think we should find a better file name for these wrappers. "EAL memory" means DPDK memory allocator in my mind. We need a file name which is about OS-independent wrappers, or libc wrappers.
What about rte_libc_mem.h? rte_mem_os.h? something else?

See above, but anyway, "libc" is non-generic.

Why libc is not generic?

Which file name can it be?

#178 lore raw

Re: [dpdk-dev] [PATCH v8 03/11] eal: introduce memory management wrappers

Quoted 4 lines
Quoted 3 lines

Not sure if they are in DPDK scope, apart from rte_mem_lock, which generalizes rte_mem_lock_page already in rte_memory.h. What may be

typical

Quoted 8 lines
Quoted 1 line

use cases for data-plane apps? I can see testpmd using mmap for

allocating

Quoted 1 line

external memory (because of possible use of hugepages), does it need

these

Quoted 1 line

functions exposed?

There is a chance the application needs such functions for another part of its dataplane.

Such reasoning can justify any API. DPDK needs compatibility layers, but collecting and providing them is not the goal of the kit. I can only think of using mmap in data plane as a high-performance IPC with non-DPDK management app.

Please add the @internal tag in doxygen to make the status clear.

Quoted 1 line

OK.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

+#include <rte_eal_memory.h>

I think we should find a better file name for these wrappers. "EAL memory" means DPDK memory allocator in my mind. We need a file name which is about OS-independent wrappers, or libc wrappers.
What about rte_libc_mem.h? rte_mem_os.h? something else?

Quoted 4 lines
Quoted 1 line

See above, but anyway, "libc" is non-generic.

Why libc is not generic?

It means nothing on Windows. Also, mmap has little to do with libc.

Which file name can it be?

Quoted 1 line

Your rte_mem_os.h sounds good, except internal header better be rte_eal_mem_os.h. Alternative: rte_eal_paging.h.

Quoted 1 line
#179 lore raw

Re: [dpdk-dev] [PATCH v8 03/11] eal: introduce memory management wrappers

12/06/2020 22:24, Dmitry Kozliuk:

Quoted 13 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

+#include <rte_eal_memory.h>

I think we should find a better file name for these wrappers. "EAL memory" means DPDK memory allocator in my mind. We need a file name which is about OS-independent wrappers, or libc wrappers.
What about rte_libc_mem.h? rte_mem_os.h? something else?

Quoted 4 lines
Quoted 1 line

See above, but anyway, "libc" is non-generic.

Why libc is not generic?

Quoted 6 lines

It means nothing on Windows. Also, mmap has little to do with libc.

Which file name can it be?

Your rte_mem_os.h sounds good, except internal header better be rte_eal_mem_os.h. Alternative: rte_eal_paging.h.

I don't think we need to use the prefix "eal" if we already have "rte_mem". I have no strong opinion.
rte_eal_paging looks also good.
Anyway, it is internal, so we could change it if needed.
Please choose one :)

#180 lore raw

PATCH v8 4/11 eal/mem: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c |  95 +++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  62 +++++++++++
 lib/librte_eal/freebsd/eal_memory.c       |  94 ++++-------------
 lib/librte_eal/linux/eal_memory.c         | 119 +++++-----------------
 4 files changed, 203 insertions(+), 167 deletions(-)
lib/librte_eal/common/eal_common_memory.c +95 -0
index f9fbd3e4e..76cf87c1f 100644
@@ -25,6 +25,7 @@
 #include "eal_private.h"
 #include "eal_internal_cfg.h"
 #include "eal_memcfg.h"
+#include "eal_options.h"
 #include "malloc_heap.h"
 
 /*
@@ -182,6 +183,100 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	return aligned_addr;
 }
 
+int
+eal_memseg_list_init_named(struct rte_memseg_list *msl, const char *name,
+		uint64_t page_sz, int n_segs, int socket_id, bool heap)
+{
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL,
+		"Memseg list allocated at socket %i, page size 0x%zxkB\n",
+		socket_id, (size_t)page_sz >> 10);
+
+	return 0;
+}
+
+int
+eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+
+	return eal_memseg_list_init_named(
+		msl, name, page_sz, n_segs, socket_id, heap);
+}
+
+int
+eal_memseg_list_alloc(struct rte_memseg_list *msl, int reserve_flags)
+{
+	size_t page_sz, mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(
+		msl->base_va, &mem_sz, page_sz, 0, reserve_flags);
+	if (addr == NULL) {
+#ifndef RTE_EXEC_ENV_WINDOWS
+		/* The hint would be misleading on Windows, but this function
+		 * is called from many places, including common code,
+		 * so don't duplicate the message.
+		 */
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+#endif
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	RTE_LOG(DEBUG, EAL, "VA reserved for memseg list at %p, size %zx\n",
+			addr, mem_sz);
+
+	return 0;
+}
+
+void
+eal_memseg_list_populate(struct rte_memseg_list *msl, void *addr, int n_segs)
+{
+	size_t page_sz = msl->page_sz;
+	int i;
+
+	for (i = 0; i < n_segs; i++) {
+		struct rte_fbarray *arr = &msl->memseg_arr;
+		struct rte_memseg *ms = rte_fbarray_get(arr, i);
+
+		if (rte_eal_iova_mode() == RTE_IOVA_VA)
+			ms->iova = (uintptr_t)addr;
+		else
+			ms->iova = RTE_BAD_IOVA;
+		ms->addr = addr;
+		ms->hugepage_sz = page_sz;
+		ms->socket_id = 0;
+		ms->len = page_sz;
+
+		rte_fbarray_set_used(arr, i);
+
+		addr = RTE_PTR_ADD(addr, page_sz);
+	}
+}
+
 static struct rte_memseg *
 virt2memseg(const void *addr, const struct rte_memseg_list *msl)
 {
lib/librte_eal/common/eal_private.h +62 -0

68 lines not on this page · raw message

lib/librte_eal/freebsd/eal_memory.c +18 -76

156 lines not on this page · raw message

lib/librte_eal/linux/eal_memory.c +28 -91

222 lines not on this page · raw message

Signature · 1 line
2.25.4
#181 lore raw

Re: [dpdk-dev] [PATCH v8 04/11] eal/mem: extract common code for memseg list initialization

10/06/2020 16:27, Dmitry Kozlyuk:

Quoted 8 lines
lib/librte_eal/common/eal_common_memory.c +8 -0
+eal_memseg_list_init_named(struct rte_memseg_list *msl, const char *name,
+		uint64_t page_sz, int n_segs, int socket_id, bool heap)
+{
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 204

Quoted 10 lines
lib/librte_eal/common/eal_common_memory.c +10 -0
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL,
+		"Memseg list allocated at socket %i, page size 0x%zxkB\n",
+		socket_id, (size_t)page_sz >> 10);

page_sz is uint64_t, so the right printf specifier is PRIx64.

[...]

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 235

Quoted 2 lines
lib/librte_eal/common/eal_common_memory.c +2 -0
+#ifndef RTE_EXEC_ENV_WINDOWS
+		/* The hint would be misleading on Windows, but this function

Would be better to explain the reason of misleading in this comment.

Quoted 8 lines
lib/librte_eal/common/eal_common_memory.c +8 -0
+		 * is called from many places, including common code,
+		 * so don't duplicate the message.
+		 */
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+#endif

[...]

lib/librte_eal/linux/eal_memory.c prealloc_segments(struct hugepage_file *hugepages, int n_pages) · line 980

Quoted 4 lines
lib/librte_eal/linux/eal_memory.c +4 -0
+			if (memseg_list_alloc(msl) < 0) {
+				RTE_LOG(ERR, EAL,
+					"Cannot preallocate %zukB hugepages\n",
+					page_sz >> 10);

page_sz is uint64_t, so the right printf specifier is PRIx64.

#182 lore raw

PATCH v8 5/11 eal/mem: extract common code for dynamic memory allocation

In reply to Dmitry Kozlyuk

Code in Linux EAL that supports dynamic memory allocation (as opposed to static allocation used by FreeBSD) is not OS-dependent and can be reused by Windows EAL. Move such code to a file compiled only for the OS that require it. Keep Anatoly Burakov maintainer of extracted code.

---
 MAINTAINERS                               |   1 +
 lib/librte_eal/common/eal_common_dynmem.c | 521 +++++++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  43 +-
 lib/librte_eal/common/meson.build         |   4 +
 lib/librte_eal/freebsd/eal_memory.c       |  12 +-
 lib/librte_eal/linux/Makefile             |   1 +
 lib/librte_eal/linux/eal_memory.c         | 523 +---------------------
 7 files changed, 582 insertions(+), 523 deletions(-)
 create mode 100644 lib/librte_eal/common/eal_common_dynmem.c
MAINTAINERS +1 -0
index 1d9aff26d..a1722ca73 100644
@@ -208,6 +208,7 @@ F: lib/librte_eal/include/rte_fbarray.h
 F: lib/librte_eal/include/rte_mem*
 F: lib/librte_eal/include/rte_malloc.h
 F: lib/librte_eal/common/*malloc*
+F: lib/librte_eal/common/eal_common_dynmem.c
 F: lib/librte_eal/common/eal_common_fbarray.c
 F: lib/librte_eal/common/eal_common_mem*
 F: lib/librte_eal/common/eal_hugepages.h
lib/librte_eal/common/eal_common_dynmem.c new file +521 -0

521 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +42 -1

61 lines not on this page · raw message

lib/librte_eal/common/meson.build +4 -0
index 55aaeb18e..d91c22220 100644
@@ -56,3 +56,7 @@ sources += files(
 	'rte_reciprocal.c',
 	'rte_service.c',
 )
+
+if is_linux
+	sources += files('eal_common_dynmem.c')
+endif
lib/librte_eal/freebsd/eal_memory.c +2 -10
index 29c3ed5a9..7106b8b84 100644
@@ -317,14 +317,6 @@ get_mem_amount(uint64_t page_sz, uint64_t max_mem)
 	return RTE_ALIGN(area_sz, page_sz);
 }
 
-static int
-memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
-		int n_segs, int socket_id, int type_msl_idx)
-{
-	return eal_memseg_list_init(
-		msl, page_sz, n_segs, socket_id, type_msl_idx, false);
-}
-
 static int
 memseg_list_alloc(struct rte_memseg_list *msl)
 {
@@ -421,8 +413,8 @@ memseg_primary_init(void)
 					cur_max_mem);
 			n_segs = cur_mem / hugepage_sz;
 
-			if (memseg_list_init(msl, hugepage_sz, n_segs,
-					0, type_msl_idx))
+			if (eal_memseg_list_init(msl, hugepage_sz, n_segs,
+					0, type_msl_idx, false))
 				return -1;
 
 			total_segs += msl->memseg_arr.len;
lib/librte_eal/linux/Makefile +1 -0
index 8febf2212..07ce643ba 100644
@@ -50,6 +50,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_timer.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memzone.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_dynmem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_mcfg.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memory.c
lib/librte_eal/linux/eal_memory.c +11 -512

586 lines not on this page · raw message

Signature · 1 line
2.25.4
#183 lore raw

PATCH v8 6/11 trace: add size_t field emitter

In reply to Dmitry Kozlyuk

It is not guaranteed that sizeof(long) == sizeof(size_t). On Windows, sizeof(long) == 4 and sizeof(size_t) == 8 for 64-bit programs. Tracepoints using "long" field emitter are therefore invalid there. Add dedicated field emitter for size_t and use it to store size_t values in all existing tracepoints.

---
 lib/librte_eal/include/rte_eal_trace.h   |  8 ++++----
 lib/librte_eal/include/rte_trace_point.h |  3 +++
 lib/librte_mempool/rte_mempool_trace.h   | 10 +++++-----
 3 files changed, 12 insertions(+), 9 deletions(-)
lib/librte_eal/include/rte_eal_trace.h +4 -4
index 1ebb2905a..bcfef0cfa 100644
@@ -143,7 +143,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(const char *type, size_t size, unsigned int align,
 		int socket, void *ptr),
 	rte_trace_point_emit_string(type);
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -154,7 +154,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(const char *type, size_t size, unsigned int align,
 		int socket, void *ptr),
 	rte_trace_point_emit_string(type);
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -164,7 +164,7 @@ RTE_TRACE_POINT(
 	rte_eal_trace_mem_realloc,
 	RTE_TRACE_POINT_ARGS(size_t size, unsigned int align, int socket,
 		void *ptr),
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -183,7 +183,7 @@ RTE_TRACE_POINT(
 		unsigned int flags, unsigned int align, unsigned int bound,
 		const void *mz),
 	rte_trace_point_emit_string(name);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_int(socket_id);
 	rte_trace_point_emit_u32(flags);
 	rte_trace_point_emit_u32(align);
lib/librte_eal/include/rte_trace_point.h +3 -0
index b45171275..377c2414a 100644
@@ -138,6 +138,8 @@ _tp _args \
 #define rte_trace_point_emit_int(val)
 /** Tracepoint function payload for long datatype */
 #define rte_trace_point_emit_long(val)
+/** Tracepoint function payload for size_t datatype */
+#define rte_trace_point_emit_size_t(val)
 /** Tracepoint function payload for float datatype */
 #define rte_trace_point_emit_float(val)
 /** Tracepoint function payload for double datatype */
@@ -395,6 +397,7 @@ do { \
 #define rte_trace_point_emit_i8(in) __rte_trace_point_emit(in, int8_t)
 #define rte_trace_point_emit_int(in) __rte_trace_point_emit(in, int32_t)
 #define rte_trace_point_emit_long(in) __rte_trace_point_emit(in, long)
+#define rte_trace_point_emit_size_t(in) __rte_trace_point_emit(in, size_t)
 #define rte_trace_point_emit_float(in) __rte_trace_point_emit(in, float)
 #define rte_trace_point_emit_double(in) __rte_trace_point_emit(in, double)
 #define rte_trace_point_emit_ptr(in) __rte_trace_point_emit(in, uintptr_t)
lib/librte_mempool/rte_mempool_trace.h +5 -5
index e776df0a6..087c913c8 100644
@@ -72,7 +72,7 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(mempool->name);
 	rte_trace_point_emit_ptr(vaddr);
 	rte_trace_point_emit_u64(iova);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_ptr(free_cb);
 	rte_trace_point_emit_ptr(opaque);
 )
@@ -84,8 +84,8 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_ptr(mempool);
 	rte_trace_point_emit_string(mempool->name);
 	rte_trace_point_emit_ptr(addr);
-	rte_trace_point_emit_long(len);
-	rte_trace_point_emit_long(pg_sz);
+	rte_trace_point_emit_size_t(len);
+	rte_trace_point_emit_size_t(pg_sz);
 	rte_trace_point_emit_ptr(free_cb);
 	rte_trace_point_emit_ptr(opaque);
 )
@@ -126,7 +126,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(struct rte_mempool *mempool, size_t pg_sz),
 	rte_trace_point_emit_ptr(mempool);
 	rte_trace_point_emit_string(mempool->name);
-	rte_trace_point_emit_long(pg_sz);
+	rte_trace_point_emit_size_t(pg_sz);
 )
 
 RTE_TRACE_POINT(
@@ -139,7 +139,7 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_u32(max_objs);
 	rte_trace_point_emit_ptr(vaddr);
 	rte_trace_point_emit_u64(iova);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_ptr(obj_cb);
 	rte_trace_point_emit_ptr(obj_cb_arg);
 )
Signature · 1 line
2.25.4
#184 lore raw

PATCH v8 7/11 eal/windows: add tracing support stubs

In reply to Dmitry Kozlyuk

EAL common code depends on tracepoint calls, but generic implementation cannot be enabled on Windows due to missing standard library facilities. Add stub functions to support tracepoint compilation, so that common code does not have to conditionally include tracepoints until proper support is added.

---
 lib/librte_eal/common/eal_common_thread.c |  5 +---
 lib/librte_eal/common/meson.build         |  1 +
 lib/librte_eal/windows/eal.c              | 34 ++++++++++++++++++++++-
 3 files changed, 35 insertions(+), 5 deletions(-)
lib/librte_eal/common/eal_common_thread.c +1 -4
index f9f588c17..370bb1b63 100644
@@ -15,9 +15,7 @@
 #include <rte_lcore.h>
 #include <rte_memory.h>
 #include <rte_log.h>
-#ifndef RTE_EXEC_ENV_WINDOWS
 #include <rte_trace_point.h>
-#endif
 
 #include "eal_internal_cfg.h"
 #include "eal_private.h"
@@ -169,9 +167,8 @@ static void *rte_thread_init(void *arg)
 		free(params);
 	}
 
-#ifndef RTE_EXEC_ENV_WINDOWS
 	__rte_trace_mem_per_thread_alloc();
-#endif
+
 	return start_routine(routine_arg);
 }
 
lib/librte_eal/common/meson.build +1 -0
index d91c22220..4e9208129 100644
@@ -14,6 +14,7 @@ if is_windows
 		'eal_common_log.c',
 		'eal_common_options.c',
 		'eal_common_thread.c',
+		'eal_common_trace_points.c',
 	)
 	subdir_done()
 endif
lib/librte_eal/windows/eal.c +33 -1
index d084606a6..e7461f731 100644
@@ -17,6 +17,7 @@
 #include <eal_filesystem.h>
 #include <eal_options.h>
 #include <eal_private.h>
+#include <rte_trace_point.h>
 
 #include "eal_windows.h"
 
@@ -221,7 +222,38 @@ rte_eal_init_alert(const char *msg)
 	RTE_LOG(ERR, EAL, "%s\n", msg);
 }
 
- /* Launch threads, called at application init(). */
+/* Stubs to enable EAL trace point compilation
+ * until eal_common_trace.c can be compiled.
+ */
+
+RTE_DEFINE_PER_LCORE(volatile int, trace_point_sz);
+RTE_DEFINE_PER_LCORE(void *, trace_mem);
+
+void
+__rte_trace_mem_per_thread_alloc(void)
+{
+}
+
+void
+__rte_trace_point_emit_field(size_t sz, const char *field,
+	const char *type)
+{
+	RTE_SET_USED(sz);
+	RTE_SET_USED(field);
+	RTE_SET_USED(type);
+}
+
+int
+__rte_trace_point_register(rte_trace_point_t *trace, const char *name,
+	void (*register_fn)(void))
+{
+	RTE_SET_USED(trace);
+	RTE_SET_USED(name);
+	RTE_SET_USED(register_fn);
+	return -ENOTSUP;
+}
+
+/* Launch threads, called at application init(). */
 int
 rte_eal_init(int argc, char **argv)
 {
Signature · 1 line
2.25.4
#185 lore raw

PATCH v8 8/11 eal/windows: replace sys/queue.h with a complete one from FreeBSD

In reply to Dmitry Kozlyuk

Limited version imported previously lacks at least SLIST macros. Import a complete file from FreeBSD, since its license exception is already approved by Technical Board.

---
 lib/librte_eal/windows/include/sys/queue.h | 663 +++++++++++++++++++--
 1 file changed, 601 insertions(+), 62 deletions(-)
lib/librte_eal/windows/include/sys/queue.h +601 -62
index a65949a78..9756bee6f 100644
@@ -8,7 +8,36 @@
 #define	_SYS_QUEUE_H_
 
 /*
- * This file defines tail queues.
+ * This file defines four types of data structures: singly-linked lists,
+ * singly-linked tail queues, lists and tail queues.
+ *
+ * A singly-linked list is headed by a single forward pointer. The elements
+ * are singly linked for minimum space and pointer manipulation overhead at
+ * the expense of O(n) removal for arbitrary elements. New elements can be
+ * added to the list after an existing element or at the head of the list.
+ * Elements being removed from the head of the list should use the explicit
+ * macro for this purpose for optimum efficiency. A singly-linked list may
+ * only be traversed in the forward direction.  Singly-linked lists are ideal
+ * for applications with large datasets and few or no removals or for
+ * implementing a LIFO queue.
+ *
+ * A singly-linked tail queue is headed by a pair of pointers, one to the
+ * head of the list and the other to the tail of the list. The elements are
+ * singly linked for minimum space and pointer manipulation overhead at the
+ * expense of O(n) removal for arbitrary elements. New elements can be added
+ * to the list after an existing element, at the head of the list, or at the
+ * end of the list. Elements being removed from the head of the tail queue
+ * should use the explicit macro for this purpose for optimum efficiency.
+ * A singly-linked tail queue may only be traversed in the forward direction.
+ * Singly-linked tail queues are ideal for applications with large datasets
+ * and few or no removals or for implementing a FIFO queue.
+ *
+ * A list is headed by a single forward pointer (or an array of forward
+ * pointers for a hash table header). The elements are doubly linked
+ * so that an arbitrary element can be removed without a need to
+ * traverse the list. New elements can be added to the list before
+ * or after an existing element or at the head of the list. A list
+ * may be traversed in either direction.
  *
  * A tail queue is headed by a pair of pointers, one to the head of the
  * list and the other to the tail of the list. The elements are doubly
@@ -17,65 +46,93 @@
  * after an existing element, at the head of the list, or at the end of
  * the list. A tail queue may be traversed in either direction.
  *
+ * For details on the use of these macros, see the queue(3) manual page.
+ *
  * Below is a summary of implemented functions where:
  *  +  means the macro is available
  *  -  means the macro is not available
  *  s  means the macro is available but is slow (runs in O(n) time)
  *
- *				TAILQ
- * _HEAD			+
- * _CLASS_HEAD			+
- * _HEAD_INITIALIZER		+
- * _ENTRY			+
- * _CLASS_ENTRY			+
- * _INIT			+
- * _EMPTY			+
- * _FIRST			+
- * _NEXT			+
- * _PREV			+
- * _LAST			+
- * _LAST_FAST			+
- * _FOREACH			+
- * _FOREACH_FROM		+
- * _FOREACH_SAFE		+
- * _FOREACH_FROM_SAFE		+
- * _FOREACH_REVERSE		+
- * _FOREACH_REVERSE_FROM	+
- * _FOREACH_REVERSE_SAFE	+
- * _FOREACH_REVERSE_FROM_SAFE	+
- * _INSERT_HEAD			+
- * _INSERT_BEFORE		+
- * _INSERT_AFTER		+
- * _INSERT_TAIL			+
- * _CONCAT			+
- * _REMOVE_AFTER		-
- * _REMOVE_HEAD			-
- * _REMOVE			+
- * _SWAP			+
+ *				SLIST	LIST	STAILQ	TAILQ
+ * _HEAD			+	+	+	+
+ * _CLASS_HEAD			+	+	+	+
+ * _HEAD_INITIALIZER		+	+	+	+
+ * _ENTRY			+	+	+	+
+ * _CLASS_ENTRY			+	+	+	+
+ * _INIT			+	+	+	+
+ * _EMPTY			+	+	+	+
+ * _FIRST			+	+	+	+
+ * _NEXT			+	+	+	+
+ * _PREV			-	+	-	+
+ * _LAST			-	-	+	+
+ * _LAST_FAST			-	-	-	+
+ * _FOREACH			+	+	+	+
+ * _FOREACH_FROM		+	+	+	+
+ * _FOREACH_SAFE		+	+	+	+
+ * _FOREACH_FROM_SAFE		+	+	+	+
+ * _FOREACH_REVERSE		-	-	-	+
+ * _FOREACH_REVERSE_FROM	-	-	-	+
+ * _FOREACH_REVERSE_SAFE	-	-	-	+
+ * _FOREACH_REVERSE_FROM_SAFE	-	-	-	+
+ * _INSERT_HEAD			+	+	+	+
+ * _INSERT_BEFORE		-	+	-	+
+ * _INSERT_AFTER		+	+	+	+
+ * _INSERT_TAIL			-	-	+	+
+ * _CONCAT			s	s	+	+
+ * _REMOVE_AFTER		+	-	+	-
+ * _REMOVE_HEAD			+	-	+	-
+ * _REMOVE			s	+	s	+
+ * _SWAP			+	+	+	+
  *
  */
-
-#ifdef __cplusplus
-extern "C" {
+#ifdef QUEUE_MACRO_DEBUG
+#warn Use QUEUE_MACRO_DEBUG_TRACE and/or QUEUE_MACRO_DEBUG_TRASH
+#define	QUEUE_MACRO_DEBUG_TRACE
+#define	QUEUE_MACRO_DEBUG_TRASH
 #endif
 
-/*
- * List definitions.
- */
-#define	LIST_HEAD(name, type)						\
-struct name {								\
-	struct type *lh_first;	/* first element */			\
-}
+#ifdef QUEUE_MACRO_DEBUG_TRACE
+/* Store the last 2 places the queue element or head was altered */
+struct qm_trace {
+	unsigned long	 lastline;
+	unsigned long	 prevline;
+	const char	*lastfile;
+	const char	*prevfile;
+};
+
+#define	TRACEBUF	struct qm_trace trace;
+#define	TRACEBUF_INITIALIZER	{ __LINE__, 0, __FILE__, NULL } ,
+
+#define	QMD_TRACE_HEAD(head) do {					\
+	(head)->trace.prevline = (head)->trace.lastline;		\
+	(head)->trace.prevfile = (head)->trace.lastfile;		\
+	(head)->trace.lastline = __LINE__;				\
+	(head)->trace.lastfile = __FILE__;				\
+} while (0)
 
+#define	QMD_TRACE_ELEM(elem) do {					\
+	(elem)->trace.prevline = (elem)->trace.lastline;		\
+	(elem)->trace.prevfile = (elem)->trace.lastfile;		\
+	(elem)->trace.lastline = __LINE__;				\
+	(elem)->trace.lastfile = __FILE__;				\
+} while (0)
+
+#else	/* !QUEUE_MACRO_DEBUG_TRACE */
 #define	QMD_TRACE_ELEM(elem)
 #define	QMD_TRACE_HEAD(head)
 #define	TRACEBUF
 #define	TRACEBUF_INITIALIZER
+#endif	/* QUEUE_MACRO_DEBUG_TRACE */
 
+#ifdef QUEUE_MACRO_DEBUG_TRASH
+#define	QMD_SAVELINK(name, link)	void **name = (void *)&(link)
+#define	TRASHIT(x)		do {(x) = (void *)-1;} while (0)
+#define	QMD_IS_TRASHED(x)	((x) == (void *)(intptr_t)-1)
+#else	/* !QUEUE_MACRO_DEBUG_TRASH */
+#define	QMD_SAVELINK(name, link)
 #define	TRASHIT(x)
 #define	QMD_IS_TRASHED(x)	0
-
-#define	QMD_SAVELINK(name, link)
+#endif	/* QUEUE_MACRO_DEBUG_TRASH */
 
 #ifdef __cplusplus
 /*
@@ -86,6 +143,445 @@ struct name {								\
 #define	QUEUE_TYPEOF(type) struct type
 #endif
 
+/*
+ * Singly-linked List declarations.
+ */
+#define	SLIST_HEAD(name, type)						\
+struct name {								\
+	struct type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	SLIST_ENTRY(type)						\
+struct {								\
+	struct type *sle_next;	/* next element */			\
+}
+
+#define	SLIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *sle_next;		/* next element */		\
+}
+
+/*
+ * Singly-linked List functions.
+ */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm) do {			\
+	if (*(prevp) != (elm))						\
+		panic("Bad prevptr *(%p) == %p != %p",			\
+		    (prevp), *(prevp), (elm));				\
+} while (0)
+#else
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm)
+#endif
+
+#define SLIST_CONCAT(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head1);		\
+	if (curelm == NULL) {						\
+		if ((SLIST_FIRST(head1) = SLIST_FIRST(head2)) != NULL)	\
+			SLIST_INIT(head2);				\
+	} else if (SLIST_FIRST(head2) != NULL) {			\
+		while (SLIST_NEXT(curelm, field) != NULL)		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_NEXT(curelm, field) = SLIST_FIRST(head2);		\
+		SLIST_INIT(head2);					\
+	}								\
+} while (0)
+
+#define	SLIST_EMPTY(head)	((head)->slh_first == NULL)
+
+#define	SLIST_FIRST(head)	((head)->slh_first)
+
+#define	SLIST_FOREACH(var, head, field)					\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			\
+	for ((varp) = &SLIST_FIRST((head));				\
+	    ((var) = *(varp)) != NULL;					\
+	    (varp) = &SLIST_NEXT((var), field))
+
+#define	SLIST_INIT(head) do {						\
+	SLIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	SLIST_INSERT_AFTER(slistelm, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_NEXT((slistelm), field);	\
+	SLIST_NEXT((slistelm), field) = (elm);				\
+} while (0)
+
+#define	SLIST_INSERT_HEAD(head, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_FIRST((head));			\
+	SLIST_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)
+
+#define	SLIST_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.sle_next);			\
+	if (SLIST_FIRST((head)) == (elm)) {				\
+		SLIST_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head);		\
+		while (SLIST_NEXT(curelm, field) != (elm))		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_REMOVE_AFTER(curelm, field);			\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define SLIST_REMOVE_AFTER(elm, field) do {				\
+	SLIST_NEXT(elm, field) =					\
+	    SLIST_NEXT(SLIST_NEXT(elm, field), field);			\
+} while (0)
+
+#define	SLIST_REMOVE_HEAD(head, field) do {				\
+	SLIST_FIRST((head)) = SLIST_NEXT(SLIST_FIRST((head)), field);	\
+} while (0)
+
+#define	SLIST_REMOVE_PREVPTR(prevp, elm, field) do {			\
+	QMD_SLIST_CHECK_PREVPTR(prevp, elm);				\
+	*(prevp) = SLIST_NEXT(elm, field);				\
+	TRASHIT((elm)->field.sle_next);					\
+} while (0)
+
+#define SLIST_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = SLIST_FIRST(head1);		\
+	SLIST_FIRST(head1) = SLIST_FIRST(head2);			\
+	SLIST_FIRST(head2) = swap_first;				\
+} while (0)
+
+/*
+ * Singly-linked Tail queue declarations.
+ */
+#define	STAILQ_HEAD(name, type)						\
+struct name {								\
+	struct type *stqh_first;/* first element */			\
+	struct type **stqh_last;/* addr of last next element */		\
+}
+
+#define	STAILQ_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *stqh_first;	/* first element */			\
+	class type **stqh_last;	/* addr of last next element */		\
+}
+
+#define	STAILQ_HEAD_INITIALIZER(head)					\
+	{ NULL, &(head).stqh_first }
+
+#define	STAILQ_ENTRY(type)						\
+struct {								\
+	struct type *stqe_next;	/* next element */			\
+}
+
+#define	STAILQ_CLASS_ENTRY(type)					\
+struct {								\
+	class type *stqe_next;	/* next element */			\
+}
+
+/*
+ * Singly-linked Tail queue functions.
+ */
+#define	STAILQ_CONCAT(head1, head2) do {				\
+	if (!STAILQ_EMPTY((head2))) {					\
+		*(head1)->stqh_last = (head2)->stqh_first;		\
+		(head1)->stqh_last = (head2)->stqh_last;		\
+		STAILQ_INIT((head2));					\
+	}								\
+} while (0)
+
+#define	STAILQ_EMPTY(head)	((head)->stqh_first == NULL)
+
+#define	STAILQ_FIRST(head)	((head)->stqh_first)
+
+#define	STAILQ_FOREACH(var, head, field)				\
+	for((var) = STAILQ_FIRST((head));				\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = STAILQ_FIRST((head));				\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_FOREACH_FROM_SAFE(var, head, field, tvar)		\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_INIT(head) do {						\
+	STAILQ_FIRST((head)) = NULL;					\
+	(head)->stqh_last = &STAILQ_FIRST((head));			\
+} while (0)
+
+#define	STAILQ_INSERT_AFTER(head, tqelm, elm, field) do {		\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_NEXT((tqelm), field)) == NULL)\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_NEXT((tqelm), field) = (elm);				\
+} while (0)
+
+#define	STAILQ_INSERT_HEAD(head, elm, field) do {			\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_FIRST((head))) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	STAILQ_INSERT_TAIL(head, elm, field) do {			\
+	STAILQ_NEXT((elm), field) = NULL;				\
+	*(head)->stqh_last = (elm);					\
+	(head)->stqh_last = &STAILQ_NEXT((elm), field);			\
+} while (0)
+
+#define	STAILQ_LAST(head, type, field)				\
+	(STAILQ_EMPTY((head)) ? NULL :				\
+	    __containerof((head)->stqh_last,			\
+	    QUEUE_TYPEOF(type), field.stqe_next))
+
+#define	STAILQ_NEXT(elm, field)	((elm)->field.stqe_next)
+
+#define	STAILQ_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.stqe_next);			\
+	if (STAILQ_FIRST((head)) == (elm)) {				\
+		STAILQ_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = STAILQ_FIRST(head);	\
+		while (STAILQ_NEXT(curelm, field) != (elm))		\
+			curelm = STAILQ_NEXT(curelm, field);		\
+		STAILQ_REMOVE_AFTER(head, curelm, field);		\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define STAILQ_REMOVE_AFTER(head, elm, field) do {			\
+	if ((STAILQ_NEXT(elm, field) =					\
+	     STAILQ_NEXT(STAILQ_NEXT(elm, field), field)) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+} while (0)
+
+#define	STAILQ_REMOVE_HEAD(head, field) do {				\
+	if ((STAILQ_FIRST((head)) =					\
+	     STAILQ_NEXT(STAILQ_FIRST((head)), field)) == NULL)		\
+		(head)->stqh_last = &STAILQ_FIRST((head));		\
+} while (0)
+
+#define STAILQ_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = STAILQ_FIRST(head1);		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->stqh_last;		\
+	STAILQ_FIRST(head1) = STAILQ_FIRST(head2);			\
+	(head1)->stqh_last = (head2)->stqh_last;			\
+	STAILQ_FIRST(head2) = swap_first;				\
+	(head2)->stqh_last = swap_last;					\
+	if (STAILQ_EMPTY(head1))					\
+		(head1)->stqh_last = &STAILQ_FIRST(head1);		\
+	if (STAILQ_EMPTY(head2))					\
+		(head2)->stqh_last = &STAILQ_FIRST(head2);		\
+} while (0)
+
+
+/*
+ * List declarations.
+ */
+#define	LIST_HEAD(name, type)						\
+struct name {								\
+	struct type *lh_first;	/* first element */			\
+}
+
+#define	LIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *lh_first;	/* first element */			\
+}
+
+#define	LIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	LIST_ENTRY(type)						\
+struct {								\
+	struct type *le_next;	/* next element */			\
+	struct type **le_prev;	/* address of previous next element */	\
+}
+
+#define	LIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *le_next;	/* next element */			\
+	class type **le_prev;	/* address of previous next element */	\
+}
+
+/*
+ * List functions.
+ */
+
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_LIST_CHECK_HEAD(LIST_HEAD *head, LIST_ENTRY NAME)
+ *
+ * If the list is non-empty, validates that the first element of the list
+ * points back at 'head.'
+ */
+#define	QMD_LIST_CHECK_HEAD(head, field) do {				\
+	if (LIST_FIRST((head)) != NULL &&				\
+	    LIST_FIRST((head))->field.le_prev !=			\
+	     &LIST_FIRST((head)))					\
+		panic("Bad list head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_NEXT(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the list, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_LIST_CHECK_NEXT(elm, field) do {				\
+	if (LIST_NEXT((elm), field) != NULL &&				\
+	    LIST_NEXT((elm), field)->field.le_prev !=			\
+	     &((elm)->field.le_next))					\
+	     	panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_PREV(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the list) points to 'elm.'
+ */
+#define	QMD_LIST_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.le_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
+#define	QMD_LIST_CHECK_HEAD(head, field)
+#define	QMD_LIST_CHECK_NEXT(elm, field)
+#define	QMD_LIST_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
+
+#define LIST_CONCAT(head1, head2, type, field) do {			      \
+	QUEUE_TYPEOF(type) *curelm = LIST_FIRST(head1);			      \
+	if (curelm == NULL) {						      \
+		if ((LIST_FIRST(head1) = LIST_FIRST(head2)) != NULL) {	      \
+			LIST_FIRST(head2)->field.le_prev =		      \
+			    &LIST_FIRST((head1));			      \
+			LIST_INIT(head2);				      \
+		}							      \
+	} else if (LIST_FIRST(head2) != NULL) {				      \
+		while (LIST_NEXT(curelm, field) != NULL)		      \
+			curelm = LIST_NEXT(curelm, field);		      \
+		LIST_NEXT(curelm, field) = LIST_FIRST(head2);		      \
+		LIST_FIRST(head2)->field.le_prev = &LIST_NEXT(curelm, field); \
+		LIST_INIT(head2);					      \
+	}								      \
+} while (0)
+
+#define	LIST_EMPTY(head)	((head)->lh_first == NULL)
+
+#define	LIST_FIRST(head)	((head)->lh_first)
+
+#define	LIST_FOREACH(var, head, field)					\
+	for ((var) = LIST_FIRST((head));				\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = LIST_FIRST((head));				\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_INIT(head) do {						\
+	LIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	LIST_INSERT_AFTER(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_NEXT(listelm, field);				\
+	if ((LIST_NEXT((elm), field) = LIST_NEXT((listelm), field)) != NULL)\
+		LIST_NEXT((listelm), field)->field.le_prev =		\
+		    &LIST_NEXT((elm), field);				\
+	LIST_NEXT((listelm), field) = (elm);				\
+	(elm)->field.le_prev = &LIST_NEXT((listelm), field);		\
+} while (0)
+
+#define	LIST_INSERT_BEFORE(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_PREV(listelm, field);				\
+	(elm)->field.le_prev = (listelm)->field.le_prev;		\
+	LIST_NEXT((elm), field) = (listelm);				\
+	*(listelm)->field.le_prev = (elm);				\
+	(listelm)->field.le_prev = &LIST_NEXT((elm), field);		\
+} while (0)
+
+#define	LIST_INSERT_HEAD(head, elm, field) do {				\
+	QMD_LIST_CHECK_HEAD((head), field);				\
+	if ((LIST_NEXT((elm), field) = LIST_FIRST((head))) != NULL)	\
+		LIST_FIRST((head))->field.le_prev = &LIST_NEXT((elm), field);\
+	LIST_FIRST((head)) = (elm);					\
+	(elm)->field.le_prev = &LIST_FIRST((head));			\
+} while (0)
+
+#define	LIST_NEXT(elm, field)	((elm)->field.le_next)
+
+#define	LIST_PREV(elm, head, type, field)			\
+	((elm)->field.le_prev == &LIST_FIRST((head)) ? NULL :	\
+	    __containerof((elm)->field.le_prev,			\
+	    QUEUE_TYPEOF(type), field.le_next))
+
+#define	LIST_REMOVE(elm, field) do {					\
+	QMD_SAVELINK(oldnext, (elm)->field.le_next);			\
+	QMD_SAVELINK(oldprev, (elm)->field.le_prev);			\
+	QMD_LIST_CHECK_NEXT(elm, field);				\
+	QMD_LIST_CHECK_PREV(elm, field);				\
+	if (LIST_NEXT((elm), field) != NULL)				\
+		LIST_NEXT((elm), field)->field.le_prev = 		\
+		    (elm)->field.le_prev;				\
+	*(elm)->field.le_prev = LIST_NEXT((elm), field);		\
+	TRASHIT(*oldnext);						\
+	TRASHIT(*oldprev);						\
+} while (0)
+
+#define LIST_SWAP(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *swap_tmp = LIST_FIRST(head1);		\
+	LIST_FIRST((head1)) = LIST_FIRST((head2));			\
+	LIST_FIRST((head2)) = swap_tmp;					\
+	if ((swap_tmp = LIST_FIRST((head1))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head1));		\
+	if ((swap_tmp = LIST_FIRST((head2))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head2));		\
+} while (0)
+
 /*
  * Tail queue declarations.
  */
@@ -123,10 +619,58 @@ struct {								\
 /*
  * Tail queue functions.
  */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_TAILQ_CHECK_HEAD(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * If the tailq is non-empty, validates that the first element of the tailq
+ * points back at 'head.'
+ */
+#define	QMD_TAILQ_CHECK_HEAD(head, field) do {				\
+	if (!TAILQ_EMPTY(head) &&					\
+	    TAILQ_FIRST((head))->field.tqe_prev !=			\
+	     &TAILQ_FIRST((head)))					\
+		panic("Bad tailq head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_TAIL(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * Validates that the tail of the tailq is a pointer to pointer to NULL.
+ */
+#define	QMD_TAILQ_CHECK_TAIL(head, field) do {				\
+	if (*(head)->tqh_last != NULL)					\
+	    	panic("Bad tailq NEXT(%p->tqh_last) != NULL", (head)); 	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_NEXT(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the tailq, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_NEXT(elm, field) do {				\
+	if (TAILQ_NEXT((elm), field) != NULL &&				\
+	    TAILQ_NEXT((elm), field)->field.tqe_prev !=			\
+	     &((elm)->field.tqe_next))					\
+		panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_PREV(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the tailq) points to 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.tqe_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
 #define	QMD_TAILQ_CHECK_HEAD(head, field)
 #define	QMD_TAILQ_CHECK_TAIL(head, headname)
 #define	QMD_TAILQ_CHECK_NEXT(elm, field)
 #define	QMD_TAILQ_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
 
 #define	TAILQ_CONCAT(head1, head2, field) do {				\
 	if (!TAILQ_EMPTY(head2)) {					\
@@ -191,9 +735,8 @@ struct {								\
 
 #define	TAILQ_INSERT_AFTER(head, listelm, elm, field) do {		\
 	QMD_TAILQ_CHECK_NEXT(listelm, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field);	\
-	if (TAILQ_NEXT((listelm), field) != NULL)			\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    &TAILQ_NEXT((elm), field);				\
 	else {								\
 		(head)->tqh_last = &TAILQ_NEXT((elm), field);		\
@@ -217,8 +760,7 @@ struct {								\
 
 #define	TAILQ_INSERT_HEAD(head, elm, field) do {			\
 	QMD_TAILQ_CHECK_HEAD(head, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_FIRST((head));			\
-	if (TAILQ_FIRST((head)) != NULL)				\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL)	\
 		TAILQ_FIRST((head))->field.tqe_prev =			\
 		    &TAILQ_NEXT((elm), field);				\
 	else								\
@@ -250,21 +792,24 @@ struct {								\
  * you may want to prefetch the last data element.
  */
 #define	TAILQ_LAST_FAST(head, type, field)			\
-	(TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last,	\
-	QUEUE_TYPEOF(type), field.tqe_next))
+    (TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last, QUEUE_TYPEOF(type), field.tqe_next))
 
 #define	TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
 
 #define	TAILQ_PREV(elm, headname, field)				\
 	(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+#define	TAILQ_PREV_FAST(elm, head, type, field)				\
+    ((elm)->field.tqe_prev == &(head)->tqh_first ? NULL :		\
+     __containerof((elm)->field.tqe_prev, QUEUE_TYPEOF(type), field.tqe_next))
+
 #define	TAILQ_REMOVE(head, elm, field) do {				\
 	QMD_SAVELINK(oldnext, (elm)->field.tqe_next);			\
 	QMD_SAVELINK(oldprev, (elm)->field.tqe_prev);			\
 	QMD_TAILQ_CHECK_NEXT(elm, field);				\
 	QMD_TAILQ_CHECK_PREV(elm, field);				\
 	if ((TAILQ_NEXT((elm), field)) != NULL)				\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    (elm)->field.tqe_prev;				\
 	else {								\
 		(head)->tqh_last = (elm)->field.tqe_prev;		\
@@ -277,26 +822,20 @@ struct {								\
 } while (0)
 
 #define TAILQ_SWAP(head1, head2, type, field) do {			\
-	QUEUE_TYPEOF(type) * swap_first = (head1)->tqh_first;		\
-	QUEUE_TYPEOF(type) * *swap_last = (head1)->tqh_last;		\
+	QUEUE_TYPEOF(type) *swap_first = (head1)->tqh_first;		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->tqh_last;		\
 	(head1)->tqh_first = (head2)->tqh_first;			\
 	(head1)->tqh_last = (head2)->tqh_last;				\
 	(head2)->tqh_first = swap_first;				\
 	(head2)->tqh_last = swap_last;					\
-	swap_first = (head1)->tqh_first;				\
-	if (swap_first != NULL)						\
+	if ((swap_first = (head1)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head1)->tqh_first;	\
 	else								\
 		(head1)->tqh_last = &(head1)->tqh_first;		\
-	swap_first = (head2)->tqh_first;				\
-	if (swap_first != NULL)			\
+	if ((swap_first = (head2)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head2)->tqh_first;	\
 	else								\
 		(head2)->tqh_last = &(head2)->tqh_first;		\
 } while (0)
 
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _SYS_QUEUE_H_ */
+#endif /* !_SYS_QUEUE_H_ */
Signature · 1 line
2.25.4
#186 lore raw

PATCH v8 9/11 eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")
---
 lib/librte_eal/windows/eal_lcore.c   | 185 +++++++++++++++++----------
 lib/librte_eal/windows/eal_windows.h |  10 ++
 2 files changed, 124 insertions(+), 71 deletions(-)
lib/librte_eal/windows/eal_lcore.c +114 -71
index 82ee45413..9d931d50a 100644
@@ -3,103 +3,146 @@
  */
 
 #include <pthread.h>
+#include <stdbool.h>
 #include <stdint.h>
 
 #include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_lcore.h>
+#include <rte_os.h>
 
 #include "eal_private.h"
 #include "eal_thread.h"
 #include "eal_windows.h"
 
-/* global data structure that contains the CPU map */
-static struct _wcpu_map {
-	unsigned int total_procs;
-	unsigned int proc_sockets;
-	unsigned int proc_cores;
-	unsigned int reserved;
-	struct _win_lcore_map {
-		uint8_t socket_id;
-		uint8_t core_id;
-	} wlcore_map[RTE_MAX_LCORE];
-} wcpu_map = { 0 };
-
-/*
- * Create a map of all processors and associated cores on the system
- */
+/** Number of logical processors (cores) in a processor group (32 or 64). */
+#define EAL_PROCESSOR_GROUP_SIZE (sizeof(KAFFINITY) * CHAR_BIT)
+
+struct lcore_map {
+	uint8_t socket_id;
+	uint8_t core_id;
+};
+
+struct socket_map {
+	uint16_t node_id;
+};
+
+struct cpu_map {
+	unsigned int socket_count;
+	unsigned int lcore_count;
+	struct lcore_map lcores[RTE_MAX_LCORE];
+	struct socket_map sockets[RTE_MAX_NUMA_NODES];
+};
+
+static struct cpu_map cpu_map = { 0 };
+
 void
-eal_create_cpu_map()
+eal_create_cpu_map(void)
 {
-	wcpu_map.total_procs =
-		GetActiveProcessorCount(ALL_PROCESSOR_GROUPS);
-
-	LOGICAL_PROCESSOR_RELATIONSHIP lprocRel;
-	DWORD lprocInfoSize = 0;
-	BOOL ht_enabled = FALSE;
-
-	/* First get the processor package information */
-	lprocRel = RelationProcessorPackage;
-	/* Determine the size of buffer we need (pass NULL) */
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_sockets = lprocInfoSize / 48;
-
-	lprocInfoSize = 0;
-	/* Next get the processor core information */
-	lprocRel = RelationProcessorCore;
-	GetLogicalProcessorInformationEx(lprocRel, NULL, &lprocInfoSize);
-	wcpu_map.proc_cores = lprocInfoSize / 48;
-
-	if (wcpu_map.total_procs > wcpu_map.proc_cores)
-		ht_enabled = TRUE;
-
-	/* Distribute the socket and core ids appropriately
-	 * across the logical cores. For now, split the cores
-	 * equally across the sockets.
-	 */
-	unsigned int lcore = 0;
-	for (unsigned int socket = 0; socket <
-			wcpu_map.proc_sockets; ++socket) {
-		for (unsigned int core = 0;
-			core < (wcpu_map.proc_cores / wcpu_map.proc_sockets);
-			++core) {
-			wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-			wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-			lcore++;
-			if (ht_enabled) {
-				wcpu_map.wlcore_map[lcore]
-					.socket_id = socket;
-				wcpu_map.wlcore_map[lcore]
-					.core_id = core;
-				lcore++;
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
+	DWORD infos_size;
+	bool full = false;
+
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			rte_panic("cannot get NUMA node info size, error %lu",
+				GetLastError());
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		rte_panic("cannot allocate memory for NUMA node information");
+		return;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		rte_panic("cannot get NUMA node information, error %lu",
+			GetLastError());
+	}
+
+	info = infos;
+	while ((uint8_t *)info - (uint8_t *)infos < infos_size) {
+		unsigned int node_id = info->NumaNode.NodeNumber;
+		GROUP_AFFINITY *cores = &info->NumaNode.GroupMask;
+		struct lcore_map *lcore;
+		unsigned int i, socket_id;
+
+		/* NUMA node may be reported multiple times if it includes
+		 * cores from different processor groups, e. g. 80 cores
+		 * of a physical processor comprise one NUMA node, but two
+		 * processor groups, because group size is limited by 32/64.
+		 */
+		for (socket_id = 0; socket_id < cpu_map.socket_count;
+		    socket_id++) {
+			if (cpu_map.sockets[socket_id].node_id == node_id)
+				break;
+		}
+
+		if (socket_id == cpu_map.socket_count) {
+			if (socket_id == RTE_DIM(cpu_map.sockets)) {
+				full = true;
+				goto exit;
 			}
+
+			cpu_map.sockets[socket_id].node_id = node_id;
+			cpu_map.socket_count++;
+		}
+
+		for (i = 0; i < EAL_PROCESSOR_GROUP_SIZE; i++) {
+			if ((cores->Mask & ((KAFFINITY)1 << i)) == 0)
+				continue;
+
+			if (cpu_map.lcore_count == RTE_DIM(cpu_map.lcores)) {
+				full = true;
+				goto exit;
+			}
+
+			lcore = &cpu_map.lcores[cpu_map.lcore_count];
+			lcore->socket_id = socket_id;
+			lcore->core_id =
+				cores->Group * EAL_PROCESSOR_GROUP_SIZE + i;
+			cpu_map.lcore_count++;
 		}
+
+		info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *)(
+			(uint8_t *)info + info->Size);
+	}
+
+exit:
+	if (full) {
+		/* RTE_LOG() may not be available, but this is important. */
+		fprintf(stderr, "Enumerated maximum of %u NUMA nodes and %u cores\n",
+			cpu_map.socket_count, cpu_map.lcore_count);
 	}
+
+	free(infos);
 }
 
-/*
- * Check if a cpu is present by the presence of the cpu information for it
- */
 int
 eal_cpu_detected(unsigned int lcore_id)
 {
-	return (lcore_id < wcpu_map.total_procs);
+	return lcore_id < cpu_map.lcore_count;
 }
 
-/*
- * Get CPU socket id for a logical core
- */
 unsigned
 eal_cpu_socket_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].socket_id;
+	return cpu_map.lcores[lcore_id].socket_id;
 }
 
-/*
- * Get CPU socket id (NUMA node) for a logical core
- */
 unsigned
 eal_cpu_core_id(unsigned int lcore_id)
 {
-	return wcpu_map.wlcore_map[lcore_id].core_id;
+	return cpu_map.lcores[lcore_id].core_id;
+}
+
+unsigned int
+eal_socket_numa_node(unsigned int socket_id)
+{
+	return cpu_map.sockets[socket_id].node_id;
 }
lib/librte_eal/windows/eal_windows.h +10 -0
index fadd676b2..390d2fd66 100644
@@ -26,4 +26,14 @@ void eal_create_cpu_map(void);
  */
 int eal_thread_create(pthread_t *thread);
 
+/**
+ * Get system NUMA node number for a socket ID.
+ *
+ * @param socket_id
+ *  Valid EAL socket ID.
+ * @return
+ *  NUMA node number to use with Win32 API.
+ */
+unsigned int eal_socket_numa_node(unsigned int socket_id);
+
 #endif /* _EAL_WINDOWS_H_ */
Signature · 1 line
2.25.4
#187 lore raw

Re: [dpdk-dev] [PATCH v8 09/11] eal/windows: improve CPU and NUMA node detection

10/06/2020 16:27, Dmitry Kozlyuk:

Quoted 6 lines

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")

Not sure we should consider it as a fix. This tag is usually required for backporting, but I don't see any need for backport here. If you think backport is required, you should add
Cc: [email protected].

#188 lore raw

Re: [dpdk-dev] [PATCH v8 09/11] eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

10/06/2020 16:27, Dmitry Kozlyuk:

Quoted 25 lines

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

Fixes 53ffd9f080fc ("eal/windows: add minimum viable code")

---

lib/librte_eal/windows/eal_lcore.c +16 -0
+eal_create_cpu_map(void)
+	SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX *infos, *info;
+	DWORD infos_size;
+	bool full = false;
+
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			rte_panic("cannot get NUMA node info size, error %lu",
+				GetLastError());
+		}
+	}
+
+	infos = malloc(infos_size);

lib/librte_eal/windows/eal_lcore.c line 66

Quoted 10 lines
lib/librte_eal/windows/eal_lcore.c +10 -0
+	if (infos == NULL) {
+		rte_panic("cannot allocate memory for NUMA node information");
+		return;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		rte_panic("cannot get NUMA node information, error %lu",
+			GetLastError());
+	}

rte_panic addition is forbidden in the libraries. An application may want to manage the error and shutdown the DPDK part gracefully.
Please can you try to return an error to rte_eal_init()?

#189 lore raw

PATCH v8 10/11 eal/windows: initialize hugepage info

In reply to Dmitry Kozlyuk

Add hugepages discovery ("large pages" in Windows terminology) and update documentation for required privilege setup. Only 2MB hugepages are supported and their number is estimated roughly due to the lack or unstable status of suitable OS APIs. Assign myself as maintainer for the implementation file.

---
 MAINTAINERS                            |   4 +
 config/meson.build                     |   2 +
 doc/guides/windows_gsg/build_dpdk.rst  |  20 -----
 doc/guides/windows_gsg/index.rst       |   1 +
 doc/guides/windows_gsg/run_apps.rst    |  47 +++++++++++
 lib/librte_eal/windows/eal.c           |  14 ++++
 lib/librte_eal/windows/eal_hugepages.c | 108 +++++++++++++++++++++++++
 lib/librte_eal/windows/meson.build     |   1 +
 8 files changed, 177 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
MAINTAINERS +4 -0
index a1722ca73..19b818f69 100644
@@ -336,6 +336,10 @@ F: lib/librte_eal/windows/
 F: lib/librte_eal/rte_eal_exports.def
 F: doc/guides/windows_gsg/
 
+Windows memory allocation
+M: Dmitry Kozlyuk <[email protected]>
+F: lib/librte_eal/eal_hugepages.c
+
 
 Core Libraries
 --------------
config/meson.build +2 -0
index 43ab11310..c1e80de4b 100644
@@ -268,6 +268,8 @@ if is_windows
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
+
+	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/build_dpdk.rst +0 -20
index d46e84e3f..650483e3b 100644
@@ -111,23 +111,3 @@ Depending on the distribution, paths in this file may need adjustments.
 
     meson --cross-file config/x86/meson_mingw.txt -Dexamples=helloworld build
     ninja -C build
-
-
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).
doc/guides/windows_gsg/index.rst +1 -0
index d9b7990a8..e94593572 100644
@@ -12,3 +12,4 @@ Getting Started Guide for Windows
 
     intro
     build_dpdk
+    run_apps
doc/guides/windows_gsg/run_apps.rst new file +47 -0

47 lines not on this page · raw message

lib/librte_eal/windows/eal.c +14 -0
index e7461f731..7c2fcc860 100644
@@ -19,8 +19,11 @@
 #include <eal_private.h>
 #include <rte_trace_point.h>
 
+#include "eal_hugepages.h"
 #include "eal_windows.h"
 
+#define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
@@ -276,6 +279,17 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
+		rte_eal_init_alert("Cannot get hugepage information");
+		rte_errno = EACCES;
+		return -1;
+	}
+
+	if (internal_config.memory == 0 && !internal_config.force_sockets) {
+		if (internal_config.no_hugetlbfs)
+			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_hugepages.c new file +108 -0

108 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index adfc8b9b7..52978e9d7 100644
@@ -6,6 +6,7 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_log.c',
 	'eal_thread.c',
Signature · 1 line
2.25.4
#190 lore raw

Re: [dpdk-dev] [PATCH v8 10/11] eal/windows: initialize hugepage info

10/06/2020 16:27, Dmitry Kozlyuk:

MAINTAINERS F: lib/librte_eal/windows/ · line 341

Quoted 5 lines
MAINTAINERS +3 -0
+Windows memory allocation
+M: Dmitry Kozlyuk <[email protected]>
+F: lib/librte_eal/eal_hugepages.c

Is it a typo? You mean lib/librte_eal/windows/eal_hugepages.c ?

[...]

Quoted 20 lines
doc/guides/windows_gsg/build_dpdk.rst +0 -18
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).

[...]

Quoted 10 lines
doc/guides/windows_gsg/run_apps.rst new file +8 -0
+Run the ``helloworld`` Example
+------------------------------
+
+Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
+
+.. code-block:: console
+
+    cd C:\Users\me\dpdk\build\examples

doc/guides/windows_gsg/run_apps.rst line 47

Quoted 10 lines
doc/guides/windows_gsg/run_apps.rst new file +10 -0
+    dpdk-helloworld.exe
+    hello from core 1
+    hello from core 3
+    hello from core 0
+    hello from core 2
+
+Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
+by default. To run the example, either add toolchain executables directory
+to the PATH or copy the library to the working directory.
+Alternatively, static linking may be used (mind the LGPLv2.1 license).

I tend thinking that such move is better understood in a separate prior patch.

#191 lore raw

PATCH v8 11/11 eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode. Multi-process mode is not implemented and is forcefully disabled at startup. Assign myself as a maintainer for Windows file and memory management implementation.

---
 MAINTAINERS                                   |   1 +
 config/meson.build                            |  12 +-
 doc/guides/windows_gsg/run_apps.rst           |  54 +-
 lib/librte_eal/common/meson.build             |  11 +
 lib/librte_eal/common/rte_malloc.c            |   1 +
 lib/librte_eal/rte_eal_exports.def            | 119 +++
 lib/librte_eal/windows/eal.c                  |  63 +-
 lib/librte_eal/windows/eal_file.c             | 125 +++
 lib/librte_eal/windows/eal_memalloc.c         | 441 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 710 ++++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  75 ++
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |  17 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   6 +
 18 files changed, 1771 insertions(+), 7 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_file.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
MAINTAINERS +1 -0
index 19b818f69..5140756b3 100644
@@ -339,6 +339,7 @@ F: doc/guides/windows_gsg/
 Windows memory allocation
 M: Dmitry Kozlyuk <[email protected]>
 F: lib/librte_eal/eal_hugepages.c
+F: lib/librte_eal/eal_mem*
 
 
 Core Libraries
config/meson.build +9 -3
index c1e80de4b..d3f05f878 100644
@@ -261,15 +261,21 @@ if is_freebsd
 endif
 
 if is_windows
-	# Minimum supported API is Windows 7.
-	add_project_arguments('-D_WIN32_WINNT=0x0601', language: 'c')
+	# VirtualAlloc2() is available since Windows 10 / Server 2016.
+	add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'c')
 
 	# Use MinGW-w64 stdio, because DPDK assumes ANSI-compliant formatting.
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	# Contrary to docs, VirtualAlloc2() is exported by mincore.lib
+	# in Windows SDK, while MinGW exports it by advapi32.a.
+	if is_ms_linker
+		add_project_link_arguments('-lmincore', language: 'c')
+	endif
+
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +51 -3

68 lines not on this page · raw message

lib/librte_eal/common/meson.build +11 -0
index 4e9208129..310844269 100644
@@ -8,13 +8,24 @@ if is_windows
 		'eal_common_bus.c',
 		'eal_common_class.c',
 		'eal_common_devargs.c',
+		'eal_common_dynmem.c',
 		'eal_common_errno.c',
+		'eal_common_fbarray.c',
 		'eal_common_launch.c',
 		'eal_common_lcore.c',
 		'eal_common_log.c',
+		'eal_common_mcfg.c',
+		'eal_common_memalloc.c',
+		'eal_common_memory.c',
+		'eal_common_memzone.c',
 		'eal_common_options.c',
+		'eal_common_string_fns.c',
+		'eal_common_tailqs.c',
 		'eal_common_thread.c',
 		'eal_common_trace_points.c',
+		'malloc_elem.c',
+		'malloc_heap.c',
+		'rte_malloc.c',
 	)
 	subdir_done()
 endif
lib/librte_eal/common/rte_malloc.c +1 -0
index f1b73168b..9d39e58c0 100644
@@ -20,6 +20,7 @@
 #include <rte_lcore.h>
 #include <rte_common.h>
 #include <rte_spinlock.h>
+
 #include <rte_eal_trace.h>
 
 #include <rte_malloc.h>
lib/librte_eal/rte_eal_exports.def +119 -0

128 lines not on this page · raw message

lib/librte_eal/windows/eal.c +62 -1

87 lines not on this page · raw message

lib/librte_eal/windows/eal_file.c new file +125 -0

125 lines not on this page · raw message

lib/librte_eal/windows/eal_memalloc.c new file +441 -0

441 lines not on this page · raw message

lib/librte_eal/windows/eal_memory.c new file +710 -0

710 lines not on this page · raw message

lib/librte_eal/windows/eal_mp.c new file +103 -0

103 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +75 -0

87 lines not on this page · raw message

lib/librte_eal/windows/include/meson.build +1 -0
index 5fb1962ac..b3534b025 100644
@@ -5,5 +5,6 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_virt2phys.h',
         'rte_windows.h',
 )
lib/librte_eal/windows/include/rte_os.h +17 -0
index 510e39e03..cb10d6494 100644
@@ -14,6 +14,7 @@
 #include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
+#include <string.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -36,6 +37,9 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
@@ -46,6 +50,7 @@ extern "C" {
 typedef long long ssize_t;
 
 #ifndef RTE_TOOLCHAIN_GCC
+
 static inline int
 asprintf(char **buffer, const char *format, ...)
 {
@@ -72,6 +77,18 @@ asprintf(char **buffer, const char *format, ...)
 	}
 	return ret;
 }
+
+static inline const char *
+eal_strerror(int code)
+{
+	static char buffer[128];
+
+	strerror_s(buffer, sizeof(buffer), code);
+	return buffer;
+}
+
+#define strerror eal_strerror
+
 #endif /* RTE_TOOLCHAIN_GCC */
 
 #ifdef __cplusplus
lib/librte_eal/windows/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/meson.build +6 -0
index 52978e9d7..ded5a2b80 100644
@@ -6,10 +6,16 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_file.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_log.c',
+	'eal_memalloc.c',
+	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'fnmatch.c',
 	'getopt.c',
 )
+
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
Signature · 1 line
2.25.4
#192 lore raw

Re: [dpdk-dev] [PATCH v8 11/11] eal/windows: implement basic memory management

2 typos below found with checkpatch.

10/06/2020 16:27, Dmitry Kozlyuk:

lib/librte_eal/windows/eal_memory.c line 254

Quoted 1 line
lib/librte_eal/windows/eal_memory.c new file +1 -0
+		/* May occcur when committing regular memory. */

Typo: occcur

lib/librte_eal/windows/eal_memory.c line 419

Quoted 1 line
lib/librte_eal/windows/eal_memory.c new file +1 -0
+	/* No need to join reserved regions adjascent to the freed one:

Typo: adjascent

#194 lore raw

11/06/2020 19:29, Thomas Monjalon:

Quoted 5 lines

10/06/2020 16:27, Dmitry Kozlyuk:

Quoted 1 line

This patchset implements basic MM with the following features:

There are some compilation issues on FreeBSD and 32-bit Linux:
http://mails.dpdk.org/archives/test-report/2020-June/135764.html

I did more comments about typos, naming, patch splitting, etc.

As soon as these comments are addressed in a v9, I think I can merge. I did not see any public formal approval, but there is no objection, so it looks good to go, and there are a lot of patches in the backlog which depend on this series.

Thanks for all your work Dmitry.

#195 lore raw

v9 cover Windows basic memory management

In reply to Dmitry Kozlyuk

Note for v9:
rte_eal_memory.h renamed, dependent patchsets have to be updated.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing without IOVA is available.

Testing revealed Windows Server 2019 does not allow allocating hugepage memory at a reserved address, despite advertised API. So allocator has to temporary free the region to be allocated. This creates in inherent race condition. This issue is being discussed with Microsoft privately.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications: rte_mem_map, rte_mem_unmap, rte_mem_lock, rte_mem_page_size.

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

To avoid code duplication between Linux and Windows EAL, common code for EALs supporting dynamic memory allocation is extracted (discussed with Anatoly Burakov in v4 thread). This is a separate patch to ease the review, but it can be merged with the previous one.

EAL tracepoints save size_t values as long, which is invalid on Windows. New size_t emitter for tracepoints is introduced (suggested by Jerin Jacob to Fady Bader, see [1]). Also, to avoid workaround in every file using the tracepoints, stubs are added to Windows EAL.

Entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file. The file is imported as-is, so it causes a bunch of checkpatch warnings.

[1]: http://mails.dpdk.org/archives/dev/2020-May/168076.html

---

v9:
* Fix build on 32-bit and FreeBSD.
* Rename rte_eal_memory.h to rte_eal_paging.h.
* Do not use rte_panic() in library code.
* Fix typos, comments, string formatting.
* Split documentation commits.

v8:
* Log eal_memseg_list_alloc() failure at caller sites (Anatoly Burakov).

v7:
* Change EAL internal file management API (Neil Horman).

v6:
* Fix 32-bit build on x86 (CI).
* Fix Makefile build (Anatoly Burakov, Thomas Monjalon).
* Restore 32-bit common code (Anatoly Burakov).
* Fix error reporting in memory management (Anatoly Burakov).
* Add Doxygen comment for size_t tracepoint emitter (Jerin Jacob).
* Update MAINTAINERS for new files and new code (Thomas Monjalon).
* Rename rte_get_page_size to rte_mem_page_size.
* Mark DPDK-only wrappers internal, move them to separate file.
* Get rid of warnings in enabled common code with Clang on Windows.

v5:
* Fix allocation and deallocation on Windows Server (Fady Bader).
* Replace remaining VirtualFree with VirtualFreeEx (Ranjit Menon).
* Fix errors in eal_get_virtual_area (Anatoly Burakov).
* Fix error handling and documentation for rte_mem_lock (Anatoly Burakov).
* Extract common code for EALs w/dynamic allocation (Anatoly Burakov).
* Use POSIX value for rte_errno after rte_mem_unmap() on Windows.
* Add stubs to use tracing functions without workarounds.

v4:
* Rebase on ToT, drop patches merged into master.
* Rearrange patches to split Windows code (Jerin).
* Fix Linux and FreeBSD build with make (Ophir).
* Use int instead of enum to hold a set of flags (Anatoly).
* Rename eal_mem_reserve items and fix their description (Anatoly).
* Add eal_mem_set_dump() wrapper around madvise (Anatoly).
* Don't claim Windows Server 2016 support due to lack of API (Tal).
* Replace enum rte_page_sizes with a set of #defines (Jerin).
* Fix documentation, SPDX tags, logging (Thomas).

v3:
* Fix Linux build on and aarch64 and 32-bit x86 (reported by CI).
* Fix logic and error handling while allocating segments.
* Fix Unix rte_mem_map(): return NULL on failure.
* Fix some checkpatch.sh issues:
* Do not return positive errno, use DWORD for GetLastError().
* Make dpdk-kmods source files non-executable.
* Improve GSG for Windows Server (suggested by Ranjit Menon).

v2:
* Rebase on ToT. Move all new code shared between Linux and FreeBSD to /unix/ subdirectory, also factor out some existing code there.
* Improve description of Clang issue with rte_page_sizes on Windows. Restore -fstrict-enum for EAL. Check running, not target compiler.
* Use EAL prefix for private facilities instead if RTE.
* Improve documentation comments for new functions.
* Remove co-installer for virt2phys. Add a typecast for clarity.
* Document virt2phys in user guide, improve its own README.
* Explicitly and forcefully disable multi-process.

Dmitry Kozlyuk (12):
eal: replace rte_page_sizes with a set of constants
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers
eal/mem: extract common code for memseg list initialization eal/mem: extract common code for dynamic memory allocation
trace: add size_t field emitter
eal/windows: add tracing support stubs
eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: improve CPU and NUMA node detection doc/windows: split build and run instructions eal/windows: initialize hugepage info
eal/windows: implement basic memory management

 MAINTAINERS                                   |   7 +
 config/meson.build                            |  12 +-
 doc/guides/rel_notes/release_20_08.rst        |   2 +
 doc/guides/windows_gsg/build_dpdk.rst         |  20 -
 doc/guides/windows_gsg/index.rst              |   1 +
 doc/guides/windows_gsg/run_apps.rst           |  95 +++
 lib/librte_eal/common/eal_common_dynmem.c     | 521 +++++++++++++
 lib/librte_eal/common/eal_common_fbarray.c    |  71 +-
 lib/librte_eal/common/eal_common_memory.c     | 157 +++-
 lib/librte_eal/common/eal_common_thread.c     |   5 +-
 lib/librte_eal/common/eal_private.h           | 254 ++++++-
 lib/librte_eal/common/meson.build             |  16 +
 lib/librte_eal/common/rte_malloc.c            |   1 +
 lib/librte_eal/freebsd/Makefile               |   5 +
 lib/librte_eal/freebsd/eal_memory.c           | 102 +--
 lib/librte_eal/include/rte_eal_paging.h       |  98 +++
 lib/librte_eal/include/rte_eal_trace.h        |   8 +-
 lib/librte_eal/include/rte_memory.h           |  23 +-
 lib/librte_eal/include/rte_trace_point.h      |   3 +
 lib/librte_eal/linux/Makefile                 |   6 +
 lib/librte_eal/linux/eal_memalloc.c           |   5 +-
 lib/librte_eal/linux/eal_memory.c             | 617 +--------------
 lib/librte_eal/meson.build                    |   4 +
 lib/librte_eal/rte_eal_exports.def            | 119 +++
 lib/librte_eal/rte_eal_version.map            |   9 +
 lib/librte_eal/unix/eal_file.c                |  80 ++
 lib/librte_eal/unix/eal_unix_memory.c         | 152 ++++
 lib/librte_eal/unix/meson.build               |   7 +
 lib/librte_eal/windows/eal.c                  | 114 ++-
 lib/librte_eal/windows/eal_file.c             | 125 +++
 lib/librte_eal/windows/eal_hugepages.c        | 108 +++
 lib/librte_eal/windows/eal_lcore.c            | 205 +++--
 lib/librte_eal/windows/eal_memalloc.c         | 441 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 710 ++++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  90 ++-
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |  17 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/sys/queue.h    | 663 ++++++++++++++--
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   7 +
 lib/librte_mempool/rte_mempool_trace.h        |  10 +-
 44 files changed, 4088 insertions(+), 945 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
 create mode 100644 lib/librte_eal/common/eal_common_dynmem.c
 create mode 100644 lib/librte_eal/include/rte_eal_paging.h
 create mode 100644 lib/librte_eal/unix/eal_file.c
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
 create mode 100644 lib/librte_eal/unix/meson.build
 create mode 100644 lib/librte_eal/windows/eal_file.c
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
Signature · 1 line
2.25.4
#196 lore raw

PATCH v9 1/12 eal: replace rte_page_sizes with a set of constants

Clang on Windows follows MS ABI where enum values are limited to 2^31-1. Enum rte_page_sizes has members valued above this limit, which get wrapped to zero, resulting in compilation error (duplicate values in enum). Using MS ABI is mandatory for Windows EAL to call Win32 APIs.

Remove rte_page_sizes and replace its values with #define's. This enumeration is not used in public API, so there's no ABI breakage. Announce API changes for 20.08 in documentation.

---
 doc/guides/rel_notes/release_20_08.rst |  2 ++
 lib/librte_eal/include/rte_memory.h    | 23 ++++++++++-------------
 2 files changed, 12 insertions(+), 13 deletions(-)
doc/guides/rel_notes/release_20_08.rst +2 -0
index dee4ccbb5..86d240213 100644
@@ -91,6 +91,8 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ``rte_page_sizes`` enumeration is replaced with ``RTE_PGSIZE_xxx`` defines.
+
 
 ABI Changes
 -----------
lib/librte_eal/include/rte_memory.h +10 -13
index 3d8d0bd69..65374d53a 100644
@@ -24,19 +24,16 @@ extern "C" {
 #include <rte_config.h>
 #include <rte_fbarray.h>
 
-__extension__
-enum rte_page_sizes {
-	RTE_PGSIZE_4K    = 1ULL << 12,
-	RTE_PGSIZE_64K   = 1ULL << 16,
-	RTE_PGSIZE_256K  = 1ULL << 18,
-	RTE_PGSIZE_2M    = 1ULL << 21,
-	RTE_PGSIZE_16M   = 1ULL << 24,
-	RTE_PGSIZE_256M  = 1ULL << 28,
-	RTE_PGSIZE_512M  = 1ULL << 29,
-	RTE_PGSIZE_1G    = 1ULL << 30,
-	RTE_PGSIZE_4G    = 1ULL << 32,
-	RTE_PGSIZE_16G   = 1ULL << 34,
-};
+#define RTE_PGSIZE_4K   (1ULL << 12)
+#define RTE_PGSIZE_64K  (1ULL << 16)
+#define RTE_PGSIZE_256K (1ULL << 18)
+#define RTE_PGSIZE_2M   (1ULL << 21)
+#define RTE_PGSIZE_16M  (1ULL << 24)
+#define RTE_PGSIZE_256M (1ULL << 28)
+#define RTE_PGSIZE_512M (1ULL << 29)
+#define RTE_PGSIZE_1G   (1ULL << 30)
+#define RTE_PGSIZE_4G   (1ULL << 32)
+#define RTE_PGSIZE_16G  (1ULL << 34)
 
 #define SOCKET_ID_ANY -1                    /**< Any NUMA socket. */
 
Signature · 1 line
2.25.4
#197 lore raw

PATCH v9 2/12 eal: introduce internal wrappers for file operations

In reply to Dmitry Kozlyuk

Introduce OS-independent wrappers in order to support common EAL code on Unix and Windows:

* eal_file_open: open or create a file.
* eal_file_lock: lock or unlock an open file.
* eal_file_truncate: enforce a given size for an open file.

Implementation for Linux and FreeBSD is placed in "unix" subdirectory, which is intended for common code between the two. These thin wrappers require no special maintenance.

Common code supporting multi-process doesn't use the new wrappers, because it is inherently Unix-specific and would impose excessive requirements on the wrappers.

---
 MAINTAINERS                                |  1 +
 lib/librte_eal/common/eal_common_fbarray.c | 31 ++++-----
 lib/librte_eal/common/eal_private.h        | 73 ++++++++++++++++++++
 lib/librte_eal/freebsd/Makefile            |  4 ++
 lib/librte_eal/linux/Makefile              |  4 ++
 lib/librte_eal/meson.build                 |  4 ++
 lib/librte_eal/unix/eal_file.c             | 80 ++++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |  6 ++
 8 files changed, 184 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_eal/unix/eal_file.c
 create mode 100644 lib/librte_eal/unix/meson.build
MAINTAINERS +1 -0
index e739b87ea..4d162efd6 100644
@@ -170,6 +170,7 @@ EAL API and common code
 F: lib/librte_eal/common/
 F: lib/librte_eal/include/
 F: lib/librte_eal/rte_eal_version.map
+F: lib/librte_eal/unix/
 F: doc/guides/prog_guide/env_abstraction_layer.rst
 F: app/test/test_alarm.c
 F: app/test/test_atomic.c
lib/librte_eal/common/eal_common_fbarray.c +12 -19
index 4f8f1af73..c52ddb967 100644
@@ -8,8 +8,8 @@
 #include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
-#include <sys/file.h>
 #include <string.h>
+#include <unistd.h>
 
 #include <rte_common.h>
 #include <rte_log.h>
@@ -85,10 +85,8 @@ resize_and_map(int fd, void *addr, size_t len)
 	char path[PATH_MAX];
 	void *map_addr;
 
-	if (ftruncate(fd, len)) {
+	if (eal_file_truncate(fd, len)) {
 		RTE_LOG(ERR, EAL, "Cannot truncate %s\n", path);
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 
@@ -772,15 +770,15 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		 * and see if we succeed. If we don't, someone else is using it
 		 * already.
 		 */
-		fd = open(path, O_CREAT | O_RDWR, 0600);
+		fd = eal_file_open(path, EAL_OPEN_CREATE | EAL_OPEN_READWRITE);
 		if (fd < 0) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't open %s: %s\n",
-					__func__, path, strerror(errno));
-			rte_errno = errno;
+				__func__, path, rte_strerror(rte_errno));
 			goto fail;
-		} else if (flock(fd, LOCK_EX | LOCK_NB)) {
+		} else if (eal_file_lock(
+				fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't lock %s: %s\n",
-					__func__, path, strerror(errno));
+				__func__, path, rte_strerror(rte_errno));
 			rte_errno = EBUSY;
 			goto fail;
 		}
@@ -789,10 +787,8 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		 * still attach to it, but no other process could reinitialize
 		 * it.
 		 */
-		if (flock(fd, LOCK_SH | LOCK_NB)) {
-			rte_errno = errno;
+		if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 			goto fail;
-		}
 
 		if (resize_and_map(fd, data, mmap_len))
 			goto fail;
@@ -888,17 +884,14 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 
 	eal_get_fbarray_path(path, sizeof(path), arr->name);
 
-	fd = open(path, O_RDWR);
+	fd = eal_file_open(path, EAL_OPEN_READWRITE);
 	if (fd < 0) {
-		rte_errno = errno;
 		goto fail;
 	}
 
 	/* lock the file, to let others know we're using it */
-	if (flock(fd, LOCK_SH | LOCK_NB)) {
-		rte_errno = errno;
+	if (eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN))
 		goto fail;
-	}
 
 	if (resize_and_map(fd, data, mmap_len))
 		goto fail;
@@ -1025,7 +1018,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		 * has been detached by all other processes
 		 */
 		fd = tmp->fd;
-		if (flock(fd, LOCK_EX | LOCK_NB)) {
+		if (eal_file_lock(fd, EAL_FLOCK_EXCLUSIVE, EAL_FLOCK_RETURN)) {
 			RTE_LOG(DEBUG, EAL, "Cannot destroy fbarray - another process is using it\n");
 			rte_errno = EBUSY;
 			ret = -1;
@@ -1042,7 +1035,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 			 * we're still holding an exclusive lock, so drop it to
 			 * shared.
 			 */
-			flock(fd, LOCK_SH | LOCK_NB);
+			eal_file_lock(fd, EAL_FLOCK_SHARED, EAL_FLOCK_RETURN);
 
 			ret = -1;
 			goto out;
lib/librte_eal/common/eal_private.h +73 -0

77 lines not on this page · raw message

lib/librte_eal/freebsd/Makefile +4 -0
index af95386d4..0f8741d96 100644
@@ -7,6 +7,7 @@ LIB = librte_eal.a
 
 ARCH_DIR ?= $(RTE_ARCH)
 VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/unix
 VPATH += $(RTE_SDK)/lib/librte_eal/common
 
 CFLAGS += -I$(SRCDIR)/include
@@ -74,6 +75,9 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_service.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_random.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
+# from unix dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_file.c
+
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_hypervisor.c
lib/librte_eal/linux/Makefile +4 -0
index 48cc34844..331489f99 100644
@@ -7,6 +7,7 @@ LIB = librte_eal.a
 
 ARCH_DIR ?= $(RTE_ARCH)
 VPATH += $(RTE_SDK)/lib/librte_eal/$(ARCH_DIR)
+VPATH += $(RTE_SDK)/lib/librte_eal/unix
 VPATH += $(RTE_SDK)/lib/librte_eal/common
 
 CFLAGS += -I$(SRCDIR)/include
@@ -81,6 +82,9 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_service.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_random.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
+# from unix dir
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_file.c
+
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_hypervisor.c
lib/librte_eal/meson.build +4 -0
index e301f4558..8d492897d 100644
@@ -6,6 +6,10 @@ subdir('include')
 
 subdir('common')
 
+if not is_windows
+	subdir('unix')
+endif
+
 dpdk_conf.set('RTE_EXEC_ENV_' + exec_env.to_upper(), 1)
 subdir(exec_env)
 
lib/librte_eal/unix/eal_file.c new file +80 -0

80 lines not on this page · raw message

lib/librte_eal/unix/meson.build new file +6 -0
new file mode 100644
index 000000000..21029ba1a
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Dmitry Kozlyuk
+
+sources += files(
+	'eal_file.c',
+)
Signature · 1 line
2.25.4
#198 lore raw

PATCH v9 3/12 eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk

Introduce OS-independent wrappers for memory management operations used across DPDK and specifically in common code of EAL:

* rte_mem_map()
* rte_mem_unmap()
* rte_mem_page_size()
* rte_mem_lock()

Windows uses different APIs for memory mapping and reservation, while Unices reserve memory by mapping it. Introduce EAL private functions to support memory reservation in common code:

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive. New symbols are internal. Being thin wrappers, they require no special maintenance.

---

Not adding rte_eal_paging.h to Doxygen index because, to my understanding, it only contains public API, and it was decided to keep rte_eal_paging.h functions private.

 lib/librte_eal/common/eal_common_fbarray.c |  40 +++---
 lib/librte_eal/common/eal_common_memory.c  |  61 ++++-----
 lib/librte_eal/common/eal_private.h        |  78 ++++++++++-
 lib/librte_eal/freebsd/Makefile            |   1 +
 lib/librte_eal/include/rte_eal_paging.h    |  98 +++++++++++++
 lib/librte_eal/linux/Makefile              |   1 +
 lib/librte_eal/linux/eal_memalloc.c        |   5 +-
 lib/librte_eal/rte_eal_version.map         |   9 ++
 lib/librte_eal/unix/eal_unix_memory.c      | 152 +++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |   1 +
 10 files changed, 381 insertions(+), 65 deletions(-)
 create mode 100644 lib/librte_eal/include/rte_eal_paging.h
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
lib/librte_eal/common/eal_common_fbarray.c +19 -21
index c52ddb967..fd0292a64 100644
@@ -5,15 +5,16 @@
 #include <fcntl.h>
 #include <inttypes.h>
 #include <limits.h>
-#include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
 #include <string.h>
 #include <unistd.h>
 
 #include <rte_common.h>
-#include <rte_log.h>
+#include <rte_eal_paging.h>
 #include <rte_errno.h>
+#include <rte_log.h>
+#include <rte_memory.h>
 #include <rte_spinlock.h>
 #include <rte_tailq.h>
 
@@ -90,12 +91,9 @@ resize_and_map(int fd, void *addr, size_t len)
 		return -1;
 	}
 
-	map_addr = mmap(addr, len, PROT_READ | PROT_WRITE,
-			MAP_SHARED | MAP_FIXED, fd, 0);
+	map_addr = rte_mem_map(addr, len, RTE_PROT_READ | RTE_PROT_WRITE,
+			RTE_MAP_SHARED | RTE_MAP_FORCE_ADDRESS, fd, 0);
 	if (map_addr != addr) {
-		RTE_LOG(ERR, EAL, "mmap() failed: %s\n", strerror(errno));
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 	return 0;
@@ -733,7 +731,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -754,11 +752,13 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 
 	if (internal_config.no_shconf) {
 		/* remap virtual area as writable */
-		void *new_data = mmap(data, mmap_len, PROT_READ | PROT_WRITE,
-				MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, fd, 0);
-		if (new_data == MAP_FAILED) {
+		static const int flags = RTE_MAP_FORCE_ADDRESS |
+			RTE_MAP_PRIVATE | RTE_MAP_ANONYMOUS;
+		void *new_data = rte_mem_map(data, mmap_len,
+			RTE_PROT_READ | RTE_PROT_WRITE, flags, fd, 0);
+		if (new_data == NULL) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't remap anonymous memory: %s\n",
-					__func__, strerror(errno));
+					__func__, rte_strerror(rte_errno));
 			goto fail;
 		}
 	} else {
@@ -820,7 +820,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -858,7 +858,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -909,7 +909,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -937,8 +937,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -957,7 +956,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 		goto out;
 	}
 
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, close fd and remove the tailq entry */
 	if (tmp->fd >= 0)
@@ -992,8 +991,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -1042,7 +1040,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		}
 		close(fd);
 	}
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, remove the tailq entry */
 	TAILQ_REMOVE(&mem_area_tailq, tmp, next);
lib/librte_eal/common/eal_common_memory.c +23 -38

143 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +75 -3

106 lines not on this page · raw message

lib/librte_eal/freebsd/Makefile +1 -0
index 0f8741d96..2374ba0b7 100644
@@ -77,6 +77,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_file.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
lib/librte_eal/include/rte_eal_paging.h new file +98 -0

98 lines not on this page · raw message

lib/librte_eal/linux/Makefile +1 -0
index 331489f99..8febf2212 100644
@@ -84,6 +84,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_file.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
lib/librte_eal/linux/eal_memalloc.c +2 -3
index 2c717f8bd..bf29b83c6 100644
@@ -630,7 +630,7 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 mapped:
 	munmap(addr, alloc_sz);
 unmapped:
-	flags = MAP_FIXED;
+	flags = EAL_RESERVE_FORCE_ADDRESS;
 	new_addr = eal_get_virtual_area(addr, &alloc_sz, alloc_sz, 0, flags);
 	if (new_addr != addr) {
 		if (new_addr != NULL)
@@ -687,8 +687,7 @@ free_seg(struct rte_memseg *ms, struct hugepage_info *hi,
 		return -1;
 	}
 
-	if (madvise(ms->addr, ms->len, MADV_DONTDUMP) != 0)
-		RTE_LOG(DEBUG, EAL, "madvise failed: %s\n", strerror(errno));
+	eal_mem_set_dump(ms->addr, ms->len, false);
 
 	exit_early = false;
 
lib/librte_eal/rte_eal_version.map +9 -0
index d8038749a..196eef5af 100644
@@ -387,3 +387,12 @@ EXPERIMENTAL {
 	rte_trace_regexp;
 	rte_trace_save;
 };
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};
lib/librte_eal/unix/eal_unix_memory.c new file +152 -0

152 lines not on this page · raw message

lib/librte_eal/unix/meson.build +1 -0
index 21029ba1a..e733910a1 100644
@@ -3,4 +3,5 @@
 
 sources += files(
 	'eal_file.c',
+	'eal_unix_memory.c',
 )
Signature · 1 line
2.25.4
#199 lore raw

Re: [dpdk-dev] [PATCH v9 03/12] eal: introduce memory management wrappers

Quoted 600 lines · Dmitry Kozlyuk

Introduce OS-independent wrappers for memory management operations used across DPDK and specifically in common code of EAL:

* rte_mem_map()
* rte_mem_unmap()
* rte_mem_page_size()
* rte_mem_lock()

Windows uses different APIs for memory mapping and reservation, while Unices reserve memory by mapping it. Introduce EAL private functions to support memory reservation in common code:

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Wrappers follow POSIX semantics limited to DPDK tasks, but their signatures deliberately differ from POSIX ones to be more safe and expressive. New symbols are internal. Being thin wrappers, they require no special maintenance.

---

Not adding rte_eal_paging.h to Doxygen index because, to my understanding, it only contains public API, and it was decided to keep rte_eal_paging.h functions private.

 lib/librte_eal/common/eal_common_fbarray.c |  40 +++---
 lib/librte_eal/common/eal_common_memory.c  |  61 ++++-----
 lib/librte_eal/common/eal_private.h        |  78 ++++++++++-
 lib/librte_eal/freebsd/Makefile            |   1 +
 lib/librte_eal/include/rte_eal_paging.h    |  98 +++++++++++++
 lib/librte_eal/linux/Makefile              |   1 +
 lib/librte_eal/linux/eal_memalloc.c        |   5 +-
 lib/librte_eal/rte_eal_version.map         |   9 ++
 lib/librte_eal/unix/eal_unix_memory.c      | 152 +++++++++++++++++++++
 lib/librte_eal/unix/meson.build            |   1 +
 10 files changed, 381 insertions(+), 65 deletions(-)
 create mode 100644 lib/librte_eal/include/rte_eal_paging.h
 create mode 100644 lib/librte_eal/unix/eal_unix_memory.c
lib/librte_eal/common/eal_common_fbarray.c +19 -21
index c52ddb967..fd0292a64 100644
@@ -5,15 +5,16 @@
 #include <fcntl.h>
 #include <inttypes.h>
 #include <limits.h>
-#include <sys/mman.h>
 #include <stdint.h>
 #include <errno.h>
 #include <string.h>
 #include <unistd.h>
 
 #include <rte_common.h>
-#include <rte_log.h>
+#include <rte_eal_paging.h>
 #include <rte_errno.h>
+#include <rte_log.h>
+#include <rte_memory.h>
 #include <rte_spinlock.h>
 #include <rte_tailq.h>
 
@@ -90,12 +91,9 @@ resize_and_map(int fd, void *addr, size_t len)
 		return -1;
 	}
 
-	map_addr = mmap(addr, len, PROT_READ | PROT_WRITE,
-			MAP_SHARED | MAP_FIXED, fd, 0);
+	map_addr = rte_mem_map(addr, len, RTE_PROT_READ | RTE_PROT_WRITE,
+			RTE_MAP_SHARED | RTE_MAP_FORCE_ADDRESS, fd, 0);
 	if (map_addr != addr) {
-		RTE_LOG(ERR, EAL, "mmap() failed: %s\n", strerror(errno));
-		/* pass errno up the chain */
-		rte_errno = errno;
 		return -1;
 	}
 	return 0;
@@ -733,7 +731,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -754,11 +752,13 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 
 	if (internal_config.no_shconf) {
 		/* remap virtual area as writable */
-		void *new_data = mmap(data, mmap_len, PROT_READ | PROT_WRITE,
-				MAP_FIXED | MAP_PRIVATE | MAP_ANONYMOUS, fd, 0);
-		if (new_data == MAP_FAILED) {
+		static const int flags = RTE_MAP_FORCE_ADDRESS |
+			RTE_MAP_PRIVATE | RTE_MAP_ANONYMOUS;
+		void *new_data = rte_mem_map(data, mmap_len,
+			RTE_PROT_READ | RTE_PROT_WRITE, flags, fd, 0);
+		if (new_data == NULL) {
 			RTE_LOG(DEBUG, EAL, "%s(): couldn't remap anonymous memory: %s\n",
-					__func__, strerror(errno));
+					__func__, rte_strerror(rte_errno));
 			goto fail;
 		}
 	} else {
@@ -820,7 +820,7 @@ rte_fbarray_init(struct rte_fbarray *arr, const char *name, unsigned int len,
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -858,7 +858,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 		return -1;
 	}
 
-	page_sz = sysconf(_SC_PAGESIZE);
+	page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1) {
 		free(ma);
 		return -1;
@@ -909,7 +909,7 @@ rte_fbarray_attach(struct rte_fbarray *arr)
 	return 0;
 fail:
 	if (data)
-		munmap(data, mmap_len);
+		rte_mem_unmap(data, mmap_len);
 	if (fd >= 0)
 		close(fd);
 	free(ma);
@@ -937,8 +937,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -957,7 +956,7 @@ rte_fbarray_detach(struct rte_fbarray *arr)
 		goto out;
 	}
 
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, close fd and remove the tailq entry */
 	if (tmp->fd >= 0)
@@ -992,8 +991,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 	 * really do anything about it, things will blow up either way.
 	 */
 
-	size_t page_sz = sysconf(_SC_PAGESIZE);
-
+	size_t page_sz = rte_mem_page_size();
 	if (page_sz == (size_t)-1)
 		return -1;
 
@@ -1042,7 +1040,7 @@ rte_fbarray_destroy(struct rte_fbarray *arr)
 		}
 		close(fd);
 	}
-	munmap(arr->data, mmap_len);
+	rte_mem_unmap(arr->data, mmap_len);
 
 	/* area is unmapped, remove the tailq entry */
 	TAILQ_REMOVE(&mem_area_tailq, tmp, next);
lib/librte_eal/common/eal_common_memory.c +23 -38
index 4c897a13f..aa377990f 100644
@@ -11,13 +11,13 @@
 #include <string.h>
 #include <unistd.h>
 #include <inttypes.h>
-#include <sys/mman.h>
 #include <sys/queue.h>
 
 #include <rte_fbarray.h>
 #include <rte_memory.h>
 #include <rte_eal.h>
 #include <rte_eal_memconfig.h>
+#include <rte_eal_paging.h>
 #include <rte_errno.h>
 #include <rte_log.h>
 
@@ -40,18 +40,10 @@
 static void *next_baseaddr;
 static uint64_t system_page_sz;
 
-#ifdef RTE_EXEC_ENV_LINUX
-#define RTE_DONTDUMP MADV_DONTDUMP
-#elif defined RTE_EXEC_ENV_FREEBSD
-#define RTE_DONTDUMP MADV_NOCORE
-#else
-#error "madvise doesn't support this OS"
-#endif
-
 #define MAX_MMAP_WITH_DEFINED_ADDR_TRIES 5
 void *
 eal_get_virtual_area(void *requested_addr, size_t *size,
-		size_t page_sz, int flags, int mmap_flags)
+	size_t page_sz, int flags, int reserve_flags)
 {
 	bool addr_is_hint, allow_shrink, unmap, no_align;
 	uint64_t map_sz;
@@ -59,9 +51,7 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	uint8_t try = 0;
 
 	if (system_page_sz == 0)
-		system_page_sz = sysconf(_SC_PAGESIZE);
-
-	mmap_flags |= MAP_PRIVATE | MAP_ANONYMOUS;
+		system_page_sz = rte_mem_page_size();
 
 	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
 
@@ -105,24 +95,24 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 			return NULL;
 		}
 
-		mapped_addr = mmap(requested_addr, (size_t)map_sz, PROT_NONE,
-				mmap_flags, -1, 0);
-		if (mapped_addr == MAP_FAILED && allow_shrink)
+		mapped_addr = eal_mem_reserve(
+			requested_addr, (size_t)map_sz, reserve_flags);
+		if ((mapped_addr == NULL) && allow_shrink)
 			*size -= page_sz;
 
-		if (mapped_addr != MAP_FAILED && addr_is_hint &&
-		    mapped_addr != requested_addr) {
+		if ((mapped_addr != NULL) && addr_is_hint &&
+				(mapped_addr != requested_addr)) {
 			try++;
 			next_baseaddr = RTE_PTR_ADD(next_baseaddr, page_sz);
 			if (try <= MAX_MMAP_WITH_DEFINED_ADDR_TRIES) {
 				/* hint was not used. Try with another offset */
-				munmap(mapped_addr, map_sz);
-				mapped_addr = MAP_FAILED;
+				eal_mem_free(mapped_addr, map_sz);
+				mapped_addr = NULL;
 				requested_addr = next_baseaddr;
 			}
 		}
 	} while ((allow_shrink || addr_is_hint) &&
-		 mapped_addr == MAP_FAILED && *size > 0);
+		(mapped_addr == NULL) && (*size > 0));
 
 	/* align resulting address - if map failed, we will ignore the value
 	 * anyway, so no need to add additional checks.
@@ -132,20 +122,17 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 
 	if (*size == 0) {
 		RTE_LOG(ERR, EAL, "Cannot get a virtual area of any size: %s\n",
-			strerror(errno));
-		rte_errno = errno;
+			rte_strerror(rte_errno));
 		return NULL;
-	} else if (mapped_addr == MAP_FAILED) {
+	} else if (mapped_addr == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot get a virtual area: %s\n",
-			strerror(errno));
-		/* pass errno up the call chain */
-		rte_errno = errno;
+			rte_strerror(rte_errno));
 		return NULL;
 	} else if (requested_addr != NULL && !addr_is_hint &&
 			aligned_addr != requested_addr) {
 		RTE_LOG(ERR, EAL, "Cannot get a virtual area at requested address: %p (got %p)\n",
 			requested_addr, aligned_addr);
-		munmap(mapped_addr, map_sz);
+		eal_mem_free(mapped_addr, map_sz);
 		rte_errno = EADDRNOTAVAIL;
 		return NULL;
 	} else if (requested_addr != NULL && addr_is_hint &&
@@ -161,7 +148,7 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 		aligned_addr, *size);
 
 	if (unmap) {
-		munmap(mapped_addr, map_sz);
+		eal_mem_free(mapped_addr, map_sz);
 	} else if (!no_align) {
 		void *map_end, *aligned_end;
 		size_t before_len, after_len;
@@ -179,19 +166,17 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 		/* unmap space before aligned mmap address */
 		before_len = RTE_PTR_DIFF(aligned_addr, mapped_addr);
 		if (before_len > 0)
-			munmap(mapped_addr, before_len);
+			eal_mem_free(mapped_addr, before_len);
 
 		/* unmap space after aligned end mmap address */
 		after_len = RTE_PTR_DIFF(map_end, aligned_end);
 		if (after_len > 0)
-			munmap(aligned_end, after_len);
+			eal_mem_free(aligned_end, after_len);
 	}
 
 	if (!unmap) {
 		/* Exclude these pages from a core dump. */
-		if (madvise(aligned_addr, *size, RTE_DONTDUMP) != 0)
-			RTE_LOG(DEBUG, EAL, "madvise failed: %s\n",
-				strerror(errno));
+		eal_mem_set_dump(aligned_addr, *size, false);
 	}
 
 	return aligned_addr;
@@ -547,10 +532,10 @@ rte_eal_memdevice_init(void)
 int
 rte_mem_lock_page(const void *virt)
 {
-	unsigned long virtual = (unsigned long)virt;
-	int page_size = getpagesize();
-	unsigned long aligned = (virtual & ~(page_size - 1));
-	return mlock((void *)aligned, page_size);
+	uintptr_t virtual = (uintptr_t)virt;
+	size_t page_size = rte_mem_page_size();
+	uintptr_t aligned = RTE_PTR_ALIGN_FLOOR(virtual, page_size);
+	return rte_mem_lock((void *)aligned, page_size);
 }
 
 int
lib/librte_eal/common/eal_private.h +75 -3
index 6733a2321..1696345c2 100644
@@ -11,6 +11,7 @@
 
 #include <rte_dev.h>
 #include <rte_lcore.h>
+#include <rte_memory.h>
 
 /**
  * Structure storing internal configuration (per-lcore)
@@ -202,6 +203,24 @@ int rte_eal_alarm_init(void);
  */
 int rte_eal_check_module(const char *module_name);
 
+/**
+ * Memory reservation flags.
+ */
+enum eal_mem_reserve_flags {
+	/**
+	 * Reserve hugepages. May be unsupported by some platforms.
+	 */
+	EAL_RESERVE_HUGEPAGES = 1 << 0,
+	/**
+	 * Force reserving memory at the requested address.
+	 * This can be a destructive action depending on the implementation.
+	 *
+	 * @see RTE_MAP_FORCE_ADDRESS for description of possible consequences
+	 *      (although implementations are not required to use it).
+	 */
+	EAL_RESERVE_FORCE_ADDRESS = 1 << 1
+};
+
 /**
  * Get virtual area of specified size from the OS.
  *
@@ -215,8 +234,8 @@ int rte_eal_check_module(const char *module_name);
  *   Page size on which to align requested virtual area.
  * @param flags
  *   EAL_VIRTUAL_AREA_* flags.
- * @param mmap_flags
- *   Extra flags passed directly to mmap().
+ * @param reserve_flags
+ *   Extra flags passed directly to eal_mem_reserve().
  *
  * @return
  *   Virtual area address if successful.
@@ -233,7 +252,7 @@ int rte_eal_check_module(const char *module_name);
 /**< immediately unmap reserved virtual area. */
 void *
 eal_get_virtual_area(void *requested_addr, size_t *size,
-		size_t page_sz, int flags, int mmap_flags);
+		size_t page_sz, int flags, int reserve_flags);
 
 /**
  * Get cpu core_id.
@@ -493,4 +512,57 @@ eal_file_lock(int fd, enum eal_flock_op op, enum eal_flock_mode mode);
 int
 eal_file_truncate(int fd, ssize_t size);
 
+/**
+ * Reserve a region of virtual memory.
+ *
+ * Use eal_mem_free() to free reserved memory.
+ *
+ * @param requested_addr
+ *  A desired reservation address which must be page-aligned.
+ *  The system might not respect it.
+ *  NULL means the address will be chosen by the system.
+ * @param size
+ *  Reservation size. Must be a multiple of system page size.
+ * @param flags
+ *  Reservation options, a combination of eal_mem_reserve_flags.
+ * @returns
+ *  Starting address of the reserved area on success, NULL on failure.
+ *  Callers must not access this memory until remapping it.
+ */
+void *
+eal_mem_reserve(void *requested_addr, size_t size, int flags);
+
+/**
+ * Free memory obtained by eal_mem_reserve() or eal_mem_alloc().
+ *
+ * If *virt* and *size* describe a part of the reserved region,
+ * only this part of the region is freed (accurately up to the system
+ * page size). If *virt* points to allocated memory, *size* must match
+ * the one specified on allocation. The behavior is undefined
+ * if the memory pointed by *virt* is obtained from another source
+ * than listed above.
+ *
+ * @param virt
+ *  A virtual address in a region previously reserved.
+ * @param size
+ *  Number of bytes to unreserve.
+ */
+void
+eal_mem_free(void *virt, size_t size);
+
+/**
+ * Configure memory region inclusion into dumps.
+ *
+ * @param virt
+ *  Starting address of the region.
+ * @param size
+ *  Size of the region.
+ * @param dump
+ *  True to include memory into dumps, false to exclude.
+ * @return
+ *  0 on success, (-1) on failure and rte_errno is set.
+ */
+int
+eal_mem_set_dump(void *virt, size_t size, bool dump);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/freebsd/Makefile +1 -0
index 0f8741d96..2374ba0b7 100644
@@ -77,6 +77,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_file.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_FREEBSD) += rte_cpuflags.c
lib/librte_eal/include/rte_eal_paging.h new file +98 -0
new file mode 100644
index 000000000..ed98e70e9
@@ -0,0 +1,98 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Dmitry Kozlyuk
+ */
+
+#include <stdint.h>
+
+#include <rte_compat.h>
+
+/**
+ * @file
+ * @internal
+ *
+ * Wrappers for OS facilities related to memory paging, used across DPDK.
+ */
+
+/** Memory protection flags. */
+enum rte_mem_prot {
+	RTE_PROT_READ = 1 << 0,   /**< Read access. */
+	RTE_PROT_WRITE = 1 << 1,  /**< Write access. */
+	RTE_PROT_EXECUTE = 1 << 2 /**< Code execution. */
+};
+
+/** Additional flags for memory mapping. */
+enum rte_map_flags {
+	/** Changes to the mapped memory are visible to other processes. */
+	RTE_MAP_SHARED = 1 << 0,
+	/** Mapping is not backed by a regular file. */
+	RTE_MAP_ANONYMOUS = 1 << 1,
+	/** Copy-on-write mapping, changes are invisible to other processes. */
+	RTE_MAP_PRIVATE = 1 << 2,
+	/**
+	 * Force mapping to the requested address. This flag should be used
+	 * with caution, because to fulfill the request implementation
+	 * may remove all other mappings in the requested region. However,
+	 * it is not required to do so, thus mapping with this flag may fail.
+	 */
+	RTE_MAP_FORCE_ADDRESS = 1 << 3
+};
+
+/**
+ * Map a portion of an opened file or the page file into memory.
+ *
+ * This function is similar to POSIX mmap(3) with common MAP_ANONYMOUS
+ * extension, except for the return value.
+ *
+ * @param requested_addr
+ *  Desired virtual address for mapping. Can be NULL to let OS choose.
+ * @param size
+ *  Size of the mapping in bytes.
+ * @param prot
+ *  Protection flags, a combination of rte_mem_prot values.
+ * @param flags
+ *  Additional mapping flags, a combination of rte_map_flags.
+ * @param fd
+ *  Mapped file descriptor. Can be negative for anonymous mapping.
+ * @param offset
+ *  Offset of the mapped region in fd. Must be 0 for anonymous mappings.
+ * @return
+ *  Mapped address or NULL on failure and rte_errno is set to OS error.
+ */
+__rte_internal
+void *
+rte_mem_map(void *requested_addr, size_t size, int prot, int flags,
+	int fd, size_t offset);
+
+/**
+ * OS-independent implementation of POSIX munmap(3).
+ */
+__rte_internal
+int
+rte_mem_unmap(void *virt, size_t size);
+
+/**
+ * Get system page size. This function never fails.
+ *
+ * @return
+ *   Page size in bytes.
+ */
+__rte_internal
+size_t
+rte_mem_page_size(void);
+
+/**
+ * Lock in physical memory all pages crossed by the address region.
+ *
+ * @param virt
+ *   Base virtual address of the region.
+ * @param size
+ *   Size of the region.
+ * @return
+ *   0 on success, negative on error.
+ *
+ * @see rte_mem_page_size() to retrieve the page size.
+ * @see rte_mem_lock_page() to lock an entire single page.
+ */
+__rte_internal
+int
+rte_mem_lock(const void *virt, size_t size);
lib/librte_eal/linux/Makefile +1 -0
index 331489f99..8febf2212 100644
@@ -84,6 +84,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_reciprocal.c
 
 # from unix dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_file.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_unix_memory.c
 
 # from arch dir
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += rte_cpuflags.c
lib/librte_eal/linux/eal_memalloc.c +2 -3
index 2c717f8bd..bf29b83c6 100644
@@ -630,7 +630,7 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
 mapped:
 	munmap(addr, alloc_sz);
 unmapped:
-	flags = MAP_FIXED;
+	flags = EAL_RESERVE_FORCE_ADDRESS;
 	new_addr = eal_get_virtual_area(addr, &alloc_sz, alloc_sz, 0, flags);
 	if (new_addr != addr) {
 		if (new_addr != NULL)
@@ -687,8 +687,7 @@ free_seg(struct rte_memseg *ms, struct hugepage_info *hi,
 		return -1;
 	}
 
-	if (madvise(ms->addr, ms->len, MADV_DONTDUMP) != 0)
-		RTE_LOG(DEBUG, EAL, "madvise failed: %s\n", strerror(errno));
+	eal_mem_set_dump(ms->addr, ms->len, false);
 
 	exit_early = false;
 
lib/librte_eal/rte_eal_version.map +0 -0
index d8038749a..196eef5af 100644
@@ -387,3 +387,12 @@ EXPERIMENTAL {
 	rte_trace_regexp;
 	rte_trace_save;

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 10 lines
lib/librte_eal/rte_eal_version.map +9 -0
 };
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};

Don't

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Belong in the map file also?

Quoted 168 lines
lib/librte_eal/unix/eal_unix_memory.c new file +152 -0
new file mode 100644
index 000000000..ec7156df9
@@ -0,0 +1,152 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2020 Dmitry Kozlyuk
+ */
+
+#include <string.h>
+#include <sys/mman.h>
+#include <unistd.h>
+
+#include <rte_eal_paging.h>
+#include <rte_errno.h>
+#include <rte_log.h>
+
+#include "eal_private.h"
+
+#ifdef RTE_EXEC_ENV_LINUX
+#define EAL_DONTDUMP MADV_DONTDUMP
+#define EAL_DODUMP   MADV_DODUMP
+#elif defined RTE_EXEC_ENV_FREEBSD
+#define EAL_DONTDUMP MADV_NOCORE
+#define EAL_DODUMP   MADV_CORE
+#else
+#error "madvise doesn't support this OS"
+#endif
+
+static void *
+mem_map(void *requested_addr, size_t size, int prot, int flags,
+	int fd, size_t offset)
+{
+	void *virt = mmap(requested_addr, size, prot, flags, fd, offset);
+	if (virt == MAP_FAILED) {
+		RTE_LOG(DEBUG, EAL,
+			"Cannot mmap(%p, 0x%zx, 0x%x, 0x%x, %d, 0x%zx): %s\n",
+			requested_addr, size, prot, flags, fd, offset,
+			strerror(errno));
+		rte_errno = errno;
+		return NULL;
+	}
+	return virt;
+}
+
+static int
+mem_unmap(void *virt, size_t size)
+{
+	int ret = munmap(virt, size);
+	if (ret < 0) {
+		RTE_LOG(DEBUG, EAL, "Cannot munmap(%p, 0x%zx): %s\n",
+			virt, size, strerror(errno));
+		rte_errno = errno;
+	}
+	return ret;
+}
+
+void *
+eal_mem_reserve(void *requested_addr, size_t size, int flags)
+{
+	int sys_flags = MAP_PRIVATE | MAP_ANONYMOUS;
+
+	if (flags & EAL_RESERVE_HUGEPAGES) {
+#ifdef MAP_HUGETLB
+		sys_flags |= MAP_HUGETLB;
+#else
+		rte_errno = ENOTSUP;
+		return NULL;
+#endif
+	}
+
+	if (flags & EAL_RESERVE_FORCE_ADDRESS)
+		sys_flags |= MAP_FIXED;
+
+	return mem_map(requested_addr, size, PROT_NONE, sys_flags, -1, 0);
+}
+
+void
+eal_mem_free(void *virt, size_t size)
+{
+	mem_unmap(virt, size);
+}
+
+int
+eal_mem_set_dump(void *virt, size_t size, bool dump)
+{
+	int flags = dump ? EAL_DODUMP : EAL_DONTDUMP;
+	int ret = madvise(virt, size, flags);
+	if (ret) {
+		RTE_LOG(DEBUG, EAL, "madvise(%p, %#zx, %d) failed: %s\n",
+				virt, size, flags, strerror(rte_errno));
+		rte_errno = errno;
+	}
+	return ret;
+}
+
+static int
+mem_rte_to_sys_prot(int prot)
+{
+	int sys_prot = PROT_NONE;
+
+	if (prot & RTE_PROT_READ)
+		sys_prot |= PROT_READ;
+	if (prot & RTE_PROT_WRITE)
+		sys_prot |= PROT_WRITE;
+	if (prot & RTE_PROT_EXECUTE)
+		sys_prot |= PROT_EXEC;
+
+	return sys_prot;
+}
+
+void *
+rte_mem_map(void *requested_addr, size_t size, int prot, int flags,
+	int fd, size_t offset)
+{
+	int sys_flags = 0;
+	int sys_prot;
+
+	sys_prot = mem_rte_to_sys_prot(prot);
+
+	if (flags & RTE_MAP_SHARED)
+		sys_flags |= MAP_SHARED;
+	if (flags & RTE_MAP_ANONYMOUS)
+		sys_flags |= MAP_ANONYMOUS;
+	if (flags & RTE_MAP_PRIVATE)
+		sys_flags |= MAP_PRIVATE;
+	if (flags & RTE_MAP_FORCE_ADDRESS)
+		sys_flags |= MAP_FIXED;
+
+	return mem_map(requested_addr, size, sys_prot, sys_flags, fd, offset);
+}
+
+int
+rte_mem_unmap(void *virt, size_t size)
+{
+	return mem_unmap(virt, size);
+}
+
+size_t
+rte_mem_page_size(void)
+{
+	static size_t page_size;
+
+	if (!page_size)
+		page_size = sysconf(_SC_PAGESIZE);
+
+	return page_size;
+}
+
+int
+rte_mem_lock(const void *virt, size_t size)
+{
+	int ret = mlock(virt, size);
+	if (ret)
+		rte_errno = errno;
+	return ret;
+}
lib/librte_eal/unix/meson.build +1 -0
index 21029ba1a..e733910a1 100644
@@ -3,4 +3,5 @@
 
 sources += files(
 	'eal_file.c',
+	'eal_unix_memory.c',
 )
#200 lore raw

Re: [dpdk-dev] [PATCH v9 03/12] eal: introduce memory management wrappers

On Mon, 15 Jun 2020 07:03:23 +0100
"Kinsella, Ray" <[email protected]> wrote:

[snip]

Quoted 12 lines

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 9 lines
lib/librte_eal/rte_eal_version.map +9 -0
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};  

Don't

Quoted 5 lines

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Belong in the map file also?

No need to export these funtions, they're only used by librte_eal.

Signature · 1 line
Dmitry Kozlyuk
#201 lore raw

Re: [dpdk-dev] [PATCH v9 03/12] eal: introduce memory management wrappers

perfect.

Quoted 22 lines · Dmitry Kozlyuk

On Mon, 15 Jun 2020 07:03:23 +0100
"Kinsella, Ray" <[email protected]> wrote:

[snip]

Quoted 11 lines

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 9 lines
lib/librte_eal/rte_eal_version.map +9 -0
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};  

Don't

Quoted 5 lines

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Belong in the map file also?

No need to export these funtions, they're only used by librte_eal.

#202 lore raw

Re: [dpdk-dev] [PATCH v9 03/12] eal: introduce memory management wrappers

In reply to Dmitry Kozlyuk
Quoted 16 lines · Dmitry Kozlyuk

On Mon, 15 Jun 2020 07:03:23 +0100
"Kinsella, Ray" <[email protected]> wrote:

[snip]

Quoted 12 lines

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 9 lines
lib/librte_eal/rte_eal_version.map +9 -0
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};  

Don't

Quoted 8 lines
Quoted 5 lines

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Belong in the map file also?

No need to export these funtions, they're only used by librte_eal.

But theres lots of locations in dpdk that could be using these functions. I count 57 calls sites in dpdk for sysconf(SC_PAGESIZE), spread throughout the library collection, as well as some others for sysconf(_SC_IOV_MAX) and sysconf(_SC_NPROCESSORS_CONF). If the goal is to abstract away the use of sysconf in dpdk, you probably at least want to export rte_mem_page_size.

the same is likely true for mmap/munmap

Neil

Quoted 3 lines
Signature · 1 line
Dmitry Kozlyuk
#203 lore raw

Re: [dpdk-dev] [PATCH v9 03/12] eal: introduce memory management wrappers

Quoted 25 lines · On Mon, 15 Jun 2020 06:53:45 -0400 Neil Horman <[email protected]> wrote:
Quoted 16 lines · Dmitry Kozlyuk

On Mon, 15 Jun 2020 07:03:23 +0100
"Kinsella, Ray" <[email protected]> wrote:

[snip]

Quoted 12 lines

lib/librte_eal/rte_eal_version.map EXPERIMENTAL { · line 398

Quoted 9 lines
lib/librte_eal/rte_eal_version.map +9 -0
+
+INTERNAL {
+	global:
+
+	rte_mem_lock;
+	rte_mem_map;
+	rte_mem_page_size;
+	rte_mem_unmap;
+};    

Don't

Quoted 8 lines
Quoted 5 lines

* eal_mem_reserve()
* eal_mem_free()
* eal_mem_set_dump()

Belong in the map file also?

No need to export these funtions, they're only used by librte_eal.

Quoted 7 lines

But theres lots of locations in dpdk that could be using these functions. I count 57 calls sites in dpdk for sysconf(SC_PAGESIZE), spread throughout the library collection, as well as some others for sysconf(_SC_IOV_MAX) and sysconf(_SC_NPROCESSORS_CONF). If the goal is to abstract away the use of sysconf in dpdk, you probably at least want to export rte_mem_page_size.

the same is likely true for mmap/munmap

My comment (and Ray's one, I believe) was about eal_mem_*() functions. Those you're talking about, rte_mem_*(), are exported from EAL, but only visible to DPDK. Everything above is true, sysconf(), etc. can be replaced as the need arises to make the calling code OS-agnostic.

Signature · 1 line
Dmitry Kozlyuk
#204 lore raw

PATCH v9 4/12 eal/mem: extract common code for memseg list initialization

In reply to Dmitry Kozlyuk

All supported OS create memory segment lists (MSL) and reserve VA space for them in a nearly identical way. Move common code into EAL private functions to reduce duplication.

---
 lib/librte_eal/common/eal_common_memory.c |  96 ++++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  62 ++++++++++++
 lib/librte_eal/freebsd/eal_memory.c       |  98 ++++--------------
 lib/librte_eal/linux/eal_memory.c         | 118 +++++-----------------
 4 files changed, 204 insertions(+), 170 deletions(-)
lib/librte_eal/common/eal_common_memory.c +96 -0
index aa377990f..1414460c7 100644
@@ -25,6 +25,7 @@
 #include "eal_private.h"
 #include "eal_internal_cfg.h"
 #include "eal_memcfg.h"
+#include "eal_options.h"
 #include "malloc_heap.h"
 
 /*
@@ -182,6 +183,101 @@ eal_get_virtual_area(void *requested_addr, size_t *size,
 	return aligned_addr;
 }
 
+int
+eal_memseg_list_init_named(struct rte_memseg_list *msl, const char *name,
+		uint64_t page_sz, int n_segs, int socket_id, bool heap)
+{
+	if (rte_fbarray_init(&msl->memseg_arr, name, n_segs,
+			sizeof(struct rte_memseg))) {
+		RTE_LOG(ERR, EAL, "Cannot allocate memseg list: %s\n",
+			rte_strerror(rte_errno));
+		return -1;
+	}
+
+	msl->page_sz = page_sz;
+	msl->socket_id = socket_id;
+	msl->base_va = NULL;
+	msl->heap = heap;
+
+	RTE_LOG(DEBUG, EAL,
+		"Memseg list allocated at socket %i, page size 0x%"PRIx64"kB\n",
+		socket_id, (size_t)page_sz >> 10);
+
+	return 0;
+}
+
+int
+eal_memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
+		int n_segs, int socket_id, int type_msl_idx, bool heap)
+{
+	char name[RTE_FBARRAY_NAME_LEN];
+
+	snprintf(name, sizeof(name), MEMSEG_LIST_FMT, page_sz >> 10, socket_id,
+		 type_msl_idx);
+
+	return eal_memseg_list_init_named(
+		msl, name, page_sz, n_segs, socket_id, heap);
+}
+
+int
+eal_memseg_list_alloc(struct rte_memseg_list *msl, int reserve_flags)
+{
+	size_t page_sz, mem_sz;
+	void *addr;
+
+	page_sz = msl->page_sz;
+	mem_sz = page_sz * msl->memseg_arr.len;
+
+	addr = eal_get_virtual_area(
+		msl->base_va, &mem_sz, page_sz, 0, reserve_flags);
+	if (addr == NULL) {
+#ifndef RTE_EXEC_ENV_WINDOWS
+		/* The hint would be misleading on Windows, because address
+		 * is by default system-selected (base VA = 0).
+		 * However, this function is called from many places,
+		 * including common code, so don't duplicate the message.
+		 */
+		if (rte_errno == EADDRNOTAVAIL)
+			RTE_LOG(ERR, EAL, "Cannot reserve %llu bytes at [%p] - "
+				"please use '--" OPT_BASE_VIRTADDR "' option\n",
+				(unsigned long long)mem_sz, msl->base_va);
+#endif
+		return -1;
+	}
+	msl->base_va = addr;
+	msl->len = mem_sz;
+
+	RTE_LOG(DEBUG, EAL, "VA reserved for memseg list at %p, size %zx\n",
+			addr, mem_sz);
+
+	return 0;
+}
+
+void
+eal_memseg_list_populate(struct rte_memseg_list *msl, void *addr, int n_segs)
+{
+	size_t page_sz = msl->page_sz;
+	int i;
+
+	for (i = 0; i < n_segs; i++) {
+		struct rte_fbarray *arr = &msl->memseg_arr;
+		struct rte_memseg *ms = rte_fbarray_get(arr, i);
+
+		if (rte_eal_iova_mode() == RTE_IOVA_VA)
+			ms->iova = (uintptr_t)addr;
+		else
+			ms->iova = RTE_BAD_IOVA;
+		ms->addr = addr;
+		ms->hugepage_sz = page_sz;
+		ms->socket_id = 0;
+		ms->len = page_sz;
+
+		rte_fbarray_set_used(arr, i);
+
+		addr = RTE_PTR_ADD(addr, page_sz);
+	}
+}
+
 static struct rte_memseg *
 virt2memseg(const void *addr, const struct rte_memseg_list *msl)
 {
lib/librte_eal/common/eal_private.h +62 -0

68 lines not on this page · raw message

lib/librte_eal/freebsd/eal_memory.c +19 -79

159 lines not on this page · raw message

lib/librte_eal/linux/eal_memory.c +27 -91

221 lines not on this page · raw message

Signature · 1 line
2.25.4
#205 lore raw

Re: [dpdk-dev] [PATCH v9 04/12] eal/mem: extract common code for memseg list initialization

15/06/2020 02:43, Dmitry Kozlyuk:

lib/librte_eal/common/eal_common_memory.c eal_get_virtual_area(void *requested_addr, size_t *size, · line 204

Quoted 5 lines
lib/librte_eal/common/eal_common_memory.c +3 -0
+	RTE_LOG(DEBUG, EAL,
+		"Memseg list allocated at socket %i, page size 0x%"PRIx64"kB\n",
+		socket_id, (size_t)page_sz >> 10);

The cast to size_t must be removed to match PRIx64 expectation. I am fixing while merging.

#206 lore raw

PATCH v9 5/12 eal/mem: extract common code for dynamic memory allocation

In reply to Dmitry Kozlyuk

Code in Linux EAL that supports dynamic memory allocation (as opposed to static allocation used by FreeBSD) is not OS-dependent and can be reused by Windows EAL. Move such code to a file compiled only for the OS that require it. Keep Anatoly Burakov maintainer of extracted code.

---
 MAINTAINERS                               |   1 +
 lib/librte_eal/common/eal_common_dynmem.c | 521 +++++++++++++++++++++
 lib/librte_eal/common/eal_private.h       |  43 +-
 lib/librte_eal/common/meson.build         |   4 +
 lib/librte_eal/freebsd/eal_memory.c       |  12 +-
 lib/librte_eal/linux/Makefile             |   1 +
 lib/librte_eal/linux/eal_memory.c         | 523 +---------------------
 7 files changed, 582 insertions(+), 523 deletions(-)
 create mode 100644 lib/librte_eal/common/eal_common_dynmem.c
MAINTAINERS +1 -0
index 4d162efd6..241dbc3d7 100644
@@ -209,6 +209,7 @@ F: lib/librte_eal/include/rte_fbarray.h
 F: lib/librte_eal/include/rte_mem*
 F: lib/librte_eal/include/rte_malloc.h
 F: lib/librte_eal/common/*malloc*
+F: lib/librte_eal/common/eal_common_dynmem.c
 F: lib/librte_eal/common/eal_common_fbarray.c
 F: lib/librte_eal/common/eal_common_mem*
 F: lib/librte_eal/common/eal_hugepages.h
lib/librte_eal/common/eal_common_dynmem.c new file +521 -0

521 lines not on this page · raw message

lib/librte_eal/common/eal_private.h +42 -1

61 lines not on this page · raw message

lib/librte_eal/common/meson.build +4 -0
index 55aaeb18e..d91c22220 100644
@@ -56,3 +56,7 @@ sources += files(
 	'rte_reciprocal.c',
 	'rte_service.c',
 )
+
+if is_linux
+	sources += files('eal_common_dynmem.c')
+endif
lib/librte_eal/freebsd/eal_memory.c +2 -10
index 2eb70c2fe..72a30f21a 100644
@@ -315,14 +315,6 @@ get_mem_amount(uint64_t page_sz, uint64_t max_mem)
 	return RTE_ALIGN(area_sz, page_sz);
 }
 
-static int
-memseg_list_init(struct rte_memseg_list *msl, uint64_t page_sz,
-		int n_segs, int socket_id, int type_msl_idx)
-{
-	return eal_memseg_list_init(
-		msl, page_sz, n_segs, socket_id, type_msl_idx, false);
-}
-
 static int
 memseg_list_alloc(struct rte_memseg_list *msl)
 {
@@ -419,8 +411,8 @@ memseg_primary_init(void)
 					cur_max_mem);
 			n_segs = cur_mem / hugepage_sz;
 
-			if (memseg_list_init(msl, hugepage_sz, n_segs,
-					0, type_msl_idx))
+			if (eal_memseg_list_init(msl, hugepage_sz, n_segs,
+					0, type_msl_idx, false))
 				return -1;
 
 			total_segs += msl->memseg_arr.len;
lib/librte_eal/linux/Makefile +1 -0
index 8febf2212..07ce643ba 100644
@@ -50,6 +50,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_timer.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memzone.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_dynmem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_mcfg.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUX) += eal_common_memory.c
lib/librte_eal/linux/eal_memory.c +11 -512

586 lines not on this page · raw message

Signature · 1 line
2.25.4
#207 lore raw

PATCH v9 6/12 trace: add size_t field emitter

In reply to Dmitry Kozlyuk

It is not guaranteed that sizeof(long) == sizeof(size_t). On Windows, sizeof(long) == 4 and sizeof(size_t) == 8 for 64-bit programs. Tracepoints using "long" field emitter are therefore invalid there. Add dedicated field emitter for size_t and use it to store size_t values in all existing tracepoints.

---
 lib/librte_eal/include/rte_eal_trace.h   |  8 ++++----
 lib/librte_eal/include/rte_trace_point.h |  3 +++
 lib/librte_mempool/rte_mempool_trace.h   | 10 +++++-----
 3 files changed, 12 insertions(+), 9 deletions(-)
lib/librte_eal/include/rte_eal_trace.h +4 -4
index 1ebb2905a..bcfef0cfa 100644
@@ -143,7 +143,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(const char *type, size_t size, unsigned int align,
 		int socket, void *ptr),
 	rte_trace_point_emit_string(type);
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -154,7 +154,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(const char *type, size_t size, unsigned int align,
 		int socket, void *ptr),
 	rte_trace_point_emit_string(type);
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -164,7 +164,7 @@ RTE_TRACE_POINT(
 	rte_eal_trace_mem_realloc,
 	RTE_TRACE_POINT_ARGS(size_t size, unsigned int align, int socket,
 		void *ptr),
-	rte_trace_point_emit_long(size);
+	rte_trace_point_emit_size_t(size);
 	rte_trace_point_emit_u32(align);
 	rte_trace_point_emit_int(socket);
 	rte_trace_point_emit_ptr(ptr);
@@ -183,7 +183,7 @@ RTE_TRACE_POINT(
 		unsigned int flags, unsigned int align, unsigned int bound,
 		const void *mz),
 	rte_trace_point_emit_string(name);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_int(socket_id);
 	rte_trace_point_emit_u32(flags);
 	rte_trace_point_emit_u32(align);
lib/librte_eal/include/rte_trace_point.h +3 -0
index b45171275..377c2414a 100644
@@ -138,6 +138,8 @@ _tp _args \
 #define rte_trace_point_emit_int(val)
 /** Tracepoint function payload for long datatype */
 #define rte_trace_point_emit_long(val)
+/** Tracepoint function payload for size_t datatype */
+#define rte_trace_point_emit_size_t(val)
 /** Tracepoint function payload for float datatype */
 #define rte_trace_point_emit_float(val)
 /** Tracepoint function payload for double datatype */
@@ -395,6 +397,7 @@ do { \
 #define rte_trace_point_emit_i8(in) __rte_trace_point_emit(in, int8_t)
 #define rte_trace_point_emit_int(in) __rte_trace_point_emit(in, int32_t)
 #define rte_trace_point_emit_long(in) __rte_trace_point_emit(in, long)
+#define rte_trace_point_emit_size_t(in) __rte_trace_point_emit(in, size_t)
 #define rte_trace_point_emit_float(in) __rte_trace_point_emit(in, float)
 #define rte_trace_point_emit_double(in) __rte_trace_point_emit(in, double)
 #define rte_trace_point_emit_ptr(in) __rte_trace_point_emit(in, uintptr_t)
lib/librte_mempool/rte_mempool_trace.h +5 -5
index e776df0a6..087c913c8 100644
@@ -72,7 +72,7 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_string(mempool->name);
 	rte_trace_point_emit_ptr(vaddr);
 	rte_trace_point_emit_u64(iova);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_ptr(free_cb);
 	rte_trace_point_emit_ptr(opaque);
 )
@@ -84,8 +84,8 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_ptr(mempool);
 	rte_trace_point_emit_string(mempool->name);
 	rte_trace_point_emit_ptr(addr);
-	rte_trace_point_emit_long(len);
-	rte_trace_point_emit_long(pg_sz);
+	rte_trace_point_emit_size_t(len);
+	rte_trace_point_emit_size_t(pg_sz);
 	rte_trace_point_emit_ptr(free_cb);
 	rte_trace_point_emit_ptr(opaque);
 )
@@ -126,7 +126,7 @@ RTE_TRACE_POINT(
 	RTE_TRACE_POINT_ARGS(struct rte_mempool *mempool, size_t pg_sz),
 	rte_trace_point_emit_ptr(mempool);
 	rte_trace_point_emit_string(mempool->name);
-	rte_trace_point_emit_long(pg_sz);
+	rte_trace_point_emit_size_t(pg_sz);
 )
 
 RTE_TRACE_POINT(
@@ -139,7 +139,7 @@ RTE_TRACE_POINT(
 	rte_trace_point_emit_u32(max_objs);
 	rte_trace_point_emit_ptr(vaddr);
 	rte_trace_point_emit_u64(iova);
-	rte_trace_point_emit_long(len);
+	rte_trace_point_emit_size_t(len);
 	rte_trace_point_emit_ptr(obj_cb);
 	rte_trace_point_emit_ptr(obj_cb_arg);
 )
Signature · 1 line
2.25.4
#208 lore raw

PATCH v9 7/12 eal/windows: add tracing support stubs

In reply to Dmitry Kozlyuk

EAL common code depends on tracepoint calls, but generic implementation cannot be enabled on Windows due to missing standard library facilities. Add stub functions to support tracepoint compilation, so that common code does not have to conditionally include tracepoints until proper support is added.

---
 lib/librte_eal/common/eal_common_thread.c |  5 +---
 lib/librte_eal/common/meson.build         |  1 +
 lib/librte_eal/windows/eal.c              | 34 ++++++++++++++++++++++-
 3 files changed, 35 insertions(+), 5 deletions(-)
lib/librte_eal/common/eal_common_thread.c +1 -4
index f9f588c17..370bb1b63 100644
@@ -15,9 +15,7 @@
 #include <rte_lcore.h>
 #include <rte_memory.h>
 #include <rte_log.h>
-#ifndef RTE_EXEC_ENV_WINDOWS
 #include <rte_trace_point.h>
-#endif
 
 #include "eal_internal_cfg.h"
 #include "eal_private.h"
@@ -169,9 +167,8 @@ static void *rte_thread_init(void *arg)
 		free(params);
 	}
 
-#ifndef RTE_EXEC_ENV_WINDOWS
 	__rte_trace_mem_per_thread_alloc();
-#endif
+
 	return start_routine(routine_arg);
 }
 
lib/librte_eal/common/meson.build +1 -0
index d91c22220..4e9208129 100644
@@ -14,6 +14,7 @@ if is_windows
 		'eal_common_log.c',
 		'eal_common_options.c',
 		'eal_common_thread.c',
+		'eal_common_trace_points.c',
 	)
 	subdir_done()
 endif
lib/librte_eal/windows/eal.c +33 -1
index d084606a6..e7461f731 100644
@@ -17,6 +17,7 @@
 #include <eal_filesystem.h>
 #include <eal_options.h>
 #include <eal_private.h>
+#include <rte_trace_point.h>
 
 #include "eal_windows.h"
 
@@ -221,7 +222,38 @@ rte_eal_init_alert(const char *msg)
 	RTE_LOG(ERR, EAL, "%s\n", msg);
 }
 
- /* Launch threads, called at application init(). */
+/* Stubs to enable EAL trace point compilation
+ * until eal_common_trace.c can be compiled.
+ */
+
+RTE_DEFINE_PER_LCORE(volatile int, trace_point_sz);
+RTE_DEFINE_PER_LCORE(void *, trace_mem);
+
+void
+__rte_trace_mem_per_thread_alloc(void)
+{
+}
+
+void
+__rte_trace_point_emit_field(size_t sz, const char *field,
+	const char *type)
+{
+	RTE_SET_USED(sz);
+	RTE_SET_USED(field);
+	RTE_SET_USED(type);
+}
+
+int
+__rte_trace_point_register(rte_trace_point_t *trace, const char *name,
+	void (*register_fn)(void))
+{
+	RTE_SET_USED(trace);
+	RTE_SET_USED(name);
+	RTE_SET_USED(register_fn);
+	return -ENOTSUP;
+}
+
+/* Launch threads, called at application init(). */
 int
 rte_eal_init(int argc, char **argv)
 {
Signature · 1 line
2.25.4
#209 lore raw

PATCH v9 8/12 eal/windows: replace sys/queue.h with a complete one from FreeBSD

In reply to Dmitry Kozlyuk

Limited version imported previously lacks at least SLIST macros. Import a complete file from FreeBSD, since its license exception is already approved by Technical Board.

---
 lib/librte_eal/windows/include/sys/queue.h | 663 +++++++++++++++++++--
 1 file changed, 601 insertions(+), 62 deletions(-)
lib/librte_eal/windows/include/sys/queue.h +601 -62
index a65949a78..9756bee6f 100644
@@ -8,7 +8,36 @@
 #define	_SYS_QUEUE_H_
 
 /*
- * This file defines tail queues.
+ * This file defines four types of data structures: singly-linked lists,
+ * singly-linked tail queues, lists and tail queues.
+ *
+ * A singly-linked list is headed by a single forward pointer. The elements
+ * are singly linked for minimum space and pointer manipulation overhead at
+ * the expense of O(n) removal for arbitrary elements. New elements can be
+ * added to the list after an existing element or at the head of the list.
+ * Elements being removed from the head of the list should use the explicit
+ * macro for this purpose for optimum efficiency. A singly-linked list may
+ * only be traversed in the forward direction.  Singly-linked lists are ideal
+ * for applications with large datasets and few or no removals or for
+ * implementing a LIFO queue.
+ *
+ * A singly-linked tail queue is headed by a pair of pointers, one to the
+ * head of the list and the other to the tail of the list. The elements are
+ * singly linked for minimum space and pointer manipulation overhead at the
+ * expense of O(n) removal for arbitrary elements. New elements can be added
+ * to the list after an existing element, at the head of the list, or at the
+ * end of the list. Elements being removed from the head of the tail queue
+ * should use the explicit macro for this purpose for optimum efficiency.
+ * A singly-linked tail queue may only be traversed in the forward direction.
+ * Singly-linked tail queues are ideal for applications with large datasets
+ * and few or no removals or for implementing a FIFO queue.
+ *
+ * A list is headed by a single forward pointer (or an array of forward
+ * pointers for a hash table header). The elements are doubly linked
+ * so that an arbitrary element can be removed without a need to
+ * traverse the list. New elements can be added to the list before
+ * or after an existing element or at the head of the list. A list
+ * may be traversed in either direction.
  *
  * A tail queue is headed by a pair of pointers, one to the head of the
  * list and the other to the tail of the list. The elements are doubly
@@ -17,65 +46,93 @@
  * after an existing element, at the head of the list, or at the end of
  * the list. A tail queue may be traversed in either direction.
  *
+ * For details on the use of these macros, see the queue(3) manual page.
+ *
  * Below is a summary of implemented functions where:
  *  +  means the macro is available
  *  -  means the macro is not available
  *  s  means the macro is available but is slow (runs in O(n) time)
  *
- *				TAILQ
- * _HEAD			+
- * _CLASS_HEAD			+
- * _HEAD_INITIALIZER		+
- * _ENTRY			+
- * _CLASS_ENTRY			+
- * _INIT			+
- * _EMPTY			+
- * _FIRST			+
- * _NEXT			+
- * _PREV			+
- * _LAST			+
- * _LAST_FAST			+
- * _FOREACH			+
- * _FOREACH_FROM		+
- * _FOREACH_SAFE		+
- * _FOREACH_FROM_SAFE		+
- * _FOREACH_REVERSE		+
- * _FOREACH_REVERSE_FROM	+
- * _FOREACH_REVERSE_SAFE	+
- * _FOREACH_REVERSE_FROM_SAFE	+
- * _INSERT_HEAD			+
- * _INSERT_BEFORE		+
- * _INSERT_AFTER		+
- * _INSERT_TAIL			+
- * _CONCAT			+
- * _REMOVE_AFTER		-
- * _REMOVE_HEAD			-
- * _REMOVE			+
- * _SWAP			+
+ *				SLIST	LIST	STAILQ	TAILQ
+ * _HEAD			+	+	+	+
+ * _CLASS_HEAD			+	+	+	+
+ * _HEAD_INITIALIZER		+	+	+	+
+ * _ENTRY			+	+	+	+
+ * _CLASS_ENTRY			+	+	+	+
+ * _INIT			+	+	+	+
+ * _EMPTY			+	+	+	+
+ * _FIRST			+	+	+	+
+ * _NEXT			+	+	+	+
+ * _PREV			-	+	-	+
+ * _LAST			-	-	+	+
+ * _LAST_FAST			-	-	-	+
+ * _FOREACH			+	+	+	+
+ * _FOREACH_FROM		+	+	+	+
+ * _FOREACH_SAFE		+	+	+	+
+ * _FOREACH_FROM_SAFE		+	+	+	+
+ * _FOREACH_REVERSE		-	-	-	+
+ * _FOREACH_REVERSE_FROM	-	-	-	+
+ * _FOREACH_REVERSE_SAFE	-	-	-	+
+ * _FOREACH_REVERSE_FROM_SAFE	-	-	-	+
+ * _INSERT_HEAD			+	+	+	+
+ * _INSERT_BEFORE		-	+	-	+
+ * _INSERT_AFTER		+	+	+	+
+ * _INSERT_TAIL			-	-	+	+
+ * _CONCAT			s	s	+	+
+ * _REMOVE_AFTER		+	-	+	-
+ * _REMOVE_HEAD			+	-	+	-
+ * _REMOVE			s	+	s	+
+ * _SWAP			+	+	+	+
  *
  */
-
-#ifdef __cplusplus
-extern "C" {
+#ifdef QUEUE_MACRO_DEBUG
+#warn Use QUEUE_MACRO_DEBUG_TRACE and/or QUEUE_MACRO_DEBUG_TRASH
+#define	QUEUE_MACRO_DEBUG_TRACE
+#define	QUEUE_MACRO_DEBUG_TRASH
 #endif
 
-/*
- * List definitions.
- */
-#define	LIST_HEAD(name, type)						\
-struct name {								\
-	struct type *lh_first;	/* first element */			\
-}
+#ifdef QUEUE_MACRO_DEBUG_TRACE
+/* Store the last 2 places the queue element or head was altered */
+struct qm_trace {
+	unsigned long	 lastline;
+	unsigned long	 prevline;
+	const char	*lastfile;
+	const char	*prevfile;
+};
+
+#define	TRACEBUF	struct qm_trace trace;
+#define	TRACEBUF_INITIALIZER	{ __LINE__, 0, __FILE__, NULL } ,
+
+#define	QMD_TRACE_HEAD(head) do {					\
+	(head)->trace.prevline = (head)->trace.lastline;		\
+	(head)->trace.prevfile = (head)->trace.lastfile;		\
+	(head)->trace.lastline = __LINE__;				\
+	(head)->trace.lastfile = __FILE__;				\
+} while (0)
 
+#define	QMD_TRACE_ELEM(elem) do {					\
+	(elem)->trace.prevline = (elem)->trace.lastline;		\
+	(elem)->trace.prevfile = (elem)->trace.lastfile;		\
+	(elem)->trace.lastline = __LINE__;				\
+	(elem)->trace.lastfile = __FILE__;				\
+} while (0)
+
+#else	/* !QUEUE_MACRO_DEBUG_TRACE */
 #define	QMD_TRACE_ELEM(elem)
 #define	QMD_TRACE_HEAD(head)
 #define	TRACEBUF
 #define	TRACEBUF_INITIALIZER
+#endif	/* QUEUE_MACRO_DEBUG_TRACE */
 
+#ifdef QUEUE_MACRO_DEBUG_TRASH
+#define	QMD_SAVELINK(name, link)	void **name = (void *)&(link)
+#define	TRASHIT(x)		do {(x) = (void *)-1;} while (0)
+#define	QMD_IS_TRASHED(x)	((x) == (void *)(intptr_t)-1)
+#else	/* !QUEUE_MACRO_DEBUG_TRASH */
+#define	QMD_SAVELINK(name, link)
 #define	TRASHIT(x)
 #define	QMD_IS_TRASHED(x)	0
-
-#define	QMD_SAVELINK(name, link)
+#endif	/* QUEUE_MACRO_DEBUG_TRASH */
 
 #ifdef __cplusplus
 /*
@@ -86,6 +143,445 @@ struct name {								\
 #define	QUEUE_TYPEOF(type) struct type
 #endif
 
+/*
+ * Singly-linked List declarations.
+ */
+#define	SLIST_HEAD(name, type)						\
+struct name {								\
+	struct type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *slh_first;	/* first element */			\
+}
+
+#define	SLIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	SLIST_ENTRY(type)						\
+struct {								\
+	struct type *sle_next;	/* next element */			\
+}
+
+#define	SLIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *sle_next;		/* next element */		\
+}
+
+/*
+ * Singly-linked List functions.
+ */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm) do {			\
+	if (*(prevp) != (elm))						\
+		panic("Bad prevptr *(%p) == %p != %p",			\
+		    (prevp), *(prevp), (elm));				\
+} while (0)
+#else
+#define	QMD_SLIST_CHECK_PREVPTR(prevp, elm)
+#endif
+
+#define SLIST_CONCAT(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head1);		\
+	if (curelm == NULL) {						\
+		if ((SLIST_FIRST(head1) = SLIST_FIRST(head2)) != NULL)	\
+			SLIST_INIT(head2);				\
+	} else if (SLIST_FIRST(head2) != NULL) {			\
+		while (SLIST_NEXT(curelm, field) != NULL)		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_NEXT(curelm, field) = SLIST_FIRST(head2);		\
+		SLIST_INIT(head2);					\
+	}								\
+} while (0)
+
+#define	SLIST_EMPTY(head)	((head)->slh_first == NULL)
+
+#define	SLIST_FIRST(head)	((head)->slh_first)
+
+#define	SLIST_FOREACH(var, head, field)					\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = SLIST_NEXT((var), field))
+
+#define	SLIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = SLIST_FIRST((head));				\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : SLIST_FIRST((head)));		\
+	    (var) && ((tvar) = SLIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			\
+	for ((varp) = &SLIST_FIRST((head));				\
+	    ((var) = *(varp)) != NULL;					\
+	    (varp) = &SLIST_NEXT((var), field))
+
+#define	SLIST_INIT(head) do {						\
+	SLIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	SLIST_INSERT_AFTER(slistelm, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_NEXT((slistelm), field);	\
+	SLIST_NEXT((slistelm), field) = (elm);				\
+} while (0)
+
+#define	SLIST_INSERT_HEAD(head, elm, field) do {			\
+	SLIST_NEXT((elm), field) = SLIST_FIRST((head));			\
+	SLIST_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)
+
+#define	SLIST_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.sle_next);			\
+	if (SLIST_FIRST((head)) == (elm)) {				\
+		SLIST_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = SLIST_FIRST(head);		\
+		while (SLIST_NEXT(curelm, field) != (elm))		\
+			curelm = SLIST_NEXT(curelm, field);		\
+		SLIST_REMOVE_AFTER(curelm, field);			\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define SLIST_REMOVE_AFTER(elm, field) do {				\
+	SLIST_NEXT(elm, field) =					\
+	    SLIST_NEXT(SLIST_NEXT(elm, field), field);			\
+} while (0)
+
+#define	SLIST_REMOVE_HEAD(head, field) do {				\
+	SLIST_FIRST((head)) = SLIST_NEXT(SLIST_FIRST((head)), field);	\
+} while (0)
+
+#define	SLIST_REMOVE_PREVPTR(prevp, elm, field) do {			\
+	QMD_SLIST_CHECK_PREVPTR(prevp, elm);				\
+	*(prevp) = SLIST_NEXT(elm, field);				\
+	TRASHIT((elm)->field.sle_next);					\
+} while (0)
+
+#define SLIST_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = SLIST_FIRST(head1);		\
+	SLIST_FIRST(head1) = SLIST_FIRST(head2);			\
+	SLIST_FIRST(head2) = swap_first;				\
+} while (0)
+
+/*
+ * Singly-linked Tail queue declarations.
+ */
+#define	STAILQ_HEAD(name, type)						\
+struct name {								\
+	struct type *stqh_first;/* first element */			\
+	struct type **stqh_last;/* addr of last next element */		\
+}
+
+#define	STAILQ_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *stqh_first;	/* first element */			\
+	class type **stqh_last;	/* addr of last next element */		\
+}
+
+#define	STAILQ_HEAD_INITIALIZER(head)					\
+	{ NULL, &(head).stqh_first }
+
+#define	STAILQ_ENTRY(type)						\
+struct {								\
+	struct type *stqe_next;	/* next element */			\
+}
+
+#define	STAILQ_CLASS_ENTRY(type)					\
+struct {								\
+	class type *stqe_next;	/* next element */			\
+}
+
+/*
+ * Singly-linked Tail queue functions.
+ */
+#define	STAILQ_CONCAT(head1, head2) do {				\
+	if (!STAILQ_EMPTY((head2))) {					\
+		*(head1)->stqh_last = (head2)->stqh_first;		\
+		(head1)->stqh_last = (head2)->stqh_last;		\
+		STAILQ_INIT((head2));					\
+	}								\
+} while (0)
+
+#define	STAILQ_EMPTY(head)	((head)->stqh_first == NULL)
+
+#define	STAILQ_FIRST(head)	((head)->stqh_first)
+
+#define	STAILQ_FOREACH(var, head, field)				\
+	for((var) = STAILQ_FIRST((head));				\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	   (var);							\
+	   (var) = STAILQ_NEXT((var), field))
+
+#define	STAILQ_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = STAILQ_FIRST((head));				\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_FOREACH_FROM_SAFE(var, head, field, tvar)		\
+	for ((var) = ((var) ? (var) : STAILQ_FIRST((head)));		\
+	    (var) && ((tvar) = STAILQ_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	STAILQ_INIT(head) do {						\
+	STAILQ_FIRST((head)) = NULL;					\
+	(head)->stqh_last = &STAILQ_FIRST((head));			\
+} while (0)
+
+#define	STAILQ_INSERT_AFTER(head, tqelm, elm, field) do {		\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_NEXT((tqelm), field)) == NULL)\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_NEXT((tqelm), field) = (elm);				\
+} while (0)
+
+#define	STAILQ_INSERT_HEAD(head, elm, field) do {			\
+	if ((STAILQ_NEXT((elm), field) = STAILQ_FIRST((head))) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+	STAILQ_FIRST((head)) = (elm);					\
+} while (0)
+
+#define	STAILQ_INSERT_TAIL(head, elm, field) do {			\
+	STAILQ_NEXT((elm), field) = NULL;				\
+	*(head)->stqh_last = (elm);					\
+	(head)->stqh_last = &STAILQ_NEXT((elm), field);			\
+} while (0)
+
+#define	STAILQ_LAST(head, type, field)				\
+	(STAILQ_EMPTY((head)) ? NULL :				\
+	    __containerof((head)->stqh_last,			\
+	    QUEUE_TYPEOF(type), field.stqe_next))
+
+#define	STAILQ_NEXT(elm, field)	((elm)->field.stqe_next)
+
+#define	STAILQ_REMOVE(head, elm, type, field) do {			\
+	QMD_SAVELINK(oldnext, (elm)->field.stqe_next);			\
+	if (STAILQ_FIRST((head)) == (elm)) {				\
+		STAILQ_REMOVE_HEAD((head), field);			\
+	}								\
+	else {								\
+		QUEUE_TYPEOF(type) *curelm = STAILQ_FIRST(head);	\
+		while (STAILQ_NEXT(curelm, field) != (elm))		\
+			curelm = STAILQ_NEXT(curelm, field);		\
+		STAILQ_REMOVE_AFTER(head, curelm, field);		\
+	}								\
+	TRASHIT(*oldnext);						\
+} while (0)
+
+#define STAILQ_REMOVE_AFTER(head, elm, field) do {			\
+	if ((STAILQ_NEXT(elm, field) =					\
+	     STAILQ_NEXT(STAILQ_NEXT(elm, field), field)) == NULL)	\
+		(head)->stqh_last = &STAILQ_NEXT((elm), field);		\
+} while (0)
+
+#define	STAILQ_REMOVE_HEAD(head, field) do {				\
+	if ((STAILQ_FIRST((head)) =					\
+	     STAILQ_NEXT(STAILQ_FIRST((head)), field)) == NULL)		\
+		(head)->stqh_last = &STAILQ_FIRST((head));		\
+} while (0)
+
+#define STAILQ_SWAP(head1, head2, type) do {				\
+	QUEUE_TYPEOF(type) *swap_first = STAILQ_FIRST(head1);		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->stqh_last;		\
+	STAILQ_FIRST(head1) = STAILQ_FIRST(head2);			\
+	(head1)->stqh_last = (head2)->stqh_last;			\
+	STAILQ_FIRST(head2) = swap_first;				\
+	(head2)->stqh_last = swap_last;					\
+	if (STAILQ_EMPTY(head1))					\
+		(head1)->stqh_last = &STAILQ_FIRST(head1);		\
+	if (STAILQ_EMPTY(head2))					\
+		(head2)->stqh_last = &STAILQ_FIRST(head2);		\
+} while (0)
+
+
+/*
+ * List declarations.
+ */
+#define	LIST_HEAD(name, type)						\
+struct name {								\
+	struct type *lh_first;	/* first element */			\
+}
+
+#define	LIST_CLASS_HEAD(name, type)					\
+struct name {								\
+	class type *lh_first;	/* first element */			\
+}
+
+#define	LIST_HEAD_INITIALIZER(head)					\
+	{ NULL }
+
+#define	LIST_ENTRY(type)						\
+struct {								\
+	struct type *le_next;	/* next element */			\
+	struct type **le_prev;	/* address of previous next element */	\
+}
+
+#define	LIST_CLASS_ENTRY(type)						\
+struct {								\
+	class type *le_next;	/* next element */			\
+	class type **le_prev;	/* address of previous next element */	\
+}
+
+/*
+ * List functions.
+ */
+
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_LIST_CHECK_HEAD(LIST_HEAD *head, LIST_ENTRY NAME)
+ *
+ * If the list is non-empty, validates that the first element of the list
+ * points back at 'head.'
+ */
+#define	QMD_LIST_CHECK_HEAD(head, field) do {				\
+	if (LIST_FIRST((head)) != NULL &&				\
+	    LIST_FIRST((head))->field.le_prev !=			\
+	     &LIST_FIRST((head)))					\
+		panic("Bad list head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_NEXT(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the list, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_LIST_CHECK_NEXT(elm, field) do {				\
+	if (LIST_NEXT((elm), field) != NULL &&				\
+	    LIST_NEXT((elm), field)->field.le_prev !=			\
+	     &((elm)->field.le_next))					\
+	     	panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_LIST_CHECK_PREV(TYPE *elm, LIST_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the list) points to 'elm.'
+ */
+#define	QMD_LIST_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.le_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
+#define	QMD_LIST_CHECK_HEAD(head, field)
+#define	QMD_LIST_CHECK_NEXT(elm, field)
+#define	QMD_LIST_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
+
+#define LIST_CONCAT(head1, head2, type, field) do {			      \
+	QUEUE_TYPEOF(type) *curelm = LIST_FIRST(head1);			      \
+	if (curelm == NULL) {						      \
+		if ((LIST_FIRST(head1) = LIST_FIRST(head2)) != NULL) {	      \
+			LIST_FIRST(head2)->field.le_prev =		      \
+			    &LIST_FIRST((head1));			      \
+			LIST_INIT(head2);				      \
+		}							      \
+	} else if (LIST_FIRST(head2) != NULL) {				      \
+		while (LIST_NEXT(curelm, field) != NULL)		      \
+			curelm = LIST_NEXT(curelm, field);		      \
+		LIST_NEXT(curelm, field) = LIST_FIRST(head2);		      \
+		LIST_FIRST(head2)->field.le_prev = &LIST_NEXT(curelm, field); \
+		LIST_INIT(head2);					      \
+	}								      \
+} while (0)
+
+#define	LIST_EMPTY(head)	((head)->lh_first == NULL)
+
+#define	LIST_FIRST(head)	((head)->lh_first)
+
+#define	LIST_FOREACH(var, head, field)					\
+	for ((var) = LIST_FIRST((head));				\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_FROM(var, head, field)				\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var);							\
+	    (var) = LIST_NEXT((var), field))
+
+#define	LIST_FOREACH_SAFE(var, head, field, tvar)			\
+	for ((var) = LIST_FIRST((head));				\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_FOREACH_FROM_SAFE(var, head, field, tvar)			\
+	for ((var) = ((var) ? (var) : LIST_FIRST((head)));		\
+	    (var) && ((tvar) = LIST_NEXT((var), field), 1);		\
+	    (var) = (tvar))
+
+#define	LIST_INIT(head) do {						\
+	LIST_FIRST((head)) = NULL;					\
+} while (0)
+
+#define	LIST_INSERT_AFTER(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_NEXT(listelm, field);				\
+	if ((LIST_NEXT((elm), field) = LIST_NEXT((listelm), field)) != NULL)\
+		LIST_NEXT((listelm), field)->field.le_prev =		\
+		    &LIST_NEXT((elm), field);				\
+	LIST_NEXT((listelm), field) = (elm);				\
+	(elm)->field.le_prev = &LIST_NEXT((listelm), field);		\
+} while (0)
+
+#define	LIST_INSERT_BEFORE(listelm, elm, field) do {			\
+	QMD_LIST_CHECK_PREV(listelm, field);				\
+	(elm)->field.le_prev = (listelm)->field.le_prev;		\
+	LIST_NEXT((elm), field) = (listelm);				\
+	*(listelm)->field.le_prev = (elm);				\
+	(listelm)->field.le_prev = &LIST_NEXT((elm), field);		\
+} while (0)
+
+#define	LIST_INSERT_HEAD(head, elm, field) do {				\
+	QMD_LIST_CHECK_HEAD((head), field);				\
+	if ((LIST_NEXT((elm), field) = LIST_FIRST((head))) != NULL)	\
+		LIST_FIRST((head))->field.le_prev = &LIST_NEXT((elm), field);\
+	LIST_FIRST((head)) = (elm);					\
+	(elm)->field.le_prev = &LIST_FIRST((head));			\
+} while (0)
+
+#define	LIST_NEXT(elm, field)	((elm)->field.le_next)
+
+#define	LIST_PREV(elm, head, type, field)			\
+	((elm)->field.le_prev == &LIST_FIRST((head)) ? NULL :	\
+	    __containerof((elm)->field.le_prev,			\
+	    QUEUE_TYPEOF(type), field.le_next))
+
+#define	LIST_REMOVE(elm, field) do {					\
+	QMD_SAVELINK(oldnext, (elm)->field.le_next);			\
+	QMD_SAVELINK(oldprev, (elm)->field.le_prev);			\
+	QMD_LIST_CHECK_NEXT(elm, field);				\
+	QMD_LIST_CHECK_PREV(elm, field);				\
+	if (LIST_NEXT((elm), field) != NULL)				\
+		LIST_NEXT((elm), field)->field.le_prev = 		\
+		    (elm)->field.le_prev;				\
+	*(elm)->field.le_prev = LIST_NEXT((elm), field);		\
+	TRASHIT(*oldnext);						\
+	TRASHIT(*oldprev);						\
+} while (0)
+
+#define LIST_SWAP(head1, head2, type, field) do {			\
+	QUEUE_TYPEOF(type) *swap_tmp = LIST_FIRST(head1);		\
+	LIST_FIRST((head1)) = LIST_FIRST((head2));			\
+	LIST_FIRST((head2)) = swap_tmp;					\
+	if ((swap_tmp = LIST_FIRST((head1))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head1));		\
+	if ((swap_tmp = LIST_FIRST((head2))) != NULL)			\
+		swap_tmp->field.le_prev = &LIST_FIRST((head2));		\
+} while (0)
+
 /*
  * Tail queue declarations.
  */
@@ -123,10 +619,58 @@ struct {								\
 /*
  * Tail queue functions.
  */
+#if (defined(_KERNEL) && defined(INVARIANTS))
+/*
+ * QMD_TAILQ_CHECK_HEAD(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * If the tailq is non-empty, validates that the first element of the tailq
+ * points back at 'head.'
+ */
+#define	QMD_TAILQ_CHECK_HEAD(head, field) do {				\
+	if (!TAILQ_EMPTY(head) &&					\
+	    TAILQ_FIRST((head))->field.tqe_prev !=			\
+	     &TAILQ_FIRST((head)))					\
+		panic("Bad tailq head %p first->prev != head", (head));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_TAIL(TAILQ_HEAD *head, TAILQ_ENTRY NAME)
+ *
+ * Validates that the tail of the tailq is a pointer to pointer to NULL.
+ */
+#define	QMD_TAILQ_CHECK_TAIL(head, field) do {				\
+	if (*(head)->tqh_last != NULL)					\
+	    	panic("Bad tailq NEXT(%p->tqh_last) != NULL", (head)); 	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_NEXT(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * If an element follows 'elm' in the tailq, validates that the next element
+ * points back at 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_NEXT(elm, field) do {				\
+	if (TAILQ_NEXT((elm), field) != NULL &&				\
+	    TAILQ_NEXT((elm), field)->field.tqe_prev !=			\
+	     &((elm)->field.tqe_next))					\
+		panic("Bad link elm %p next->prev != elm", (elm));	\
+} while (0)
+
+/*
+ * QMD_TAILQ_CHECK_PREV(TYPE *elm, TAILQ_ENTRY NAME)
+ *
+ * Validates that the previous element (or head of the tailq) points to 'elm.'
+ */
+#define	QMD_TAILQ_CHECK_PREV(elm, field) do {				\
+	if (*(elm)->field.tqe_prev != (elm))				\
+		panic("Bad link elm %p prev->next != elm", (elm));	\
+} while (0)
+#else
 #define	QMD_TAILQ_CHECK_HEAD(head, field)
 #define	QMD_TAILQ_CHECK_TAIL(head, headname)
 #define	QMD_TAILQ_CHECK_NEXT(elm, field)
 #define	QMD_TAILQ_CHECK_PREV(elm, field)
+#endif /* (_KERNEL && INVARIANTS) */
 
 #define	TAILQ_CONCAT(head1, head2, field) do {				\
 	if (!TAILQ_EMPTY(head2)) {					\
@@ -191,9 +735,8 @@ struct {								\
 
 #define	TAILQ_INSERT_AFTER(head, listelm, elm, field) do {		\
 	QMD_TAILQ_CHECK_NEXT(listelm, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field);	\
-	if (TAILQ_NEXT((listelm), field) != NULL)			\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_NEXT((listelm), field)) != NULL)\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    &TAILQ_NEXT((elm), field);				\
 	else {								\
 		(head)->tqh_last = &TAILQ_NEXT((elm), field);		\
@@ -217,8 +760,7 @@ struct {								\
 
 #define	TAILQ_INSERT_HEAD(head, elm, field) do {			\
 	QMD_TAILQ_CHECK_HEAD(head, field);				\
-	TAILQ_NEXT((elm), field) = TAILQ_FIRST((head));			\
-	if (TAILQ_FIRST((head)) != NULL)				\
+	if ((TAILQ_NEXT((elm), field) = TAILQ_FIRST((head))) != NULL)	\
 		TAILQ_FIRST((head))->field.tqe_prev =			\
 		    &TAILQ_NEXT((elm), field);				\
 	else								\
@@ -250,21 +792,24 @@ struct {								\
  * you may want to prefetch the last data element.
  */
 #define	TAILQ_LAST_FAST(head, type, field)			\
-	(TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last,	\
-	QUEUE_TYPEOF(type), field.tqe_next))
+    (TAILQ_EMPTY(head) ? NULL : __containerof((head)->tqh_last, QUEUE_TYPEOF(type), field.tqe_next))
 
 #define	TAILQ_NEXT(elm, field) ((elm)->field.tqe_next)
 
 #define	TAILQ_PREV(elm, headname, field)				\
 	(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+#define	TAILQ_PREV_FAST(elm, head, type, field)				\
+    ((elm)->field.tqe_prev == &(head)->tqh_first ? NULL :		\
+     __containerof((elm)->field.tqe_prev, QUEUE_TYPEOF(type), field.tqe_next))
+
 #define	TAILQ_REMOVE(head, elm, field) do {				\
 	QMD_SAVELINK(oldnext, (elm)->field.tqe_next);			\
 	QMD_SAVELINK(oldprev, (elm)->field.tqe_prev);			\
 	QMD_TAILQ_CHECK_NEXT(elm, field);				\
 	QMD_TAILQ_CHECK_PREV(elm, field);				\
 	if ((TAILQ_NEXT((elm), field)) != NULL)				\
-		TAILQ_NEXT((elm), field)->field.tqe_prev =		\
+		TAILQ_NEXT((elm), field)->field.tqe_prev = 		\
 		    (elm)->field.tqe_prev;				\
 	else {								\
 		(head)->tqh_last = (elm)->field.tqe_prev;		\
@@ -277,26 +822,20 @@ struct {								\
 } while (0)
 
 #define TAILQ_SWAP(head1, head2, type, field) do {			\
-	QUEUE_TYPEOF(type) * swap_first = (head1)->tqh_first;		\
-	QUEUE_TYPEOF(type) * *swap_last = (head1)->tqh_last;		\
+	QUEUE_TYPEOF(type) *swap_first = (head1)->tqh_first;		\
+	QUEUE_TYPEOF(type) **swap_last = (head1)->tqh_last;		\
 	(head1)->tqh_first = (head2)->tqh_first;			\
 	(head1)->tqh_last = (head2)->tqh_last;				\
 	(head2)->tqh_first = swap_first;				\
 	(head2)->tqh_last = swap_last;					\
-	swap_first = (head1)->tqh_first;				\
-	if (swap_first != NULL)						\
+	if ((swap_first = (head1)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head1)->tqh_first;	\
 	else								\
 		(head1)->tqh_last = &(head1)->tqh_first;		\
-	swap_first = (head2)->tqh_first;				\
-	if (swap_first != NULL)			\
+	if ((swap_first = (head2)->tqh_first) != NULL)			\
 		swap_first->field.tqe_prev = &(head2)->tqh_first;	\
 	else								\
 		(head2)->tqh_last = &(head2)->tqh_first;		\
 } while (0)
 
-#ifdef __cplusplus
-}
-#endif
-
-#endif /* _SYS_QUEUE_H_ */
+#endif /* !_SYS_QUEUE_H_ */
Signature · 1 line
2.25.4
#210 lore raw

PATCH v9 9/12 eal/windows: improve CPU and NUMA node detection

In reply to Dmitry Kozlyuk

1. Map CPU cores to their respective NUMA nodes as reported by system.
2. Support systems with more than 64 cores (multiple processor groups).
3. Fix magic constants, styling issues, and compiler warnings.
4. Add EAL private function to map DPDK socket ID to NUMA node number.

---
 lib/librte_eal/windows/eal.c         |   7 +-
 lib/librte_eal/windows/eal_lcore.c   | 205 +++++++++++++++++----------
 lib/librte_eal/windows/eal_windows.h |  15 +-
 3 files changed, 152 insertions(+), 75 deletions(-)
lib/librte_eal/windows/eal.c +5 -2
index e7461f731..dfc10b494 100644
@@ -263,8 +263,11 @@ rte_eal_init(int argc, char **argv)
 
 	eal_log_level_parse(argc, argv);
 
-	/* create a map of all processors in the system */
-	eal_create_cpu_map();
+	if (eal_create_cpu_map() < 0) {
+		rte_eal_init_alert("Cannot discover CPU and NUMA.");
+		/* rte_errno is set */
+		return -1;
+	}
 
 	if (rte_eal_cpu_init() < 0) {
 		rte_eal_init_alert("Cannot detect lcores.");
lib/librte_eal/windows/eal_lcore.c +133 -72

236 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +14 -1
index fadd676b2..f3ed8c37f 100644
@@ -13,8 +13,11 @@
 
 /**
  * Create a map of processors and cores on the system.
+ *
+ * @return
+ *  0 on success, (-1) on failure and rte_errno is set.
  */
-void eal_create_cpu_map(void);
+int eal_create_cpu_map(void);
 
 /**
  * Create a thread.
@@ -26,4 +29,14 @@ void eal_create_cpu_map(void);
  */
 int eal_thread_create(pthread_t *thread);
 
+/**
+ * Get system NUMA node number for a socket ID.
+ *
+ * @param socket_id
+ *  Valid EAL socket ID.
+ * @return
+ *  NUMA node number to use with Win32 API.
+ */
+unsigned int eal_socket_numa_node(unsigned int socket_id);
+
 #endif /* _EAL_WINDOWS_H_ */
Signature · 1 line
2.25.4
#211 lore raw

Re: [dpdk-dev] [PATCH v9 09/12] eal/windows: improve CPU and NUMA node detection

15/06/2020 02:43, Dmitry Kozlyuk:

Quoted 16 lines
(quoted diff) +16 -0
+	infos_size = 0;
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, NULL, &infos_size)) {
+		DWORD error = GetLastError();
+		if (error != ERROR_INSUFFICIENT_BUFFER) {
+			log_early("Cannot get NUMA node info size, error %lu\n",
+				GetLastError());
+			rte_errno = ENOMEM;
+			return -1;
+		}
+	}
+
+	infos = malloc(infos_size);
+	if (infos == NULL) {
+		log_early("Cannot allocate memory for NUMA node information\n");
+		rte_errno = ENOMEM;

Review

Quoted 10 lines
(quoted diff) +10 -0
+		return -1;
+	}
+
+	if (!GetLogicalProcessorInformationEx(
+			RelationNumaNode, infos, &infos_size)) {
+		log_early("Cannot get NUMA node information, error %lu\n",
+			GetLastError());
+		rte_errno = EINVAL;
+		return -1;
+	}

rte_errno is unknown

It seems to be fixed in patch 12:

lib/librte_eal/windows/eal_windows.h +1 -0
@@ -9,8 +9,24 @@
  * @file Facilities private to Windows EAL
  */
 
+#include <rte_errno.h>
 #include <rte_windows.h>
 
 

I'll merge it in patch 9

#212 lore raw

Re: [dpdk-dev] [PATCH v9 09/12] eal/windows: improve CPU and NUMA node detection

[snip]

Quoted 18 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+		rte_errno = EINVAL;
+		return -1;
+	}  

rte_errno is unknown

It seems to be fixed in patch 12:

lib/librte_eal/windows/eal_windows.h +1 -0
@@ -9,8 +9,24 @@
  * @file Facilities private to Windows EAL
  */
 
+#include <rte_errno.h>
 #include <rte_windows.h>
 
 
Quoted 1 line

I'll merge it in patch 9

OK. Thanks for both fixes while merging and sorry for the mess. I'll try to automate local per-commit testing next time (test-meson-builds, etc).

Signature · 1 line
Dmitry Kozlyuk
#213 lore raw

PATCH v9 10/12 doc/windows: split build and run instructions

In reply to Dmitry Kozlyuk

With memory management implemented for Windows, the guide for running sample applications is going to be extended with hugepages and driver setup. Move run instructions to a separate file to give space for planned expansion.

---
 doc/guides/windows_gsg/build_dpdk.rst | 20 --------------------
 doc/guides/windows_gsg/index.rst      |  1 +
 doc/guides/windows_gsg/run_apps.rst   | 24 ++++++++++++++++++++++++
 3 files changed, 25 insertions(+), 20 deletions(-)
 create mode 100644 doc/guides/windows_gsg/run_apps.rst
doc/guides/windows_gsg/build_dpdk.rst +0 -20
index d46e84e3f..650483e3b 100644
@@ -111,23 +111,3 @@ Depending on the distribution, paths in this file may need adjustments.
 
     meson --cross-file config/x86/meson_mingw.txt -Dexamples=helloworld build
     ninja -C build
-
-
-Run the helloworld example
-==========================
-
-Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
-
-.. code-block:: console
-
-    cd C:\Users\me\dpdk\build\examples
-    dpdk-helloworld.exe
-    hello from core 1
-    hello from core 3
-    hello from core 0
-    hello from core 2
-
-Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
-by default. To run the example, either add toolchain executables directory
-to the PATH or copy the library to the working directory.
-Alternatively, static linking may be used (mind the LGPLv2.1 license).
doc/guides/windows_gsg/index.rst +1 -0
index d9b7990a8..e94593572 100644
@@ -12,3 +12,4 @@ Getting Started Guide for Windows
 
     intro
     build_dpdk
+    run_apps
doc/guides/windows_gsg/run_apps.rst new file +24 -0
new file mode 100644
index 000000000..ff4c4654f
@@ -0,0 +1,24 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2020 Dmitry Kozlyuk
+
+Running DPDK Applications
+=========================
+
+Run the ``helloworld`` Example
+------------------------------
+
+Navigate to the examples in the build directory and run `dpdk-helloworld.exe`.
+
+.. code-block:: console
+
+    cd C:\Users\me\dpdk\build\examples
+    dpdk-helloworld.exe
+    hello from core 1
+    hello from core 3
+    hello from core 0
+    hello from core 2
+
+Note for MinGW-w64: applications are linked to ``libwinpthread-1.dll``
+by default. To run the example, either add toolchain executables directory
+to the PATH or copy the library to the working directory.
+Alternatively, static linking may be used (mind the LGPLv2.1 license).
Signature · 1 line
2.25.4
#214 lore raw

PATCH v9 11/12 eal/windows: initialize hugepage info

In reply to Dmitry Kozlyuk

Add hugepages discovery ("large pages" in Windows terminology) and update documentation for required privilege setup. Only 2MB hugepages are supported and their number is estimated roughly due to the lack or unstable status of suitable OS APIs. Assign myself as maintainer for the implementation file.

---
 MAINTAINERS                            |   4 +
 config/meson.build                     |   2 +
 doc/guides/windows_gsg/run_apps.rst    |  23 ++++++
 lib/librte_eal/windows/eal.c           |  14 ++++
 lib/librte_eal/windows/eal_hugepages.c | 108 +++++++++++++++++++++++++
 lib/librte_eal/windows/meson.build     |   1 +
 6 files changed, 152 insertions(+)
 create mode 100644 lib/librte_eal/windows/eal_hugepages.c
MAINTAINERS +4 -0
index 241dbc3d7..9d5dacc23 100644
@@ -334,6 +334,10 @@ F: lib/librte_eal/windows/
 F: lib/librte_eal/rte_eal_exports.def
 F: doc/guides/windows_gsg/
 
+Windows memory allocation
+M: Dmitry Kozlyuk <[email protected]>
+F: lib/librte_eal/windows/eal_hugepages.c
+
 
 Core Libraries
 --------------
config/meson.build +2 -0
index 43ab11310..c1e80de4b 100644
@@ -268,6 +268,8 @@ if is_windows
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
+
+	add_project_link_arguments('-ladvapi32', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +23 -0
index ff4c4654f..21ac7f6c1 100644
@@ -4,6 +4,29 @@
 Running DPDK Applications
 =========================
 
+Grant *Lock pages in memory* Privilege
+--------------------------------------
+
+Use of hugepages ("large pages" in Windows terminolocy) requires
+``SeLockMemoryPrivilege`` for the user running an application.
+
+1. Open *Local Security Policy* snap in, either:
+
+   * Control Panel / Computer Management / Local Security Policy;
+   * or Win+R, type ``secpol``, press Enter.
+
+2. Open *Local Policies / User Rights Assignment / Lock pages in memory.*
+
+3. Add desired users or groups to the list of grantees.
+
+4. Privilege is applied upon next logon. In particular, if privilege has been
+   granted to current user, a logoff is required before it is available.
+
+See `Large-Page Support`_ in MSDN for details.
+
+.. _Large-page Support: https://docs.microsoft.com/en-us/windows/win32/memory/large-page-support
+
+
 Run the ``helloworld`` Example
 ------------------------------
 
lib/librte_eal/windows/eal.c +14 -0
index dfc10b494..759bf4be5 100644
@@ -19,8 +19,11 @@
 #include <eal_private.h>
 #include <rte_trace_point.h>
 
+#include "eal_hugepages.h"
 #include "eal_windows.h"
 
+#define MEMSIZE_IF_NO_HUGE_PAGE (64ULL * 1024ULL * 1024ULL)
+
  /* Allow the application to print its usage message too if set */
 static rte_usage_hook_t	rte_application_usage_hook;
 
@@ -279,6 +282,17 @@ rte_eal_init(int argc, char **argv)
 	if (fctret < 0)
 		exit(1);
 
+	if (!internal_config.no_hugetlbfs && (eal_hugepage_info_init() < 0)) {
+		rte_eal_init_alert("Cannot get hugepage information");
+		rte_errno = EACCES;
+		return -1;
+	}
+
+	if (internal_config.memory == 0 && !internal_config.force_sockets) {
+		if (internal_config.no_hugetlbfs)
+			internal_config.memory = MEMSIZE_IF_NO_HUGE_PAGE;
+	}
+
 	eal_thread_init_master(rte_config.master_lcore);
 
 	RTE_LCORE_FOREACH_SLAVE(i) {
lib/librte_eal/windows/eal_hugepages.c new file +108 -0

108 lines not on this page · raw message

lib/librte_eal/windows/meson.build +1 -0
index adfc8b9b7..52978e9d7 100644
@@ -6,6 +6,7 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_log.c',
 	'eal_thread.c',
Signature · 1 line
2.25.4
#215 lore raw

PATCH v9 12/12 eal/windows: implement basic memory management

In reply to Dmitry Kozlyuk

Basic memory management supports core libraries and PMDs operating in IOVA as PA mode. It uses a kernel-mode driver, virt2phys, to obtain IOVAs of hugepages allocated from user-mode. Multi-process mode is not implemented and is forcefully disabled at startup. Assign myself as a maintainer for Windows file and memory management implementation.

---
 MAINTAINERS                                   |   1 +
 config/meson.build                            |  12 +-
 doc/guides/windows_gsg/run_apps.rst           |  54 +-
 lib/librte_eal/common/meson.build             |  11 +
 lib/librte_eal/common/rte_malloc.c            |   1 +
 lib/librte_eal/rte_eal_exports.def            | 119 +++
 lib/librte_eal/windows/eal.c                  |  63 +-
 lib/librte_eal/windows/eal_file.c             | 125 +++
 lib/librte_eal/windows/eal_memalloc.c         | 441 +++++++++++
 lib/librte_eal/windows/eal_memory.c           | 710 ++++++++++++++++++
 lib/librte_eal/windows/eal_mp.c               | 103 +++
 lib/librte_eal/windows/eal_windows.h          |  75 ++
 lib/librte_eal/windows/include/meson.build    |   1 +
 lib/librte_eal/windows/include/rte_os.h       |  17 +
 .../windows/include/rte_virt2phys.h           |  34 +
 lib/librte_eal/windows/include/rte_windows.h  |   2 +
 lib/librte_eal/windows/include/unistd.h       |   3 +
 lib/librte_eal/windows/meson.build            |   6 +
 18 files changed, 1771 insertions(+), 7 deletions(-)
 create mode 100644 lib/librte_eal/windows/eal_file.c
 create mode 100644 lib/librte_eal/windows/eal_memalloc.c
 create mode 100644 lib/librte_eal/windows/eal_memory.c
 create mode 100644 lib/librte_eal/windows/eal_mp.c
 create mode 100644 lib/librte_eal/windows/include/rte_virt2phys.h
MAINTAINERS +1 -0
index 9d5dacc23..a80a3b904 100644
@@ -337,6 +337,7 @@ F: doc/guides/windows_gsg/
 Windows memory allocation
 M: Dmitry Kozlyuk <[email protected]>
 F: lib/librte_eal/windows/eal_hugepages.c
+F: lib/librte_eal/windows/eal_mem*
 
 
 Core Libraries
config/meson.build +9 -3
index c1e80de4b..d3f05f878 100644
@@ -261,15 +261,21 @@ if is_freebsd
 endif
 
 if is_windows
-	# Minimum supported API is Windows 7.
-	add_project_arguments('-D_WIN32_WINNT=0x0601', language: 'c')
+	# VirtualAlloc2() is available since Windows 10 / Server 2016.
+	add_project_arguments('-D_WIN32_WINNT=0x0A00', language: 'c')
 
 	# Use MinGW-w64 stdio, because DPDK assumes ANSI-compliant formatting.
 	if cc.get_id() == 'gcc'
 		add_project_arguments('-D__USE_MINGW_ANSI_STDIO', language: 'c')
 	endif
 
-	add_project_link_arguments('-ladvapi32', language: 'c')
+	# Contrary to docs, VirtualAlloc2() is exported by mincore.lib
+	# in Windows SDK, while MinGW exports it by advapi32.a.
+	if is_ms_linker
+		add_project_link_arguments('-lmincore', language: 'c')
+	endif
+
+	add_project_link_arguments('-ladvapi32', '-lsetupapi', language: 'c')
 endif
 
 if get_option('b_lto')
doc/guides/windows_gsg/run_apps.rst +51 -3

68 lines not on this page · raw message

lib/librte_eal/common/meson.build +11 -0
index 4e9208129..310844269 100644
@@ -8,13 +8,24 @@ if is_windows
 		'eal_common_bus.c',
 		'eal_common_class.c',
 		'eal_common_devargs.c',
+		'eal_common_dynmem.c',
 		'eal_common_errno.c',
+		'eal_common_fbarray.c',
 		'eal_common_launch.c',
 		'eal_common_lcore.c',
 		'eal_common_log.c',
+		'eal_common_mcfg.c',
+		'eal_common_memalloc.c',
+		'eal_common_memory.c',
+		'eal_common_memzone.c',
 		'eal_common_options.c',
+		'eal_common_string_fns.c',
+		'eal_common_tailqs.c',
 		'eal_common_thread.c',
 		'eal_common_trace_points.c',
+		'malloc_elem.c',
+		'malloc_heap.c',
+		'rte_malloc.c',
 	)
 	subdir_done()
 endif
lib/librte_eal/common/rte_malloc.c +1 -0
index f1b73168b..9d39e58c0 100644
@@ -20,6 +20,7 @@
 #include <rte_lcore.h>
 #include <rte_common.h>
 #include <rte_spinlock.h>
+
 #include <rte_eal_trace.h>
 
 #include <rte_malloc.h>
lib/librte_eal/rte_eal_exports.def +119 -0

128 lines not on this page · raw message

lib/librte_eal/windows/eal.c +62 -1

87 lines not on this page · raw message

lib/librte_eal/windows/eal_file.c new file +125 -0

125 lines not on this page · raw message

lib/librte_eal/windows/eal_memalloc.c new file +441 -0

441 lines not on this page · raw message

lib/librte_eal/windows/eal_memory.c new file +710 -0

710 lines not on this page · raw message

lib/librte_eal/windows/eal_mp.c new file +103 -0

103 lines not on this page · raw message

lib/librte_eal/windows/eal_windows.h +75 -0

87 lines not on this page · raw message

lib/librte_eal/windows/include/meson.build +1 -0
index 5fb1962ac..b3534b025 100644
@@ -5,5 +5,6 @@ includes += include_directories('.')
 
 headers += files(
         'rte_os.h',
+        'rte_virt2phys.h',
         'rte_windows.h',
 )
lib/librte_eal/windows/include/rte_os.h +17 -0
index 510e39e03..cb10d6494 100644
@@ -14,6 +14,7 @@
 #include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
+#include <string.h>
 
 #ifdef __cplusplus
 extern "C" {
@@ -36,6 +37,9 @@ extern "C" {
 
 #define strncasecmp(s1, s2, count)        _strnicmp(s1, s2, count)
 
+#define close _close
+#define unlink _unlink
+
 /* cpu_set macros implementation */
 #define RTE_CPU_AND(dst, src1, src2) CPU_AND(dst, src1, src2)
 #define RTE_CPU_OR(dst, src1, src2) CPU_OR(dst, src1, src2)
@@ -46,6 +50,7 @@ extern "C" {
 typedef long long ssize_t;
 
 #ifndef RTE_TOOLCHAIN_GCC
+
 static inline int
 asprintf(char **buffer, const char *format, ...)
 {
@@ -72,6 +77,18 @@ asprintf(char **buffer, const char *format, ...)
 	}
 	return ret;
 }
+
+static inline const char *
+eal_strerror(int code)
+{
+	static char buffer[128];
+
+	strerror_s(buffer, sizeof(buffer), code);
+	return buffer;
+}
+
+#define strerror eal_strerror
+
 #endif /* RTE_TOOLCHAIN_GCC */
 
 #ifdef __cplusplus
lib/librte_eal/windows/include/rte_virt2phys.h new file +34 -0
new file mode 100644
index 000000000..4bb2b4aaf
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright (c) 2020 Dmitry Kozlyuk
+ */
+
+/**
+ * @file virt2phys driver interface
+ */
+
+/**
+ * Driver device interface GUID {539c2135-793a-4926-afec-d3a1b61bbc8a}.
+ */
+DEFINE_GUID(GUID_DEVINTERFACE_VIRT2PHYS,
+	0x539c2135, 0x793a, 0x4926,
+	0xaf, 0xec, 0xd3, 0xa1, 0xb6, 0x1b, 0xbc, 0x8a);
+
+/**
+ * Driver device type for IO control codes.
+ */
+#define VIRT2PHYS_DEVTYPE 0x8000
+
+/**
+ * Translate a valid non-paged virtual address to a physical address.
+ *
+ * Note: A physical address zero (0) is reported if input address
+ * is paged out or not mapped. However, if input is a valid mapping
+ * of I/O port 0x0000, output is also zero. There is no way
+ * to distinguish between these cases by return value only.
+ *
+ * Input: a non-paged virtual address (PVOID).
+ *
+ * Output: the corresponding physical address (LARGE_INTEGER).
+ */
+#define IOCTL_VIRT2PHYS_TRANSLATE CTL_CODE( \
+	VIRT2PHYS_DEVTYPE, 0x800, METHOD_BUFFERED, FILE_ANY_ACCESS)
lib/librte_eal/windows/include/rte_windows.h +2 -0
index ed6e4c148..899ed7d87 100644
@@ -23,6 +23,8 @@
 
 #include <basetsd.h>
 #include <psapi.h>
+#include <setupapi.h>
+#include <winioctl.h>
 
 /* Have GUIDs defined. */
 #ifndef INITGUID
lib/librte_eal/windows/include/unistd.h +3 -0
index 757b7f3c5..6b33005b2 100644
@@ -9,4 +9,7 @@
  * as Microsoft libc does not contain unistd.h. This may be removed
  * in future releases.
  */
+
+#include <io.h>
+
 #endif /* _UNISTD_H_ */
lib/librte_eal/windows/meson.build +6 -0
index 52978e9d7..ded5a2b80 100644
@@ -6,10 +6,16 @@ subdir('include')
 sources += files(
 	'eal.c',
 	'eal_debug.c',
+	'eal_file.c',
 	'eal_hugepages.c',
 	'eal_lcore.c',
 	'eal_log.c',
+	'eal_memalloc.c',
+	'eal_memory.c',
+	'eal_mp.c',
 	'eal_thread.c',
 	'fnmatch.c',
 	'getopt.c',
 )
+
+dpdk_conf.set10('RTE_EAL_NUMA_AWARE_HUGEPAGES', true)
Signature · 1 line
2.25.4
#216 lore raw
In reply to Dmitry Kozlyuk

15/06/2020 02:43, Dmitry Kozlyuk:

Quoted 41 lines

Note for v9:
rte_eal_memory.h renamed, dependent patchsets have to be updated.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing without IOVA is available.

Testing revealed Windows Server 2019 does not allow allocating hugepage memory at a reserved address, despite advertised API. So allocator has to temporary free the region to be allocated. This creates in inherent race condition. This issue is being discussed with Microsoft privately.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications: rte_mem_map, rte_mem_unmap, rte_mem_lock, rte_mem_page_size.

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

To avoid code duplication between Linux and Windows EAL, common code for EALs supporting dynamic memory allocation is extracted (discussed with Anatoly Burakov in v4 thread). This is a separate patch to ease the review, but it can be merged with the previous one.

EAL tracepoints save size_t values as long, which is invalid on Windows. New size_t emitter for tracepoints is introduced (suggested by Jerin Jacob to Fady Bader, see [1]). Also, to avoid workaround in every file using the tracepoints, stubs are added to Windows EAL.

Entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file. The file is imported as-is, so it causes a bunch of checkpatch warnings.

[1]: http://mails.dpdk.org/archives/dev/2020-May/168076.html

[...]

Quoted 5 lines

Dmitry Kozlyuk (12):
eal: replace rte_page_sizes with a set of constants
eal: introduce internal wrappers for file operations
eal: introduce memory management wrappers eal/mem: extract common code for memseg list initialization

Quoted 8 lines

eal/mem: extract common code for dynamic memory allocation
trace: add size_t field emitter
eal/windows: add tracing support stubs
eal/windows: replace sys/queue.h with a complete one from FreeBSD eal/windows: improve CPU and NUMA node detection doc/windows: split build and run instructions eal/windows: initialize hugepage info
eal/windows: implement basic memory management

Applied, thanks for the huge work!

Acked-by on the v9 series
#217 lore raw
In reply to Dmitry Kozlyuk
Quoted 42 lines · Dmitry Kozlyuk

Note for v9:
rte_eal_memory.h renamed, dependent patchsets have to be updated.

This patchset implements basic MM with the following features:

* Hugepages are dynamically allocated in user-mode.
* Only 2MB hugepages are supported.
* IOVA is always PA, obtained through kernel-mode driver.
* No 32-bit support (presumably not demanded).
* Ni multi-process support (it is forcefully disabled).
* No-huge mode for testing without IOVA is available.

Testing revealed Windows Server 2019 does not allow allocating hugepage memory at a reserved address, despite advertised API. So allocator has to temporary free the region to be allocated. This creates in inherent race condition. This issue is being discussed with Microsoft privately.

New EAL public functions for memory mapping are introduced to mitigate OS differences in DPDK libraries and applications: rte_mem_map, rte_mem_unmap, rte_mem_lock, rte_mem_page_size.

To support common MM routines, internal wrappers for low-level memory reservation and file management are introduced. These changes affect Linux and FreeBSD EAL. Shared code is placed unded /unix/ subdirectory (suggested by Thomas).

To avoid code duplication between Linux and Windows EAL, common code for EALs supporting dynamic memory allocation is extracted (discussed with Anatoly Burakov in v4 thread). This is a separate patch to ease the review, but it can be merged with the previous one.

EAL tracepoints save size_t values as long, which is invalid on Windows. New size_t emitter for tracepoints is introduced (suggested by Jerin Jacob to Fady Bader, see [1]). Also, to avoid workaround in every file using the tracepoints, stubs are added to Windows EAL.

Entire <sys/queue.h> is imported from FreeBSD, replacing existing partial import. There is already a license exception for this file. The file is imported as-is, so it causes a bunch of checkpatch warnings.

[1]: http://mails.dpdk.org/archives/dev/2020-May/168076.html

Quoted 9 lines

---

v9:
* Fix build on 32-bit and FreeBSD.
* Rename rte_eal_memory.h to rte_eal_paging.h.
* Do not use rte_panic() in library code.
* Fix typos, comments, string formatting.
* Split documentation commits.

Great work on this, Dmitry!

I know the patch has already been applied, but: