Skip to contentSkip to conversation

Dynamic memory allocation for DPDK

PATCH Superseded Anatoly Burakov 1 participant 44 messages Dec 19, 2017 Network drivers

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Dynamic memory allocation for DPDK

This patchset introduces a prototype implementation of dynamic memory allocation for DPDK. It is intended to start a conversation and build consensus on the best way to implement this functionality. The patchset works well enough to pass all unit tests, and to work with traffic forwarding, provided the device drivers are adjusted to ensure contiguous memory allocation where it matters.

The vast majority of changes are in the EAL and malloc, the external API disruption is minimal: a new set of API's are added for contiguous memory allocation (for rte_malloc and rte_memzone), and a few API additions in rte_memory. Every other API change is internal to EAL, and all of the memory allocation/freeing is handled through rte_malloc, with no externally visible API changes, aside from a call to get physmem layout, which no longer makes sense given that there are multiple memseg lists.

Quick outline of all changes done as part of this patchset:

* Malloc heap adjusted to handle holes in address space
* Single memseg list replaced by multiple expandable memseg lists
* VA space for hugepages is preallocated in advance
* Added dynamic alloc/free for pages, happening as needed on malloc/free
* Added contiguous memory allocation API's for rte_malloc and rte_memzone
* Integrated Pawel Wodkowski's patch [1] for registering/unregistering memory with VFIO

The biggest difference is a "memseg" now represents a single page (as opposed to being a big contiguous block of pages). As a consequence, both memzones and malloc elements are no longer guaranteed to be physically contiguous, unless the user asks for it. To preserve whatever functionality that was dependent on previous behavior, a legacy memory option is also provided, however it is expected to be temporary solution. The drivers weren't adjusted in this patchset, and it is expected that whoever shall test the drivers with this patchset will modify their relevant drivers to support the new set of API's. Basic testing with forwarding traffic was performed, both with UIO and VFIO, and no performance degradation was observed.

Why multiple memseg lists instead of one? It makes things easier on a number of fronts. Since memseg is a single page now, the list will get quite big, and we need to locate pages somehow when we allocate and free them. We could of course just walk the list and allocate one contiguous chunk of VA space for memsegs, but i chose to use separate lists instead, to speed up many operations with the list.

It would be great to see the following discussions within the community regarding both current implementation and future work:

* Any suggestions to improve current implementation. The whole system with multiple memseg lists is kind of unweildy, so maybe there are better ways to do the same thing. Maybe use a single list after all? We're not expecting malloc/free on hot path, so maybe it doesn't matter that we have to walk the list of potentially thousands of pages?
* Pluggable memory allocators. Right now, allocators are hardcoded, but down the line it would be great to have custom allocators (e.g. for externally allocated memory). I've tried to keep the memalloc API minimal and generic enough to be able to easily change it down the line, but suggestions are welcome. Memory drivers, with ops for alloc/free etc.?
* Memory tagging. This is related to previous item. Right now, we can only ask malloc to allocate memory by page size, but one could potentially have different memory regions backed by pages of similar sizes (for example, locked 1G pages, to completely avoid TLB misses, alongside regular 1G pages), and it would be good to have that kind of mechanism to distinguish between different memory types available to a DPDK application. One could, for example, tag memory by "purpose" (i.e. "fast", "slow"), or in other ways.
* Secondary process implementation, in particular when it comes to allocating/ freeing new memory. Current plan is to make use of RPC mechanism proposed by Jianfeng [2] to communicate between primary and secondary processes, however other suggestions are welcome.
* Support for non-hugepage memory. This work is planned down the line. Aside from obvious concerns about physical addresses, 4K pages are small and will eat up enormous amounts of memseg list space, so my proposal would be to allocate 4K pages in bigger blocks (say, 2MB).
* 32-bit support. Current implementation lacks it, and i don't see a trivial way to make it work if we are to preallocate huge chunks of VA space in advance. We could limit it to 1G per page size, but even that, on multiple sockets, won't work that well, and we can't know in advance what kind of memory user will try to allocate. Drop it? Leave it in legacy mode only?
* Preallocation. Right now, malloc will free any and all memory that it can, which could lead to a (perhaps counterintuitive) situation where a user calls DPDK with --socket-mem=1024,1024, does a single "rte_free" and loses all of the preallocated memory in the process. Would preallocating memory *and keeping it no matter what* be a valid use case? E.g. if DPDK was run without any memory requirements specified, grow and shrink as needed, but DPDK was asked to preallocate memory, we can grow but we can't shrink past the preallocated amount?

Any other feedback about things i didn't think of or missed is greatly appreciated.

[1] http://dpdk.org/dev/patchwork/patch/24484/
[2] http://dpdk.org/dev/patchwork/patch/31838/

Anatoly Burakov (23):
eal: move get_virtual_area out of linuxapp eal_memory.c
eal: add function to report number of detected sockets
eal: add rte_fbarray
eal: move all locking to heap
eal: protect malloc heap stats with a lock
eal: make malloc a doubly-linked list
eal: make malloc_elem_join_adjacent_free public
eal: add "single file segments" command-line option
eal: add "legacy memory" option
eal: read hugepage counts from node-specific sysfs path
eal: replace memseg with memseg lists
eal: add support for dynamic memory allocation
eal: make use of dynamic memory allocation for init
eal: add support for dynamic unmapping of pages
eal: add API to check if memory is physically contiguous
eal: enable dynamic memory allocation/free on malloc/free
eal: add backend support for contiguous memory allocation
eal: add rte_malloc support for allocating contiguous memory
eal: enable reserving physically contiguous memzones
eal: make memzones use rte_fbarray
mempool: add support for the new memory allocation methods
vfio: allow to map other memory regions
eal: map/unmap memory with VFIO when alloc/free pages

 config/common_base                                |   5 +-
 drivers/bus/pci/linux/pci.c                       |  29 +-
 drivers/net/ena/ena_ethdev.c                      |  10 +-
 drivers/net/virtio/virtio_user/vhost_kernel.c     | 106 ++--
 lib/librte_eal/common/Makefile                    |   2 +-
 lib/librte_eal/common/eal_common_fbarray.c        | 585 ++++++++++++++++++++++
 lib/librte_eal/common/eal_common_lcore.c          |  11 +
 lib/librte_eal/common/eal_common_memalloc.c       |  79 +++
 lib/librte_eal/common/eal_common_memory.c         | 315 +++++++++++-
 lib/librte_eal/common/eal_common_memzone.c        | 250 ++++++---
 lib/librte_eal/common/eal_common_options.c        |   8 +
 lib/librte_eal/common/eal_filesystem.h            |  13 +
 lib/librte_eal/common/eal_hugepages.h             |   1 +
 lib/librte_eal/common/eal_internal_cfg.h          |   6 +
 lib/librte_eal/common/eal_memalloc.h              |  55 ++
 lib/librte_eal/common/eal_options.h               |   4 +
 lib/librte_eal/common/eal_private.h               |  29 ++
 lib/librte_eal/common/include/rte_eal.h           |   1 +
 lib/librte_eal/common/include/rte_eal_memconfig.h |  26 +-
 lib/librte_eal/common/include/rte_fbarray.h       |  98 ++++
 lib/librte_eal/common/include/rte_lcore.h         |   8 +
 lib/librte_eal/common/include/rte_malloc.h        | 181 +++++++
 lib/librte_eal/common/include/rte_malloc_heap.h   |   6 +
 lib/librte_eal/common/include/rte_memory.h        |  16 +
 lib/librte_eal/common/include/rte_memzone.h       | 158 ++++++
 lib/librte_eal/common/malloc_elem.c               | 411 ++++++++++++---
 lib/librte_eal/common/malloc_elem.h               |  30 +-
 lib/librte_eal/common/malloc_heap.c               | 433 ++++++++++++++--
 lib/librte_eal/common/malloc_heap.h               |  14 +-
 lib/librte_eal/common/rte_malloc.c                | 139 +++--
 lib/librte_eal/linuxapp/eal/Makefile              |   4 +
 lib/librte_eal/linuxapp/eal/eal.c                 |  23 +-
 lib/librte_eal/linuxapp/eal/eal_hugepage_info.c   |  73 ++-
 lib/librte_eal/linuxapp/eal/eal_memalloc.c        | 556 ++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal_memory.c          | 452 ++++++++++-------
 lib/librte_eal/linuxapp/eal/eal_vfio.c            | 280 ++++++++---
 lib/librte_eal/linuxapp/eal/eal_vfio.h            |  11 +
 lib/librte_mempool/rte_mempool.c                  |  84 +++-
 test/test/test_malloc.c                           |  29 +-
 test/test/test_memory.c                           |  44 +-
 test/test/test_memzone.c                          |  17 +-
 41 files changed, 3999 insertions(+), 603 deletions(-)
 create mode 100755 lib/librte_eal/common/eal_common_fbarray.c
 create mode 100755 lib/librte_eal/common/eal_common_memalloc.c
 create mode 100755 lib/librte_eal/common/eal_memalloc.h
 create mode 100755 lib/librte_eal/common/include/rte_fbarray.h
 create mode 100755 lib/librte_eal/linuxapp/eal/eal_memalloc.c
Signature · 1 line
2.7.4
#1 lore raw

RFC 1/23 eal/memory: move get_virtual_area out of linuxapp eal_memory.c

Move get_virtual_area out of linuxapp EAL memory and make it common to EAL, so that other code could reserve virtual areas as well.

---
 lib/librte_eal/common/eal_common_memory.c | 70 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_private.h       | 29 +++++++++++++
 lib/librte_eal/linuxapp/eal/eal_memory.c  | 71 ++-----------------------------
 3 files changed, 102 insertions(+), 68 deletions(-)
lib/librte_eal/common/eal_common_memory.c +70 -0
index fc6c44d..96570a7 100644
@@ -31,6 +31,8 @@
  *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
+#include <errno.h>
+#include <string.h>
 #include <stdio.h>
 #include <stdint.h>
 #include <stdlib.h>
@@ -49,6 +51,74 @@
 #include "eal_internal_cfg.h"
 
 /*
+ * Try to mmap *size bytes in /dev/zero. If it is successful, return the
+ * pointer to the mmap'd area and keep *size unmodified. Else, retry
+ * with a smaller zone: decrease *size by hugepage_sz until it reaches
+ * 0. In this case, return NULL. Note: this function returns an address
+ * which is a multiple of hugepage size.
+ */
+
+static uint64_t baseaddr_offset;
+
+void *
+eal_get_virtual_area(void *requested_addr, uint64_t *size,
+		uint64_t page_sz, int flags)
+{
+	bool addr_is_hint, allow_shrink;
+	void *addr;
+
+	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
+
+	addr_is_hint = (flags & EAL_VIRTUAL_AREA_ADDR_IS_HINT) > 0;
+	allow_shrink = (flags & EAL_VIRTUAL_AREA_ALLOW_SHRINK) > 0;
+
+	if (requested_addr == NULL && internal_config.base_virtaddr != 0) {
+		requested_addr = (void*) (internal_config.base_virtaddr +
+				baseaddr_offset);
+		addr_is_hint = true;
+	}
+
+	do {
+		// TODO: we may not necessarily be using memory mapped by this
+		// function for hugepage mapping, so... HUGETLB flag?
+
+		addr = mmap(requested_addr,
+				(*size) + page_sz, PROT_READ,
+#ifdef RTE_ARCH_PPC_64
+				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
+#else
+				MAP_PRIVATE | MAP_ANONYMOUS,
+#endif
+				-1, 0);
+		if (addr == MAP_FAILED && allow_shrink)
+			*size -= page_sz;
+	} while (allow_shrink && addr == MAP_FAILED && *size > 0);
+
+	if (addr == MAP_FAILED) {
+		RTE_LOG(ERR, EAL, "Cannot get a virtual area: %s\n",
+			strerror(errno));
+		return NULL;
+	} else if (requested_addr != NULL && !addr_is_hint &&
+			addr != requested_addr) {
+		RTE_LOG(ERR, EAL, "Cannot get a virtual area at requested address: %p\n",
+			requested_addr);
+		munmap(addr, (*size) + page_sz);
+		return NULL;
+	}
+
+	/* align addr to page size boundary */
+	addr = RTE_PTR_ALIGN(addr, page_sz);
+
+	RTE_LOG(DEBUG, EAL, "Virtual area found at %p (size = 0x%zx)\n",
+		addr, *size);
+
+	baseaddr_offset += *size;
+
+	return addr;
+}
+
+
+/*
  * Return a pointer to a read-only table of struct rte_physmem_desc
  * elements, containing the layout of all addressable physical
  * memory. The last element of the table contains a NULL address.
lib/librte_eal/common/eal_private.h +29 -0
index 462226f..5d57fc1 100644
@@ -34,6 +34,7 @@
 #ifndef _EAL_PRIVATE_H_
 #define _EAL_PRIVATE_H_
 
+#include <stdint.h>
 #include <stdbool.h>
 #include <stdint.h>
 #include <stdio.h>
@@ -224,4 +225,32 @@ int rte_eal_hugepage_attach(void);
  */
 struct rte_bus *rte_bus_find_by_device_name(const char *str);
 
+/**
+ * Get virtual area of specified size from the OS.
+ *
+ * This function is private to the EAL.
+ *
+ * @param requested_addr
+ *   Address where to request address space.
+ * @param size
+ *   Size of requested area.
+ * @param page_sz
+ *   Page size on which to align requested virtual area.
+ * @param flags
+ *   EAL_VIRTUAL_AREA_* flags.
+ *
+ * @return
+ *   Virtual area address if successful.
+ *   NULL if unsuccessful.
+ */
+
+#define EAL_VIRTUAL_AREA_ADDR_IS_HINT 0x1
+/**< don't fail if cannot get exact requested address */
+#define EAL_VIRTUAL_AREA_ALLOW_SHRINK 0x2
+/**< try getting smaller sized (decrement by page size) virtual areas if cannot
+ * get area of requested size. */
+void *
+eal_get_virtual_area(void *requested_addr, uint64_t *size,
+		uint64_t page_sz, int flags);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/linuxapp/eal/eal_memory.c +3 -68

89 lines not on this page · raw message

Signature · 1 line
2.7.4
#2 lore raw

Re: [RFC 01/23] eal/memory: move get_virtual_area out of linuxapp eal_memory.c

Quoted 6 lines · Anatoly Burakov

Move get_virtual_area out of linuxapp EAL memory and make it common to EAL, so that other code could reserve virtual areas as well.

---

My apologies, this patchset was sent erroneously. Please look at the version tagged as "RFC v2".

Signature · 2 lines
Thanks,
Anatoly
#3 lore raw

PATCH 1/23 eal: move get_virtual_area out of linuxapp eal_memory.c

In reply to Anatoly Burakov

Move get_virtual_area out of linuxapp EAL memory and make it common to EAL, so that other code could reserve virtual areas as well.

---
 lib/librte_eal/common/eal_common_memory.c | 70 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_private.h       | 29 +++++++++++++
 lib/librte_eal/linuxapp/eal/eal_memory.c  | 71 ++-----------------------------
 3 files changed, 102 insertions(+), 68 deletions(-)
lib/librte_eal/common/eal_common_memory.c +70 -0
index fc6c44d..96570a7 100644
@@ -31,6 +31,8 @@
  *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  */
 
+#include <errno.h>
+#include <string.h>
 #include <stdio.h>
 #include <stdint.h>
 #include <stdlib.h>
@@ -49,6 +51,74 @@
 #include "eal_internal_cfg.h"
 
 /*
+ * Try to mmap *size bytes in /dev/zero. If it is successful, return the
+ * pointer to the mmap'd area and keep *size unmodified. Else, retry
+ * with a smaller zone: decrease *size by hugepage_sz until it reaches
+ * 0. In this case, return NULL. Note: this function returns an address
+ * which is a multiple of hugepage size.
+ */
+
+static uint64_t baseaddr_offset;
+
+void *
+eal_get_virtual_area(void *requested_addr, uint64_t *size,
+		uint64_t page_sz, int flags)
+{
+	bool addr_is_hint, allow_shrink;
+	void *addr;
+
+	RTE_LOG(DEBUG, EAL, "Ask a virtual area of 0x%zx bytes\n", *size);
+
+	addr_is_hint = (flags & EAL_VIRTUAL_AREA_ADDR_IS_HINT) > 0;
+	allow_shrink = (flags & EAL_VIRTUAL_AREA_ALLOW_SHRINK) > 0;
+
+	if (requested_addr == NULL && internal_config.base_virtaddr != 0) {
+		requested_addr = (void*) (internal_config.base_virtaddr +
+				baseaddr_offset);
+		addr_is_hint = true;
+	}
+
+	do {
+		// TODO: we may not necessarily be using memory mapped by this
+		// function for hugepage mapping, so... HUGETLB flag?
+
+		addr = mmap(requested_addr,
+				(*size) + page_sz, PROT_READ,
+#ifdef RTE_ARCH_PPC_64
+				MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
+#else
+				MAP_PRIVATE | MAP_ANONYMOUS,
+#endif
+				-1, 0);
+		if (addr == MAP_FAILED && allow_shrink)
+			*size -= page_sz;
+	} while (allow_shrink && addr == MAP_FAILED && *size > 0);
+
+	if (addr == MAP_FAILED) {
+		RTE_LOG(ERR, EAL, "Cannot get a virtual area: %s\n",
+			strerror(errno));
+		return NULL;
+	} else if (requested_addr != NULL && !addr_is_hint &&
+			addr != requested_addr) {
+		RTE_LOG(ERR, EAL, "Cannot get a virtual area at requested address: %p\n",
+			requested_addr);
+		munmap(addr, (*size) + page_sz);
+		return NULL;
+	}
+
+	/* align addr to page size boundary */
+	addr = RTE_PTR_ALIGN(addr, page_sz);
+
+	RTE_LOG(DEBUG, EAL, "Virtual area found at %p (size = 0x%zx)\n",
+		addr, *size);
+
+	baseaddr_offset += *size;
+
+	return addr;
+}
+
+
+/*
  * Return a pointer to a read-only table of struct rte_physmem_desc
  * elements, containing the layout of all addressable physical
  * memory. The last element of the table contains a NULL address.
lib/librte_eal/common/eal_private.h +29 -0
index 462226f..5d57fc1 100644
@@ -34,6 +34,7 @@
 #ifndef _EAL_PRIVATE_H_
 #define _EAL_PRIVATE_H_
 
+#include <stdint.h>
 #include <stdbool.h>
 #include <stdint.h>
 #include <stdio.h>
@@ -224,4 +225,32 @@ int rte_eal_hugepage_attach(void);
  */
 struct rte_bus *rte_bus_find_by_device_name(const char *str);
 
+/**
+ * Get virtual area of specified size from the OS.
+ *
+ * This function is private to the EAL.
+ *
+ * @param requested_addr
+ *   Address where to request address space.
+ * @param size
+ *   Size of requested area.
+ * @param page_sz
+ *   Page size on which to align requested virtual area.
+ * @param flags
+ *   EAL_VIRTUAL_AREA_* flags.
+ *
+ * @return
+ *   Virtual area address if successful.
+ *   NULL if unsuccessful.
+ */
+
+#define EAL_VIRTUAL_AREA_ADDR_IS_HINT 0x1
+/**< don't fail if cannot get exact requested address */
+#define EAL_VIRTUAL_AREA_ALLOW_SHRINK 0x2
+/**< try getting smaller sized (decrement by page size) virtual areas if cannot
+ * get area of requested size. */
+void *
+eal_get_virtual_area(void *requested_addr, uint64_t *size,
+		uint64_t page_sz, int flags);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/linuxapp/eal/eal_memory.c +3 -68

89 lines not on this page · raw message

Signature · 1 line
2.7.4
#4 lore raw

PATCH 2/23 eal: add function to report number of detected sockets

In reply to Anatoly Burakov

At the moment, we always rely on scanning everything for every socket up until RTE_MAX_NUMA_NODES and checking if there's a memseg associated with each socket if we want to find out how many sockets we actually have. This becomes a problem when we may have memory on socket but it's not allocated yet, so we do the detection on lcore scan instead, and store the value for later use.

---
 lib/librte_eal/common/eal_common_lcore.c  | 11 +++++++++++
 lib/librte_eal/common/include/rte_eal.h   |  1 +
 lib/librte_eal/common/include/rte_lcore.h |  8 ++++++++
 3 files changed, 20 insertions(+)
lib/librte_eal/common/eal_common_lcore.c +11 -0
index 0db1555..7566a6b 100644
@@ -57,6 +57,7 @@ rte_eal_cpu_init(void)
 	struct rte_config *config = rte_eal_get_configuration();
 	unsigned lcore_id;
 	unsigned count = 0;
+	unsigned max_socket_id = 0;
 
 	/*
 	 * Parse the maximum set of logical cores, detect the subset of running
@@ -100,6 +101,8 @@ rte_eal_cpu_init(void)
 				lcore_id, lcore_config[lcore_id].core_id,
 				lcore_config[lcore_id].socket_id);
 		count++;
+		max_socket_id = RTE_MAX(max_socket_id,
+					lcore_config[lcore_id].socket_id);
 	}
 	/* Set the count of enabled logical cores of the EAL configuration */
 	config->lcore_count = count;
@@ -108,5 +111,13 @@ rte_eal_cpu_init(void)
 		RTE_MAX_LCORE);
 	RTE_LOG(INFO, EAL, "Detected %u lcore(s)\n", config->lcore_count);
 
+	config->numa_node_count = max_socket_id + 1;
+	RTE_LOG(INFO, EAL, "Detected %u NUMA nodes\n", config->numa_node_count);
+
 	return 0;
 }
+
+unsigned rte_num_sockets(void) {
+	const struct rte_config *config = rte_eal_get_configuration();
+	return config->numa_node_count;
+}
lib/librte_eal/common/include/rte_eal.h +1 -0
index 8e4e71c..5b12914 100644
@@ -83,6 +83,7 @@ enum rte_proc_type_t {
 struct rte_config {
 	uint32_t master_lcore;       /**< Id of the master lcore */
 	uint32_t lcore_count;        /**< Number of available logical cores. */
+	uint32_t numa_node_count;    /**< Number of detected NUMA nodes. */
 	uint32_t service_lcore_count;/**< Number of available service cores. */
 	enum rte_lcore_role_t lcore_role[RTE_MAX_LCORE]; /**< State of cores. */
 
lib/librte_eal/common/include/rte_lcore.h +8 -0
index c89e6ba..6a75c9b 100644
@@ -148,6 +148,14 @@ rte_lcore_index(int lcore_id)
 unsigned rte_socket_id(void);
 
 /**
+ * Return number of physical sockets on the system.
+ * @return
+ *   the number of physical sockets as recognized by EAL
+ *
+ */
+unsigned rte_num_sockets(void);
+
+/**
  * Get the ID of the physical socket of the specified lcore
  *
  * @param lcore_id
Signature · 1 line
2.7.4
#5 lore raw

RFC 2/23 eal/lcore: add function to report number of detected sockets

In reply to Anatoly Burakov

At the moment, we always rely on scanning everything for every socket up until RTE_MAX_NUMA_NODES and checking if there's a memseg associated with each socket if we want to find out how many sockets we actually have. This becomes a problem when we may have memory on socket but it's not allocated yet, so we do the detection on lcore scan instead, and store the value for later use.

---
 lib/librte_eal/common/eal_common_lcore.c  | 11 +++++++++++
 lib/librte_eal/common/include/rte_eal.h   |  1 +
 lib/librte_eal/common/include/rte_lcore.h |  8 ++++++++
 3 files changed, 20 insertions(+)
lib/librte_eal/common/eal_common_lcore.c +11 -0
index 0db1555..7566a6b 100644
@@ -57,6 +57,7 @@ rte_eal_cpu_init(void)
 	struct rte_config *config = rte_eal_get_configuration();
 	unsigned lcore_id;
 	unsigned count = 0;
+	unsigned max_socket_id = 0;
 
 	/*
 	 * Parse the maximum set of logical cores, detect the subset of running
@@ -100,6 +101,8 @@ rte_eal_cpu_init(void)
 				lcore_id, lcore_config[lcore_id].core_id,
 				lcore_config[lcore_id].socket_id);
 		count++;
+		max_socket_id = RTE_MAX(max_socket_id,
+					lcore_config[lcore_id].socket_id);
 	}
 	/* Set the count of enabled logical cores of the EAL configuration */
 	config->lcore_count = count;
@@ -108,5 +111,13 @@ rte_eal_cpu_init(void)
 		RTE_MAX_LCORE);
 	RTE_LOG(INFO, EAL, "Detected %u lcore(s)\n", config->lcore_count);
 
+	config->numa_node_count = max_socket_id + 1;
+	RTE_LOG(INFO, EAL, "Detected %u NUMA nodes\n", config->numa_node_count);
+
 	return 0;
 }
+
+unsigned rte_num_sockets(void) {
+	const struct rte_config *config = rte_eal_get_configuration();
+	return config->numa_node_count;
+}
lib/librte_eal/common/include/rte_eal.h +1 -0
index 8e4e71c..5b12914 100644
@@ -83,6 +83,7 @@ enum rte_proc_type_t {
 struct rte_config {
 	uint32_t master_lcore;       /**< Id of the master lcore */
 	uint32_t lcore_count;        /**< Number of available logical cores. */
+	uint32_t numa_node_count;    /**< Number of detected NUMA nodes. */
 	uint32_t service_lcore_count;/**< Number of available service cores. */
 	enum rte_lcore_role_t lcore_role[RTE_MAX_LCORE]; /**< State of cores. */
 
lib/librte_eal/common/include/rte_lcore.h +8 -0
index c89e6ba..6a75c9b 100644
@@ -148,6 +148,14 @@ rte_lcore_index(int lcore_id)
 unsigned rte_socket_id(void);
 
 /**
+ * Return number of physical sockets on the system.
+ * @return
+ *   the number of physical sockets as recognized by EAL
+ *
+ */
+unsigned rte_num_sockets(void);
+
+/**
  * Get the ID of the physical socket of the specified lcore
  *
  * @param lcore_id
Signature · 1 line
2.7.4
#6 lore raw

PATCH 3/23 eal: add rte_fbarray

In reply to Anatoly Burakov

rte_fbarray is a simple resizable array, not unlike vectors in higher-level languages. Rationale for its existence is the
following: since we are going to map memory page-by-page, there could be quite a lot of memory segments to keep track of (for smaller page sizes, page count can easily reach thousands). We can't really make page lists truly dynamic and infinitely expandable, because that involves reallocating memory (which is a big no-no in multiprocess). What we can do instead is have a maximum capacity as something really, really large, and preallocate address space for that, but only use a small portion of that memory as needed, via mmap()'ing portions of the address space to an actual file. This also doubles as a mechanism to share fbarrays between processes (although multiprocess is neither implemented nor tested at the moment). Hence the name: file-backed array.

In addition, in understanding that we will frequently need to scan this array for free space and iterating over array linearly can become slow, rte_fbarray provides facilities to index array's usage. The following use cases are covered:
- find next free/used slot (useful either for adding new elements to fbarray, or walking the list)
- find starting index for next N free/used slots (useful for when we want to allocate chunk of VA-contiguous memory composed of several pages)
- find how many contiguous free/used slots there are, starting from specified index (useful for when we want to figure out how many pages we have until next hole in allocated memory, to speed up some bulk operations where we would otherwise have to walk the array and add pages one by one)

This is accomplished by storing a usage mask in-memory, right after the data section of the array, and using some bit-level magic to figure out the info we need.

rte_fbarray is a bit clunky to use and its primary purpose is to be used within EAL for certain things, but hopefully it is (or can be made so) generic enough to be useful in other contexts.

Note that current implementation is leaking fd's whenever new allocations happen.

---
 lib/librte_eal/common/Makefile              |   2 +-
 lib/librte_eal/common/eal_common_fbarray.c  | 585 ++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_filesystem.h      |  13 +
 lib/librte_eal/common/include/rte_fbarray.h |  98 +++++
 lib/librte_eal/linuxapp/eal/Makefile        |   1 +
 5 files changed, 698 insertions(+), 1 deletion(-)
 create mode 100755 lib/librte_eal/common/eal_common_fbarray.c
 create mode 100755 lib/librte_eal/common/include/rte_fbarray.h
lib/librte_eal/common/Makefile +1 -1
index 9effd0d..7868698 100644
@@ -43,7 +43,7 @@ INC += rte_hexdump.h rte_devargs.h rte_bus.h rte_dev.h
 INC += rte_pci_dev_feature_defs.h rte_pci_dev_features.h
 INC += rte_malloc.h rte_keepalive.h rte_time.h
 INC += rte_service.h rte_service_component.h
-INC += rte_bitmap.h rte_vfio.h
+INC += rte_bitmap.h rte_vfio.h rte_fbarray.h
 
 GENERIC_INC := rte_atomic.h rte_byteorder.h rte_cycles.h rte_prefetch.h
 GENERIC_INC += rte_spinlock.h rte_memcpy.h rte_cpuflags.h rte_rwlock.h
lib/librte_eal/common/eal_common_fbarray.c new file +585 -0

585 lines not on this page · raw message

lib/librte_eal/common/eal_filesystem.h +13 -0
index 8acbd99..10e7474 100644
@@ -42,6 +42,7 @@
 
 /** Path of rte config file. */
 #define RUNTIME_CONFIG_FMT "%s/.%s_config"
+#define FBARRAY_FMT "%s/%s_%s"
 
 #include <stdint.h>
 #include <limits.h>
@@ -67,6 +68,18 @@ eal_runtime_config_path(void)
 	return buffer;
 }
 
+static inline const char *
+eal_get_fbarray_path(char *buffer, size_t buflen, const char *name) {
+	const char *directory = default_config_dir;
+	const char *home_dir = getenv("HOME");
+
+	if (getuid() != 0 && home_dir != NULL)
+		directory = home_dir;
+	snprintf(buffer, buflen - 1, FBARRAY_FMT, directory,
+			internal_config.hugefile_prefix, name);
+	return buffer;
+}
+
 /** Path of hugepage info file. */
 #define HUGEPAGE_INFO_FMT "%s/.%s_hugepage_info"
 
lib/librte_eal/common/include/rte_fbarray.h new file +98 -0

98 lines not on this page · raw message

lib/librte_eal/linuxapp/eal/Makefile +1 -0
index 5a7b8b2..782e1ad 100644
@@ -86,6 +86,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_dev.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_options.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
Signature · 1 line
2.7.4
#7 lore raw

PATCH 4/23 eal: move all locking to heap

In reply to Anatoly Burakov

Down the line, we will need to do everything from the heap as any alloc or free may trigger alloc/free OS memory, which would involve growing/shrinking heap.

---
 lib/librte_eal/common/malloc_elem.c | 16 ++--------------
 lib/librte_eal/common/malloc_heap.c | 36 ++++++++++++++++++++++++++++++++++++
 lib/librte_eal/common/malloc_heap.h |  6 ++++++
 lib/librte_eal/common/rte_malloc.c  |  4 ++--
 4 files changed, 46 insertions(+), 16 deletions(-)
lib/librte_eal/common/malloc_elem.c +2 -14
index 98bcd37..6b4f2a5 100644
@@ -271,10 +271,6 @@ join_elem(struct malloc_elem *elem1, struct malloc_elem *elem2)
 int
 malloc_elem_free(struct malloc_elem *elem)
 {
-	if (!malloc_elem_cookies_ok(elem) || elem->state != ELEM_BUSY)
-		return -1;
-
-	rte_spinlock_lock(&(elem->heap->lock));
 	size_t sz = elem->size - sizeof(*elem) - MALLOC_ELEM_TRAILER_LEN;
 	uint8_t *ptr = (uint8_t *)&elem[1];
 	struct malloc_elem *next = RTE_PTR_ADD(elem, elem->size);
@@ -302,8 +298,6 @@ malloc_elem_free(struct malloc_elem *elem)
 
 	memset(ptr, 0, sz);
 
-	rte_spinlock_unlock(&(elem->heap->lock));
-
 	return 0;
 }
 
@@ -320,11 +314,10 @@ malloc_elem_resize(struct malloc_elem *elem, size_t size)
 		return 0;
 
 	struct malloc_elem *next = RTE_PTR_ADD(elem, elem->size);
-	rte_spinlock_lock(&elem->heap->lock);
 	if (next ->state != ELEM_FREE)
-		goto err_return;
+		return -1;
 	if (elem->size + next->size < new_size)
-		goto err_return;
+		return -1;
 
 	/* we now know the element fits, so remove from free list,
 	 * join the two
@@ -339,10 +332,5 @@ malloc_elem_resize(struct malloc_elem *elem, size_t size)
 		split_elem(elem, split_pt);
 		malloc_elem_free_list_insert(split_pt);
 	}
-	rte_spinlock_unlock(&elem->heap->lock);
 	return 0;
-
-err_return:
-	rte_spinlock_unlock(&elem->heap->lock);
-	return -1;
 }
lib/librte_eal/common/malloc_heap.c +36 -0
index 267a4c6..099e448 100644
@@ -174,6 +174,42 @@ malloc_heap_alloc(struct malloc_heap *heap,
 	return elem == NULL ? NULL : (void *)(&elem[1]);
 }
 
+int
+malloc_heap_free(struct malloc_elem *elem) {
+	struct malloc_heap *heap;
+	int ret;
+
+	if (!malloc_elem_cookies_ok(elem) || elem->state != ELEM_BUSY)
+		return -1;
+
+	/* elem may be merged with previous element, so keep heap address */
+	heap = elem->heap;
+
+	rte_spinlock_lock(&(heap->lock));
+
+	ret = malloc_elem_free(elem);
+
+	rte_spinlock_unlock(&(heap->lock));
+
+	return ret;
+}
+
+int
+malloc_heap_resize(struct malloc_elem *elem, size_t size) {
+	int ret;
+
+	if (!malloc_elem_cookies_ok(elem) || elem->state != ELEM_BUSY)
+		return -1;
+
+	rte_spinlock_lock(&(elem->heap->lock));
+
+	ret = malloc_elem_resize(elem, size);
+
+	rte_spinlock_unlock(&(elem->heap->lock));
+
+	return ret;
+}
+
 /*
  * Function to retrieve data for heap on given socket
  */
lib/librte_eal/common/malloc_heap.h +6 -0
index 3ccbef0..3767ef3 100644
@@ -57,6 +57,12 @@ malloc_heap_alloc(struct malloc_heap *heap,	const char *type, size_t size,
 		unsigned flags, size_t align, size_t bound);
 
 int
+malloc_heap_free(struct malloc_elem *elem);
+
+int
+malloc_heap_resize(struct malloc_elem *elem, size_t size);
+
+int
 malloc_heap_get_stats(const struct malloc_heap *heap,
 		struct rte_malloc_socket_stats *socket_stats);
 
lib/librte_eal/common/rte_malloc.c +2 -2
index fe2278b..74b5417 100644
@@ -58,7 +58,7 @@
 void rte_free(void *addr)
 {
 	if (addr == NULL) return;
-	if (malloc_elem_free(malloc_elem_from_data(addr)) < 0)
+	if (malloc_heap_free(malloc_elem_from_data(addr)) < 0)
 		rte_panic("Fatal error: Invalid memory\n");
 }
 
@@ -169,7 +169,7 @@ rte_realloc(void *ptr, size_t size, unsigned align)
 	size = RTE_CACHE_LINE_ROUNDUP(size), align = RTE_CACHE_LINE_ROUNDUP(align);
 	/* check alignment matches first, and if ok, see if we can resize block */
 	if (RTE_PTR_ALIGN(ptr,align) == ptr &&
-			malloc_elem_resize(elem, size) == 0)
+			malloc_heap_resize(elem, size) == 0)
 		return ptr;
 
 	/* either alignment is off, or we have no room to expand,
Signature · 1 line
2.7.4
#8 lore raw

RFC 4/23 eal/malloc: move all locking to heap

In reply to Anatoly Burakov

Down the line, we will need to do everything from the heap as any alloc or free may trigger alloc/free OS memory, which would involve growing/shrinking heap.

---
 lib/librte_eal/common/malloc_elem.c | 16 ++--------------
 lib/librte_eal/common/malloc_heap.c | 36 ++++++++++++++++++++++++++++++++++++
 lib/librte_eal/common/malloc_heap.h |  6 ++++++
 lib/librte_eal/common/rte_malloc.c  |  4 ++--
 4 files changed, 46 insertions(+), 16 deletions(-)
lib/librte_eal/common/malloc_elem.c +2 -14
index 98bcd37..6b4f2a5 100644
@@ -271,10 +271,6 @@ join_elem(struct malloc_elem *elem1, struct malloc_elem *elem2)
 int
 malloc_elem_free(struct malloc_elem *elem)
 {
-	if (!malloc_elem_cookies_ok(elem) || elem->state != ELEM_BUSY)
-		return -1;
-
-	rte_spinlock_lock(&(elem->heap->lock));
 	size_t sz = elem->size - sizeof(*elem) - MALLOC_ELEM_TRAILER_LEN;
 	uint8_t *ptr = (uint8_t *)&elem[1];
 	struct malloc_elem *next = RTE_PTR_ADD(elem, elem->size);
@@ -302,8 +298,6 @@ malloc_elem_free(struct malloc_elem *elem)
 
 	memset(ptr, 0, sz);
 
-	rte_spinlock_unlock(&(elem->heap->lock));
-
 	return 0;
 }
 
@@ -320,11 +314,10 @@ malloc_elem_resize(struct malloc_elem *elem, size_t size)
 		return 0;
 
 	struct malloc_elem *next = RTE_PTR_ADD(elem, elem->size);
-	rte_spinlock_lock(&elem->heap->lock);
 	if (next ->state != ELEM_FREE)
-		goto err_return;
+		return -1;
 	if (elem->size + next->size < new_size)
-		goto err_return;
+		return -1;
 
 	/* we now know the element fits, so remove from free list,
 	 * join the two
@@ -339,10 +332,5 @@ malloc_elem_resize(struct malloc_elem *elem, size_t size)
 		split_elem(elem, split_pt);
 		malloc_elem_free_list_insert(split_pt);
 	}
-	rte_spinlock_unlock(&elem->heap->lock);
 	return 0;
-
-err_return:
-	rte_spinlock_unlock(&elem->heap->lock);
-	return -1;
 }
lib/librte_eal/common/malloc_heap.c +36 -0
index 267a4c6..099e448 100644
@@ -174,6 +174,42 @@ malloc_heap_alloc(struct malloc_heap *heap,
 	return elem == NULL ? NULL : (void *)(&elem[1]);
 }
 
+int
+malloc_heap_free(struct malloc_elem *elem) {
+	struct malloc_heap *heap;
+	int ret;
+
+	if (!malloc_elem_cookies_ok(elem) || elem->state != ELEM_BUSY)
+		return -1;
+
+	/* elem may be merged with previous element, so keep heap address */
+	heap = elem->heap;
+
+	rte_spinlock_lock(&(heap->lock));
+
+	ret = malloc_elem_free(elem);
+
+	rte_spinlock_unlock(&(heap->lock));
+
+	return ret;
+}
+
+int
+malloc_heap_resize(struct malloc_elem *elem, size_t size) {
+	int ret;
+
+	if (!malloc_elem_cookies_ok(elem) || elem->state != ELEM_BUSY)
+		return -1;
+
+	rte_spinlock_lock(&(elem->heap->lock));
+
+	ret = malloc_elem_resize(elem, size);
+
+	rte_spinlock_unlock(&(elem->heap->lock));
+
+	return ret;
+}
+
 /*
  * Function to retrieve data for heap on given socket
  */
lib/librte_eal/common/malloc_heap.h +6 -0
index 3ccbef0..3767ef3 100644
@@ -57,6 +57,12 @@ malloc_heap_alloc(struct malloc_heap *heap,	const char *type, size_t size,
 		unsigned flags, size_t align, size_t bound);
 
 int
+malloc_heap_free(struct malloc_elem *elem);
+
+int
+malloc_heap_resize(struct malloc_elem *elem, size_t size);
+
+int
 malloc_heap_get_stats(const struct malloc_heap *heap,
 		struct rte_malloc_socket_stats *socket_stats);
 
lib/librte_eal/common/rte_malloc.c +2 -2
index fe2278b..74b5417 100644
@@ -58,7 +58,7 @@
 void rte_free(void *addr)
 {
 	if (addr == NULL) return;
-	if (malloc_elem_free(malloc_elem_from_data(addr)) < 0)
+	if (malloc_heap_free(malloc_elem_from_data(addr)) < 0)
 		rte_panic("Fatal error: Invalid memory\n");
 }
 
@@ -169,7 +169,7 @@ rte_realloc(void *ptr, size_t size, unsigned align)
 	size = RTE_CACHE_LINE_ROUNDUP(size), align = RTE_CACHE_LINE_ROUNDUP(align);
 	/* check alignment matches first, and if ok, see if we can resize block */
 	if (RTE_PTR_ALIGN(ptr,align) == ptr &&
-			malloc_elem_resize(elem, size) == 0)
+			malloc_heap_resize(elem, size) == 0)
 		return ptr;
 
 	/* either alignment is off, or we have no room to expand,
Signature · 1 line
2.7.4
#9 lore raw

RFC 5/23 eal/malloc: protect malloc heap stats with a lock

In reply to Anatoly Burakov

This does not change the public API, as this API is not meant to be called directly.

---
 lib/librte_eal/common/malloc_heap.c | 7 ++++++-
 lib/librte_eal/common/malloc_heap.h | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)
lib/librte_eal/common/malloc_heap.c +6 -1
index 099e448..b3a1043 100644
@@ -214,12 +214,14 @@ malloc_heap_resize(struct malloc_elem *elem, size_t size) {
  * Function to retrieve data for heap on given socket
  */
 int
-malloc_heap_get_stats(const struct malloc_heap *heap,
+malloc_heap_get_stats(struct malloc_heap *heap,
 		struct rte_malloc_socket_stats *socket_stats)
 {
 	size_t idx;
 	struct malloc_elem *elem;
 
+	rte_spinlock_lock(&(heap->lock));
+
 	/* Initialise variables for heap */
 	socket_stats->free_count = 0;
 	socket_stats->heap_freesz_bytes = 0;
@@ -241,6 +243,9 @@ malloc_heap_get_stats(const struct malloc_heap *heap,
 	socket_stats->heap_allocsz_bytes = (socket_stats->heap_totalsz_bytes -
 			socket_stats->heap_freesz_bytes);
 	socket_stats->alloc_count = heap->alloc_count;
+
+	rte_spinlock_unlock(&(heap->lock));
+
 	return 0;
 }
 
lib/librte_eal/common/malloc_heap.h +1 -1
index 3767ef3..df04dd8 100644
@@ -63,7 +63,7 @@ int
 malloc_heap_resize(struct malloc_elem *elem, size_t size);
 
 int
-malloc_heap_get_stats(const struct malloc_heap *heap,
+malloc_heap_get_stats(struct malloc_heap *heap,
 		struct rte_malloc_socket_stats *socket_stats);
 
 int
Signature · 1 line
2.7.4
#10 lore raw

PATCH 5/23 eal: protect malloc heap stats with a lock

In reply to Anatoly Burakov

This does not change the public API, as this API is not meant to be called directly.

---
 lib/librte_eal/common/malloc_heap.c | 7 ++++++-
 lib/librte_eal/common/malloc_heap.h | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)
lib/librte_eal/common/malloc_heap.c +6 -1
index 099e448..b3a1043 100644
@@ -214,12 +214,14 @@ malloc_heap_resize(struct malloc_elem *elem, size_t size) {
  * Function to retrieve data for heap on given socket
  */
 int
-malloc_heap_get_stats(const struct malloc_heap *heap,
+malloc_heap_get_stats(struct malloc_heap *heap,
 		struct rte_malloc_socket_stats *socket_stats)
 {
 	size_t idx;
 	struct malloc_elem *elem;
 
+	rte_spinlock_lock(&(heap->lock));
+
 	/* Initialise variables for heap */
 	socket_stats->free_count = 0;
 	socket_stats->heap_freesz_bytes = 0;
@@ -241,6 +243,9 @@ malloc_heap_get_stats(const struct malloc_heap *heap,
 	socket_stats->heap_allocsz_bytes = (socket_stats->heap_totalsz_bytes -
 			socket_stats->heap_freesz_bytes);
 	socket_stats->alloc_count = heap->alloc_count;
+
+	rte_spinlock_unlock(&(heap->lock));
+
 	return 0;
 }
 
lib/librte_eal/common/malloc_heap.h +1 -1
index 3767ef3..df04dd8 100644
@@ -63,7 +63,7 @@ int
 malloc_heap_resize(struct malloc_elem *elem, size_t size);
 
 int
-malloc_heap_get_stats(const struct malloc_heap *heap,
+malloc_heap_get_stats(struct malloc_heap *heap,
 		struct rte_malloc_socket_stats *socket_stats);
 
 int
Signature · 1 line
2.7.4
#11 lore raw

PATCH 6/23 eal: make malloc a doubly-linked list

In reply to Anatoly Burakov

As we are preparing for dynamic memory allocation, we need to be able to handle holes in our malloc heap, hence we're switching to doubly linked list, and prepare infrastructure to support it.

Since our heap is now aware where are our first and last elements, there is no longer any need to have a dummy element at the end of each heap, so get rid of that as well. Instead, let insert/remove/ join/split operations handle end-of-list conditions automatically.

---
 lib/librte_eal/common/include/rte_malloc_heap.h |   6 +
 lib/librte_eal/common/malloc_elem.c             | 196 +++++++++++++++++++-----
 lib/librte_eal/common/malloc_elem.h             |   7 +-
 lib/librte_eal/common/malloc_heap.c             |   8 +-
 4 files changed, 170 insertions(+), 47 deletions(-)
lib/librte_eal/common/include/rte_malloc_heap.h +6 -0
index b270356..48a46c9 100644
@@ -42,12 +42,18 @@
 /* Number of free lists per heap, grouped by size. */
 #define RTE_HEAP_NUM_FREELISTS  13
 
+/* dummy definition, for pointers */
+struct malloc_elem;
+
 /**
  * Structure to hold malloc heap
  */
 struct malloc_heap {
 	rte_spinlock_t lock;
 	LIST_HEAD(, malloc_elem) free_head[RTE_HEAP_NUM_FREELISTS];
+	struct malloc_elem *first;
+	struct malloc_elem *last;
+
 	unsigned alloc_count;
 	size_t total_size;
 } __rte_cache_aligned;
lib/librte_eal/common/malloc_elem.c +160 -36

267 lines not on this page · raw message

lib/librte_eal/common/malloc_elem.h +2 -5
index ce39129..b3d39c0 100644
@@ -48,6 +48,7 @@ enum elem_state {
 struct malloc_elem {
 	struct malloc_heap *heap;
 	struct malloc_elem *volatile prev;      /* points to prev elem in memseg */
+	struct malloc_elem *volatile next;      /* points to next elem in memseg */
 	LIST_ENTRY(malloc_elem) free_list;      /* list of free elements in heap */
 	const struct rte_memseg *ms;
 	volatile enum elem_state state;
@@ -139,12 +140,8 @@ malloc_elem_init(struct malloc_elem *elem,
 		const struct rte_memseg *ms,
 		size_t size);
 
-/*
- * initialise a dummy malloc_elem header for the end-of-memseg marker
- */
 void
-malloc_elem_mkend(struct malloc_elem *elem,
-		struct malloc_elem *prev_free);
+malloc_elem_insert(struct malloc_elem *elem);
 
 /*
  * return true if the current malloc_elem can hold a block of data
lib/librte_eal/common/malloc_heap.c +2 -6
index b3a1043..1b35468 100644
@@ -99,15 +99,11 @@ check_hugepage_sz(unsigned flags, uint64_t hugepage_sz)
 static void
 malloc_heap_add_memseg(struct malloc_heap *heap, struct rte_memseg *ms)
 {
-	/* allocate the memory block headers, one at end, one at start */
 	struct malloc_elem *start_elem = (struct malloc_elem *)ms->addr;
-	struct malloc_elem *end_elem = RTE_PTR_ADD(ms->addr,
-			ms->len - MALLOC_ELEM_OVERHEAD);
-	end_elem = RTE_PTR_ALIGN_FLOOR(end_elem, RTE_CACHE_LINE_SIZE);
-	const size_t elem_size = (uintptr_t)end_elem - (uintptr_t)start_elem;
+	const size_t elem_size = ms->len - MALLOC_ELEM_OVERHEAD;
 
 	malloc_elem_init(start_elem, heap, ms, elem_size);
-	malloc_elem_mkend(end_elem, start_elem);
+	malloc_elem_insert(start_elem);
 	malloc_elem_free_list_insert(start_elem);
 
 	heap->total_size += elem_size;
Signature · 1 line
2.7.4
#12 lore raw

RFC 6/23 eal/malloc: make malloc a doubly-linked list

In reply to Anatoly Burakov

As we are preparing for dynamic memory allocation, we need to be able to handle holes in our malloc heap, hence we're switching to doubly linked list, and prepare infrastructure to support it.

Since our heap is now aware where are our first and last elements, there is no longer any need to have a dummy element at the end of each heap, so get rid of that as well. Instead, let insert/remove/ join/split operations handle end-of-list conditions automatically.

---
 lib/librte_eal/common/include/rte_malloc_heap.h |   6 +
 lib/librte_eal/common/malloc_elem.c             | 196 +++++++++++++++++++-----
 lib/librte_eal/common/malloc_elem.h             |   7 +-
 lib/librte_eal/common/malloc_heap.c             |   8 +-
 4 files changed, 170 insertions(+), 47 deletions(-)
lib/librte_eal/common/include/rte_malloc_heap.h +6 -0
index b270356..48a46c9 100644
@@ -42,12 +42,18 @@
 /* Number of free lists per heap, grouped by size. */
 #define RTE_HEAP_NUM_FREELISTS  13
 
+/* dummy definition, for pointers */
+struct malloc_elem;
+
 /**
  * Structure to hold malloc heap
  */
 struct malloc_heap {
 	rte_spinlock_t lock;
 	LIST_HEAD(, malloc_elem) free_head[RTE_HEAP_NUM_FREELISTS];
+	struct malloc_elem *first;
+	struct malloc_elem *last;
+
 	unsigned alloc_count;
 	size_t total_size;
 } __rte_cache_aligned;
lib/librte_eal/common/malloc_elem.c +160 -36

267 lines not on this page · raw message

lib/librte_eal/common/malloc_elem.h +2 -5
index ce39129..b3d39c0 100644
@@ -48,6 +48,7 @@ enum elem_state {
 struct malloc_elem {
 	struct malloc_heap *heap;
 	struct malloc_elem *volatile prev;      /* points to prev elem in memseg */
+	struct malloc_elem *volatile next;      /* points to next elem in memseg */
 	LIST_ENTRY(malloc_elem) free_list;      /* list of free elements in heap */
 	const struct rte_memseg *ms;
 	volatile enum elem_state state;
@@ -139,12 +140,8 @@ malloc_elem_init(struct malloc_elem *elem,
 		const struct rte_memseg *ms,
 		size_t size);
 
-/*
- * initialise a dummy malloc_elem header for the end-of-memseg marker
- */
 void
-malloc_elem_mkend(struct malloc_elem *elem,
-		struct malloc_elem *prev_free);
+malloc_elem_insert(struct malloc_elem *elem);
 
 /*
  * return true if the current malloc_elem can hold a block of data
lib/librte_eal/common/malloc_heap.c +2 -6
index b3a1043..1b35468 100644
@@ -99,15 +99,11 @@ check_hugepage_sz(unsigned flags, uint64_t hugepage_sz)
 static void
 malloc_heap_add_memseg(struct malloc_heap *heap, struct rte_memseg *ms)
 {
-	/* allocate the memory block headers, one at end, one at start */
 	struct malloc_elem *start_elem = (struct malloc_elem *)ms->addr;
-	struct malloc_elem *end_elem = RTE_PTR_ADD(ms->addr,
-			ms->len - MALLOC_ELEM_OVERHEAD);
-	end_elem = RTE_PTR_ALIGN_FLOOR(end_elem, RTE_CACHE_LINE_SIZE);
-	const size_t elem_size = (uintptr_t)end_elem - (uintptr_t)start_elem;
+	const size_t elem_size = ms->len - MALLOC_ELEM_OVERHEAD;
 
 	malloc_elem_init(start_elem, heap, ms, elem_size);
-	malloc_elem_mkend(end_elem, start_elem);
+	malloc_elem_insert(start_elem);
 	malloc_elem_free_list_insert(start_elem);
 
 	heap->total_size += elem_size;
Signature · 1 line
2.7.4
#13 lore raw

PATCH 7/23 eal: make malloc_elem_join_adjacent_free public

In reply to Anatoly Burakov

We need this function to join newly allocated segments with the heap.

---
 lib/librte_eal/common/malloc_elem.c | 6 +++---
 lib/librte_eal/common/malloc_elem.h | 3 +++
 2 files changed, 6 insertions(+), 3 deletions(-)
lib/librte_eal/common/malloc_elem.c +3 -3
index 7609a9b..782aaa7 100644
@@ -349,8 +349,8 @@ join_elem(struct malloc_elem *elem1, struct malloc_elem *elem2)
 	elem1->next = next;
 }
 
-static struct malloc_elem *
-elem_join_adjacent_free(struct malloc_elem *elem) {
+struct malloc_elem *
+malloc_elem_join_adjacent_free(struct malloc_elem *elem) {
 	/*
 	 * check if next element exists, is adjacent and is free, if so join
 	 * with it, need to remove from free list.
@@ -411,7 +411,7 @@ malloc_elem_free(struct malloc_elem *elem)
 	ptr = RTE_PTR_ADD(elem, sizeof(*elem));
 	data_len = elem->size - MALLOC_ELEM_OVERHEAD;
 
-	elem = elem_join_adjacent_free(elem);
+	elem = malloc_elem_join_adjacent_free(elem);
 
 	malloc_elem_free_list_insert(elem);
 
lib/librte_eal/common/malloc_elem.h +3 -0
index b3d39c0..cf27b59 100644
@@ -167,6 +167,9 @@ malloc_elem_alloc(struct malloc_elem *elem, size_t size,
 int
 malloc_elem_free(struct malloc_elem *elem);
 
+struct malloc_elem *
+malloc_elem_join_adjacent_free(struct malloc_elem *elem);
+
 /*
  * attempt to resize a malloc_elem by expanding into any free space
  * immediately after it in memory.
Signature · 1 line
2.7.4
#14 lore raw

RFC 7/23 eal/malloc: make malloc_elem_join_adjacent_free public

In reply to Anatoly Burakov

We need this function to join newly allocated segments with the heap.

---
 lib/librte_eal/common/malloc_elem.c | 6 +++---
 lib/librte_eal/common/malloc_elem.h | 3 +++
 2 files changed, 6 insertions(+), 3 deletions(-)
lib/librte_eal/common/malloc_elem.c +3 -3
index 7609a9b..782aaa7 100644
@@ -349,8 +349,8 @@ join_elem(struct malloc_elem *elem1, struct malloc_elem *elem2)
 	elem1->next = next;
 }
 
-static struct malloc_elem *
-elem_join_adjacent_free(struct malloc_elem *elem) {
+struct malloc_elem *
+malloc_elem_join_adjacent_free(struct malloc_elem *elem) {
 	/*
 	 * check if next element exists, is adjacent and is free, if so join
 	 * with it, need to remove from free list.
@@ -411,7 +411,7 @@ malloc_elem_free(struct malloc_elem *elem)
 	ptr = RTE_PTR_ADD(elem, sizeof(*elem));
 	data_len = elem->size - MALLOC_ELEM_OVERHEAD;
 
-	elem = elem_join_adjacent_free(elem);
+	elem = malloc_elem_join_adjacent_free(elem);
 
 	malloc_elem_free_list_insert(elem);
 
lib/librte_eal/common/malloc_elem.h +3 -0
index b3d39c0..cf27b59 100644
@@ -167,6 +167,9 @@ malloc_elem_alloc(struct malloc_elem *elem, size_t size,
 int
 malloc_elem_free(struct malloc_elem *elem);
 
+struct malloc_elem *
+malloc_elem_join_adjacent_free(struct malloc_elem *elem);
+
 /*
  * attempt to resize a malloc_elem by expanding into any free space
  * immediately after it in memory.
Signature · 1 line
2.7.4
#15 lore raw

PATCH 8/23 eal: add "single file segments" command-line option

In reply to Anatoly Burakov

For now, this option does nothing, but it will be useful in dynamic memory allocation down the line. Currently, DPDK stores all pages as separate files in hugetlbfs. This option will allow storing all pages in one file (one file per socket, per page size).

---
 lib/librte_eal/common/eal_common_options.c | 4 ++++
 lib/librte_eal/common/eal_internal_cfg.h   | 3 +++
 lib/librte_eal/common/eal_options.h        | 2 ++
 lib/librte_eal/linuxapp/eal/eal.c          | 1 +
 4 files changed, 10 insertions(+)
lib/librte_eal/common/eal_common_options.c +4 -0
index 996a034..c3f7c41 100644
@@ -98,6 +98,7 @@ eal_long_options[] = {
 	{OPT_VDEV,              1, NULL, OPT_VDEV_NUM             },
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
+	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1158,6 +1159,9 @@ eal_parse_common_option(int opt, const char *optarg,
 		}
 		core_parsed = 1;
 		break;
+	case OPT_SINGLE_FILE_SEGMENTS_NUM:
+		conf->single_file_segments = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
lib/librte_eal/common/eal_internal_cfg.h +3 -0
index fa6ccbe..484a32e 100644
@@ -76,6 +76,9 @@ struct internal_config {
 	volatile unsigned force_sockets;
 	volatile uint64_t socket_mem[RTE_MAX_NUMA_NODES]; /**< amount of memory per socket */
 	uintptr_t base_virtaddr;          /**< base address to try and reserve memory from */
+	volatile unsigned single_file_segments;
+	/**< true if storing all pages within single files (per-page-size,
+	 * per-node). */
 	volatile int syslog_facility;	  /**< facility passed to openlog() */
 	/** default interrupt mode for VFIO */
 	volatile enum rte_intr_mode vfio_intr_mode;
lib/librte_eal/common/eal_options.h +2 -0
index 30e6bb4..26a682a 100644
@@ -83,6 +83,8 @@ enum {
 	OPT_VFIO_INTR_NUM,
 #define OPT_VMWARE_TSC_MAP    "vmware-tsc-map"
 	OPT_VMWARE_TSC_MAP_NUM,
+#define OPT_SINGLE_FILE_SEGMENTS    "single-file-segments"
+	OPT_SINGLE_FILE_SEGMENTS_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/linuxapp/eal/eal.c +1 -0
index 229eec9..2a3127f 100644
@@ -366,6 +366,7 @@ eal_usage(const char *prgname)
 	       "  --"OPT_BASE_VIRTADDR"     Base virtual address\n"
 	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
 	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
+	       "  --"OPT_SINGLE_FILE_SEGMENTS" Put all hugepage memory in single files\n"
 	       "\n");
 	/* Allow the application to print its usage message too if hook is set */
 	if ( rte_application_usage_hook ) {
Signature · 1 line
2.7.4
#16 lore raw

PATCH 9/23 eal: add "legacy memory" option

In reply to Anatoly Burakov

This adds a "--legacy-mem" command-line switch. It will be used to go back to the old memory behavior, one where we can't dynamically allocate/free memory (the downside), but one where the user can get physically contiguous memory, like before (the upside).

For now, nothing but the legacy behavior exists, non-legacy memory init sequence will be added later.

---
 lib/librte_eal/common/eal_common_options.c |  4 ++++
 lib/librte_eal/common/eal_internal_cfg.h   |  3 +++
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/linuxapp/eal/eal.c          |  1 +
 lib/librte_eal/linuxapp/eal/eal_memory.c   | 22 ++++++++++++++++++----
 5 files changed, 28 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_options.c +4 -0
index c3f7c41..88ff35a 100644
@@ -99,6 +99,7 @@ eal_long_options[] = {
 	{OPT_VFIO_INTR,         1, NULL, OPT_VFIO_INTR_NUM        },
 	{OPT_VMWARE_TSC_MAP,    0, NULL, OPT_VMWARE_TSC_MAP_NUM   },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
+	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1162,6 +1163,9 @@ eal_parse_common_option(int opt, const char *optarg,
 	case OPT_SINGLE_FILE_SEGMENTS_NUM:
 		conf->single_file_segments = 1;
 		break;
+	case OPT_LEGACY_MEM_NUM:
+		conf->legacy_mem = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
lib/librte_eal/common/eal_internal_cfg.h +3 -0
index 484a32e..62ab15b 100644
@@ -79,6 +79,9 @@ struct internal_config {
 	volatile unsigned single_file_segments;
 	/**< true if storing all pages within single files (per-page-size,
 	 * per-node). */
+	volatile unsigned legacy_mem;
+	/**< true to enable legacy memory behavior (no dynamic allocation,
+	 * contiguous segments). */
 	volatile int syslog_facility;	  /**< facility passed to openlog() */
 	/** default interrupt mode for VFIO */
 	volatile enum rte_intr_mode vfio_intr_mode;
lib/librte_eal/common/eal_options.h +2 -0
index 26a682a..d09b034 100644
@@ -85,6 +85,8 @@ enum {
 	OPT_VMWARE_TSC_MAP_NUM,
 #define OPT_SINGLE_FILE_SEGMENTS    "single-file-segments"
 	OPT_SINGLE_FILE_SEGMENTS_NUM,
+#define OPT_LEGACY_MEM    "legacy-mem"
+	OPT_LEGACY_MEM_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/linuxapp/eal/eal.c +1 -0
index 2a3127f..37ae8e0 100644
@@ -367,6 +367,7 @@ eal_usage(const char *prgname)
 	       "  --"OPT_CREATE_UIO_DEV"    Create /dev/uioX (usually done by hotplug)\n"
 	       "  --"OPT_VFIO_INTR"         Interrupt mode for VFIO (legacy|msi|msix)\n"
 	       "  --"OPT_SINGLE_FILE_SEGMENTS" Put all hugepage memory in single files\n"
+	       "  --"OPT_LEGACY_MEM"        Legacy memory mode (no dynamic allocation, contiguous segments)\n"
 	       "\n");
 	/* Allow the application to print its usage message too if hook is set */
 	if ( rte_application_usage_hook ) {
lib/librte_eal/linuxapp/eal/eal_memory.c +18 -4
index dd18d98..5b18af9 100644
@@ -940,8 +940,8 @@ huge_recover_sigbus(void)
  *  6. unmap the first mapping
  *  7. fill memsegs in configuration with contiguous zones
  */
-int
-rte_eal_hugepage_init(void)
+static int
+eal_legacy_hugepage_init(void)
 {
 	struct rte_mem_config *mcfg;
 	struct hugepage_file *hugepage = NULL, *tmp_hp = NULL;
@@ -1283,8 +1283,8 @@ getFileSize(int fd)
  * configuration and finds the hugepages which form that segment, mapping them
  * in order to form a contiguous block in the virtual memory space
  */
-int
-rte_eal_hugepage_attach(void)
+static int
+eal_legacy_hugepage_attach(void)
 {
 	const struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
 	struct hugepage_file *hp = NULL;
@@ -1435,6 +1435,20 @@ rte_eal_hugepage_attach(void)
 }
 
 int
+rte_eal_hugepage_init(void) {
+	if (internal_config.legacy_mem)
+		return eal_legacy_hugepage_init();
+	return -1;
+}
+
+int
+rte_eal_hugepage_attach(void) {
+	if (internal_config.legacy_mem)
+		return eal_legacy_hugepage_attach();
+	return -1;
+}
+
+int
 rte_eal_using_phys_addrs(void)
 {
 	return phys_addrs_available;
Signature · 1 line
2.7.4
#17 lore raw

RFC 10/23 eal: populate hugepage counts from socket-specific sysfs path

In reply to Anatoly Burakov

For non-legacy memory init mode, instead of looking at generic sysfs path, look at sysfs paths pertaining to each NUMA node for hugepage counts. Note that per-NUMA node path does not provide information regarding reserved pages, so we might not get the best info from these paths, but this saves us from the whole mapping/remapping business before we're actually able to tell which page is on which socket, because we no longer require our memory to be physically contiguous.

Legacy memory init will not use this.

---
 lib/librte_eal/linuxapp/eal/eal_hugepage_info.c | 73 +++++++++++++++++++++++--
 1 file changed, 67 insertions(+), 6 deletions(-)
lib/librte_eal/linuxapp/eal/eal_hugepage_info.c +67 -6
index 86e174f..a85c15a 100644
@@ -59,6 +59,7 @@
 #include "eal_filesystem.h"
 
 static const char sys_dir_path[] = "/sys/kernel/mm/hugepages";
+static const char sys_pages_numa_dir_path[] = "/sys/devices/system/node";
 
 /* this function is only called from eal_hugepage_info_init which itself
  * is only called from a primary process */
@@ -99,6 +100,42 @@ get_num_hugepages(const char *subdir)
 	return num_pages;
 }
 
+static uint32_t
+get_num_hugepages_on_node(const char *subdir, unsigned socket) {
+	char path[PATH_MAX], socketpath[PATH_MAX];
+	DIR *socketdir;
+	long unsigned num_pages = 0;
+	const char *nr_hp_file = "free_hugepages";
+
+	snprintf(socketpath, sizeof(socketpath), "%s/node%u/hugepages",
+		 sys_pages_numa_dir_path, socket);
+
+	socketdir = opendir(socketpath);
+	if (socketdir) {
+		/* Keep calm and carry on */
+		closedir(socketdir);
+	} else {
+		/* Can't find socket dir, so ignore it */
+		return 0;
+	}
+
+	snprintf(path, sizeof(path), "%s/%s/%s",
+			socketpath, subdir, nr_hp_file);
+	if (eal_parse_sysfs_value(path, &num_pages) < 0)
+		return 0;
+
+	if (num_pages == 0)
+		RTE_LOG(WARNING, EAL, "No free hugepages reported in %s\n",
+				subdir);
+
+	/* we want to return a uint32_t and more than this looks suspicious
+	 * anyway ... */
+	if (num_pages > UINT32_MAX)
+		num_pages = UINT32_MAX;
+
+	return num_pages;
+}
+
 static uint64_t
 get_default_hp_size(void)
 {
@@ -277,7 +314,7 @@ eal_hugepage_info_init(void)
 {
 	const char dirent_start_text[] = "hugepages-";
 	const size_t dirent_start_len = sizeof(dirent_start_text) - 1;
-	unsigned i, num_sizes = 0;
+	unsigned i, total_pages, num_sizes = 0;
 	DIR *dir;
 	struct dirent *dirent;
 
@@ -331,9 +368,24 @@ eal_hugepage_info_init(void)
 		if (clear_hugedir(hpi->hugedir) == -1)
 			break;
 
-		/* for now, put all pages into socket 0,
-		 * later they will be sorted */
-		hpi->num_pages[0] = get_num_hugepages(dirent->d_name);
+		/* first, try to put all hugepages into relevant sockets, but
+		 * if first attempts fails, fall back to collecting all pages
+		 * in one socket and sorting them later */
+		total_pages = 0;
+		/* we also don't want to do this for legacy init */
+		if (!internal_config.legacy_mem)
+			for (i = 0; i < rte_num_sockets(); i++) {
+				unsigned num_pages =
+						get_num_hugepages_on_node(
+							dirent->d_name, i);
+				hpi->num_pages[i] = num_pages;
+				total_pages += num_pages;
+			}
+		/* we failed to sort memory from the get go, so fall
+		 * back to old way */
+		if (total_pages == 0) {
+			hpi->num_pages[0] = get_num_hugepages(dirent->d_name);
+		}
 
 #ifndef RTE_ARCH_64
 		/* for 32-bit systems, limit number of hugepages to
@@ -357,10 +409,19 @@ eal_hugepage_info_init(void)
 	      sizeof(internal_config.hugepage_info[0]), compare_hpi);
 
 	/* now we have all info, check we have at least one valid size */
-	for (i = 0; i < num_sizes; i++)
+	for (i = 0; i < num_sizes; i++) {
+		/* pages may no longer all be on socket 0, so check all */
+		unsigned j, num_pages = 0;
+
+		for (j = 0; j < RTE_MAX_NUMA_NODES; j++) {
+			struct hugepage_info *hpi =
+					&internal_config.hugepage_info[i];
+			num_pages += hpi->num_pages[j];
+		}
 		if (internal_config.hugepage_info[i].hugedir != NULL &&
-		    internal_config.hugepage_info[i].num_pages[0] > 0)
+				num_pages > 0)
 			return 0;
+	}
 
 	/* no valid hugepage mounts available, return error */
 	return -1;
Signature · 1 line
2.7.4
#18 lore raw

PATCH 10/23 eal: read hugepage counts from node-specific sysfs path

In reply to Anatoly Burakov

For non-legacy memory init mode, instead of looking at generic sysfs path, look at sysfs paths pertaining to each NUMA node for hugepage counts. Note that per-NUMA node path does not provide information regarding reserved pages, so we might not get the best info from these paths, but this saves us from the whole mapping/remapping business before we're actually able to tell which page is on which socket, because we no longer require our memory to be physically contiguous.

Legacy memory init will not use this.

---
 lib/librte_eal/linuxapp/eal/eal_hugepage_info.c | 73 +++++++++++++++++++++++--
 1 file changed, 67 insertions(+), 6 deletions(-)
lib/librte_eal/linuxapp/eal/eal_hugepage_info.c +67 -6
index 86e174f..a85c15a 100644
@@ -59,6 +59,7 @@
 #include "eal_filesystem.h"
 
 static const char sys_dir_path[] = "/sys/kernel/mm/hugepages";
+static const char sys_pages_numa_dir_path[] = "/sys/devices/system/node";
 
 /* this function is only called from eal_hugepage_info_init which itself
  * is only called from a primary process */
@@ -99,6 +100,42 @@ get_num_hugepages(const char *subdir)
 	return num_pages;
 }
 
+static uint32_t
+get_num_hugepages_on_node(const char *subdir, unsigned socket) {
+	char path[PATH_MAX], socketpath[PATH_MAX];
+	DIR *socketdir;
+	long unsigned num_pages = 0;
+	const char *nr_hp_file = "free_hugepages";
+
+	snprintf(socketpath, sizeof(socketpath), "%s/node%u/hugepages",
+		 sys_pages_numa_dir_path, socket);
+
+	socketdir = opendir(socketpath);
+	if (socketdir) {
+		/* Keep calm and carry on */
+		closedir(socketdir);
+	} else {
+		/* Can't find socket dir, so ignore it */
+		return 0;
+	}
+
+	snprintf(path, sizeof(path), "%s/%s/%s",
+			socketpath, subdir, nr_hp_file);
+	if (eal_parse_sysfs_value(path, &num_pages) < 0)
+		return 0;
+
+	if (num_pages == 0)
+		RTE_LOG(WARNING, EAL, "No free hugepages reported in %s\n",
+				subdir);
+
+	/* we want to return a uint32_t and more than this looks suspicious
+	 * anyway ... */
+	if (num_pages > UINT32_MAX)
+		num_pages = UINT32_MAX;
+
+	return num_pages;
+}
+
 static uint64_t
 get_default_hp_size(void)
 {
@@ -277,7 +314,7 @@ eal_hugepage_info_init(void)
 {
 	const char dirent_start_text[] = "hugepages-";
 	const size_t dirent_start_len = sizeof(dirent_start_text) - 1;
-	unsigned i, num_sizes = 0;
+	unsigned i, total_pages, num_sizes = 0;
 	DIR *dir;
 	struct dirent *dirent;
 
@@ -331,9 +368,24 @@ eal_hugepage_info_init(void)
 		if (clear_hugedir(hpi->hugedir) == -1)
 			break;
 
-		/* for now, put all pages into socket 0,
-		 * later they will be sorted */
-		hpi->num_pages[0] = get_num_hugepages(dirent->d_name);
+		/* first, try to put all hugepages into relevant sockets, but
+		 * if first attempts fails, fall back to collecting all pages
+		 * in one socket and sorting them later */
+		total_pages = 0;
+		/* we also don't want to do this for legacy init */
+		if (!internal_config.legacy_mem)
+			for (i = 0; i < rte_num_sockets(); i++) {
+				unsigned num_pages =
+						get_num_hugepages_on_node(
+							dirent->d_name, i);
+				hpi->num_pages[i] = num_pages;
+				total_pages += num_pages;
+			}
+		/* we failed to sort memory from the get go, so fall
+		 * back to old way */
+		if (total_pages == 0) {
+			hpi->num_pages[0] = get_num_hugepages(dirent->d_name);
+		}
 
 #ifndef RTE_ARCH_64
 		/* for 32-bit systems, limit number of hugepages to
@@ -357,10 +409,19 @@ eal_hugepage_info_init(void)
 	      sizeof(internal_config.hugepage_info[0]), compare_hpi);
 
 	/* now we have all info, check we have at least one valid size */
-	for (i = 0; i < num_sizes; i++)
+	for (i = 0; i < num_sizes; i++) {
+		/* pages may no longer all be on socket 0, so check all */
+		unsigned j, num_pages = 0;
+
+		for (j = 0; j < RTE_MAX_NUMA_NODES; j++) {
+			struct hugepage_info *hpi =
+					&internal_config.hugepage_info[i];
+			num_pages += hpi->num_pages[j];
+		}
 		if (internal_config.hugepage_info[i].hugedir != NULL &&
-		    internal_config.hugepage_info[i].num_pages[0] > 0)
+				num_pages > 0)
 			return 0;
+	}
 
 	/* no valid hugepage mounts available, return error */
 	return -1;
Signature · 1 line
2.7.4
#19 lore raw

PATCH 11/23 eal: replace memseg with memseg lists

In reply to Anatoly Burakov

Before, we were aggregating multiple pages into one memseg, so the number of memsegs was small. Now, each page gets its own memseg, so the list of memsegs is huge. To accommodate the new memseg list size and to keep the under-the-hood workings sane, the memseg list is now not just a single list, but multiple lists. To be precise, each hugepage size available on the system gets a memseg list per socket (so, for example, on a 2-socket system with 2M and 1G hugepages, we will get 4 memseg lists).

In order to support dynamic memory allocation, we reserve all memory in advance. As in, we do an anonymous mmap() of the entire maximum size of memory per hugepage size (which is limited to either RTE_MAX_MEMSEG_PER_LIST or 128G worth of memory, whichever is the smaller one). The limit is arbitrary.

So, for each hugepage size, we get (by default) up to 128G worth of memory, per socket. The address space is claimed at the start, in eal_common_memory.c. The actual page allocation code is in eal_memalloc.c (Linux-only for now), and largely consists of moved EAL memory init code.

Pages in the list are also indexed by address. That is, for non-legacy mode, in order to figure out where the page belongs, one can simply look at base address for a memseg list. Similarly, figuring out IOVA address of a memzone is a matter of finding the right memseg list, getting offset and dividing by page size to get the appropriate memseg. For legacy mode, old behavior of walking the memseg list remains.

Due to switch to fbarray, secondary processes are not currently supported nor tested. Also, one particular API call (dump physmem layout) no longer makes sense not only becase there can now be holes in memseg list, but also because there are several memseg lists to choose from.

In legacy mode, nothing is preallocated, and all memsegs are in a list like before, but each segment still resides in an appropriate memseg list.

The rest of the changes are really ripple effects from the memseg change - heap changes, compile fixes, and rewrites to support fbarray-backed memseg lists.

---
 config/common_base                                |   3 +-
 drivers/bus/pci/linux/pci.c                       |  29 ++-
 drivers/net/virtio/virtio_user/vhost_kernel.c     | 106 +++++---
 lib/librte_eal/common/eal_common_memory.c         | 245 ++++++++++++++++--
 lib/librte_eal/common/eal_common_memzone.c        |   5 +-
 lib/librte_eal/common/eal_hugepages.h             |   1 +
 lib/librte_eal/common/include/rte_eal_memconfig.h |  22 +-
 lib/librte_eal/common/include/rte_memory.h        |  16 ++
 lib/librte_eal/common/malloc_elem.c               |   8 +-
 lib/librte_eal/common/malloc_elem.h               |   6 +-
 lib/librte_eal/common/malloc_heap.c               |  88 +++++--
 lib/librte_eal/common/rte_malloc.c                |  20 +-
 lib/librte_eal/linuxapp/eal/eal.c                 |  21 +-
 lib/librte_eal/linuxapp/eal/eal_memory.c          | 299 ++++++++++++++--------
 lib/librte_eal/linuxapp/eal/eal_vfio.c            | 162 ++++++++----
 test/test/test_malloc.c                           |  29 ++-
 test/test/test_memory.c                           |  44 +++-
 test/test/test_memzone.c                          |  17 +-
 18 files changed, 815 insertions(+), 306 deletions(-)
config/common_base +2 -1
index e74febe..9730d4c 100644
@@ -90,7 +90,8 @@ CONFIG_RTE_CACHE_LINE_SIZE=64
 CONFIG_RTE_LIBRTE_EAL=y
 CONFIG_RTE_MAX_LCORE=128
 CONFIG_RTE_MAX_NUMA_NODES=8
-CONFIG_RTE_MAX_MEMSEG=256
+CONFIG_RTE_MAX_MEMSEG_LISTS=16
+CONFIG_RTE_MAX_MEMSEG_PER_LIST=32768
 CONFIG_RTE_MAX_MEMZONE=2560
 CONFIG_RTE_MAX_TAILQ=32
 CONFIG_RTE_ENABLE_ASSERT=n
drivers/bus/pci/linux/pci.c +20 -9
index 5da6728..6d3100f 100644
@@ -148,19 +148,30 @@ rte_pci_unmap_device(struct rte_pci_device *dev)
 void *
 pci_find_max_end_va(void)
 {
-	const struct rte_memseg *seg = rte_eal_get_physmem_layout();
-	const struct rte_memseg *last = seg;
-	unsigned i = 0;
+	void *cur_end, *max_end = NULL;
+	int i = 0;
 
-	for (i = 0; i < RTE_MAX_MEMSEG; i++, seg++) {
-		if (seg->addr == NULL)
-			break;
+	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+		const struct rte_mem_config *mcfg =
+				rte_eal_get_configuration()->mem_config;
+		const struct rte_memseg_list *msl = &mcfg->memsegs[i];
+		const struct rte_fbarray *arr = &msl->memseg_arr;
 
-		if (seg->addr > last->addr)
-			last = seg;
+		if (arr->capacity == 0)
+			continue;
 
+		/*
+		 * we need to handle legacy mem case, so don't rely on page size
+		 * to calculate max VA end
+		 */
+		while ((i = rte_fbarray_find_next_used(arr, i)) >= 0) {
+			struct rte_memseg *ms = rte_fbarray_get(arr, i);
+			cur_end = RTE_PTR_ADD(ms->addr, ms->len);
+			if (cur_end > max_end)
+				max_end = cur_end;
+		}
 	}
-	return RTE_PTR_ADD(last->addr, last->len);
+	return max_end;
 }
 
 /* parse one line of the "resource" sysfs file (note that the 'line'
drivers/net/virtio/virtio_user/vhost_kernel.c +64 -42

130 lines not on this page · raw message

lib/librte_eal/common/eal_common_memory.c +224 -21

290 lines not on this page · raw message

lib/librte_eal/common/eal_common_memzone.c +2 -3
index ea072a2..f558ac2 100644
@@ -254,10 +254,9 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 	mz->iova = rte_malloc_virt2iova(mz_addr);
 	mz->addr = mz_addr;
 	mz->len = (requested_len == 0 ? elem->size : requested_len);
-	mz->hugepage_sz = elem->ms->hugepage_sz;
-	mz->socket_id = elem->ms->socket_id;
+	mz->hugepage_sz = elem->msl->hugepage_sz;
+	mz->socket_id = elem->msl->socket_id;
 	mz->flags = 0;
-	mz->memseg_id = elem->ms - rte_eal_get_configuration()->mem_config->memseg;
 
 	return mz;
 }
lib/librte_eal/common/eal_hugepages.h +1 -0
index 68369f2..cf91009 100644
@@ -52,6 +52,7 @@ struct hugepage_file {
 	int socket_id;      /**< NUMA socket ID */
 	int file_id;        /**< the '%d' in HUGEFILE_FMT */
 	int memseg_id;      /**< the memory segment to which page belongs */
+	int memseg_list_id; /**< the memory segment list to which page belongs */
 	char filepath[MAX_HUGEPAGE_PATH]; /**< path to backing file on filesystem */
 };
 
lib/librte_eal/common/include/rte_eal_memconfig.h +21 -1
index b9eee70..c9b57a4 100644
@@ -40,12 +40,30 @@
 #include <rte_malloc_heap.h>
 #include <rte_rwlock.h>
 #include <rte_pause.h>
+#include <rte_fbarray.h>
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
 /**
+ * memseg list is a special case as we need to store a bunch of other data
+ * together with the array itself.
+ */
+struct rte_memseg_list {
+	RTE_STD_C11
+	union {
+		void *base_va;
+		/**< Base virtual address for this memseg list. */
+		uint64_t addr_64;
+		/**< Makes sure addr is always 64-bits */
+	};
+	int socket_id; /**< Socket ID for all memsegs in this list. */
+	uint64_t hugepage_sz; /**< page size for all memsegs in this list. */
+	struct rte_fbarray memseg_arr;
+};
+
+/**
  * the structure for the memory configuration for the RTE.
  * Used by the rte_config structure. It is separated out, as for multi-process
  * support, the memory details should be shared across instances
@@ -71,9 +89,11 @@ struct rte_mem_config {
 	uint32_t memzone_cnt; /**< Number of allocated memzones */
 
 	/* memory segments and zones */
-	struct rte_memseg memseg[RTE_MAX_MEMSEG];    /**< Physmem descriptors. */
 	struct rte_memzone memzone[RTE_MAX_MEMZONE]; /**< Memzone descriptors. */
 
+	struct rte_memseg_list memsegs[RTE_MAX_MEMSEG_LISTS];
+	/**< list of dynamic arrays holding memsegs */
+
 	struct rte_tailq_head tailq_head[RTE_MAX_TAILQ]; /**< Tailqs for objects */
 
 	/* Heaps of Malloc per socket */
lib/librte_eal/common/include/rte_memory.h +16 -0
index 14aacea..f005716 100644
@@ -50,6 +50,9 @@ extern "C" {
 
 #include <rte_common.h>
 
+/* forward declaration for pointers */
+struct rte_memseg_list;
+
 __extension__
 enum rte_page_sizes {
 	RTE_PGSIZE_4K    = 1ULL << 12,
@@ -158,6 +161,19 @@ phys_addr_t rte_mem_virt2phy(const void *virt);
 rte_iova_t rte_mem_virt2iova(const void *virt);
 
 /**
+ * Get memseg corresponding to virtual memory address.
+ *
+ * @param virt
+ *   The virtual address.
+ * @param msl
+ *   Memseg list in which to look for memsegs (can be NULL).
+ * @return
+ *   Memseg to which this virtual address belongs to.
+ */
+const struct rte_memseg *rte_mem_virt2memseg(const void *virt,
+		const struct rte_memseg_list *msl);
+
+/**
  * Get the layout of the available physical memory.
  *
  * It can be useful for an application to have the full physical
lib/librte_eal/common/malloc_elem.c +4 -4
index 782aaa7..ab09b94 100644
@@ -54,11 +54,11 @@
  * Initialize a general malloc_elem header structure
  */
 void
-malloc_elem_init(struct malloc_elem *elem,
-		struct malloc_heap *heap, const struct rte_memseg *ms, size_t size)
+malloc_elem_init(struct malloc_elem *elem, struct malloc_heap *heap,
+		const struct rte_memseg_list *msl, size_t size)
 {
 	elem->heap = heap;
-	elem->ms = ms;
+	elem->msl = msl;
 	elem->prev = NULL;
 	elem->next = NULL;
 	memset(&elem->free_list, 0, sizeof(elem->free_list));
@@ -172,7 +172,7 @@ split_elem(struct malloc_elem *elem, struct malloc_elem *split_pt)
 	const size_t old_elem_size = (uintptr_t)split_pt - (uintptr_t)elem;
 	const size_t new_elem_size = elem->size - old_elem_size;
 
-	malloc_elem_init(split_pt, elem->heap, elem->ms, new_elem_size);
+	malloc_elem_init(split_pt, elem->heap, elem->msl, new_elem_size);
 	split_pt->prev = elem;
 	split_pt->next = next_elem;
 	if (next_elem)
lib/librte_eal/common/malloc_elem.h +3 -3
index cf27b59..330bddc 100644
@@ -34,7 +34,7 @@
 #ifndef MALLOC_ELEM_H_
 #define MALLOC_ELEM_H_
 
-#include <rte_memory.h>
+#include <rte_eal_memconfig.h>
 
 /* dummy definition of struct so we can use pointers to it in malloc_elem struct */
 struct malloc_heap;
@@ -50,7 +50,7 @@ struct malloc_elem {
 	struct malloc_elem *volatile prev;      /* points to prev elem in memseg */
 	struct malloc_elem *volatile next;      /* points to next elem in memseg */
 	LIST_ENTRY(malloc_elem) free_list;      /* list of free elements in heap */
-	const struct rte_memseg *ms;
+	const struct rte_memseg_list *msl;
 	volatile enum elem_state state;
 	uint32_t pad;
 	size_t size;
@@ -137,7 +137,7 @@ malloc_elem_from_data(const void *data)
 void
 malloc_elem_init(struct malloc_elem *elem,
 		struct malloc_heap *heap,
-		const struct rte_memseg *ms,
+		const struct rte_memseg_list *msl,
 		size_t size);
 
 void
lib/librte_eal/common/malloc_heap.c +69 -19

120 lines not on this page · raw message

lib/librte_eal/common/rte_malloc.c +12 -8
index 74b5417..92cd7d8 100644
@@ -251,17 +251,21 @@ rte_malloc_set_limit(__rte_unused const char *type,
 rte_iova_t
 rte_malloc_virt2iova(const void *addr)
 {
-	rte_iova_t iova;
+	const struct rte_memseg *ms;
 	const struct malloc_elem *elem = malloc_elem_from_data(addr);
+
 	if (elem == NULL)
 		return RTE_BAD_IOVA;
-	if (elem->ms->iova == RTE_BAD_IOVA)
-		return RTE_BAD_IOVA;
 
 	if (rte_eal_iova_mode() == RTE_IOVA_VA)
-		iova = (uintptr_t)addr;
-	else
-		iova = elem->ms->iova +
-			RTE_PTR_DIFF(addr, elem->ms->addr);
-	return iova;
+		return (uintptr_t) addr;
+
+	ms = rte_mem_virt2memseg(addr, elem->msl);
+	if (ms == NULL)
+		return RTE_BAD_IOVA;
+
+	if (ms->iova == RTE_BAD_IOVA)
+		return RTE_BAD_IOVA;
+
+	return ms->iova + RTE_PTR_DIFF(addr, ms->addr);
 }
lib/librte_eal/linuxapp/eal/eal.c +12 -9
index 37ae8e0..a27536f 100644
@@ -102,8 +102,8 @@ static int mem_cfg_fd = -1;
 static struct flock wr_lock = {
 		.l_type = F_WRLCK,
 		.l_whence = SEEK_SET,
-		.l_start = offsetof(struct rte_mem_config, memseg),
-		.l_len = sizeof(early_mem_config.memseg),
+		.l_start = offsetof(struct rte_mem_config, memsegs),
+		.l_len = sizeof(early_mem_config.memsegs),
 };
 
 /* Address of global and public configuration */
@@ -661,17 +661,20 @@ eal_parse_args(int argc, char **argv)
 static void
 eal_check_mem_on_local_socket(void)
 {
-	const struct rte_memseg *ms;
+	const struct rte_memseg_list *msl;
 	int i, socket_id;
 
 	socket_id = rte_lcore_to_socket_id(rte_config.master_lcore);
 
-	ms = rte_eal_get_physmem_layout();
-
-	for (i = 0; i < RTE_MAX_MEMSEG; i++)
-		if (ms[i].socket_id == socket_id &&
-				ms[i].len > 0)
-			return;
+	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+		msl = &rte_eal_get_configuration()->mem_config->memsegs[i];
+		if (msl->socket_id != socket_id)
+			continue;
+		/* for legacy memory, check if there's anything allocated */
+		if (internal_config.legacy_mem && msl->memseg_arr.count == 0)
+			continue;
+		return;
+	}
 
 	RTE_LOG(WARNING, EAL, "WARNING: Master core has no "
 			"memory on local socket!\n");
lib/librte_eal/linuxapp/eal/eal_memory.c +197 -102

383 lines not on this page · raw message

lib/librte_eal/linuxapp/eal/eal_vfio.c +107 -55

206 lines not on this page · raw message

test/test/test_malloc.c +15 -14
index 4572caf..ae24c33 100644
@@ -41,6 +41,7 @@
 
 #include <rte_common.h>
 #include <rte_memory.h>
+#include <rte_eal_memconfig.h>
 #include <rte_per_lcore.h>
 #include <rte_launch.h>
 #include <rte_eal.h>
@@ -734,15 +735,23 @@ test_malloc_bad_params(void)
 	return -1;
 }
 
-/* Check if memory is available on a specific socket */
+/* Check if memory is avilable on a specific socket */
 static int
 is_mem_on_socket(int32_t socket)
 {
-	const struct rte_memseg *ms = rte_eal_get_physmem_layout();
+	const struct rte_mem_config *mcfg =
+			rte_eal_get_configuration()->mem_config;
 	unsigned i;
 
-	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
-		if (socket == ms[i].socket_id)
+	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+		const struct rte_memseg_list *msl =
+				&mcfg->memsegs[i];
+		const struct rte_fbarray *arr = &msl->memseg_arr;
+
+		if (msl->socket_id != socket)
+			continue;
+
+		if (arr->count)
 			return 1;
 	}
 	return 0;
@@ -755,16 +764,8 @@ is_mem_on_socket(int32_t socket)
 static int32_t
 addr_to_socket(void * addr)
 {
-	const struct rte_memseg *ms = rte_eal_get_physmem_layout();
-	unsigned i;
-
-	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
-		if ((ms[i].addr <= addr) &&
-				((uintptr_t)addr <
-				((uintptr_t)ms[i].addr + (uintptr_t)ms[i].len)))
-			return ms[i].socket_id;
-	}
-	return -1;
+	const struct rte_memseg *ms = rte_mem_virt2memseg(addr, NULL);
+	return ms == NULL ? -1 : ms->socket_id;
 }
 
 /* Test using rte_[c|m|zm]alloc_socket() on a specific socket */
test/test/test_memory.c +36 -8

74 lines not on this page · raw message

test/test/test_memzone.c +10 -7
index 1cf235a..47af721 100644
@@ -132,22 +132,25 @@ static int
 test_memzone_reserve_flags(void)
 {
 	const struct rte_memzone *mz;
-	const struct rte_memseg *ms;
 	int hugepage_2MB_avail = 0;
 	int hugepage_1GB_avail = 0;
 	int hugepage_16MB_avail = 0;
 	int hugepage_16GB_avail = 0;
 	const size_t size = 100;
 	int i = 0;
-	ms = rte_eal_get_physmem_layout();
-	for (i = 0; i < RTE_MAX_MEMSEG; i++) {
-		if (ms[i].hugepage_sz == RTE_PGSIZE_2M)
+
+	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+		struct rte_mem_config *mcfg =
+				rte_eal_get_configuration()->mem_config;
+		struct rte_memseg_list *msl = &mcfg->memsegs[i];
+
+		if (msl->hugepage_sz == RTE_PGSIZE_2M)
 			hugepage_2MB_avail = 1;
-		if (ms[i].hugepage_sz == RTE_PGSIZE_1G)
+		if (msl->hugepage_sz == RTE_PGSIZE_1G)
 			hugepage_1GB_avail = 1;
-		if (ms[i].hugepage_sz == RTE_PGSIZE_16M)
+		if (msl->hugepage_sz == RTE_PGSIZE_16M)
 			hugepage_16MB_avail = 1;
-		if (ms[i].hugepage_sz == RTE_PGSIZE_16G)
+		if (msl->hugepage_sz == RTE_PGSIZE_16G)
 			hugepage_16GB_avail = 1;
 	}
 	/* Display the availability of 2MB ,1GB, 16MB, 16GB pages */
Signature · 1 line
2.7.4
#20 lore raw

PATCH 12/23 eal: add support for dynamic memory allocation

In reply to Anatoly Burakov

Nothing uses that code yet. The bulk of it is copied from old memory allocation stuff (eal_memory.c). We provide an API to allocate either one page or multiple pages, guaranteeing that we'll get contiguous VA for all of the pages that we requested.

---
 lib/librte_eal/common/eal_memalloc.h       |  47 ++++
 lib/librte_eal/linuxapp/eal/Makefile       |   2 +
 lib/librte_eal/linuxapp/eal/eal_memalloc.c | 416 +++++++++++++++++++++++++++++
 3 files changed, 465 insertions(+)
 create mode 100755 lib/librte_eal/common/eal_memalloc.h
 create mode 100755 lib/librte_eal/linuxapp/eal/eal_memalloc.c
lib/librte_eal/common/eal_memalloc.h new file +47 -0
new file mode 100755
index 0000000..59fd330
@@ -0,0 +1,47 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef EAL_MEMALLOC_H
+#define EAL_MEMALLOC_H
+
+#include <stdbool.h>
+
+#include <rte_memory.h>
+
+struct rte_memseg *
+eal_memalloc_alloc_page(uint64_t size, int socket);
+
+int
+eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n, uint64_t size,
+		int socket, bool exact);
+
+#endif // EAL_MEMALLOC_H
lib/librte_eal/linuxapp/eal/Makefile +2 -0
index 782e1ad..88f10e9 100644
@@ -62,6 +62,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_vfio.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_vfio_mp_sync.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_debug.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_lcore.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_timer.c
@@ -105,6 +106,7 @@ CFLAGS_eal_interrupts.o := -D_GNU_SOURCE
 CFLAGS_eal_vfio_mp_sync.o := -D_GNU_SOURCE
 CFLAGS_eal_timer.o := -D_GNU_SOURCE
 CFLAGS_eal_lcore.o := -D_GNU_SOURCE
+CFLAGS_eal_memalloc.o := -D_GNU_SOURCE
 CFLAGS_eal_thread.o := -D_GNU_SOURCE
 CFLAGS_eal_log.o := -D_GNU_SOURCE
 CFLAGS_eal_common_log.o := -D_GNU_SOURCE
lib/librte_eal/linuxapp/eal/eal_memalloc.c new file +416 -0

416 lines not on this page · raw message

Signature · 1 line
2.7.4
#21 lore raw

RFC 12/23 eal/memalloc: add support for dynamic memory allocation

In reply to Anatoly Burakov

Nothing uses that code yet. The bulk of it is copied from old memory allocation stuff (eal_memory.c). We provide an API to allocate either one page or multiple pages, guaranteeing that we'll get contiguous VA for all of the pages that we requested.

---
 lib/librte_eal/common/eal_memalloc.h       |  47 ++++
 lib/librte_eal/linuxapp/eal/Makefile       |   2 +
 lib/librte_eal/linuxapp/eal/eal_memalloc.c | 416 +++++++++++++++++++++++++++++
 3 files changed, 465 insertions(+)
 create mode 100755 lib/librte_eal/common/eal_memalloc.h
 create mode 100755 lib/librte_eal/linuxapp/eal/eal_memalloc.c
lib/librte_eal/common/eal_memalloc.h new file +47 -0
new file mode 100755
index 0000000..59fd330
@@ -0,0 +1,47 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef EAL_MEMALLOC_H
+#define EAL_MEMALLOC_H
+
+#include <stdbool.h>
+
+#include <rte_memory.h>
+
+struct rte_memseg *
+eal_memalloc_alloc_page(uint64_t size, int socket);
+
+int
+eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n, uint64_t size,
+		int socket, bool exact);
+
+#endif // EAL_MEMALLOC_H
lib/librte_eal/linuxapp/eal/Makefile +2 -0
index 782e1ad..88f10e9 100644
@@ -62,6 +62,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_vfio.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_vfio_mp_sync.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_debug.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_lcore.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_timer.c
@@ -105,6 +106,7 @@ CFLAGS_eal_interrupts.o := -D_GNU_SOURCE
 CFLAGS_eal_vfio_mp_sync.o := -D_GNU_SOURCE
 CFLAGS_eal_timer.o := -D_GNU_SOURCE
 CFLAGS_eal_lcore.o := -D_GNU_SOURCE
+CFLAGS_eal_memalloc.o := -D_GNU_SOURCE
 CFLAGS_eal_thread.o := -D_GNU_SOURCE
 CFLAGS_eal_log.o := -D_GNU_SOURCE
 CFLAGS_eal_common_log.o := -D_GNU_SOURCE
lib/librte_eal/linuxapp/eal/eal_memalloc.c new file +416 -0

416 lines not on this page · raw message

Signature · 1 line
2.7.4
#22 lore raw

PATCH 13/23 eal: make use of dynamic memory allocation for init

In reply to Anatoly Burakov

Add a new (non-legacy) memory init path for EAL. It uses the new dynamic allocation facilities, although it's only being run at startup.

If no -m or --socket-mem switches were specified, the new init will not allocate anything, whereas if those switches were passed, appropriate amounts of pages would be requested, just like for legacy init.

Since rte_malloc support for dynamic allocation comes in later patches, running DPDK without --socket-mem or -m switches will fail in this patch.

Also, allocated pages will be physically discontiguous (or rather, they're not guaranteed to be physically contiguous - they may still be, by accident) unless IOVA_AS_VA mode is used.

---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 60 ++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
lib/librte_eal/linuxapp/eal/eal_memory.c +60 -0
index 59f6889..7cc4a55 100644
@@ -68,6 +68,7 @@
 #include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "eal_memalloc.h"
 #include "eal_internal_cfg.h"
 #include "eal_filesystem.h"
 #include "eal_hugepages.h"
@@ -1322,6 +1323,61 @@ eal_legacy_hugepage_init(void)
 	return -1;
 }
 
+static int
+eal_hugepage_init(void) {
+	struct hugepage_info used_hp[MAX_HUGEPAGE_SIZES];
+	uint64_t memory[RTE_MAX_NUMA_NODES];
+	int hp_sz_idx, socket_id;
+
+	test_phys_addrs_available();
+
+	memset(used_hp, 0, sizeof(used_hp));
+
+	for (hp_sz_idx = 0;
+			hp_sz_idx < (int) internal_config.num_hugepage_sizes;
+			hp_sz_idx++) {
+		/* meanwhile, also initialize used_hp hugepage sizes in used_hp */
+		struct hugepage_info *hpi;
+		hpi = &internal_config.hugepage_info[hp_sz_idx];
+		used_hp[hp_sz_idx].hugepage_sz = hpi->hugepage_sz;
+	}
+
+	/* make a copy of socket_mem, needed for balanced allocation. */
+	for (hp_sz_idx = 0; hp_sz_idx < RTE_MAX_NUMA_NODES; hp_sz_idx++)
+		memory[hp_sz_idx] = internal_config.socket_mem[hp_sz_idx];
+
+	/* calculate final number of pages */
+	if (calc_num_pages_per_socket(memory,
+			internal_config.hugepage_info, used_hp,
+			internal_config.num_hugepage_sizes) < 0)
+		return -1;
+
+	for (int hp_sz_idx = 0;
+			hp_sz_idx < (int) internal_config.num_hugepage_sizes;
+			hp_sz_idx++) {
+		for (socket_id = 0; socket_id < RTE_MAX_NUMA_NODES;
+				socket_id++) {
+			struct hugepage_info *hpi = &used_hp[hp_sz_idx];
+			unsigned num_pages = hpi->num_pages[socket_id];
+			int num_pages_alloc;
+
+			if (num_pages == 0)
+				continue;
+
+			RTE_LOG(DEBUG, EAL, "Allocating %u pages of size %luM on socket %i\n",
+				num_pages, hpi->hugepage_sz >> 20, socket_id);
+
+			num_pages_alloc = eal_memalloc_alloc_page_bulk(NULL,
+					num_pages,
+					hpi->hugepage_sz, socket_id,
+					true);
+			if (num_pages_alloc < 0)
+				return -1;
+		}
+	}
+	return 0;
+}
+
 /*
  * uses fstat to report the size of a file on disk
  */
@@ -1533,6 +1589,8 @@ int
 rte_eal_hugepage_init(void) {
 	if (internal_config.legacy_mem)
 		return eal_legacy_hugepage_init();
+	else
+		return eal_hugepage_init();
 	return -1;
 }
 
@@ -1540,6 +1598,8 @@ int
 rte_eal_hugepage_attach(void) {
 	if (internal_config.legacy_mem)
 		return eal_legacy_hugepage_attach();
+	else
+		RTE_LOG(ERR, EAL, "Secondary processes aren't supported yet\n");
 	return -1;
 }
 
Signature · 1 line
2.7.4
#23 lore raw

RFC 13/23 eal/memory: make use of dynamic memory allocation for init

In reply to Anatoly Burakov

Add a new (non-legacy) memory init path for EAL. It uses the new dynamic allocation facilities, although it's only being run at startup.

If no -m or --socket-mem switches were specified, the new init will not allocate anything, whereas if those switches were passed, appropriate amounts of pages would be requested, just like for legacy init.

Since rte_malloc support for dynamic allocation comes in later patches, running DPDK without --socket-mem or -m switches will fail in this patch.

Also, allocated pages will be physically discontiguous (or rather, they're not guaranteed to be physically contiguous - they may still be, by accident) unless IOVA_AS_VA mode is used.

---
 lib/librte_eal/linuxapp/eal/eal_memory.c | 60 ++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
lib/librte_eal/linuxapp/eal/eal_memory.c +60 -0
index 59f6889..7cc4a55 100644
@@ -68,6 +68,7 @@
 #include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "eal_memalloc.h"
 #include "eal_internal_cfg.h"
 #include "eal_filesystem.h"
 #include "eal_hugepages.h"
@@ -1322,6 +1323,61 @@ eal_legacy_hugepage_init(void)
 	return -1;
 }
 
+static int
+eal_hugepage_init(void) {
+	struct hugepage_info used_hp[MAX_HUGEPAGE_SIZES];
+	uint64_t memory[RTE_MAX_NUMA_NODES];
+	int hp_sz_idx, socket_id;
+
+	test_phys_addrs_available();
+
+	memset(used_hp, 0, sizeof(used_hp));
+
+	for (hp_sz_idx = 0;
+			hp_sz_idx < (int) internal_config.num_hugepage_sizes;
+			hp_sz_idx++) {
+		/* meanwhile, also initialize used_hp hugepage sizes in used_hp */
+		struct hugepage_info *hpi;
+		hpi = &internal_config.hugepage_info[hp_sz_idx];
+		used_hp[hp_sz_idx].hugepage_sz = hpi->hugepage_sz;
+	}
+
+	/* make a copy of socket_mem, needed for balanced allocation. */
+	for (hp_sz_idx = 0; hp_sz_idx < RTE_MAX_NUMA_NODES; hp_sz_idx++)
+		memory[hp_sz_idx] = internal_config.socket_mem[hp_sz_idx];
+
+	/* calculate final number of pages */
+	if (calc_num_pages_per_socket(memory,
+			internal_config.hugepage_info, used_hp,
+			internal_config.num_hugepage_sizes) < 0)
+		return -1;
+
+	for (int hp_sz_idx = 0;
+			hp_sz_idx < (int) internal_config.num_hugepage_sizes;
+			hp_sz_idx++) {
+		for (socket_id = 0; socket_id < RTE_MAX_NUMA_NODES;
+				socket_id++) {
+			struct hugepage_info *hpi = &used_hp[hp_sz_idx];
+			unsigned num_pages = hpi->num_pages[socket_id];
+			int num_pages_alloc;
+
+			if (num_pages == 0)
+				continue;
+
+			RTE_LOG(DEBUG, EAL, "Allocating %u pages of size %luM on socket %i\n",
+				num_pages, hpi->hugepage_sz >> 20, socket_id);
+
+			num_pages_alloc = eal_memalloc_alloc_page_bulk(NULL,
+					num_pages,
+					hpi->hugepage_sz, socket_id,
+					true);
+			if (num_pages_alloc < 0)
+				return -1;
+		}
+	}
+	return 0;
+}
+
 /*
  * uses fstat to report the size of a file on disk
  */
@@ -1533,6 +1589,8 @@ int
 rte_eal_hugepage_init(void) {
 	if (internal_config.legacy_mem)
 		return eal_legacy_hugepage_init();
+	else
+		return eal_hugepage_init();
 	return -1;
 }
 
@@ -1540,6 +1598,8 @@ int
 rte_eal_hugepage_attach(void) {
 	if (internal_config.legacy_mem)
 		return eal_legacy_hugepage_attach();
+	else
+		RTE_LOG(ERR, EAL, "Secondary processes aren't supported yet\n");
 	return -1;
 }
 
Signature · 1 line
2.7.4
#24 lore raw

RFC 14/23 eal/memory: add support for dynamic unmapping of pages

In reply to Anatoly Burakov

This isn't used anywhere yet, but the support is now there. Also, adding cleanup to allocation procedures, so that if we fail to allocate everything we asked for, we can free all of it back.

---
 lib/librte_eal/common/eal_memalloc.h       |   3 +
 lib/librte_eal/linuxapp/eal/eal_memalloc.c | 131 ++++++++++++++++++++++++++++-
 2 files changed, 133 insertions(+), 1 deletion(-)
lib/librte_eal/common/eal_memalloc.h +3 -0
index 59fd330..47e4367 100755
@@ -44,4 +44,7 @@ int
 eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n, uint64_t size,
 		int socket, bool exact);
 
+int
+eal_memalloc_free_page(struct rte_memseg *ms);
+
 #endif // EAL_MEMALLOC_H
lib/librte_eal/linuxapp/eal/eal_memalloc.c +130 -1

164 lines not on this page · raw message

Signature · 1 line
2.7.4
#25 lore raw

PATCH 14/23 eal: add support for dynamic unmapping of pages

In reply to Anatoly Burakov

This isn't used anywhere yet, but the support is now there. Also, adding cleanup to allocation procedures, so that if we fail to allocate everything we asked for, we can free all of it back.

---
 lib/librte_eal/common/eal_memalloc.h       |   3 +
 lib/librte_eal/linuxapp/eal/eal_memalloc.c | 131 ++++++++++++++++++++++++++++-
 2 files changed, 133 insertions(+), 1 deletion(-)
lib/librte_eal/common/eal_memalloc.h +3 -0
index 59fd330..47e4367 100755
@@ -44,4 +44,7 @@ int
 eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n, uint64_t size,
 		int socket, bool exact);
 
+int
+eal_memalloc_free_page(struct rte_memseg *ms);
+
 #endif // EAL_MEMALLOC_H
lib/librte_eal/linuxapp/eal/eal_memalloc.c +130 -1

164 lines not on this page · raw message

Signature · 1 line
2.7.4
#26 lore raw

PATCH 15/23 eal: add API to check if memory is physically contiguous

In reply to Anatoly Burakov

This will be helpful down the line when we implement support for allocating physically contiguous memory. We can no longer guarantee physically contiguous memory unless we're in IOVA_AS_VA mode, but we can certainly try and see if we succeed. In addition, this would be useful for e.g. PMD's who may allocate chunks that are smaller than the pagesize, but they must not cross the page boundary, in which case we will be able to accommodate that request.

---
 lib/librte_eal/common/eal_common_memalloc.c | 79 +++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_memalloc.h        |  5 ++
 lib/librte_eal/linuxapp/eal/Makefile        |  1 +
 3 files changed, 85 insertions(+)
 create mode 100755 lib/librte_eal/common/eal_common_memalloc.c
lib/librte_eal/common/eal_common_memalloc.c new file +79 -0
new file mode 100755
index 0000000..395753a
@@ -0,0 +1,79 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_lcore.h>
+#include <rte_fbarray.h>
+#include <rte_memzone.h>
+#include <rte_memory.h>
+#include <rte_eal_memconfig.h>
+
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+#include "eal_memalloc.h"
+
+// TODO: secondary
+// TODO: 32-bit
+
+bool
+eal_memalloc_is_contig(const struct rte_memseg_list *msl, void *start,
+		size_t len) {
+	const struct rte_memseg *ms;
+	uint64_t page_sz;
+	void *end;
+	int start_page, end_page, cur_page;
+	rte_iova_t expected;
+
+	/* for legacy memory, it's always contiguous */
+	if (internal_config.legacy_mem)
+		return true;
+
+	/* figure out how many pages we need to fit in current data */
+	page_sz = msl->hugepage_sz;
+	end = RTE_PTR_ADD(start, len);
+
+	start_page = RTE_PTR_DIFF(start, msl->base_va) / page_sz;
+	end_page = RTE_PTR_DIFF(end, msl->base_va) / page_sz;
+
+	/* now, look for contiguous memory */
+	ms = rte_fbarray_get(&msl->memseg_arr, start_page);
+	expected = ms->iova + page_sz;
+
+	for (cur_page = start_page + 1; cur_page < end_page;
+			cur_page++, expected += page_sz) {
+		ms = rte_fbarray_get(&msl->memseg_arr, cur_page);
+
+		if (ms->iova != expected)
+			return false;
+	}
+
+	return true;
+}
lib/librte_eal/common/eal_memalloc.h +5 -0
index 47e4367..04f9b72 100755
@@ -36,6 +36,7 @@
 #include <stdbool.h>
 
 #include <rte_memory.h>
+#include <rte_eal_memconfig.h>
 
 struct rte_memseg *
 eal_memalloc_alloc_page(uint64_t size, int socket);
@@ -47,4 +48,8 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n, uint64_t size,
 int
 eal_memalloc_free_page(struct rte_memseg *ms);
 
+bool
+eal_memalloc_is_contig(const struct rte_memseg_list *msl, void *start,
+		size_t len);
+
 #endif // EAL_MEMALLOC_H
lib/librte_eal/linuxapp/eal/Makefile +1 -0
index 88f10e9..c1fc557 100644
@@ -75,6 +75,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_timer.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_memzone.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_memory.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_tailqs.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_errno.c
Signature · 1 line
2.7.4
#27 lore raw

RFC 15/23 eal/memalloc: add function to check if memory is physically contiguous

In reply to Anatoly Burakov

This will be helpful down the line when we implement support for allocating physically contiguous memory. We can no longer guarantee physically contiguous memory unless we're in IOVA_AS_VA mode, but we can certainly try and see if we succeed. In addition, this would be useful for e.g. PMD's who may allocate chunks that are smaller than the pagesize, but they must not cross the page boundary, in which case we will be able to accommodate that request.

---
 lib/librte_eal/common/eal_common_memalloc.c | 79 +++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_memalloc.h        |  5 ++
 lib/librte_eal/linuxapp/eal/Makefile        |  1 +
 3 files changed, 85 insertions(+)
 create mode 100755 lib/librte_eal/common/eal_common_memalloc.c
lib/librte_eal/common/eal_common_memalloc.c new file +79 -0
new file mode 100755
index 0000000..395753a
@@ -0,0 +1,79 @@
+/*-
+ *   BSD LICENSE
+ *
+ *   Copyright(c) 2017 Intel Corporation. All rights reserved.
+ *
+ *   Redistribution and use in source and binary forms, with or without
+ *   modification, are permitted provided that the following conditions
+ *   are met:
+ *
+ *     * Redistributions of source code must retain the above copyright
+ *       notice, this list of conditions and the following disclaimer.
+ *     * Redistributions in binary form must reproduce the above copyright
+ *       notice, this list of conditions and the following disclaimer in
+ *       the documentation and/or other materials provided with the
+ *       distribution.
+ *     * Neither the name of Intel Corporation nor the names of its
+ *       contributors may be used to endorse or promote products derived
+ *       from this software without specific prior written permission.
+ *
+ *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <rte_lcore.h>
+#include <rte_fbarray.h>
+#include <rte_memzone.h>
+#include <rte_memory.h>
+#include <rte_eal_memconfig.h>
+
+#include "eal_private.h"
+#include "eal_internal_cfg.h"
+#include "eal_memalloc.h"
+
+// TODO: secondary
+// TODO: 32-bit
+
+bool
+eal_memalloc_is_contig(const struct rte_memseg_list *msl, void *start,
+		size_t len) {
+	const struct rte_memseg *ms;
+	uint64_t page_sz;
+	void *end;
+	int start_page, end_page, cur_page;
+	rte_iova_t expected;
+
+	/* for legacy memory, it's always contiguous */
+	if (internal_config.legacy_mem)
+		return true;
+
+	/* figure out how many pages we need to fit in current data */
+	page_sz = msl->hugepage_sz;
+	end = RTE_PTR_ADD(start, len);
+
+	start_page = RTE_PTR_DIFF(start, msl->base_va) / page_sz;
+	end_page = RTE_PTR_DIFF(end, msl->base_va) / page_sz;
+
+	/* now, look for contiguous memory */
+	ms = rte_fbarray_get(&msl->memseg_arr, start_page);
+	expected = ms->iova + page_sz;
+
+	for (cur_page = start_page + 1; cur_page < end_page;
+			cur_page++, expected += page_sz) {
+		ms = rte_fbarray_get(&msl->memseg_arr, cur_page);
+
+		if (ms->iova != expected)
+			return false;
+	}
+
+	return true;
+}
lib/librte_eal/common/eal_memalloc.h +5 -0
index 47e4367..04f9b72 100755
@@ -36,6 +36,7 @@
 #include <stdbool.h>
 
 #include <rte_memory.h>
+#include <rte_eal_memconfig.h>
 
 struct rte_memseg *
 eal_memalloc_alloc_page(uint64_t size, int socket);
@@ -47,4 +48,8 @@ eal_memalloc_alloc_page_bulk(struct rte_memseg **ms, int n, uint64_t size,
 int
 eal_memalloc_free_page(struct rte_memseg *ms);
 
+bool
+eal_memalloc_is_contig(const struct rte_memseg_list *msl, void *start,
+		size_t len);
+
 #endif // EAL_MEMALLOC_H
lib/librte_eal/linuxapp/eal/Makefile +1 -0
index 88f10e9..c1fc557 100644
@@ -75,6 +75,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_timer.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_memzone.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_log.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_launch.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_memalloc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_memory.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_tailqs.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_errno.c
Signature · 1 line
2.7.4
#28 lore raw

RFC 16/23 eal/malloc: enable dynamic memory allocation/free on malloc/free

In reply to Anatoly Burakov

This set of changes enables rte_malloc to allocate and free memory as needed. The way it works is, first malloc checks if there is enough memory already allocated to satisfy user's request. If there isn't, we try and allocate more memory. The reverse happens with free - we free an element, check its size (including free element merging due to adjacency) and see if it's bigger than hugepage size and that its start and end span a hugepage or more. Then we remove the area from malloc heap (adjusting element lengths where appropriate), and deallocate the page.

For legacy mode, dynamic alloc/free is disabled.

It is worth noting that memseg lists are being sorted by page size, and that we try our best to satisfy user's request. That is, if the user requests an element from a 2MB page memory, we will check if we can satisfy that request from existing memory, if not we try and allocate more 2MB pages. If that fails and user also specified a "size is hint" flag, we then check other page sizes and try to allocate from there. If that fails too, then, depending on flags, we may try allocating from other sockets. In other words, we try our best to give the user what they asked for, but going to other sockets is last resort - first we try to allocate more memory on the same socket.

---
 lib/librte_eal/common/eal_common_memzone.c |  19 +-
 lib/librte_eal/common/malloc_elem.c        |  96 +++++++++-
 lib/librte_eal/common/malloc_elem.h        |   8 +-
 lib/librte_eal/common/malloc_heap.c        | 280 +++++++++++++++++++++++++++--
 lib/librte_eal/common/malloc_heap.h        |   4 +-
 lib/librte_eal/common/rte_malloc.c         |  24 +--
 6 files changed, 373 insertions(+), 58 deletions(-)
lib/librte_eal/common/eal_common_memzone.c +3 -16
index f558ac2..c571145 100644
@@ -132,7 +132,7 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 	struct rte_memzone *mz;
 	struct rte_mem_config *mcfg;
 	size_t requested_len;
-	int socket, i;
+	int socket;
 
 	/* get pointer to global configuration */
 	mcfg = rte_eal_get_configuration()->mem_config;
@@ -216,21 +216,8 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 		socket = socket_id;
 
 	/* allocate memory on heap */
-	void *mz_addr = malloc_heap_alloc(&mcfg->malloc_heaps[socket], NULL,
-			requested_len, flags, align, bound);
-
-	if ((mz_addr == NULL) && (socket_id == SOCKET_ID_ANY)) {
-		/* try other heaps */
-		for (i = 0; i < RTE_MAX_NUMA_NODES; i++) {
-			if (socket == i)
-				continue;
-
-			mz_addr = malloc_heap_alloc(&mcfg->malloc_heaps[i],
-					NULL, requested_len, flags, align, bound);
-			if (mz_addr != NULL)
-				break;
-		}
-	}
+	void *mz_addr = malloc_heap_alloc(NULL, requested_len, socket, flags,
+			align, bound);
 
 	if (mz_addr == NULL) {
 		rte_errno = ENOMEM;
lib/librte_eal/common/malloc_elem.c +88 -8

138 lines not on this page · raw message

lib/librte_eal/common/malloc_elem.h +7 -1
index 330bddc..b47c55e 100644
@@ -164,7 +164,7 @@ malloc_elem_alloc(struct malloc_elem *elem, size_t size,
  * blocks either immediately before or immediately after newly freed block
  * are also free, the blocks are merged together.
  */
-int
+struct malloc_elem *
 malloc_elem_free(struct malloc_elem *elem);
 
 struct malloc_elem *
@@ -177,6 +177,12 @@ malloc_elem_join_adjacent_free(struct malloc_elem *elem);
 int
 malloc_elem_resize(struct malloc_elem *elem, size_t size);
 
+void
+malloc_elem_hide_region(struct malloc_elem *elem, void *start, size_t len);
+
+void
+malloc_elem_free_list_remove(struct malloc_elem *elem);
+
 /*
  * Given an element size, compute its freelist index.
  */
lib/librte_eal/common/malloc_heap.c +270 -10

324 lines not on this page · raw message

lib/librte_eal/common/malloc_heap.h +2 -2
index df04dd8..3fcd14f 100644
@@ -53,8 +53,8 @@ malloc_get_numa_socket(void)
 }
 
 void *
-malloc_heap_alloc(struct malloc_heap *heap,	const char *type, size_t size,
-		unsigned flags, size_t align, size_t bound);
+malloc_heap_alloc(const char *type, size_t size, int socket, unsigned flags,
+		size_t align, size_t bound);
 
 int
 malloc_heap_free(struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +3 -21
index 92cd7d8..dc3199a 100644
@@ -68,9 +68,7 @@ void rte_free(void *addr)
 void *
 rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
 {
-	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
-	int socket, i;
-	void *ret;
+	int socket;
 
 	/* return NULL if size is 0 or alignment is not power-of-2 */
 	if (size == 0 || (align && !rte_is_power_of_2(align)))
@@ -88,24 +86,8 @@ rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
 	if (socket >= RTE_MAX_NUMA_NODES)
 		return NULL;
 
-	ret = malloc_heap_alloc(&mcfg->malloc_heaps[socket], type,
-				size, 0, align == 0 ? 1 : align, 0);
-	if (ret != NULL || socket_arg != SOCKET_ID_ANY)
-		return ret;
-
-	/* try other heaps */
-	for (i = 0; i < RTE_MAX_NUMA_NODES; i++) {
-		/* we already tried this one */
-		if (i == socket)
-			continue;
-
-		ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type,
-					size, 0, align == 0 ? 1 : align, 0);
-		if (ret != NULL)
-			return ret;
-	}
-
-	return NULL;
+	return malloc_heap_alloc(type, size, socket_arg, 0,
+			align == 0 ? 1 : align, 0);
 }
 
 /*
Signature · 1 line
2.7.4
#29 lore raw

PATCH 16/23 eal: enable dynamic memory allocation/free on malloc/free

In reply to Anatoly Burakov

This set of changes enables rte_malloc to allocate and free memory as needed. The way it works is, first malloc checks if there is enough memory already allocated to satisfy user's request. If there isn't, we try and allocate more memory. The reverse happens with free - we free an element, check its size (including free element merging due to adjacency) and see if it's bigger than hugepage size and that its start and end span a hugepage or more. Then we remove the area from malloc heap (adjusting element lengths where appropriate), and deallocate the page.

For legacy mode, dynamic alloc/free is disabled.

It is worth noting that memseg lists are being sorted by page size, and that we try our best to satisfy user's request. That is, if the user requests an element from a 2MB page memory, we will check if we can satisfy that request from existing memory, if not we try and allocate more 2MB pages. If that fails and user also specified a "size is hint" flag, we then check other page sizes and try to allocate from there. If that fails too, then, depending on flags, we may try allocating from other sockets. In other words, we try our best to give the user what they asked for, but going to other sockets is last resort - first we try to allocate more memory on the same socket.

---
 lib/librte_eal/common/eal_common_memzone.c |  19 +-
 lib/librte_eal/common/malloc_elem.c        |  96 +++++++++-
 lib/librte_eal/common/malloc_elem.h        |   8 +-
 lib/librte_eal/common/malloc_heap.c        | 280 +++++++++++++++++++++++++++--
 lib/librte_eal/common/malloc_heap.h        |   4 +-
 lib/librte_eal/common/rte_malloc.c         |  24 +--
 6 files changed, 373 insertions(+), 58 deletions(-)
lib/librte_eal/common/eal_common_memzone.c +3 -16
index f558ac2..c571145 100644
@@ -132,7 +132,7 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 	struct rte_memzone *mz;
 	struct rte_mem_config *mcfg;
 	size_t requested_len;
-	int socket, i;
+	int socket;
 
 	/* get pointer to global configuration */
 	mcfg = rte_eal_get_configuration()->mem_config;
@@ -216,21 +216,8 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 		socket = socket_id;
 
 	/* allocate memory on heap */
-	void *mz_addr = malloc_heap_alloc(&mcfg->malloc_heaps[socket], NULL,
-			requested_len, flags, align, bound);
-
-	if ((mz_addr == NULL) && (socket_id == SOCKET_ID_ANY)) {
-		/* try other heaps */
-		for (i = 0; i < RTE_MAX_NUMA_NODES; i++) {
-			if (socket == i)
-				continue;
-
-			mz_addr = malloc_heap_alloc(&mcfg->malloc_heaps[i],
-					NULL, requested_len, flags, align, bound);
-			if (mz_addr != NULL)
-				break;
-		}
-	}
+	void *mz_addr = malloc_heap_alloc(NULL, requested_len, socket, flags,
+			align, bound);
 
 	if (mz_addr == NULL) {
 		rte_errno = ENOMEM;
lib/librte_eal/common/malloc_elem.c +88 -8

138 lines not on this page · raw message

lib/librte_eal/common/malloc_elem.h +7 -1
index 330bddc..b47c55e 100644
@@ -164,7 +164,7 @@ malloc_elem_alloc(struct malloc_elem *elem, size_t size,
  * blocks either immediately before or immediately after newly freed block
  * are also free, the blocks are merged together.
  */
-int
+struct malloc_elem *
 malloc_elem_free(struct malloc_elem *elem);
 
 struct malloc_elem *
@@ -177,6 +177,12 @@ malloc_elem_join_adjacent_free(struct malloc_elem *elem);
 int
 malloc_elem_resize(struct malloc_elem *elem, size_t size);
 
+void
+malloc_elem_hide_region(struct malloc_elem *elem, void *start, size_t len);
+
+void
+malloc_elem_free_list_remove(struct malloc_elem *elem);
+
 /*
  * Given an element size, compute its freelist index.
  */
lib/librte_eal/common/malloc_heap.c +270 -10

324 lines not on this page · raw message

lib/librte_eal/common/malloc_heap.h +2 -2
index df04dd8..3fcd14f 100644
@@ -53,8 +53,8 @@ malloc_get_numa_socket(void)
 }
 
 void *
-malloc_heap_alloc(struct malloc_heap *heap,	const char *type, size_t size,
-		unsigned flags, size_t align, size_t bound);
+malloc_heap_alloc(const char *type, size_t size, int socket, unsigned flags,
+		size_t align, size_t bound);
 
 int
 malloc_heap_free(struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +3 -21
index 92cd7d8..dc3199a 100644
@@ -68,9 +68,7 @@ void rte_free(void *addr)
 void *
 rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
 {
-	struct rte_mem_config *mcfg = rte_eal_get_configuration()->mem_config;
-	int socket, i;
-	void *ret;
+	int socket;
 
 	/* return NULL if size is 0 or alignment is not power-of-2 */
 	if (size == 0 || (align && !rte_is_power_of_2(align)))
@@ -88,24 +86,8 @@ rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
 	if (socket >= RTE_MAX_NUMA_NODES)
 		return NULL;
 
-	ret = malloc_heap_alloc(&mcfg->malloc_heaps[socket], type,
-				size, 0, align == 0 ? 1 : align, 0);
-	if (ret != NULL || socket_arg != SOCKET_ID_ANY)
-		return ret;
-
-	/* try other heaps */
-	for (i = 0; i < RTE_MAX_NUMA_NODES; i++) {
-		/* we already tried this one */
-		if (i == socket)
-			continue;
-
-		ret = malloc_heap_alloc(&mcfg->malloc_heaps[i], type,
-					size, 0, align == 0 ? 1 : align, 0);
-		if (ret != NULL)
-			return ret;
-	}
-
-	return NULL;
+	return malloc_heap_alloc(type, size, socket_arg, 0,
+			align == 0 ? 1 : align, 0);
 }
 
 /*
Signature · 1 line
2.7.4
#30 lore raw

RFC 17/23 eal/malloc: add backend support for contiguous memory allocation

In reply to Anatoly Burakov

No major changes, just add some checks in a few key places, and a new parameter to pass around.

---
 lib/librte_eal/common/eal_common_memzone.c |  16 +++--
 lib/librte_eal/common/malloc_elem.c        | 105 +++++++++++++++++++++++------
 lib/librte_eal/common/malloc_elem.h        |   6 +-
 lib/librte_eal/common/malloc_heap.c        |  54 +++++++++------
 lib/librte_eal/common/malloc_heap.h        |   6 +-
 lib/librte_eal/common/rte_malloc.c         |  38 +++++++----
 6 files changed, 158 insertions(+), 67 deletions(-)
lib/librte_eal/common/eal_common_memzone.c +9 -7
index c571145..542ae90 100644
@@ -127,7 +127,8 @@ find_heap_max_free_elem(int *s, unsigned align)
 
 static const struct rte_memzone *
 memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
-		int socket_id, unsigned flags, unsigned align, unsigned bound)
+		int socket_id, unsigned flags, unsigned align, unsigned bound,
+		bool contig)
 {
 	struct rte_memzone *mz;
 	struct rte_mem_config *mcfg;
@@ -217,7 +218,7 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 
 	/* allocate memory on heap */
 	void *mz_addr = malloc_heap_alloc(NULL, requested_len, socket, flags,
-			align, bound);
+			align, bound, contig);
 
 	if (mz_addr == NULL) {
 		rte_errno = ENOMEM;
@@ -251,7 +252,7 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 static const struct rte_memzone *
 rte_memzone_reserve_thread_safe(const char *name, size_t len,
 				int socket_id, unsigned flags, unsigned align,
-				unsigned bound)
+				unsigned bound, bool contig)
 {
 	struct rte_mem_config *mcfg;
 	const struct rte_memzone *mz = NULL;
@@ -262,7 +263,7 @@ rte_memzone_reserve_thread_safe(const char *name, size_t len,
 	rte_rwlock_write_lock(&mcfg->mlock);
 
 	mz = memzone_reserve_aligned_thread_unsafe(
-		name, len, socket_id, flags, align, bound);
+		name, len, socket_id, flags, align, bound, contig);
 
 	rte_rwlock_write_unlock(&mcfg->mlock);
 
@@ -279,7 +280,7 @@ rte_memzone_reserve_bounded(const char *name, size_t len, int socket_id,
 			    unsigned flags, unsigned align, unsigned bound)
 {
 	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
-					       align, bound);
+					       align, bound, false);
 }
 
 /*
@@ -291,7 +292,7 @@ rte_memzone_reserve_aligned(const char *name, size_t len, int socket_id,
 			    unsigned flags, unsigned align)
 {
 	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
-					       align, 0);
+					       align, 0, false);
 }
 
 /*
@@ -303,7 +304,8 @@ rte_memzone_reserve(const char *name, size_t len, int socket_id,
 		    unsigned flags)
 {
 	return rte_memzone_reserve_thread_safe(name, len, socket_id,
-					       flags, RTE_CACHE_LINE_SIZE, 0);
+					       flags, RTE_CACHE_LINE_SIZE, 0,
+					       false);
 }
 
 int
lib/librte_eal/common/malloc_elem.c +83 -22

154 lines not on this page · raw message

lib/librte_eal/common/malloc_elem.h +3 -3
index b47c55e..02d6bd7 100644
@@ -149,7 +149,7 @@ malloc_elem_insert(struct malloc_elem *elem);
  */
 int
 malloc_elem_can_hold(struct malloc_elem *elem, size_t size,
-		unsigned align, size_t bound);
+		unsigned align, size_t bound, bool contig);
 
 /*
  * reserve a block of data in an existing malloc_elem. If the malloc_elem
@@ -157,7 +157,7 @@ malloc_elem_can_hold(struct malloc_elem *elem, size_t size,
  */
 struct malloc_elem *
 malloc_elem_alloc(struct malloc_elem *elem, size_t size,
-		unsigned align, size_t bound);
+		unsigned align, size_t bound, bool contig);
 
 /*
  * free a malloc_elem block by adding it to the free list. If the
@@ -175,7 +175,7 @@ malloc_elem_join_adjacent_free(struct malloc_elem *elem);
  * immediately after it in memory.
  */
 int
-malloc_elem_resize(struct malloc_elem *elem, size_t size);
+malloc_elem_resize(struct malloc_elem *elem, size_t size, bool contig);
 
 void
 malloc_elem_hide_region(struct malloc_elem *elem, void *start, size_t len);
lib/librte_eal/common/malloc_heap.c +33 -21

173 lines not on this page · raw message

lib/librte_eal/common/malloc_heap.h +4 -2
index 3fcd14f..e95b526 100644
@@ -34,6 +34,8 @@
 #ifndef MALLOC_HEAP_H_
 #define MALLOC_HEAP_H_
 
+#include <stdbool.h>
+
 #include <rte_malloc.h>
 #include <rte_malloc_heap.h>
 
@@ -54,13 +56,13 @@ malloc_get_numa_socket(void)
 
 void *
 malloc_heap_alloc(const char *type, size_t size, int socket, unsigned flags,
-		size_t align, size_t bound);
+		size_t align, size_t bound, bool contig);
 
 int
 malloc_heap_free(struct malloc_elem *elem);
 
 int
-malloc_heap_resize(struct malloc_elem *elem, size_t size);
+malloc_heap_resize(struct malloc_elem *elem, size_t size, bool contig);
 
 int
 malloc_heap_get_stats(struct malloc_heap *heap,
lib/librte_eal/common/rte_malloc.c +26 -12
index dc3199a..623725e 100644
@@ -62,12 +62,9 @@ void rte_free(void *addr)
 		rte_panic("Fatal error: Invalid memory\n");
 }
 
-/*
- * Allocate memory on specified heap.
- */
-void *
-rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
-{
+static void *
+malloc_socket(const char *type, size_t size, unsigned align, int socket_arg,
+		bool contig) {
 	int socket;
 
 	/* return NULL if size is 0 or alignment is not power-of-2 */
@@ -86,8 +83,16 @@ rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
 	if (socket >= RTE_MAX_NUMA_NODES)
 		return NULL;
 
-	return malloc_heap_alloc(type, size, socket_arg, 0,
-			align == 0 ? 1 : align, 0);
+	return malloc_heap_alloc(type, size, socket_arg, 0, align, 0, contig);
+}
+
+/*
+ * Allocate memory on specified heap.
+ */
+void *
+rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
+{
+	return malloc_socket(type, size, align, socket_arg, false);
 }
 
 /*
@@ -138,8 +143,8 @@ rte_calloc(const char *type, size_t num, size_t size, unsigned align)
 /*
  * Resize allocated memory.
  */
-void *
-rte_realloc(void *ptr, size_t size, unsigned align)
+static void *
+do_realloc(void *ptr, size_t size, unsigned align, bool contig)
 {
 	if (ptr == NULL)
 		return rte_malloc(NULL, size, align);
@@ -151,12 +156,12 @@ rte_realloc(void *ptr, size_t size, unsigned align)
 	size = RTE_CACHE_LINE_ROUNDUP(size), align = RTE_CACHE_LINE_ROUNDUP(align);
 	/* check alignment matches first, and if ok, see if we can resize block */
 	if (RTE_PTR_ALIGN(ptr,align) == ptr &&
-			malloc_heap_resize(elem, size) == 0)
+			malloc_heap_resize(elem, size, contig) == 0)
 		return ptr;
 
 	/* either alignment is off, or we have no room to expand,
 	 * so move data. */
-	void *new_ptr = rte_malloc(NULL, size, align);
+	void *new_ptr = malloc_socket(NULL, size, align, SOCKET_ID_ANY, contig);
 	if (new_ptr == NULL)
 		return NULL;
 	const unsigned old_size = elem->size - MALLOC_ELEM_OVERHEAD;
@@ -166,6 +171,15 @@ rte_realloc(void *ptr, size_t size, unsigned align)
 	return new_ptr;
 }
 
+/*
+ * Resize allocated memory.
+ */
+void *
+rte_realloc(void *ptr, size_t size, unsigned align)
+{
+	return do_realloc(ptr, size, align, false);
+}
+
 int
 rte_malloc_validate(const void *ptr, size_t *size)
 {
Signature · 1 line
2.7.4
#31 lore raw

PATCH 17/23 eal: add backend support for contiguous memory allocation

In reply to Anatoly Burakov

No major changes, just add some checks in a few key places, and a new parameter to pass around.

---
 lib/librte_eal/common/eal_common_memzone.c |  16 +++--
 lib/librte_eal/common/malloc_elem.c        | 105 +++++++++++++++++++++++------
 lib/librte_eal/common/malloc_elem.h        |   6 +-
 lib/librte_eal/common/malloc_heap.c        |  54 +++++++++------
 lib/librte_eal/common/malloc_heap.h        |   6 +-
 lib/librte_eal/common/rte_malloc.c         |  38 +++++++----
 6 files changed, 158 insertions(+), 67 deletions(-)
lib/librte_eal/common/eal_common_memzone.c +9 -7
index c571145..542ae90 100644
@@ -127,7 +127,8 @@ find_heap_max_free_elem(int *s, unsigned align)
 
 static const struct rte_memzone *
 memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
-		int socket_id, unsigned flags, unsigned align, unsigned bound)
+		int socket_id, unsigned flags, unsigned align, unsigned bound,
+		bool contig)
 {
 	struct rte_memzone *mz;
 	struct rte_mem_config *mcfg;
@@ -217,7 +218,7 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 
 	/* allocate memory on heap */
 	void *mz_addr = malloc_heap_alloc(NULL, requested_len, socket, flags,
-			align, bound);
+			align, bound, contig);
 
 	if (mz_addr == NULL) {
 		rte_errno = ENOMEM;
@@ -251,7 +252,7 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 static const struct rte_memzone *
 rte_memzone_reserve_thread_safe(const char *name, size_t len,
 				int socket_id, unsigned flags, unsigned align,
-				unsigned bound)
+				unsigned bound, bool contig)
 {
 	struct rte_mem_config *mcfg;
 	const struct rte_memzone *mz = NULL;
@@ -262,7 +263,7 @@ rte_memzone_reserve_thread_safe(const char *name, size_t len,
 	rte_rwlock_write_lock(&mcfg->mlock);
 
 	mz = memzone_reserve_aligned_thread_unsafe(
-		name, len, socket_id, flags, align, bound);
+		name, len, socket_id, flags, align, bound, contig);
 
 	rte_rwlock_write_unlock(&mcfg->mlock);
 
@@ -279,7 +280,7 @@ rte_memzone_reserve_bounded(const char *name, size_t len, int socket_id,
 			    unsigned flags, unsigned align, unsigned bound)
 {
 	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
-					       align, bound);
+					       align, bound, false);
 }
 
 /*
@@ -291,7 +292,7 @@ rte_memzone_reserve_aligned(const char *name, size_t len, int socket_id,
 			    unsigned flags, unsigned align)
 {
 	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
-					       align, 0);
+					       align, 0, false);
 }
 
 /*
@@ -303,7 +304,8 @@ rte_memzone_reserve(const char *name, size_t len, int socket_id,
 		    unsigned flags)
 {
 	return rte_memzone_reserve_thread_safe(name, len, socket_id,
-					       flags, RTE_CACHE_LINE_SIZE, 0);
+					       flags, RTE_CACHE_LINE_SIZE, 0,
+					       false);
 }
 
 int
lib/librte_eal/common/malloc_elem.c +83 -22

154 lines not on this page · raw message

lib/librte_eal/common/malloc_elem.h +3 -3
index b47c55e..02d6bd7 100644
@@ -149,7 +149,7 @@ malloc_elem_insert(struct malloc_elem *elem);
  */
 int
 malloc_elem_can_hold(struct malloc_elem *elem, size_t size,
-		unsigned align, size_t bound);
+		unsigned align, size_t bound, bool contig);
 
 /*
  * reserve a block of data in an existing malloc_elem. If the malloc_elem
@@ -157,7 +157,7 @@ malloc_elem_can_hold(struct malloc_elem *elem, size_t size,
  */
 struct malloc_elem *
 malloc_elem_alloc(struct malloc_elem *elem, size_t size,
-		unsigned align, size_t bound);
+		unsigned align, size_t bound, bool contig);
 
 /*
  * free a malloc_elem block by adding it to the free list. If the
@@ -175,7 +175,7 @@ malloc_elem_join_adjacent_free(struct malloc_elem *elem);
  * immediately after it in memory.
  */
 int
-malloc_elem_resize(struct malloc_elem *elem, size_t size);
+malloc_elem_resize(struct malloc_elem *elem, size_t size, bool contig);
 
 void
 malloc_elem_hide_region(struct malloc_elem *elem, void *start, size_t len);
lib/librte_eal/common/malloc_heap.c +33 -21

173 lines not on this page · raw message

lib/librte_eal/common/malloc_heap.h +4 -2
index 3fcd14f..e95b526 100644
@@ -34,6 +34,8 @@
 #ifndef MALLOC_HEAP_H_
 #define MALLOC_HEAP_H_
 
+#include <stdbool.h>
+
 #include <rte_malloc.h>
 #include <rte_malloc_heap.h>
 
@@ -54,13 +56,13 @@ malloc_get_numa_socket(void)
 
 void *
 malloc_heap_alloc(const char *type, size_t size, int socket, unsigned flags,
-		size_t align, size_t bound);
+		size_t align, size_t bound, bool contig);
 
 int
 malloc_heap_free(struct malloc_elem *elem);
 
 int
-malloc_heap_resize(struct malloc_elem *elem, size_t size);
+malloc_heap_resize(struct malloc_elem *elem, size_t size, bool contig);
 
 int
 malloc_heap_get_stats(struct malloc_heap *heap,
lib/librte_eal/common/rte_malloc.c +26 -12
index dc3199a..623725e 100644
@@ -62,12 +62,9 @@ void rte_free(void *addr)
 		rte_panic("Fatal error: Invalid memory\n");
 }
 
-/*
- * Allocate memory on specified heap.
- */
-void *
-rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
-{
+static void *
+malloc_socket(const char *type, size_t size, unsigned align, int socket_arg,
+		bool contig) {
 	int socket;
 
 	/* return NULL if size is 0 or alignment is not power-of-2 */
@@ -86,8 +83,16 @@ rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
 	if (socket >= RTE_MAX_NUMA_NODES)
 		return NULL;
 
-	return malloc_heap_alloc(type, size, socket_arg, 0,
-			align == 0 ? 1 : align, 0);
+	return malloc_heap_alloc(type, size, socket_arg, 0, align, 0, contig);
+}
+
+/*
+ * Allocate memory on specified heap.
+ */
+void *
+rte_malloc_socket(const char *type, size_t size, unsigned align, int socket_arg)
+{
+	return malloc_socket(type, size, align, socket_arg, false);
 }
 
 /*
@@ -138,8 +143,8 @@ rte_calloc(const char *type, size_t num, size_t size, unsigned align)
 /*
  * Resize allocated memory.
  */
-void *
-rte_realloc(void *ptr, size_t size, unsigned align)
+static void *
+do_realloc(void *ptr, size_t size, unsigned align, bool contig)
 {
 	if (ptr == NULL)
 		return rte_malloc(NULL, size, align);
@@ -151,12 +156,12 @@ rte_realloc(void *ptr, size_t size, unsigned align)
 	size = RTE_CACHE_LINE_ROUNDUP(size), align = RTE_CACHE_LINE_ROUNDUP(align);
 	/* check alignment matches first, and if ok, see if we can resize block */
 	if (RTE_PTR_ALIGN(ptr,align) == ptr &&
-			malloc_heap_resize(elem, size) == 0)
+			malloc_heap_resize(elem, size, contig) == 0)
 		return ptr;
 
 	/* either alignment is off, or we have no room to expand,
 	 * so move data. */
-	void *new_ptr = rte_malloc(NULL, size, align);
+	void *new_ptr = malloc_socket(NULL, size, align, SOCKET_ID_ANY, contig);
 	if (new_ptr == NULL)
 		return NULL;
 	const unsigned old_size = elem->size - MALLOC_ELEM_OVERHEAD;
@@ -166,6 +171,15 @@ rte_realloc(void *ptr, size_t size, unsigned align)
 	return new_ptr;
 }
 
+/*
+ * Resize allocated memory.
+ */
+void *
+rte_realloc(void *ptr, size_t size, unsigned align)
+{
+	return do_realloc(ptr, size, align, false);
+}
+
 int
 rte_malloc_validate(const void *ptr, size_t *size)
 {
Signature · 1 line
2.7.4
#32 lore raw

RFC 18/23 eal/malloc: add rte_malloc support for allocating contiguous memory

In reply to Anatoly Burakov

This adds a new set of _contig API's to rte_malloc.

---
 lib/librte_eal/common/include/rte_malloc.h | 181 +++++++++++++++++++++++++++++
 lib/librte_eal/common/rte_malloc.c         |  63 ++++++++++
 2 files changed, 244 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +181 -0
index 5d4c11a..c132d33 100644
@@ -242,6 +242,187 @@ void *
 rte_calloc_socket(const char *type, size_t num, size_t size, unsigned align, int socket);
 
 /**
+ * This function allocates memory from the huge-page area of memory. The memory
+ * is not cleared. In NUMA systems, the memory allocated resides on the same
+ * NUMA socket as the core that calls this function.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_malloc_contig(const char *type, size_t size, unsigned align);
+
+/**
+ * Allocate zero'ed memory from the heap.
+ *
+ * Equivalent to rte_malloc() except that the memory zone is
+ * initialised with zeros. In NUMA systems, the memory allocated resides on the
+ * same NUMA socket as the core that calls this function.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_zmalloc_contig(const char *type, size_t size, unsigned align);
+
+/**
+ * Replacement function for calloc(), using huge-page memory. Memory area is
+ * initialised with zeros. In NUMA systems, the memory allocated resides on the
+ * same NUMA socket as the core that calls this function.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param num
+ *   Number of elements to be allocated.
+ * @param size
+ *   Size (in bytes) of a single element.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_calloc_contig(const char *type, size_t num, size_t size, unsigned align);
+
+/**
+ * Replacement function for realloc(), using huge-page memory. Reserved area
+ * memory is resized, preserving contents. In NUMA systems, the new area
+ * resides on the same NUMA socket as the old area.
+ *
+ * @param ptr
+ *   Pointer to already allocated memory
+ * @param size
+ *   Size (in bytes) of new area. If this is 0, memory is freed.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the reallocated memory.
+ */
+void *
+rte_realloc_contig(void *ptr, size_t size, unsigned align);
+
+/**
+ * This function allocates memory from the huge-page area of memory. The memory
+ * is not cleared.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @param socket
+ *   NUMA socket to allocate memory on. If SOCKET_ID_ANY is used, this function
+ *   will behave the same as rte_malloc().
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_malloc_socket_contig(const char *type, size_t size, unsigned align, int socket);
+
+/**
+ * Allocate zero'ed memory from the heap.
+ *
+ * Equivalent to rte_malloc() except that the memory zone is
+ * initialised with zeros.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @param socket
+ *   NUMA socket to allocate memory on. If SOCKET_ID_ANY is used, this function
+ *   will behave the same as rte_zmalloc().
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_zmalloc_socket_contig(const char *type, size_t size, unsigned align, int socket);
+
+/**
+ * Replacement function for calloc(), using huge-page memory. Memory area is
+ * initialised with zeros.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param num
+ *   Number of elements to be allocated.
+ * @param size
+ *   Size (in bytes) of a single element.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @param socket
+ *   NUMA socket to allocate memory on. If SOCKET_ID_ANY is used, this function
+ *   will behave the same as rte_calloc().
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_calloc_socket_contig(const char *type, size_t num, size_t size, unsigned align, int socket);
+
+/**
  * Frees the memory space pointed to by the provided pointer.
  *
  * This pointer must have been returned by a previous call to
lib/librte_eal/common/rte_malloc.c +63 -0

105 lines not on this page · raw message

Signature · 1 line
2.7.4
#33 lore raw

PATCH 18/23 eal: add rte_malloc support for allocating contiguous memory

In reply to Anatoly Burakov

This adds a new set of _contig API's to rte_malloc.

---
 lib/librte_eal/common/include/rte_malloc.h | 181 +++++++++++++++++++++++++++++
 lib/librte_eal/common/rte_malloc.c         |  63 ++++++++++
 2 files changed, 244 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +181 -0
index 5d4c11a..c132d33 100644
@@ -242,6 +242,187 @@ void *
 rte_calloc_socket(const char *type, size_t num, size_t size, unsigned align, int socket);
 
 /**
+ * This function allocates memory from the huge-page area of memory. The memory
+ * is not cleared. In NUMA systems, the memory allocated resides on the same
+ * NUMA socket as the core that calls this function.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_malloc_contig(const char *type, size_t size, unsigned align);
+
+/**
+ * Allocate zero'ed memory from the heap.
+ *
+ * Equivalent to rte_malloc() except that the memory zone is
+ * initialised with zeros. In NUMA systems, the memory allocated resides on the
+ * same NUMA socket as the core that calls this function.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_zmalloc_contig(const char *type, size_t size, unsigned align);
+
+/**
+ * Replacement function for calloc(), using huge-page memory. Memory area is
+ * initialised with zeros. In NUMA systems, the memory allocated resides on the
+ * same NUMA socket as the core that calls this function.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param num
+ *   Number of elements to be allocated.
+ * @param size
+ *   Size (in bytes) of a single element.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_calloc_contig(const char *type, size_t num, size_t size, unsigned align);
+
+/**
+ * Replacement function for realloc(), using huge-page memory. Reserved area
+ * memory is resized, preserving contents. In NUMA systems, the new area
+ * resides on the same NUMA socket as the old area.
+ *
+ * @param ptr
+ *   Pointer to already allocated memory
+ * @param size
+ *   Size (in bytes) of new area. If this is 0, memory is freed.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the reallocated memory.
+ */
+void *
+rte_realloc_contig(void *ptr, size_t size, unsigned align);
+
+/**
+ * This function allocates memory from the huge-page area of memory. The memory
+ * is not cleared.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @param socket
+ *   NUMA socket to allocate memory on. If SOCKET_ID_ANY is used, this function
+ *   will behave the same as rte_malloc().
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_malloc_socket_contig(const char *type, size_t size, unsigned align, int socket);
+
+/**
+ * Allocate zero'ed memory from the heap.
+ *
+ * Equivalent to rte_malloc() except that the memory zone is
+ * initialised with zeros.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param size
+ *   Size (in bytes) to be allocated.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @param socket
+ *   NUMA socket to allocate memory on. If SOCKET_ID_ANY is used, this function
+ *   will behave the same as rte_zmalloc().
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_zmalloc_socket_contig(const char *type, size_t size, unsigned align, int socket);
+
+/**
+ * Replacement function for calloc(), using huge-page memory. Memory area is
+ * initialised with zeros.
+ *
+ * @param type
+ *   A string identifying the type of allocated objects (useful for debug
+ *   purposes, such as identifying the cause of a memory leak). Can be NULL.
+ * @param num
+ *   Number of elements to be allocated.
+ * @param size
+ *   Size (in bytes) of a single element.
+ * @param align
+ *   If 0, the return is a pointer that is suitably aligned for any kind of
+ *   variable (in the same manner as malloc()).
+ *   Otherwise, the return is a pointer that is a multiple of *align*. In
+ *   this case, it must obviously be a power of two. (Minimum alignment is the
+ *   cacheline size, i.e. 64-bytes)
+ * @param socket
+ *   NUMA socket to allocate memory on. If SOCKET_ID_ANY is used, this function
+ *   will behave the same as rte_calloc().
+ * @return
+ *   - NULL on error. Not enough memory, or invalid arguments (size is 0,
+ *     align is not a power of two).
+ *   - Otherwise, the pointer to the allocated object.
+ */
+void *
+rte_calloc_socket_contig(const char *type, size_t num, size_t size, unsigned align, int socket);
+
+/**
  * Frees the memory space pointed to by the provided pointer.
  *
  * This pointer must have been returned by a previous call to
lib/librte_eal/common/rte_malloc.c +63 -0

105 lines not on this page · raw message

Signature · 1 line
2.7.4
#34 lore raw

PATCH 19/23 eal: enable reserving physically contiguous memzones

In reply to Anatoly Burakov

This adds a new set of _contig API's to rte_memzone.

---
 lib/librte_eal/common/eal_common_memzone.c  |  44 ++++++++
 lib/librte_eal/common/include/rte_memzone.h | 158 ++++++++++++++++++++++++++++
 2 files changed, 202 insertions(+)
lib/librte_eal/common/eal_common_memzone.c +44 -0
index 542ae90..a9a4bef 100644
@@ -200,6 +200,12 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 		socket_id = SOCKET_ID_ANY;
 
 	if (len == 0) {
+		/* len == 0 is only allowed for non-contiguous zones */
+		// TODO: technically, we can make it work, is it worth it?
+		if (contig) {
+			rte_errno = EINVAL;
+			return NULL;
+		}
 		if (bound != 0)
 			requested_len = bound;
 		else {
@@ -285,6 +291,19 @@ rte_memzone_reserve_bounded(const char *name, size_t len, int socket_id,
 
 /*
  * Return a pointer to a correctly filled memzone descriptor (with a
+ * specified alignment and boundary). If the allocation cannot be done,
+ * return NULL.
+ */
+const struct rte_memzone *
+rte_memzone_reserve_bounded_contig(const char *name, size_t len, int socket_id,
+			    unsigned flags, unsigned align, unsigned bound)
+{
+	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
+					       align, bound, true);
+}
+
+/*
+ * Return a pointer to a correctly filled memzone descriptor (with a
  * specified alignment). If the allocation cannot be done, return NULL.
  */
 const struct rte_memzone *
@@ -296,6 +315,18 @@ rte_memzone_reserve_aligned(const char *name, size_t len, int socket_id,
 }
 
 /*
+ * Return a pointer to a correctly filled memzone descriptor (with a
+ * specified alignment). If the allocation cannot be done, return NULL.
+ */
+const struct rte_memzone *
+rte_memzone_reserve_aligned_contig(const char *name, size_t len, int socket_id,
+			    unsigned flags, unsigned align)
+{
+	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
+					       align, 0, true);
+}
+
+/*
  * Return a pointer to a correctly filled memzone descriptor. If the
  * allocation cannot be done, return NULL.
  */
@@ -308,6 +339,19 @@ rte_memzone_reserve(const char *name, size_t len, int socket_id,
 					       false);
 }
 
+/*
+ * Return a pointer to a correctly filled memzone descriptor. If the
+ * allocation cannot be done, return NULL.
+ */
+const struct rte_memzone *
+rte_memzone_reserve_contig(const char *name, size_t len, int socket_id,
+		    unsigned flags)
+{
+	return rte_memzone_reserve_thread_safe(name, len, socket_id,
+					       flags, RTE_CACHE_LINE_SIZE, 0,
+					       true);
+}
+
 int
 rte_memzone_free(const struct rte_memzone *mz)
 {
lib/librte_eal/common/include/rte_memzone.h +158 -0

164 lines not on this page · raw message

Signature · 1 line
2.7.4
#35 lore raw

RFC 19/23 eal/memzone: add support for reserving physically contiguous memzones

In reply to Anatoly Burakov

This adds a new set of _contig API's to rte_memzone.

---
 lib/librte_eal/common/eal_common_memzone.c  |  44 ++++++++
 lib/librte_eal/common/include/rte_memzone.h | 158 ++++++++++++++++++++++++++++
 2 files changed, 202 insertions(+)
lib/librte_eal/common/eal_common_memzone.c +44 -0
index 542ae90..a9a4bef 100644
@@ -200,6 +200,12 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
 		socket_id = SOCKET_ID_ANY;
 
 	if (len == 0) {
+		/* len == 0 is only allowed for non-contiguous zones */
+		// TODO: technically, we can make it work, is it worth it?
+		if (contig) {
+			rte_errno = EINVAL;
+			return NULL;
+		}
 		if (bound != 0)
 			requested_len = bound;
 		else {
@@ -285,6 +291,19 @@ rte_memzone_reserve_bounded(const char *name, size_t len, int socket_id,
 
 /*
  * Return a pointer to a correctly filled memzone descriptor (with a
+ * specified alignment and boundary). If the allocation cannot be done,
+ * return NULL.
+ */
+const struct rte_memzone *
+rte_memzone_reserve_bounded_contig(const char *name, size_t len, int socket_id,
+			    unsigned flags, unsigned align, unsigned bound)
+{
+	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
+					       align, bound, true);
+}
+
+/*
+ * Return a pointer to a correctly filled memzone descriptor (with a
  * specified alignment). If the allocation cannot be done, return NULL.
  */
 const struct rte_memzone *
@@ -296,6 +315,18 @@ rte_memzone_reserve_aligned(const char *name, size_t len, int socket_id,
 }
 
 /*
+ * Return a pointer to a correctly filled memzone descriptor (with a
+ * specified alignment). If the allocation cannot be done, return NULL.
+ */
+const struct rte_memzone *
+rte_memzone_reserve_aligned_contig(const char *name, size_t len, int socket_id,
+			    unsigned flags, unsigned align)
+{
+	return rte_memzone_reserve_thread_safe(name, len, socket_id, flags,
+					       align, 0, true);
+}
+
+/*
  * Return a pointer to a correctly filled memzone descriptor. If the
  * allocation cannot be done, return NULL.
  */
@@ -308,6 +339,19 @@ rte_memzone_reserve(const char *name, size_t len, int socket_id,
 					       false);
 }
 
+/*
+ * Return a pointer to a correctly filled memzone descriptor. If the
+ * allocation cannot be done, return NULL.
+ */
+const struct rte_memzone *
+rte_memzone_reserve_contig(const char *name, size_t len, int socket_id,
+		    unsigned flags)
+{
+	return rte_memzone_reserve_thread_safe(name, len, socket_id,
+					       flags, RTE_CACHE_LINE_SIZE, 0,
+					       true);
+}
+
 int
 rte_memzone_free(const struct rte_memzone *mz)
 {
lib/librte_eal/common/include/rte_memzone.h +158 -0

164 lines not on this page · raw message

Signature · 1 line
2.7.4
#36 lore raw

PATCH 20/23 eal: make memzones use rte_fbarray

In reply to Anatoly Burakov

We greatly expand memzone list, and it makes some operations faster. Plus, it's there, so we might as well use it.

As part of this commit, a potential memory leak is fixed (when we allocate a memzone but there's no room in config, we don't free it back), and there's a compile fix for ENA driver.

---
 config/common_base                                |   2 +-
 drivers/net/ena/ena_ethdev.c                      |  10 +-
 lib/librte_eal/common/eal_common_memzone.c        | 168 ++++++++++++++++------
 lib/librte_eal/common/include/rte_eal_memconfig.h |   4 +-
 4 files changed, 137 insertions(+), 47 deletions(-)
config/common_base +1 -1
index 9730d4c..cce464d 100644
@@ -92,7 +92,7 @@ CONFIG_RTE_MAX_LCORE=128
 CONFIG_RTE_MAX_NUMA_NODES=8
 CONFIG_RTE_MAX_MEMSEG_LISTS=16
 CONFIG_RTE_MAX_MEMSEG_PER_LIST=32768
-CONFIG_RTE_MAX_MEMZONE=2560
+CONFIG_RTE_MAX_MEMZONE=32768
 CONFIG_RTE_MAX_TAILQ=32
 CONFIG_RTE_ENABLE_ASSERT=n
 CONFIG_RTE_LOG_LEVEL=RTE_LOG_INFO
drivers/net/ena/ena_ethdev.c +7 -3
index 22db895..aa37cad 100644
@@ -249,11 +249,15 @@ static const struct eth_dev_ops ena_dev_ops = {
 static inline int ena_cpu_to_node(int cpu)
 {
 	struct rte_config *config = rte_eal_get_configuration();
+	const struct rte_fbarray *arr = &config->mem_config->memzones;
+	const struct rte_memzone *mz;
 
-	if (likely(cpu < RTE_MAX_MEMZONE))
-		return config->mem_config->memzone[cpu].socket_id;
+	if (unlikely(cpu >= RTE_MAX_MEMZONE))
+		return NUMA_NO_NODE;
 
-	return NUMA_NO_NODE;
+	mz = rte_fbarray_get(arr, cpu);
+
+	return mz->socket_id;
 }
 
 static inline void ena_rx_mbuf_prepare(struct rte_mbuf *mbuf,
lib/librte_eal/common/eal_common_memzone.c +128 -40

273 lines not on this page · raw message

lib/librte_eal/common/include/rte_eal_memconfig.h +1 -3
index c9b57a4..8f4cc34 100644
@@ -86,10 +86,8 @@ struct rte_mem_config {
 	rte_rwlock_t qlock;   /**< used for tailq operation for thread safe. */
 	rte_rwlock_t mplock;  /**< only used by mempool LIB for thread-safe. */
 
-	uint32_t memzone_cnt; /**< Number of allocated memzones */
-
 	/* memory segments and zones */
-	struct rte_memzone memzone[RTE_MAX_MEMZONE]; /**< Memzone descriptors. */
+	struct rte_fbarray memzones; /**< Memzone descriptors. */
 
 	struct rte_memseg_list memsegs[RTE_MAX_MEMSEG_LISTS];
 	/**< list of dynamic arrays holding memsegs */
Signature · 1 line
2.7.4
#37 lore raw

RFC 20/23 eal/memzone: make memzones use rte_fbarray

In reply to Anatoly Burakov

We greatly expand memzone list, and it makes some operations faster. Plus, it's there, so we might as well use it.

As part of this commit, a potential memory leak is fixed (when we allocate a memzone but there's no room in config, we don't free it back), and there's a compile fix for ENA driver.

---
 config/common_base                                |   2 +-
 drivers/net/ena/ena_ethdev.c                      |  10 +-
 lib/librte_eal/common/eal_common_memzone.c        | 168 ++++++++++++++++------
 lib/librte_eal/common/include/rte_eal_memconfig.h |   4 +-
 4 files changed, 137 insertions(+), 47 deletions(-)
config/common_base +1 -1
index 9730d4c..cce464d 100644
@@ -92,7 +92,7 @@ CONFIG_RTE_MAX_LCORE=128
 CONFIG_RTE_MAX_NUMA_NODES=8
 CONFIG_RTE_MAX_MEMSEG_LISTS=16
 CONFIG_RTE_MAX_MEMSEG_PER_LIST=32768
-CONFIG_RTE_MAX_MEMZONE=2560
+CONFIG_RTE_MAX_MEMZONE=32768
 CONFIG_RTE_MAX_TAILQ=32
 CONFIG_RTE_ENABLE_ASSERT=n
 CONFIG_RTE_LOG_LEVEL=RTE_LOG_INFO
drivers/net/ena/ena_ethdev.c +7 -3
index 22db895..aa37cad 100644
@@ -249,11 +249,15 @@ static const struct eth_dev_ops ena_dev_ops = {
 static inline int ena_cpu_to_node(int cpu)
 {
 	struct rte_config *config = rte_eal_get_configuration();
+	const struct rte_fbarray *arr = &config->mem_config->memzones;
+	const struct rte_memzone *mz;
 
-	if (likely(cpu < RTE_MAX_MEMZONE))
-		return config->mem_config->memzone[cpu].socket_id;
+	if (unlikely(cpu >= RTE_MAX_MEMZONE))
+		return NUMA_NO_NODE;
 
-	return NUMA_NO_NODE;
+	mz = rte_fbarray_get(arr, cpu);
+
+	return mz->socket_id;
 }
 
 static inline void ena_rx_mbuf_prepare(struct rte_mbuf *mbuf,
lib/librte_eal/common/eal_common_memzone.c +128 -40

273 lines not on this page · raw message

lib/librte_eal/common/include/rte_eal_memconfig.h +1 -3
index c9b57a4..8f4cc34 100644
@@ -86,10 +86,8 @@ struct rte_mem_config {
 	rte_rwlock_t qlock;   /**< used for tailq operation for thread safe. */
 	rte_rwlock_t mplock;  /**< only used by mempool LIB for thread-safe. */
 
-	uint32_t memzone_cnt; /**< Number of allocated memzones */
-
 	/* memory segments and zones */
-	struct rte_memzone memzone[RTE_MAX_MEMZONE]; /**< Memzone descriptors. */
+	struct rte_fbarray memzones; /**< Memzone descriptors. */
 
 	struct rte_memseg_list memsegs[RTE_MAX_MEMSEG_LISTS];
 	/**< list of dynamic arrays holding memsegs */
Signature · 1 line
2.7.4
#38 lore raw

PATCH 21/23 mempool: add support for the new memory allocation methods

In reply to Anatoly Burakov

If a user has specified that the zone should have contiguous memory, use the new _contig allocation API's instead of normal ones. Otherwise, account for the fact that unless we're in IOVA_AS_VA mode, we cannot guarantee that the pages would be physically contiguous, so we calculate the memzone size and alignments as if we were getting the smallest page size available.

---
 lib/librte_mempool/rte_mempool.c | 84 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 75 insertions(+), 9 deletions(-)
lib/librte_mempool/rte_mempool.c +75 -9
index d50dba4..4b9ab22 100644
@@ -127,6 +127,26 @@ static unsigned optimize_object_size(unsigned obj_size)
 	return new_obj_size * RTE_MEMPOOL_ALIGN;
 }
 
+static size_t
+get_min_page_size(void) {
+	const struct rte_mem_config *mcfg =
+			rte_eal_get_configuration()->mem_config;
+	int i;
+	size_t min_pagesz = SIZE_MAX;
+
+	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+		const struct rte_memseg_list *msl = &mcfg->memsegs[i];
+
+		if (msl->base_va == NULL)
+			continue;
+
+		if (msl->hugepage_sz < min_pagesz)
+			min_pagesz = msl->hugepage_sz;
+	}
+
+	return min_pagesz == SIZE_MAX ? (size_t) getpagesize() : min_pagesz;
+}
+
 static void
 mempool_add_elem(struct rte_mempool *mp, void *obj, rte_iova_t iova)
 {
@@ -568,6 +588,7 @@ rte_mempool_populate_default(struct rte_mempool *mp)
 	unsigned mz_id, n;
 	unsigned int mp_flags;
 	int ret;
+	bool force_contig, no_contig;
 
 	/* mempool must not be populated */
 	if (mp->nb_mem_chunks != 0)
@@ -582,10 +603,46 @@ rte_mempool_populate_default(struct rte_mempool *mp)
 	/* update mempool capabilities */
 	mp->flags |= mp_flags;
 
-	if (rte_eal_has_hugepages()) {
-		pg_shift = 0; /* not needed, zone is physically contiguous */
+	no_contig = mp->flags & MEMPOOL_F_NO_PHYS_CONTIG;
+	force_contig = mp->flags & MEMPOOL_F_CAPA_PHYS_CONTIG;
+
+	/*
+	 * there are several considerations for page size and page shift here.
+	 *
+	 * if we don't need our mempools to have physically contiguous objects,
+	 * then just set page shift and page size to 0, because the user has
+	 * indicated that there's no need to care about anything.
+	 *
+	 * if we do need contiguous objects, there is also an option to reserve
+	 * the entire mempool memory as one contiguous block of memory, in
+	 * which case the page shift and alignment wouldn't matter as well.
+	 *
+	 * if we require contiguous objects, but not necessarily the entire
+	 * mempool reserved space to be contiguous, then there are two options.
+	 *
+	 * if our IO addresses are virtual, not actual physical (IOVA as VA
+	 * case), then no page shift needed - our memory allocation will give us
+	 * contiguous physical memory as far as the hardware is concerned, so
+	 * act as if we're getting contiguous memory.
+	 *
+	 * if our IO addresses are physical, we may get memory from bigger
+	 * pages, or we might get memory from smaller pages, and how much of it
+	 * we require depends on whether we want bigger or smaller pages.
+	 * However, requesting each and every memory size is too much work, so
+	 * what we'll do instead is walk through the page sizes available, pick
+	 * the smallest one and set up page shift to match that one. We will be
+	 * wasting some space this way, but it's much nicer than looping around
+	 * trying to reserve each and every page size.
+	 */
+
+	if (no_contig || force_contig || rte_eal_iova_mode() == RTE_IOVA_VA) {
 		pg_sz = 0;
+		pg_shift = 0;
 		align = RTE_CACHE_LINE_SIZE;
+	} else if (rte_eal_has_hugepages()) {
+		pg_sz = get_min_page_size();
+		pg_shift = rte_bsf32(pg_sz);
+		align = pg_sz;
 	} else {
 		pg_sz = getpagesize();
 		pg_shift = rte_bsf32(pg_sz);
@@ -604,23 +661,32 @@ rte_mempool_populate_default(struct rte_mempool *mp)
 			goto fail;
 		}
 
-		mz = rte_memzone_reserve_aligned(mz_name, size,
-			mp->socket_id, mz_flags, align);
-		/* not enough memory, retry with the biggest zone we have */
-		if (mz == NULL)
-			mz = rte_memzone_reserve_aligned(mz_name, 0,
+		if (force_contig) {
+			/*
+			 * if contiguous memory for entire mempool memory was
+			 * requested, don't try reserving again if we fail.
+			 */
+			mz = rte_memzone_reserve_aligned_contig(mz_name, size,
+				mp->socket_id, mz_flags, align);
+		} else {
+			mz = rte_memzone_reserve_aligned(mz_name, size,
 				mp->socket_id, mz_flags, align);
+			/* not enough memory, retry with the biggest zone we have */
+			if (mz == NULL)
+				mz = rte_memzone_reserve_aligned(mz_name, 0,
+					mp->socket_id, mz_flags, align);
+		}
 		if (mz == NULL) {
 			ret = -rte_errno;
 			goto fail;
 		}
 
-		if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG)
+		if (no_contig)
 			iova = RTE_BAD_IOVA;
 		else
 			iova = mz->iova;
 
-		if (rte_eal_has_hugepages())
+		if (rte_eal_has_hugepages() && force_contig)
 			ret = rte_mempool_populate_iova(mp, mz->addr,
 				iova, mz->len,
 				rte_mempool_memchunk_mz_free,
Signature · 1 line
2.7.4
#39 lore raw

RFC 21/23 lib/mempool: add support for the new memory allocation methods

In reply to Anatoly Burakov

If a user has specified that the zone should have contiguous memory, use the new _contig allocation API's instead of normal ones. Otherwise, account for the fact that unless we're in IOVA_AS_VA mode, we cannot guarantee that the pages would be physically contiguous, so we calculate the memzone size and alignments as if we were getting the smallest page size available.

---
 lib/librte_mempool/rte_mempool.c | 84 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 75 insertions(+), 9 deletions(-)
lib/librte_mempool/rte_mempool.c +75 -9
index d50dba4..4b9ab22 100644
@@ -127,6 +127,26 @@ static unsigned optimize_object_size(unsigned obj_size)
 	return new_obj_size * RTE_MEMPOOL_ALIGN;
 }
 
+static size_t
+get_min_page_size(void) {
+	const struct rte_mem_config *mcfg =
+			rte_eal_get_configuration()->mem_config;
+	int i;
+	size_t min_pagesz = SIZE_MAX;
+
+	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
+		const struct rte_memseg_list *msl = &mcfg->memsegs[i];
+
+		if (msl->base_va == NULL)
+			continue;
+
+		if (msl->hugepage_sz < min_pagesz)
+			min_pagesz = msl->hugepage_sz;
+	}
+
+	return min_pagesz == SIZE_MAX ? (size_t) getpagesize() : min_pagesz;
+}
+
 static void
 mempool_add_elem(struct rte_mempool *mp, void *obj, rte_iova_t iova)
 {
@@ -568,6 +588,7 @@ rte_mempool_populate_default(struct rte_mempool *mp)
 	unsigned mz_id, n;
 	unsigned int mp_flags;
 	int ret;
+	bool force_contig, no_contig;
 
 	/* mempool must not be populated */
 	if (mp->nb_mem_chunks != 0)
@@ -582,10 +603,46 @@ rte_mempool_populate_default(struct rte_mempool *mp)
 	/* update mempool capabilities */
 	mp->flags |= mp_flags;
 
-	if (rte_eal_has_hugepages()) {
-		pg_shift = 0; /* not needed, zone is physically contiguous */
+	no_contig = mp->flags & MEMPOOL_F_NO_PHYS_CONTIG;
+	force_contig = mp->flags & MEMPOOL_F_CAPA_PHYS_CONTIG;
+
+	/*
+	 * there are several considerations for page size and page shift here.
+	 *
+	 * if we don't need our mempools to have physically contiguous objects,
+	 * then just set page shift and page size to 0, because the user has
+	 * indicated that there's no need to care about anything.
+	 *
+	 * if we do need contiguous objects, there is also an option to reserve
+	 * the entire mempool memory as one contiguous block of memory, in
+	 * which case the page shift and alignment wouldn't matter as well.
+	 *
+	 * if we require contiguous objects, but not necessarily the entire
+	 * mempool reserved space to be contiguous, then there are two options.
+	 *
+	 * if our IO addresses are virtual, not actual physical (IOVA as VA
+	 * case), then no page shift needed - our memory allocation will give us
+	 * contiguous physical memory as far as the hardware is concerned, so
+	 * act as if we're getting contiguous memory.
+	 *
+	 * if our IO addresses are physical, we may get memory from bigger
+	 * pages, or we might get memory from smaller pages, and how much of it
+	 * we require depends on whether we want bigger or smaller pages.
+	 * However, requesting each and every memory size is too much work, so
+	 * what we'll do instead is walk through the page sizes available, pick
+	 * the smallest one and set up page shift to match that one. We will be
+	 * wasting some space this way, but it's much nicer than looping around
+	 * trying to reserve each and every page size.
+	 */
+
+	if (no_contig || force_contig || rte_eal_iova_mode() == RTE_IOVA_VA) {
 		pg_sz = 0;
+		pg_shift = 0;
 		align = RTE_CACHE_LINE_SIZE;
+	} else if (rte_eal_has_hugepages()) {
+		pg_sz = get_min_page_size();
+		pg_shift = rte_bsf32(pg_sz);
+		align = pg_sz;
 	} else {
 		pg_sz = getpagesize();
 		pg_shift = rte_bsf32(pg_sz);
@@ -604,23 +661,32 @@ rte_mempool_populate_default(struct rte_mempool *mp)
 			goto fail;
 		}
 
-		mz = rte_memzone_reserve_aligned(mz_name, size,
-			mp->socket_id, mz_flags, align);
-		/* not enough memory, retry with the biggest zone we have */
-		if (mz == NULL)
-			mz = rte_memzone_reserve_aligned(mz_name, 0,
+		if (force_contig) {
+			/*
+			 * if contiguous memory for entire mempool memory was
+			 * requested, don't try reserving again if we fail.
+			 */
+			mz = rte_memzone_reserve_aligned_contig(mz_name, size,
+				mp->socket_id, mz_flags, align);
+		} else {
+			mz = rte_memzone_reserve_aligned(mz_name, size,
 				mp->socket_id, mz_flags, align);
+			/* not enough memory, retry with the biggest zone we have */
+			if (mz == NULL)
+				mz = rte_memzone_reserve_aligned(mz_name, 0,
+					mp->socket_id, mz_flags, align);
+		}
 		if (mz == NULL) {
 			ret = -rte_errno;
 			goto fail;
 		}
 
-		if (mp->flags & MEMPOOL_F_NO_PHYS_CONTIG)
+		if (no_contig)
 			iova = RTE_BAD_IOVA;
 		else
 			iova = mz->iova;
 
-		if (rte_eal_has_hugepages())
+		if (rte_eal_has_hugepages() && force_contig)
 			ret = rte_mempool_populate_iova(mp, mz->addr,
 				iova, mz->len,
 				rte_mempool_memchunk_mz_free,
Signature · 1 line
2.7.4
#40 lore raw

PATCH 22/23 vfio: allow to map other memory regions

In reply to Anatoly Burakov

Currently it is not possible to use memory that is not owned by DPDK to perform DMA. This scenarion might be used in vhost applications (like SPDK) where guest send its own memory table. To fill this gap provide API to allow registering arbitrary address in VFIO container.

---
 lib/librte_eal/linuxapp/eal/eal_vfio.c | 150 ++++++++++++++++++++++++++++-----
 lib/librte_eal/linuxapp/eal/eal_vfio.h |  11 +++
 2 files changed, 140 insertions(+), 21 deletions(-)
lib/librte_eal/linuxapp/eal/eal_vfio.c +129 -21
index 09dfc68..15d28ad 100644
@@ -40,6 +40,7 @@
 #include <rte_memory.h>
 #include <rte_eal_memconfig.h>
 #include <rte_vfio.h>
+#include <rte_iommu.h>
 
 #include "eal_filesystem.h"
 #include "eal_vfio.h"
@@ -51,17 +52,35 @@
 static struct vfio_config vfio_cfg;
 
 static int vfio_type1_dma_map(int);
+static int vfio_type1_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
 static int vfio_spapr_dma_map(int);
 static int vfio_noiommu_dma_map(int);
+static int vfio_noiommu_dma_mem_map(int, uint64_t, uint64_t, uint64_t, int);
 
 /* IOMMU types we support */
 static const struct vfio_iommu_type iommu_types[] = {
 	/* x86 IOMMU, otherwise known as type 1 */
-	{ RTE_VFIO_TYPE1, "Type 1", &vfio_type1_dma_map},
+	{
+		.type_id = RTE_VFIO_TYPE1,
+		.name = "Type 1",
+		.dma_map_func = &vfio_type1_dma_map,
+		.dma_user_map_func = &vfio_type1_dma_mem_map
+	},
 	/* ppc64 IOMMU, otherwise known as spapr */
-	{ RTE_VFIO_SPAPR, "sPAPR", &vfio_spapr_dma_map},
+	{
+		.type_id = RTE_VFIO_SPAPR,
+		.name = "sPAPR",
+		.dma_map_func = &vfio_spapr_dma_map,
+		.dma_user_map_func = NULL
+		// TODO: work with PPC64 people on enabling this, window size!
+	},
 	/* IOMMU-less mode */
-	{ RTE_VFIO_NOIOMMU, "No-IOMMU", &vfio_noiommu_dma_map},
+	{
+		.type_id = RTE_VFIO_NOIOMMU,
+		.name = "No-IOMMU",
+		.dma_map_func = &vfio_noiommu_dma_map,
+		.dma_user_map_func = &vfio_noiommu_dma_mem_map
+	},
 };
 
 int
@@ -362,9 +381,10 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
 		 */
 		if (internal_config.process_type == RTE_PROC_PRIMARY &&
 				vfio_cfg.vfio_active_groups == 1) {
+			const struct vfio_iommu_type *t;
+
 			/* select an IOMMU type which we will be using */
-			const struct vfio_iommu_type *t =
-				vfio_set_iommu_type(vfio_cfg.vfio_container_fd);
+			t = vfio_set_iommu_type(vfio_cfg.vfio_container_fd);
 			if (!t) {
 				RTE_LOG(ERR, EAL,
 					"  %s failed to select IOMMU type\n",
@@ -382,6 +402,8 @@ rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr,
 				clear_group(vfio_group_fd);
 				return -1;
 			}
+
+			vfio_cfg.vfio_iommu_type = t;
 		}
 	}
 
@@ -694,13 +716,52 @@ vfio_get_group_no(const char *sysfs_base,
 }
 
 static int
+vfio_type1_dma_mem_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova,
+		uint64_t len, int do_map)
+{
+	struct vfio_iommu_type1_dma_map dma_map;
+	struct vfio_iommu_type1_dma_unmap dma_unmap;
+	int ret;
+
+	if (do_map != 0) {
+		memset(&dma_map, 0, sizeof(dma_map));
+		dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+		dma_map.vaddr = vaddr;
+		dma_map.size = len;
+		dma_map.iova = iova;
+		dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+		ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "  cannot set up DMA remapping, error %i (%s)\n",
+				errno, strerror(errno));
+				return -1;
+		}
+
+	} else {
+		memset(&dma_unmap, 0, sizeof(dma_unmap));
+		dma_unmap.argsz = sizeof(struct vfio_iommu_type1_dma_unmap);
+		dma_unmap.size = len;
+		dma_unmap.iova = iova;
+
+		ret = ioctl(vfio_container_fd, VFIO_IOMMU_UNMAP_DMA, &dma_unmap);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "  cannot clear DMA remapping, error %i (%s)\n",
+					errno, strerror(errno));
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static int
 vfio_type1_dma_map(int vfio_container_fd)
 {
-	int i, ret;
+	int i;
 
 	/* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
 	for (i = 0; i < RTE_MAX_MEMSEG_LISTS; i++) {
-		struct vfio_iommu_type1_dma_map dma_map;
 		const struct rte_memseg_list *msl;
 		const struct rte_fbarray *arr;
 		int ms_idx, next_idx;
@@ -727,21 +788,9 @@ vfio_type1_dma_map(int vfio_container_fd)
 			len = ms->hugepage_sz;
 			hw_addr = ms->iova;
 
-			memset(&dma_map, 0, sizeof(dma_map));
-			dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
-			dma_map.vaddr = addr;
-			dma_map.size = len;
-			dma_map.iova = hw_addr;
-			dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
-
-			ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
-
-			if (ret) {
-				RTE_LOG(ERR, EAL, "  cannot set up DMA remapping, "
-						  "error %i (%s)\n", errno,
-						  strerror(errno));
+			if (vfio_type1_dma_mem_map(vfio_container_fd, addr,
+					hw_addr, len, 1))
 				return -1;
-			}
 		}
 	}
 
@@ -892,6 +941,49 @@ vfio_noiommu_dma_map(int __rte_unused vfio_container_fd)
 	return 0;
 }
 
+static int
+vfio_noiommu_dma_mem_map(int __rte_unused vfio_container_fd,
+			 uint64_t __rte_unused vaddr,
+			 uint64_t __rte_unused iova, uint64_t __rte_unused len,
+			 int __rte_unused do_map)
+{
+	/* No-IOMMU mode does not need DMA mapping */
+	return 0;
+}
+
+static int
+vfio_dma_mem_map(uint64_t vaddr, uint64_t iova, uint64_t len, int do_map)
+{
+	const struct vfio_iommu_type *t = vfio_cfg.vfio_iommu_type;
+
+	if (!t) {
+		RTE_LOG(ERR, EAL, "  VFIO support not initialized\n");
+		return -1;
+	}
+
+	if (!t->dma_user_map_func) {
+		RTE_LOG(ERR, EAL,
+			"  VFIO custom DMA region maping not supported by IOMMU %s\n",
+			t->name);
+		return -1;
+	}
+
+	return t->dma_user_map_func(vfio_cfg.vfio_container_fd, vaddr, iova,
+			len, do_map);
+}
+
+int
+rte_iommu_dma_map(uint64_t vaddr, uint64_t iova, uint64_t len)
+{
+	return vfio_dma_mem_map(vaddr, iova, len, 1);
+}
+
+int
+rte_iommu_dma_unmap(uint64_t vaddr, uint64_t iova, uint64_t len)
+{
+	return vfio_dma_mem_map(vaddr, iova, len, 0);
+}
+
 int
 rte_vfio_noiommu_is_enabled(void)
 {
@@ -911,4 +1003,20 @@ rte_vfio_noiommu_is_enabled(void)
 	return ret;
 }
 
+#else
+
+int
+rte_iommu_dma_map(uint64_t __rte_unused vaddr, __rte_unused uint64_t iova,
+		  __rte_unused uint64_t len)
+{
+	return 0;
+}
+
+int
+rte_iommu_dma_unmap(uint64_t __rte_unused vaddr, uint64_t __rte_unused iova,
+		    __rte_unused uint64_t len)
+{
+	return 0;
+}
+
 #endif
lib/librte_eal/linuxapp/eal/eal_vfio.h +11 -0
index ba7892b..bb669f0 100644
@@ -48,6 +48,7 @@
 
 #ifdef VFIO_PRESENT
 
+#include <stdint.h>
 #include <linux/vfio.h>
 
 #define RTE_VFIO_TYPE1 VFIO_TYPE1_IOMMU
@@ -139,6 +140,7 @@ struct vfio_config {
 	int vfio_enabled;
 	int vfio_container_fd;
 	int vfio_active_groups;
+	const struct vfio_iommu_type *vfio_iommu_type;
 	struct vfio_group vfio_groups[VFIO_MAX_GROUPS];
 };
 
@@ -148,9 +150,18 @@ struct vfio_config {
  * */
 typedef int (*vfio_dma_func_t)(int);
 
+/* Custom memory region DMA mapping function prototype.
+ * Takes VFIO container fd, virtual address, phisical address, length and
+ * operation type (0 to unmap 1 for map) as a parameters.
+ * Returns 0 on success, -1 on error.
+ **/
+typedef int (*vfio_dma_user_func_t)(int fd, uint64_t vaddr, uint64_t iova,
+		uint64_t len, int do_map);
+
 struct vfio_iommu_type {
 	int type_id;
 	const char *name;
+	vfio_dma_user_func_t dma_user_map_func;
 	vfio_dma_func_t dma_map_func;
 };
 
Signature · 1 line
2.7.4
#41 lore raw

PATCH 23/23 eal: map/unmap memory with VFIO when alloc/free pages

In reply to Anatoly Burakov
---
 lib/librte_eal/linuxapp/eal/eal_memalloc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
lib/librte_eal/linuxapp/eal/eal_memalloc.c +11 -0
index 13172a0..8b3f219 100755
@@ -61,6 +61,7 @@
 #include <rte_eal_memconfig.h>
 #include <rte_eal.h>
 #include <rte_memory.h>
+#include <rte_iommu.h>
 
 #include "eal_filesystem.h"
 #include "eal_internal_cfg.h"
@@ -259,6 +260,11 @@ alloc_page(struct rte_memseg *ms, void *addr, uint64_t size, int socket_id,
 	ms->iova = iova;
 	ms->socket_id = socket_id;
 
+	/* map the segment so that VFIO has access to it */
+	if (rte_iommu_dma_map(ms->addr_64, iova, size)) {
+		RTE_LOG(DEBUG, EAL, "Cannot register segment with VFIO\n");
+	}
+
 	goto out;
 
 mapped:
@@ -295,6 +301,11 @@ free_page(struct rte_memseg *ms, struct hugepage_info *hi, unsigned list_idx,
 				list_idx * RTE_MAX_MEMSEG_PER_LIST + seg_idx);
 	}
 
+	/* unmap the segment from VFIO */
+	if (rte_iommu_dma_unmap(ms->addr_64, ms->iova, ms->len)) {
+		RTE_LOG(DEBUG, EAL, "Cannot unregister segment with VFIO\n");
+	}
+
 	munmap(ms->addr, ms->hugepage_sz);
 
 	// TODO: race condition?
Signature · 1 line
2.7.4
#42 lore raw

RFC 23/23 eal/memalloc: register/unregister memory with VFIO when alloc/free pages

In reply to Anatoly Burakov
---
 lib/librte_eal/linuxapp/eal/eal_memalloc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
lib/librte_eal/linuxapp/eal/eal_memalloc.c +11 -0
index 13172a0..8b3f219 100755
@@ -61,6 +61,7 @@
 #include <rte_eal_memconfig.h>
 #include <rte_eal.h>
 #include <rte_memory.h>
+#include <rte_iommu.h>
 
 #include "eal_filesystem.h"
 #include "eal_internal_cfg.h"
@@ -259,6 +260,11 @@ alloc_page(struct rte_memseg *ms, void *addr, uint64_t size, int socket_id,
 	ms->iova = iova;
 	ms->socket_id = socket_id;
 
+	/* map the segment so that VFIO has access to it */
+	if (rte_iommu_dma_map(ms->addr_64, iova, size)) {
+		RTE_LOG(DEBUG, EAL, "Cannot register segment with VFIO\n");
+	}
+
 	goto out;
 
 mapped:
@@ -295,6 +301,11 @@ free_page(struct rte_memseg *ms, struct hugepage_info *hi, unsigned list_idx,
 				list_idx * RTE_MAX_MEMSEG_PER_LIST + seg_idx);
 	}
 
+	/* unmap the segment from VFIO */
+	if (rte_iommu_dma_unmap(ms->addr_64, ms->iova, ms->len)) {
+		RTE_LOG(DEBUG, EAL, "Cannot unregister segment with VFIO\n");
+	}
+
 	munmap(ms->addr, ms->hugepage_sz);
 
 	// TODO: race condition?
Signature · 1 line
2.7.4
#43 lore raw
In reply to Anatoly Burakov
Quoted 6 lines · Anatoly Burakov

This patchset introduces a prototype implementation of dynamic memory allocation for DPDK. It is intended to start a conversation and build consensus on the best way to implement this functionality. The patchset works well enough to pass all unit tests, and to work with traffic forwarding, provided the device drivers are adjusted to ensure contiguous memory allocation where it matters.

Quoted 7 lines

The vast majority of changes are in the EAL and malloc, the external API disruption is minimal: a new set of API's are added for contiguous memory allocation (for rte_malloc and rte_memzone), and a few API additions in rte_memory. Every other API change is internal to EAL, and all of the memory allocation/freeing is handled through rte_malloc, with no externally visible API changes, aside from a call to get physmem layout, which no longer makes sense given that there are multiple memseg lists.

My apologies, this patchset was sent erroneously. Please look at the version tagged as "RFC v2".

Signature · 2 lines
Thanks,
Anatoly