Support externally allocated memory in DPDK
Subsystems (2)
PATCH v5 10/21 malloc: add function to check if socket is external
In reply to Anatoly BurakovAn API is needed to check whether a particular socket ID belongs to an internal or external heap. Prime user of this would be mempool allocator, because normal assumptions of IOVA contiguousness in IOVA as VA mode do not hold in case of externally allocated memory.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 15 +++++++++++++ lib/librte_eal/common/rte_malloc.c | 25 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + lib/librte_mempool/rte_mempool.c | 22 ++++++++++++++++--- 4 files changed, 60 insertions(+), 3 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +15 -0
@@ -277,6 +277,21 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_get_socket (const char *name); +/** +* Check if a given socket ID refers to externally allocated memory. +* +* @note Passing SOCKET_ID_ANY will return 0. +* +* @param socket_id +* Socket ID to check +* @return +* 1 if socket ID refers to externally allocated memory +* 0 if socket ID refers to internal DPDK memory +* -1 if socket ID is invalid +*/ +int __rte_experimental +rte_malloc_heap_socket_is_external (int socket_id); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +25 -0
@@ -220,6 +220,31 @@ rte_malloc_heap_get_socket(const char *name)return ret; } +int +rte_malloc_heap_socket_is_external (int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int idx; +int ret =-1 ; + +if (socket_id ==SOCKET_ID_ANY ) +return 0 ; + +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if ((int )tmp->socket_id == socket_id) { +/* external memory always has large socket ID's */ + ret = tmp->socket_id >=RTE_MAX_NUMA_NODES ; +break ; + } + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_get_socket; + rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
lib/librte_mempool/rte_mempool.c +19 -3
@@ -428,12 +428,18 @@ rte_mempool_populate_default(struct rte_mempool *mp)rte_iova_t iova;unsigned mz_id, n;int ret; -bool no_contig, try_contig, no_pageshift; +bool no_contig, try_contig, no_pageshift, external; ret =mempool_ops_alloc_once (mp);if (ret !=0 )return ret; +/* check if we can retrieve a valid socket ID */ + ret =rte_malloc_heap_socket_is_external (mp->socket_id); +if (ret <0 ) +return -EINVAL ; + external = ret; +/* mempool must not be populated */ if (mp->nb_mem_chunks !=0 )return -EEXIST ;
@@ -481,9 +487,19 @@ rte_mempool_populate_default(struct rte_mempool *mp)* in one contiguous chunk as well (otherwise we might end up wasting a * 1G page on a 10MB memzone). If we fail to get enough contiguous * memory, then we'll go and reserve space page-by-page. +* +* We also have to take into account the fact that memory that we're +* going to allocate from can belong to an externally allocated memory +* area, in which case the assumption of IOVA as VA mode being +* synonymous with IOVA contiguousness will not hold. We should also try +* to go for contiguous memory even if we're in no-huge mode, because +* external memory may in fact be IOVA-contiguous. */ - no_pageshift = no_contig ||rte_eal_iova_mode () ==RTE_IOVA_VA ; - try_contig = !no_contig && !no_pageshift &&rte_eal_has_hugepages (); + external =rte_malloc_heap_socket_is_external (mp->socket_id) ==1 ; + no_pageshift = no_contig || + (!external &&rte_eal_iova_mode () ==RTE_IOVA_VA ); + try_contig = !no_contig && !no_pageshift && + (rte_eal_has_hugepages () || external);if (no_pageshift) { pg_sz =0 ;
Signature · 1 line
2.17.1
PATCH v5 11/21 malloc: allow creating malloc heaps
In reply to Anatoly BurakovAdd API to allow creating new malloc heaps. They will be created with socket ID's going above RTE_MAX_NUMA_NODES, to avoid clashing with internal heaps.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 2 + .../common/include/rte_eal_memconfig.h | 3 ++ lib/librte_eal/common/include/rte_malloc.h | 19 +++++++ lib/librte_eal/common/malloc_heap.c | 37 +++++++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 52 +++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 7 files changed, 117 insertions(+)
doc/guides/rel_notes/release_18_11.rst +2 -0
@@ -125,6 +125,8 @@ ABI Changes supporting external memory in DPDK. Structure ``rte_memseg_list`` now has a new flag indicating whether the memseg list refers to external memory. Structure ``rte_malloc_heap`` now has a ``heap_name`` string member. + Structure ``rte_eal_memconfig`` has been extended to contain next socket + ID for externally allocated memory segments. Removed Items -------------
lib/librte_eal/common/include/rte_eal_memconfig.h +3 -0
@@ -75,6 +75,9 @@ struct rte_mem_config {/* Heaps of Malloc */ struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ]; +/* next socket ID for external malloc heap */ +int next_socket_id; +/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it. */
lib/librte_eal/common/include/rte_malloc.h +19 -0
@@ -263,6 +263,25 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Creates a new empty malloc heap with a specified name. +* +* @note Heaps created via this call will automatically get assigned a unique +* socket ID, which can be found using ``rte_malloc_heap_get_socket()`` +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on successful creation +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* EEXIST - heap by name of ``heap_name`` already exists +* ENOSPC - no more space in internal config to store a new heap +*/ +int __rte_experimental +rte_malloc_heap_create (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +37 -0
@@ -29,6 +29,10 @@#include "malloc_heap.h" #include "malloc_mp.h" +/* start external socket ID's at a very high number */ +#define CONST_MAX (a, b) (a > b ? a : b)/* RTE_MAX is not a constant */ +#define EXTERNAL_HEAP_MIN_SOCKET_ID (CONST_MAX((1 << 8), RTE_MAX_NUMA_NODES)) +static unsigned check_hugepage_sz (unsigned flags,uint64_t hugepage_sz) {
@@ -1015,6 +1019,36 @@ malloc_heap_dump(struct malloc_heap *heap, FILE *f)rte_spinlock_unlock (&heap->lock); } +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +uint32_t next_socket_id = mcfg->next_socket_id; + +/* prevent overflow. did you really create 2 billion heaps??? */ +if (next_socket_id >INT32_MAX ) { +RTE_LOG (ERR ,EAL ,"Cannot assign new socket ID's\n" ); + rte_errno =ENOSPC ; +return -1 ; + } + +/* initialize empty heap */ + heap->alloc_count =0 ; + heap->first =NULL ; + heap->last =NULL ; +LIST_INIT (heap->free_head); +rte_spinlock_init (&heap->lock); + heap->total_size =0 ; + heap->socket_id = next_socket_id; + +/* we hold a global mem hotplug writelock, so it's safe to increment */ + mcfg->next_socket_id++; + +/* set up name */ +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
@@ -1022,6 +1056,9 @@ rte_eal_malloc_heap_init(void)unsigned int i;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign min socket ID to external heaps */ + mcfg->next_socket_id =EXTERNAL_HEAP_MIN_SOCKET_ID ; +/* assign names to default DPDK heaps */ for (i =0 ; i <rte_socket_count (); i++) {struct malloc_heap *heap = &mcfg->malloc_heaps[i];
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -33,6 +33,9 @@ void * malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags, size_t align, bool contig); +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +52 -0
61 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_create; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v5 12/21 malloc: allow destroying heaps
In reply to Anatoly BurakovAdd an API to destroy specified heap.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 23 +++++++++ lib/librte_eal/common/malloc_heap.c | 22 ++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 58 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 107 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +23 -0
@@ -282,6 +282,29 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_create (const char *heap_name); +/** +* Destroys a previously created malloc heap with specified name. +* +* @note This function will return a failure result if not all memory allocated +* from the heap has been freed back to the heap +* +* @note This function will return a failure result if not all memory segments +* were removed from the heap prior to its destruction +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* ENOENT - heap by the name of ``heap_name`` was not found +* EPERM - attempting to destroy reserved heap +* EBUSY - heap still contains data +*/ +int __rte_experimental +rte_malloc_heap_destroy (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +22 -0
@@ -1049,6 +1049,28 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name)return 0 ; } +int +malloc_heap_destroy (struct malloc_heap *heap) +{ +if (heap->alloc_count !=0 ) { +RTE_LOG (ERR ,EAL ,"Heap is still in use\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->first !=NULL || heap->last !=NULL ) { +RTE_LOG (ERR ,EAL ,"Heap still contains memory segments\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->total_size !=0 ) +RTE_LOG (ERR ,EAL ,"Total size not zero, heap is likely corrupt\n" ); + +/* after this, the lock will be dropped */ +memset (heap,0 ,sizeof (*heap)); + +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -36,6 +36,9 @@ malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags,int malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int +malloc_heap_destroy (struct malloc_heap *heap); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +58 -0
67 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_create; + rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v5 13/21 malloc: allow adding memory to named heaps
In reply to Anatoly BurakovAdd an API to add externally allocated memory to malloc heap. The memory will be stored in memseg lists like regular DPDK memory. Multiple segments are allowed within a heap. If IOVA table is not provided, IOVA addresses are filled in with RTE_BAD_IOVA.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 39 ++++++++++++ lib/librte_eal/common/malloc_heap.c | 74 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 51 +++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 169 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +39 -0
@@ -263,6 +263,45 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Add memory chunk to a heap with specified name. +* +* @note Multiple memory chunks can be added to the same heap +* +* @note Memory must be previously allocated for DPDK to be able to use it as a +* malloc heap. Failing to do so will result in undefined behavior, up to and +* including segmentation faults. +* +* @note Calling this function will erase any contents already present at the +* supplied memory address. +* +* @param heap_name +* Name of the heap to add memory chunk to +* @param va_addr +* Start of virtual area to add to the heap +* @param len +* Length of virtual area to add to the heap +* @param iova_addrs +* Array of page IOVA addresses corresponding to each page in this memory +* area. Can be NULL, in which case page IOVA addresses will be set to +* RTE_BAD_IOVA. +* @param n_pages +* Number of elements in the iova_addrs array. Ignored if ``iova_addrs`` +* is NULL. +* @param page_sz +* Page size of the underlying memory +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to add memory to a reserved heap +* ENOSPC - no more space in internal config to store a new memory chunk +*/ +int __rte_experimental +rte_malloc_heap_memory_add (const char *heap_name,void *va_addr,size_t len, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +74 -0
80 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -39,6 +39,10 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name);int malloc_heap_destroy (struct malloc_heap *heap); +int +malloc_heap_add_external_memory (struct malloc_heap *heap,void *va_addr, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +51 -0
57 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -321,6 +321,7 @@ EXPERIMENTAL { rte_malloc_heap_create; rte_malloc_heap_destroy; rte_malloc_heap_get_socket; + rte_malloc_heap_memory_add; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v5 14/21 malloc: allow removing memory from named heaps
In reply to Anatoly BurakovAdd an API to remove memory from specified heaps. This will first check if all elements within the region are free, and that the region is the original region that was added to the heap (by comparing its length to length of memory addressed by the underlying memseg list).
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++ lib/librte_eal/common/malloc_heap.c | 54 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 39 ++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 125 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -302,6 +302,33 @@ int __rte_experimental rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +/** +* Remove memory chunk from heap with specified name. +* +* @note Memory chunk being removed must be the same as one that was added; +* partially removing memory chunks is not supported +* +* @note Memory area must not contain any allocated elements to allow its +* removal from the heap +* +* @param heap_name +* Name of the heap to remove memory from +* @param va_addr +* Virtual address to remove from the heap +* @param len +* Length of virtual area to remove from the heap +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to remove memory from a reserved heap +* ENOENT - heap or memory chunk was not found +* EBUSY - memory chunk still contains data +*/ +int __rte_experimental +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +54 -0
66 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -43,6 +43,10 @@ int malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +int +malloc_heap_remove_external_memory (struct malloc_heap *heap,void *va_addr, +size_t len); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +39 -0
@@ -379,6 +379,45 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,return ret; } +int +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +int ret; + +if (heap_name ==NULL || va_addr ==NULL || len ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_write_lock (&mcfg->memory_hotplug_lock); +/* find our heap */ + heap =find_named_heap (heap_name); +if (heap ==NULL ) { + rte_errno =ENOENT ; + ret =-1 ; + goto unlock; + } +if (heap->socket_id <RTE_MAX_NUMA_NODES ) { +/* cannot remove memory from internal heaps */ + rte_errno =EPERM ; + ret =-1 ; + goto unlock; + } + +rte_spinlock_lock (&heap->lock); + ret =malloc_heap_remove_external_memory (heap, va_addr, len); +rte_spinlock_unlock (&heap->lock); + +unlock: +rte_rwlock_write_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v5 15/21 malloc: allow attaching to external memory chunks
In reply to Anatoly BurakovIn order to use external memory in multiple processes, we need to attach to primary process's memseg lists, so add a new API to do that. It is the responsibility of the user to ensure that memory is accessible and that it has been previously added to the malloc heap by another process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 28 ++++++++ lib/librte_eal/common/rte_malloc.c | 83 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 112 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +28 -0
@@ -268,6 +268,10 @@ rte_malloc_get_socket_stats(int socket,* * @note Multiple memory chunks can be added to the same heap * +* @note Before accessing this memory in other processes, it needs to be +* attached in each of those processes by calling +* ``rte_malloc_heap_memory_attach`` in each other process. +* * @note Memory must be previously allocated for DPDK to be able to use it as a * malloc heap. Failing to do so will result in undefined behavior, up to and * including segmentation faults.
@@ -329,6 +333,30 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, int __rte_experimentalrte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** +* Attach to an already existing chunk of external memory in another process. +* +* @note This function must be called before any attempt is made to use an +* already existing external memory chunk. This function does *not* need to +* be called if a call to ``rte_malloc_heap_memory_add`` was made in the +* current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful attach +* -1 on unsuccessful attach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to attach memory to a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +83 -0
89 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_attach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v5 16/21 malloc: allow detaching from external memory
In reply to Anatoly BurakovAdd API to detach from existing chunk of external memory in a process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++++++++++ lib/librte_eal/common/rte_malloc.c | 31 +++++++++++++++++----- lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+), 7 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -315,6 +315,9 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,* @note Memory area must not contain any allocated elements to allow its * removal from the heap * +* @note All other processes must detach from the memory chunk prior to it being +* removed from the heap. +* * @param heap_name * Name of the heap to remove memory from * @param va_addr
@@ -357,6 +360,30 @@ rte_malloc_heap_memory_remove(const char *heap_name, void *va_addr, size_t len); int __rte_experimentalrte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** +* Detach from a chunk of external memory in secondary process. +* +* @note This function must be called in before any attempt is made to remove +* external memory from the heap in another process. This function does *not* +* need to be called if a call to ``rte_malloc_heap_memory_remove`` will be +* called in current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful detach +* -1 on unsuccessful detach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to detach memory from a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +24 -7
@@ -422,10 +422,11 @@ struct sync_mem_walk_arg {void *va_addr;size_t len;int result; +bool attach; };static int -attach_mem_walk (const struct rte_memseg_list *msl,void *arg) +sync_mem_walk (const struct rte_memseg_list *msl,void *arg) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct sync_mem_walk_arg *wa = arg;
@@ -440,7 +441,10 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; - ret =rte_fbarray_attach (&found_msl->memseg_arr); +if (wa->attach) + ret =rte_fbarray_attach (&found_msl->memseg_arr); +else + ret =rte_fbarray_detach (&found_msl->memseg_arr);if (ret <0 ) wa->result = -rte_errno;
@@ -451,8 +455,8 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg)return 0 ; } -int -rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +static int +sync_memory (const char *heap_name,void *va_addr,size_t len,bool attach) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct malloc_heap *heap =NULL ;
@@ -475,20 +479,21 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len) ret =-1 ; goto unlock; } -/* we shouldn't be able to attach to internal heaps */ +/* we shouldn't be able to sync to internal heaps */ if (heap->socket_id <RTE_MAX_NUMA_NODES ) { rte_errno =EPERM ; ret =-1 ; goto unlock; } -/* find corresponding memseg list to attach to */ +/* find corresponding memseg list to sync to */ wa.va_addr = va_addr; wa.len = len; wa.result = -ENOENT ;/* fail unless explicitly told to succeed */ + wa.attach = attach;/* we're already holding a read lock */ -rte_memseg_list_walk_thread_unsafe (attach_mem_walk, &wa); +rte_memseg_list_walk_thread_unsafe (sync_mem_walk, &wa);if (wa.result <0 ) { rte_errno = -wa.result;
@@ -501,6 +506,18 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len)return ret; } +int +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, true); +} + +int +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, false); +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -323,6 +323,7 @@ EXPERIMENTAL { rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; rte_malloc_heap_memory_attach; + rte_malloc_heap_memory_detach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v5 17/21 malloc: enable event callbacks for external memory
In reply to Anatoly BurakovWhen adding or removing external memory from the memory map, there may be actions that need to be taken on account of this memory (e.g. DMA mapping). Add support for triggering callbacks when adding, removing, attaching or detaching external memory.
Some memory event callback handlers will need additional logic to handle external memory regions. For example, virtio callback has to completely ignore externally allocated memory, because there is no way to find file descriptors backing the memory address in a generic fashion. All other callbacks have also been adjusted to handle RTE_BAD_IOVA as IOVA address, as this is one of the expected use cases for external memory support.
Signed-off-by Anatoly Burakov <[email protected]>
--- drivers/bus/fslmc/fslmc_vfio.c | 7 +++++ .../net/virtio/virtio_user/virtio_user_dev.c | 8 ++++++ lib/librte_eal/common/malloc_heap.c | 7 +++++ lib/librte_eal/common/rte_malloc.c | 27 ++++++++++++++++--- lib/librte_eal/linuxapp/eal/eal_vfio.c | 10 +++++-- 5 files changed, 53 insertions(+), 6 deletions(-)
drivers/bus/fslmc/fslmc_vfio.c +7 -0
@@ -221,6 +221,13 @@ fslmc_memevent_cb(enum rte_mem_event type, const void *addr, size_t len, "alloc" : "dealloc", va, virt_addr, iova_addr, map_len); +/* iova_addr may be set to RTE_BAD_IOVA */ +if (iova_addr ==RTE_BAD_IOVA ) { +DPAA2_BUS_DEBUG ("Segment has invalid iova, skipping\n" ); + cur_len += map_len; +continue ; + } +if (type ==RTE_MEM_EVENT_ALLOC ) ret =fslmc_map_dma (virt_addr, iova_addr, map_len);else
drivers/net/virtio/virtio_user/virtio_user_dev.c +8 -0
@@ -13,6 +13,8 @@#include <sys/types.h> #include <sys/stat.h> +#include <rte_eal_memconfig.h> +#include "vhost.h" #include "virtio_user_dev.h" #include "../virtio_ethdev.h"
@@ -282,8 +284,14 @@ virtio_user_mem_event_cb(enum rte_mem_event type __rte_unused, void *arg) {struct virtio_user_dev *dev = arg; +struct rte_memseg_list *msl;uint16_t i; +/* ignore externally allocated memory */ + msl =rte_mem_virt2memseg_list (addr); +if (msl->external) +return ; +pthread_mutex_lock (&dev->mutex);if (dev->started == false)
lib/librte_eal/common/malloc_heap.c +7 -0
@@ -1027,6 +1027,9 @@ destroy_seg(struct malloc_elem *elem, size_t len) msl = elem->msl; +/* notify all subscribers that a memory area is going to be removed */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , elem, len); +/* this element can be removed */ malloc_elem_free_list_remove (elem);malloc_elem_hide_region (elem, elem, len);
@@ -1116,6 +1119,10 @@ malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr,RTE_LOG (DEBUG ,EAL ,"Added segment for heap %s starting at %p\n" , heap->name, va_addr); +/* notify all subscribers that a new memory area has been added */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, seg_len); +return 0 ; }
lib/librte_eal/common/rte_malloc.c +23 -4
@@ -25,6 +25,7 @@#include <rte_malloc.h> #include "malloc_elem.h" #include "malloc_heap.h" +#include "eal_memalloc.h" /* Free the memory space back to heap */
@@ -441,15 +442,29 @@ sync_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; -if (wa->attach) +if (wa->attach) { ret =rte_fbarray_attach (&found_msl->memseg_arr); -else + }else { +/* notify all subscribers that a memory area is about to +* be removed +*/ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , + msl->base_va, msl->len); ret =rte_fbarray_detach (&found_msl->memseg_arr); + } -if (ret <0 ) +if (ret <0 ) { wa->result = -rte_errno; -else + }else { +/* notify all subscribers that a new memory area was +* added +*/ +if (wa->attach) +eal_memalloc_mem_event_notify ( +RTE_MEM_EVENT_ALLOC , + msl->base_va, msl->len); wa->result =0 ; + }return 1 ; }return 0 ;
@@ -499,6 +514,10 @@ sync_memory(const char *heap_name, void *va_addr, size_t len, bool attach) rte_errno = -wa.result; ret =-1 ; }else { +/* notify all subscribers that a new memory area was added */ +if (attach) +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, len); ret =0 ; } unlock:
lib/librte_eal/linuxapp/eal/eal_vfio.c +8 -2
@@ -509,7 +509,7 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, msl =rte_mem_virt2memseg_list (addr);/* for IOVA as VA mode, no need to care for IOVA addresses */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) { +if (rte_eal_iova_mode () ==RTE_IOVA_VA && msl->external ==0 ) {uint64_t vfio_va = (uint64_t )(uintptr_t )addr;if (type ==RTE_MEM_EVENT_ALLOC ) vfio_dma_mem_map(default_vfio_cfg, vfio_va, vfio_va,
@@ -523,13 +523,19 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,/* memsegs are contiguous in memory */ ms =rte_mem_virt2memseg (addr, msl);while (cur_len < len) { +/* some memory segments may have invalid IOVA */ +if (ms->iova ==RTE_BAD_IOVA ) { +RTE_LOG (DEBUG ,EAL ,"Memory segment at %p has bad IOVA, skipping\n" , + ms->addr); + goto next; + }if (type ==RTE_MEM_EVENT_ALLOC )vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,1 );else vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,0 ); - +next: cur_len += ms->len; ++ms; }
Signature · 1 line
2.17.1
PATCH v5 18/21 test: add unit tests for external memory support
In reply to Anatoly BurakovAdd simple unit tests to test external memory support. The tests are pretty basic and mostly consist of checking if invalid API calls are handled correctly, plus a simple allocation/deallocation test for malloc and memzone.
Signed-off-by Anatoly Burakov <[email protected]>
--- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++++++++++++++++++++ 4 files changed, 401 insertions(+), 4 deletions(-) create mode 100644 test/test/test_external_mem.c
test/test/Makefile +1 -0
@@ -71,6 +71,7 @@ SRCS-y += test_bitmap.c SRCS-y += test_reciprocal_division.c SRCS-y += test_reciprocal_division_perf.c SRCS-y += test_fbarray.c +SRCS-y += test_external_mem.c SRCS-y += test_ring.c SRCS-y += test_ring_perf.c
test/test/autotest_data.py +10 -4
@@ -477,10 +477,16 @@ "Report": None, }, { - "Name": "Fbarray autotest", - "Command": "fbarray_autotest", - "Func": default_autotest, - "Report": None, + "Name": "Fbarray autotest", + "Command": "fbarray_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "External memory autotest", + "Command": "external_mem_autotest", + "Func": default_autotest, + "Report": None, }, # #Please always keep all dump tests at the end and together!
test/test/meson.build +1 -0
@@ -155,6 +155,7 @@ test_names = [ 'eventdev_common_autotest', 'eventdev_octeontx_autotest', 'eventdev_sw_autotest', + 'external_mem_autotest', 'func_reentrancy_autotest', 'flow_classify_autotest', 'hash_scaling_autotest',
test/test/test_external_mem.c new file +389 -0
389 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v5 19/21 app/testpmd: add support for external memory
In reply to Anatoly BurakovCurrently, mempools can only be allocated either using native DPDK memory, or anonymous memory. This patch will add two new methods to allocate mempool using external memory (regular or hugepage memory), and add documentation about it to testpmd user guide.
It adds a new flag "--mp-alloc", with four possible values:
native (use regular DPDK allocator), anon (use anonymous mempool), xmem (use externally allocated memory area), and xmemhuge (use externally allocated hugepage memory area). Old flag "--mp-anon" is kept for compatibility.
All external memory is allocated using the same external heap, but each will allocate and add a new memory area.
Signed-off-by Anatoly Burakov <[email protected]>
Suggested-by Konstantin Ananyev <[email protected]>
--- app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 305 ++++++++++++++++++++++++-- app/test-pmd/testpmd.h | 13 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + 5 files changed, 349 insertions(+), 25 deletions(-)
app/test-pmd/config.c +19 -2
@@ -2413,6 +2413,23 @@ fwd_config_setup(void)simple_fwd_config_setup (); } +static const char * +mp_alloc_to_str (uint8_t mode) +{ +switch (mode) { +case MP_ALLOC_NATIVE : +return "native" ; +case MP_ALLOC_ANON : +return "anon" ; +case MP_ALLOC_XMEM : +return "xmem" ; +case MP_ALLOC_XMEM_HUGE : +return "xmemhuge" ; +default : +return "invalid" ; + } +} +void pkt_fwd_config_display (struct fwd_config *cfg) {
@@ -2421,12 +2438,12 @@ pkt_fwd_config_display(struct fwd_config *cfg)streamid_t sm_id;printf ("%s packet forwarding%s - ports=%d - cores=%d - streams=%d - " -"NUMA support %s, MP over anonymous pages %s\n" , +"NUMA support %s, MP allocation mode: %s\n" , cfg->fwd_eng->fwd_mode_name, retry_enabled ==0 ?"" :" with retry" , cfg->nb_fwd_ports, cfg->nb_fwd_lcores, cfg->nb_fwd_streams, numa_support ==1 ?"enabled" :"disabled" , - mp_anon !=0 ?"enabled" :"disabled" ); +mp_alloc_to_str (mp_alloc_type));if (retry_enabled) printf("TX retry num: %u, delay between TX retries: %uus\n",
app/test-pmd/parameters.c +22 -1
@@ -190,6 +190,11 @@ usage(char* progname)printf (" --vxlan-gpe-port=N: UPD port of tunnel VXLAN-GPE\n" );printf (" --mlockall: lock all memory\n" );printf (" --no-mlockall: do not lock all memory\n" ); +printf (" --mp-alloc <native|anon|xmem|xmemhuge>: mempool allocation method.\n" +" native: use regular DPDK memory to create and populate mempool\n" +" anon: use regular DPDK memory to create and anonymous memory to populate mempool\n" +" xmem: use anonymous memory to create and populate mempool\n" +" xmemhuge: use anonymous hugepage memory to create and populate mempool\n" ); }#ifdef RTE_LIBRTE_CMDLINE
@@ -625,6 +630,7 @@ launch_args_parse(int argc, char** argv) { "vxlan-gpe-port", 1, 0, 0 }, { "mlockall", 0, 0, 0 }, { "no-mlockall", 0, 0, 0 }, + { "mp-alloc", 1, 0, 0 }, { 0, 0, 0, 0 }, };
@@ -743,7 +749,22 @@ launch_args_parse(int argc, char** argv)if (!strcmp (lgopts[opt_idx].name,"numa" )) numa_support =1 ;if (!strcmp (lgopts[opt_idx].name,"mp-anon" )) { - mp_anon =1 ; + mp_alloc_type =MP_ALLOC_ANON ; + } +if (!strcmp (lgopts[opt_idx].name,"mp-alloc" )) { +if (!strcmp (optarg,"native" )) + mp_alloc_type =MP_ALLOC_NATIVE ; +else if (!strcmp (optarg,"anon" )) + mp_alloc_type =MP_ALLOC_ANON ; +else if (!strcmp (optarg,"xmem" )) + mp_alloc_type =MP_ALLOC_XMEM ; +else if (!strcmp (optarg,"xmemhuge" )) + mp_alloc_type =MP_ALLOC_XMEM_HUGE ; +else +rte_exit (EXIT_FAILURE , +"mp-alloc %s invalid - must be: " +"native, anon or xmem\n" , + optarg); }if (!strcmp (lgopts[opt_idx].name,"port-numa-config" )) {if (parse_portnuma_config (optarg))
app/test-pmd/testpmd.c +284 -21
338 lines not on this page · raw message
app/test-pmd/testpmd.h +12 -1
@@ -69,6 +69,16 @@ enum { PORT_TOPOLOGY_LOOP, }; +enum { +MP_ALLOC_NATIVE ,/**< allocate and populate mempool natively */ +MP_ALLOC_ANON , +/**< allocate mempool natively, but populate using anonymous memory */ +MP_ALLOC_XMEM , +/**< allocate and populate mempool using anonymous memory */ +MP_ALLOC_XMEM_HUGE +/**< allocate and populate mempool using anonymous hugepage memory */ +}; +#ifdef RTE_TEST_PMD_RECORD_BURST_STATS /** * The data structure associated with RX and TX packet burst statistics
@@ -304,7 +314,8 @@ extern uint8_t numa_support; /**< set by "--numa" parameter */extern uint16_t port_topology;/**< set by "--port-topology" parameter */ extern uint8_t no_flush_rx;/**<set by "--no-flush-rx" parameter */ extern uint8_t flow_isolate_all;/**< set by "--flow-isolate-all */ -extern uint8_t mp_anon;/**< set by "--mp-anon" parameter */ +extern uint8_t mp_alloc_type; +/**< set by "--mp-anon" or "--mp-alloc" parameter */ extern uint8_t no_link_check;/**<set by "--disable-link-check" parameter */ extern volatile int test_done;/* stop packet forwarding when set to 1. */ extern uint8_t lsc_interrupt;/**< disabled by "--no-lsc-interrupt" parameter */
doc/guides/testpmd_app_ug/run_app.rst +12 -0
@@ -498,3 +498,15 @@ The commandline options are: * ``--no-mlockall`` Disable locking all memory. + +* ``--mp-alloc <native|anon|xmem|xmemhuge>`` + + Select mempool allocation mode: + + * native: create and populate mempool using native DPDK memory + * anon: create mempool using native DPDK memory, but populate using + anonymous memory + * xmem: create and populate mempool using externally and anonymously + allocated area + * xmemhuge: create and populate mempool using externally and anonymously + allocated hugepage area
Signature · 1 line
2.17.1
PATCH v5 20/21 doc: add external memory feature to the release notes
In reply to Anatoly BurakovDocument the addition of external memory support to DPDK.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 5 +++++ 1 file changed, 5 insertions(+)
doc/guides/rel_notes/release_18_11.rst +5 -0
@@ -67,6 +67,11 @@ New Features SR-IOV option in Hyper-V and Azure. This is an alternative to the previous vdev_netvsc, tap, and failsafe drivers combination. +* **Added support for using externally allocated memory in DPDK.** + + DPDK has gained support for creating new ``rte_malloc`` heaps referencing + memory that was created outside of DPDK's own page allocator, and using that + memory natively with any other DPDK library or data structure. API Changes -----------
Signature · 1 line
2.17.1
PATCH v5 21/21 doc: add external memory feature to programmer's guide
In reply to Anatoly BurakovAdd a short chapter on usage of external memory in DPDK to the Programmer's Guide.
Signed-off-by Anatoly Burakov <[email protected]>
--- .../prog_guide/env_abstraction_layer.rst | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +37 -0
@@ -213,6 +213,43 @@ Normally, these options do not need to be changed. can later be mapped into that preallocated VA space (if dynamic memory mode is enabled), and can optionally be mapped into it at startup. +Support for Externally Allocated Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is possible to use externally allocated memory in DPDK, using a set of malloc +heap API's. Support for externally allocated memory is implemented through +overloading the socket ID - externally allocated heaps will have socket ID's +that would be considered invalid under normal circumstances. Requesting an +allocation to take place from a specified externally allocated memory is a +matter of supplying the correct socket ID to DPDK allocator, either directly +(e.g. through a call to ``rte_malloc``) or indirectly (through data +structure-specific allocation API's such as ``rte_ring_create``). + +Since there is no way DPDK can verify whether memory are is available or valid, +this responsibility falls on the shoulders of the user. All multiprocess +synchronization is also user's responsibility, as well as ensuring that all +calls to add/attach/detach/remove memory are done in the correct order. It is +not required to attach to a memory area in all processes - only attach to memory +areas as needed. + +The expected workflow is as follows: + +* Get a pointer to memory area +* Create a named heap +* Add memory area(s) to the heap + * If IOVA table is not specified, IOVA addresses will be assumed to be + unavailable, and DMA mappings will not be performed + * Other processes must attach to the memory area before they can use it +* Get socket ID used for the heap +* Use normal DPDK allocation procedures, using supplied socket ID +* If memory area is no longer needed, it can be removed from the heap + * Other processes must detach from this memory area before it can be removed +* If heap is no longer needed, remove it + * Socket ID will become invalid and will not be reused + +For more information, please refer to ``rte_malloc`` API documentation, +specifically the ``rte_malloc_heap_*`` family of function calls. + PCI Access ~~~~~~~~~~
Signature · 1 line
2.17.1
Re: [PATCH v5 21/21] doc: add external memory feature to programmer's guide
Quoted 14 lines
Add a short chapter on usage of external memory in DPDK to the Programmer's Guide.
Signed-off-by Anatoly Burakov <[email protected]>--- .../prog_guide/env_abstraction_layer.rst | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+)doc/guides/prog_guide/env_abstraction_layer.rst +6
-0+The expected workflow is as follows: + +* Get a pointer to memory area +* Create a named heap +* Add memory area(s) to the heap + * If IOVA table is not specified, IOVA addresses will be assumed to be
doc/guides/prog_guide/env_abstraction_layer.rst Normally, these options do not need to be changed. · line 248
Quoted 10 lines
doc/guides/prog_guide/env_abstraction_layer.rst +10
-0+ unavailable, and DMA mappings will not be performed + * Other processes must attach to the memory area before they can use +it +* Get socket ID used for the heap +* Use normal DPDK allocation procedures, using supplied socket ID +* If memory area is no longer needed, it can be removed from the heap + * Other processes must detach from this memory area before it can be +removed +* If heap is no longer needed, remove it + * Socket ID will become invalid and will not be reused
Hi Anatoly,
Im getting an error when doing
make-doc-guides-html
/dpdk/doc/guides/prog_guide/env_abstraction_layer.rst:241: WARNING: Unexpected indentation. /dpdk/doc/guides/prog_guide/env_abstraction_layer.rst:242: WARNING: Block quote ends without a blank line; unexpected unindent.
This is due to the indentation of your inner bullet points instead of two spaces put 4 For all four that you have.
Thanks,
Marko K
Re: [PATCH v5 21/21] doc: add external memory feature to programmer's guide
Quoted 32 lines · Marko
Quoted 14 lines
Add a short chapter on usage of external memory in DPDK to the Programmer's Guide.
Signed-off-by Anatoly Burakov <[email protected]>--- .../prog_guide/env_abstraction_layer.rst | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+)doc/guides/prog_guide/env_abstraction_layer.rst +6
-0+The expected workflow is as follows: + +* Get a pointer to memory area +* Create a named heap +* Add memory area(s) to the heap + * If IOVA table is not specified, IOVA addresses will be assumed to bedoc/guides/prog_guide/env_abstraction_layer.rst Normally, these options do not need to be changed. · line 248
Quoted 10 lines
doc/guides/prog_guide/env_abstraction_layer.rst +10
-0+ unavailable, and DMA mappings will not be performed + * Other processes must attach to the memory area before they can use +it +* Get socket ID used for the heap +* Use normal DPDK allocation procedures, using supplied socket ID +* If memory area is no longer needed, it can be removed from the heap + * Other processes must detach from this memory area before it can be +removed +* If heap is no longer needed, remove it + * Socket ID will become invalid and will not be reusedHi Anatoly,
Im getting an error when doing
make-doc-guides-html
Quoted 9 lines
/dpdk/doc/guides/prog_guide/env_abstraction_layer.rst:241: WARNING: Unexpected indentation. /dpdk/doc/guides/prog_guide/env_abstraction_layer.rst:242: WARNING: Block quote ends without a blank line; unexpected unindent.
This is due to the indentation of your inner bullet points instead of two spaces put 4 For all four that you have.
Thanks,
Marko K
Hi Marko,
Those are supposed to be sub-bullet points, i.e. 2nd level. Does it have to be six?
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v5 21/21] doc: add external memory feature to programmer's guide
Quoted 17 lines
Quoted 7 lines
Hi Anatoly,
Im getting an error when doing
make-doc-guides-html
/dpdk/doc/guides/prog_guide/env_abstraction_layer.rst:241: WARNING:
Unexpected indentation.
Quoted 1 line
/dpdk/doc/guides/prog_guide/env_abstraction_layer.rst:242: WARNING:
Block quote ends without a blank line; unexpected unindent.
Quoted 7 lines
This is due to the indentation of your inner bullet points instead of two spaces put 4 For all four that you have.
Thanks,
Marko K
Quoted 8 lines
Hi Marko,
Those are supposed to be sub-bullet points, i.e. 2nd level. Does it have to be six?
--
Thanks,
Anatoly
Yeah I understand that it is supposed to be sub-bullets but the output is skewed when you only have 2 spaces for them instead of having 2 have 4 not 6
v6 cover Support externally allocated memory in DPDK
In reply to Anatoly BurakovThis is a proposal to enable using externally allocated memory in DPDK.
In a nutshell, here is what is being done here:
- Index internal malloc heaps by NUMA node index, rather than NUMA node itself (external heaps will have ID's in order of creation)
- Add identifier string to malloc heap, to uniquely identify it
- Each new heap will receive a unique socket ID that will be used by allocator to decide from which heap (internal or external) to allocate requested amount of memory
- Allow creating named heaps and add/remove memory to/from those heaps
- Allocate memseg lists at runtime, to keep track of IOVA addresses of externally allocated memory
- If IOVA addresses aren't provided, use RTE_BAD_IOVA
- Allow malloc and memzones to allocate from external heaps
- Allow other data structures to allocate from externall heaps
The responsibility to ensure memory is accessible before using it is on the shoulders of the user - there is no checking done with regards to validity of the memory (nor could there be...).
The general approach is to create heap and add memory into it. For any other process wishing to use the same memory, said memory must first be attached (otherwise some things will not work).
A design decision was made to make multiprocess synchronization a manual process. Due to underlying issues with attaching to fbarrays in secondary processes, this design was deemed to be better because we don't want to fail to create external heap in the primary because something in the secondary has failed when in fact we may not eve have wanted this memory to be accessible in the secondary in the first place.
Using external memory in multiprocess is *hard*, because not only memory space needs to be preallocated, but it also needs to be attached in each process to allow other processes to access the page table. The attach API call may or may not succeed, depending on memory layout, for reasons similar to other multiprocess failures. This is treated as a "known issue" for this release.
v6 -> v5 changes:
- Fixed documentation formatting as per Marko's comments
v5 -> v4 changes:
- All processes are now able to create and destroy malloc heaps
- Memory is automatically mapped for DMA on adding it to heap
- Mem event callbacks are triggered on adding/removing memory
- Fixed compile issues on FreeBSD
- Better documentation on API/ABI changes
v4 -> v3 changes:
- Dropped sample application in favor of new testpmd flag
- Added new flag to testpmd, with four options of mempool allocation
- Added new API to check if a socket ID belongs to an external heap
- Adjusted malloc and mempool code to not make any assumptions about IOVA-contiguousness when dealing with externally allocated memory
v3 -> v2 changes:
- Rebase on top of latest master
- Clarifications added to mempool code as per Andrew Rynchenko's comments
v2 -> v1 changes:
- Fixed NULL dereference on heap socket ID lookup
- Fixed memseg offset calculation on adding memory to heap
- Improved unit test to test for above bugfixes
- Restricted heap creation to primary processes only
- Added sample application
- Added documentation
RFC -> v1 changes:
- Removed the "named heaps" API, allocate using fake socket ID instead
- Added multiprocess support
- Everything is now thread-safe
- Numerous bugfixes and API improvements
Anatoly Burakov (21):
mem: add length to memseg list
mem: allow memseg lists to be marked as external
malloc: index heaps using heap ID rather than NUMA node
mem: do not check for invalid socket ID
flow_classify: do not check for invalid socket ID
pipeline: do not check for invalid socket ID
sched: do not check for invalid socket ID
malloc: add name to malloc heaps
malloc: add function to query socket ID of named heap
malloc: add function to check if socket is external
malloc: allow creating malloc heaps
malloc: allow destroying heaps
malloc: allow adding memory to named heaps
malloc: allow removing memory from named heaps
malloc: allow attaching to external memory chunks
malloc: allow detaching from external memory
malloc: enable event callbacks for external memory
test: add unit tests for external memory support app/testpmd: add support for external memory
doc: add external memory feature to the release notes
doc: add external memory feature to programmer's guide
app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 305 ++++++++++++- app/test-pmd/testpmd.h | 13 +- config/common_base | 1 + config/rte_config.h | 1 + .../prog_guide/env_abstraction_layer.rst | 37 ++ doc/guides/rel_notes/deprecation.rst | 15 - doc/guides/rel_notes/release_18_11.rst | 28 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + drivers/bus/fslmc/fslmc_vfio.c | 14 +- drivers/bus/pci/linux/pci.c | 2 +- drivers/net/mlx4/mlx4_mr.c | 3 + drivers/net/mlx5/mlx5.c | 5 +- drivers/net/mlx5/mlx5_mr.c | 3 + drivers/net/virtio/virtio_user/vhost_kernel.c | 5 +- .../net/virtio/virtio_user/virtio_user_dev.c | 8 + lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal.c | 3 + lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +- lib/librte_eal/common/eal_common_memory.c | 8 +- lib/librte_eal/common/eal_common_memzone.c | 8 +- .../common/include/rte_eal_memconfig.h | 9 +- lib/librte_eal/common/include/rte_malloc.h | 192 ++++++++ .../common/include/rte_malloc_heap.h | 3 + lib/librte_eal/common/include/rte_memory.h | 9 + lib/librte_eal/common/malloc_elem.c | 10 +- lib/librte_eal/common/malloc_heap.c | 316 +++++++++++-- lib/librte_eal/common/malloc_heap.h | 17 + lib/librte_eal/common/rte_malloc.c | 429 +++++++++++++++++- lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +- lib/librte_eal/linuxapp/eal/eal_vfio.c | 27 +- lib/librte_eal/meson.build | 2 +- lib/librte_eal/rte_eal_version.map | 8 + lib/librte_flow_classify/rte_flow_classify.c | 3 +- lib/librte_mempool/rte_mempool.c | 57 ++- lib/librte_pipeline/rte_pipeline.c | 3 +- lib/librte_sched/rte_sched.c | 2 +- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++ test/test/test_malloc.c | 3 + test/test/test_memzone.c | 3 + 47 files changed, 1913 insertions(+), 139 deletions(-) create mode 100644 test/test/test_external_mem.c
Signature · 1 line
2.17.1
PATCH v6 1/21 mem: add length to memseg list
Previously, to calculate length of memory area covered by a memseg list, we would've needed to multiply page size by length of fbarray backing that memseg list. This is not obvious and unnecessarily low level, so store length in the memseg list itself.
Signed-off-by Anatoly Burakov <[email protected]>
--- drivers/bus/pci/linux/pci.c | 2 +- lib/librte_eal/bsdapp/eal/eal_memory.c | 2 ++ lib/librte_eal/common/eal_common_memory.c | 5 ++--- lib/librte_eal/common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/linuxapp/eal/eal_memalloc.c | 3 ++- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +++- 6 files changed, 11 insertions(+), 6 deletions(-)
drivers/bus/pci/linux/pci.c +1 -1
@@ -119,7 +119,7 @@ rte_pci_unmap_device(struct rte_pci_device *dev)static int find_max_end_va (const struct rte_memseg_list *msl,void *arg) { -size_t sz = msl->memseg_arr.len * msl->page_sz; +size_t sz = msl->len;void *end_va =RTE_PTR_ADD (msl->base_va, sz);void **max_va = arg;
lib/librte_eal/bsdapp/eal/eal_memory.c +2 -0
@@ -79,6 +79,7 @@ rte_eal_hugepage_init(void) } msl->base_va = addr; msl->page_sz = page_sz; + msl->len = internal_config.memory; msl->socket_id =0 ;/* populate memsegs. each memseg is 1 page long */
@@ -370,6 +371,7 @@ alloc_va_space(struct rte_memseg_list *msl)return -1 ; } msl->base_va = addr; + msl->len = mem_sz;return 0 ; }
lib/librte_eal/common/eal_common_memory.c +2 -3
@@ -171,7 +171,7 @@ virt2memseg(const void *addr, const struct rte_memseg_list *msl)/* a memseg list was specified, check if it's the right one */ start = msl->base_va; - end =RTE_PTR_ADD (start, (size_t )msl->page_sz * msl->memseg_arr.len); + end =RTE_PTR_ADD (start, msl->len);if (addr < start || addr >= end)return NULL ;
@@ -194,8 +194,7 @@ virt2memseg_list(const void *addr) msl = &mcfg->memsegs[msl_idx]; start = msl->base_va; - end =RTE_PTR_ADD (start, - (size_t )msl->page_sz * msl->memseg_arr.len); + end =RTE_PTR_ADD (start, msl->len);if (addr >= start && addr < end)break ; }
lib/librte_eal/common/include/rte_eal_memconfig.h +1 -0
@@ -30,6 +30,7 @@ struct rte_memseg_list {uint64_t addr_64;/**< Makes sure addr is always 64-bits */ }; +size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ volatile uint32_t version;/**< version number for multiprocess sync. */
lib/librte_eal/linuxapp/eal/eal_memalloc.c +2 -1
@@ -986,7 +986,7 @@ free_seg_walk(const struct rte_memseg_list *msl, void *arg)int msl_idx, seg_idx, ret, dir_fd =-1 ; start_addr = (uintptr_t ) msl->base_va; - end_addr = start_addr + msl->memseg_arr.len * (size_t )msl->page_sz; + end_addr = start_addr + msl->len;if ((uintptr_t )wa->ms->addr < start_addr || (uintptr_t )wa->ms->addr >= end_addr)
@@ -1472,6 +1472,7 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,return -1 ; } local_msl->base_va = primary_msl->base_va; + local_msl->len = primary_msl->len;return 0 ; }
lib/librte_eal/linuxapp/eal/eal_memory.c +3 -1
@@ -861,6 +861,7 @@ alloc_va_space(struct rte_memseg_list *msl)return -1 ; } msl->base_va = addr; + msl->len = mem_sz;return 0 ; }
@@ -1369,6 +1370,7 @@ eal_legacy_hugepage_init(void) msl->base_va = addr; msl->page_sz = page_sz; msl->socket_id =0 ; + msl->len = internal_config.memory;/* populate memsegs. each memseg is one page long */ for (cur_seg =0 ; cur_seg < n_segs; cur_seg++) {
@@ -1615,7 +1617,7 @@ eal_legacy_hugepage_init(void)if (msl->memseg_arr.count >0 )continue ;/* this is an unused list, deallocate it */ - mem_sz = (size_t )msl->page_sz * msl->memseg_arr.len; + mem_sz = msl->len;munmap (msl->base_va, mem_sz); msl->base_va =NULL ;
Signature · 1 line
2.17.1
Re: [PATCH v6 01/21] mem: add length to memseg list
Quoted 6 lines · Anatoly Burakov
Previously, to calculate length of memory area covered by a memseg list, we would've needed to multiply page size by length of fbarray backing that memseg list. This is not obvious and unnecessarily low level, so store length in the memseg list itself.
Signed-off-by Anatoly Burakov <[email protected]>
Quoted 9 lines
--- drivers/bus/pci/linux/pci.c | 2 +- lib/librte_eal/bsdapp/eal/eal_memory.c | 2 ++ lib/librte_eal/common/eal_common_memory.c | 5 ++--- lib/librte_eal/common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/linuxapp/eal/eal_memalloc.c | 3 ++- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +++- 6 files changed, 11 insertions(+), 6 deletions(-)
Acked-by Shreyansh Jain <[email protected]>
PATCH v6 2/21 mem: allow memseg lists to be marked as external
In reply to Anatoly BurakovWhen we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Signed-off-by Anatoly Burakov <[email protected]>
Acked-by Andrew Rybchenko <[email protected]>
---
Notes:
v3:
- Add comment to explain the process of picking up minimum page sizes for mempool
v2:
- Add documentation changes and ABI break
v1:
- Adjust all calls to memseg walk functions to ignore external segments where it made sense to do so
doc/guides/rel_notes/deprecation.rst | 15 -------- doc/guides/rel_notes/release_18_11.rst | 13 ++++++- drivers/bus/fslmc/fslmc_vfio.c | 7 ++-- drivers/net/mlx4/mlx4_mr.c | 3 ++ drivers/net/mlx5/mlx5.c | 5 ++- drivers/net/mlx5/mlx5_mr.c | 3 ++ drivers/net/virtio/virtio_user/vhost_kernel.c | 5 ++- lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal.c | 3 ++ lib/librte_eal/bsdapp/eal/eal_memory.c | 7 ++-- lib/librte_eal/common/eal_common_memory.c | 3 ++ .../common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/common/include/rte_memory.h | 9 +++++ lib/librte_eal/common/malloc_elem.c | 10 ++++-- lib/librte_eal/common/malloc_heap.c | 9 +++-- lib/librte_eal/common/rte_malloc.c | 2 +- lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +++++- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 9 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 ++++++--- lib/librte_eal/meson.build | 2 +- lib/librte_mempool/rte_mempool.c | 35 ++++++++++++++----- test/test/test_malloc.c | 3 ++ test/test/test_memzone.c | 3 ++ 24 files changed, 134 insertions(+), 44 deletions(-)
doc/guides/rel_notes/deprecation.rst +0 -15
@@ -11,21 +11,6 @@ API and ABI deprecation notices are to be posted here. Deprecation Notices ------------------- -* eal: certain structures will change in EAL on account of upcoming external - memory support. Aside from internal changes leading to an ABI break, the - following externally visible changes will also be implemented: - - - ``rte_memseg_list`` will change to include a boolean flag indicating - whether a particular memseg list is externally allocated. This will have - implications for any users of memseg-walk-related functions, as they will - now have to skip externally allocated segments in most cases if the intent - is to only iterate over internal DPDK memory. - - ``socket_id`` parameter across the entire DPDK will gain additional meaning, - as some socket ID's will now be representing externally allocated memory. No - changes will be required for existing code as backwards compatibility will - be kept, and those who do not use this feature will not see these extra - socket ID's. - * eal: both declaring and identifying devices will be streamlined in v18.11. New functions will appear to query a specific port from buses, classes of device and device drivers. Device declaration will be made coherent with the
doc/guides/rel_notes/release_18_11.rst +12 -1
@@ -91,6 +91,13 @@ API Changes flag the MAC can be properly configured in any case. This is particularly important for bonding. +* eal: The following API changes were made in 18.11: + + - ``rte_memseg_list`` structure now has an additional flag indicating whether + the memseg list is externally allocated. This will have implications for any + users of memseg-walk-related functions, as they will now have to skip + externally allocated segments in most cases if the intent is to only iterate + over internal DPDK memory. ABI Changes -----------
@@ -107,6 +114,10 @@ ABI Changes ========================================================= +* eal: EAL library ABI version was changed due to previously announced work on + supporting external memory in DPDK. Structure ``rte_memseg_list`` now has + a new flag indicating whether the memseg list refers to external memory. + Removed Items -------------
@@ -152,7 +163,7 @@ The libraries prepended with a plus sign were incremented in this version. librte_compressdev.so.1 librte_cryptodev.so.5 librte_distributor.so.1 - librte_eal.so.8 + + librte_eal.so.9 librte_ethdev.so.10 librte_eventdev.so.4 librte_flow_classify.so.1
drivers/bus/fslmc/fslmc_vfio.c +5 -2
@@ -317,12 +317,15 @@ fslmc_unmap_dma(uint64_t vaddr, uint64_t iovaddr __rte_unused, size_t len) }static int -fslmc_dmamap_seg(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +fslmc_dmamap_seg (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {int *n_segs = arg;int ret; +if (msl->external) +return 0 ; + ret =fslmc_map_dma (ms->addr_64, ms->iova, ms->len);if (ret) DPAA2_BUS_ERR("Unable to VFIO map (addr=%p, len=%zu)",
drivers/net/mlx4/mlx4_mr.c +3 -0
@@ -496,6 +496,9 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl, {struct mr_find_contig_memsegs_data *data = arg; +if (msl->external) +return 0 ; +if (data->addr < ms->addr_64 || data->addr >= ms->addr_64 + len)return 0 ;/* Found, save it and stop walking. */
drivers/net/mlx5/mlx5.c +4 -1
@@ -568,11 +568,14 @@ static struct rte_pci_driver mlx5_driver;static void *uar_base;static int -find_lower_va_bound(const struct rte_memseg_list *msl __rte_unused, +find_lower_va_bound (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {void **addr = arg; +if (msl->external) +return 0 ; +if (*addr ==NULL ) *addr = ms->addr;else
drivers/net/mlx5/mlx5_mr.c +3 -0
@@ -486,6 +486,9 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl, {struct mr_find_contig_memsegs_data *data = arg; +if (msl->external) +return 0 ; +if (data->addr < ms->addr_64 || data->addr >= ms->addr_64 + len)return 0 ;/* Found, save it and stop walking. */
drivers/net/virtio/virtio_user/vhost_kernel.c +4 -1
@@ -75,13 +75,16 @@ struct walk_arg {uint32_t region_nr; };static int -add_memory_region(const struct rte_memseg_list *msl __rte_unused, +add_memory_region (const struct rte_memseg_list *msl,const struct rte_memseg *ms,size_t len,void *arg) {struct walk_arg *wa = arg;struct vhost_memory_region *mr;void *start_addr; +if (msl->external) +return 0 ; +if (wa->region_nr >= max_regions)return -1 ;
lib/librte_eal/bsdapp/eal/Makefile +1 -1
@@ -22,7 +22,7 @@ LDLIBS += -lrte_kvargs EXPORT_MAP := ../../rte_eal_version.map -LIBABIVER := 8 +LIBABIVER := 9 # specific to bsdapp exec-env SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) := eal.c
lib/librte_eal/bsdapp/eal/eal.c +3 -0
@@ -502,6 +502,9 @@ check_socket(const struct rte_memseg_list *msl, void *arg) {int *socket_id = arg; +if (msl->external) +return 0 ; +if (msl->socket_id == *socket_id && msl->memseg_arr.count !=0 )return 1 ;
lib/librte_eal/bsdapp/eal/eal_memory.c +5 -2
@@ -236,12 +236,15 @@ struct attach_walk_args {int seg_idx; };static int -attach_segment(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +attach_segment (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {struct attach_walk_args *wa = arg;void *addr; +if (msl->external) +return 0 ; + addr =mmap (ms->addr, ms->len,PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_FIXED , wa->fd_hugepage, wa->seg_idx *EAL_PAGE_SIZE );
lib/librte_eal/common/eal_common_memory.c +3 -0
@@ -272,6 +272,9 @@ physmem_size(const struct rte_memseg_list *msl, void *arg) {uint64_t *total_len = arg; +if (msl->external) +return 0 ; + *total_len += msl->memseg_arr.count * msl->page_sz;return 0 ;
lib/librte_eal/common/include/rte_eal_memconfig.h +1 -0
@@ -33,6 +33,7 @@ struct rte_memseg_list {size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ +unsigned int external;/**< 1 if this list points to external memory */ volatile uint32_t version;/**< version number for multiprocess sync. */ struct rte_fbarray memseg_arr; };
lib/librte_eal/common/include/rte_memory.h +9 -0
@@ -215,6 +215,9 @@ typedef int (*rte_memseg_list_walk_t)(const struct rte_memseg_list *msl,* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
@@ -233,6 +236,9 @@ rte_memseg_walk(rte_memseg_walk_t func, void *arg);* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
@@ -251,6 +257,9 @@ rte_memseg_contig_walk(rte_memseg_contig_walk_t func, void *arg);* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
lib/librte_eal/common/malloc_elem.c +7 -3
@@ -39,10 +39,14 @@ malloc_elem_find_max_iova_contig(struct malloc_elem *elem, size_t align) contig_seg_start =RTE_PTR_ALIGN_CEIL (data_start, align);/* if we're in IOVA as VA mode, or if we're in legacy mode with -* hugepages, all elements are IOVA-contiguous. +* hugepages, all elements are IOVA-contiguous. however, we can only +* make these assumptions about internal memory - externally allocated +* segments have to be checked. */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA || - (internal_config.legacy_mem &&rte_eal_has_hugepages ())) +if (!elem->msl->external && + (rte_eal_iova_mode () ==RTE_IOVA_VA || + (internal_config.legacy_mem && +rte_eal_has_hugepages ())))return RTE_PTR_DIFF (data_end, contig_seg_start); cur_page =RTE_PTR_ALIGN_FLOOR (contig_seg_start, page_sz);
lib/librte_eal/common/malloc_heap.c +7 -2
@@ -95,6 +95,9 @@ malloc_add_seg(const struct rte_memseg_list *msl,struct malloc_heap *heap;int msl_idx; +if (msl->external) +return 0 ; + heap = &mcfg->malloc_heaps[msl->socket_id];/* msl is const, so find it */
@@ -754,8 +757,10 @@ malloc_heap_free(struct malloc_elem *elem)/* anything after this is a bonus */ ret =0 ; -/* ...of which we can't avail if we are in legacy mode */ -if (internal_config.legacy_mem) +/* ...of which we can't avail if we are in legacy mode, or if this is an +* externally allocated segment. +*/ +if (internal_config.legacy_mem || msl->external) goto free_unlock;/* check if we can free any memory back to the system */
lib/librte_eal/common/rte_malloc.c +1 -1
@@ -223,7 +223,7 @@ rte_malloc_virt2iova(const void *addr)if (elem ==NULL )return RTE_BAD_IOVA ; -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) +if (!elem->msl->external &&rte_eal_iova_mode () ==RTE_IOVA_VA )return (uintptr_t ) addr; ms =rte_mem_virt2memseg (addr, elem->msl);
lib/librte_eal/linuxapp/eal/Makefile +1 -1
@@ -10,7 +10,7 @@ ARCH_DIR ?= $(RTE_ARCH) EXPORT_MAP := ../../rte_eal_version.map VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR) -LIBABIVER := 8 +LIBABIVER := 9 VPATH += $(RTE_SDK)/lib/librte_eal/common
lib/librte_eal/linuxapp/eal/eal.c +9 -1
@@ -725,6 +725,9 @@ check_socket(const struct rte_memseg_list *msl, void *arg) {int *socket_id = arg; +if (msl->external) +return 0 ; +return *socket_id == msl->socket_id; }
@@ -1059,7 +1062,12 @@ mark_freeable(const struct rte_memseg_list *msl, const struct rte_memseg *ms, void *arg __rte_unused) {/* ms is const, so find this memseg */ -struct rte_memseg *found =rte_mem_virt2memseg (ms->addr, msl); +struct rte_memseg *found; + +if (msl->external) +return 0 ; + + found =rte_mem_virt2memseg (ms->addr, msl); found->flags &= ~RTE_MEMSEG_FLAG_DO_NOT_FREE ;
lib/librte_eal/linuxapp/eal/eal_memalloc.c +9 -0
@@ -1408,6 +1408,9 @@ sync_walk(const struct rte_memseg_list *msl, void *arg __rte_unused)unsigned int i;int msl_idx; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; primary_msl = &mcfg->memsegs[msl_idx]; local_msl = &local_memsegs[msl_idx];
@@ -1456,6 +1459,9 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,char name[PATH_MAX ];int msl_idx, ret; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; primary_msl = &mcfg->memsegs[msl_idx]; local_msl = &local_memsegs[msl_idx];
@@ -1509,6 +1515,9 @@ fd_list_create_walk(const struct rte_memseg_list *msl,unsigned int len;int msl_idx; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; len = msl->memseg_arr.len;
lib/librte_eal/linuxapp/eal/eal_vfio.c +13 -4
@@ -1082,11 +1082,14 @@ rte_vfio_get_group_num(const char *sysfs_base, }static int -type1_map(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +type1_map (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {int *vfio_container_fd = arg; +if (msl->external) +return 0 ; +return vfio_type1_dma_mem_map (*vfio_container_fd, ms->addr_64, ms->iova, ms->len,1 ); }
@@ -1196,11 +1199,14 @@ vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova, }static int -vfio_spapr_map_walk(const struct rte_memseg_list *msl __rte_unused, +vfio_spapr_map_walk (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {int *vfio_container_fd = arg; +if (msl->external) +return 0 ; +return vfio_spapr_dma_mem_map (*vfio_container_fd, ms->addr_64, ms->iova, ms->len,1 ); }
@@ -1210,12 +1216,15 @@ struct spapr_walk_param {uint64_t hugepage_sz; };static int -vfio_spapr_window_size_walk(const struct rte_memseg_list *msl __rte_unused, +vfio_spapr_window_size_walk (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {struct spapr_walk_param *param = arg;uint64_t max = ms->iova + ms->len; +if (msl->external) +return 0 ; +if (max > param->window_size) { param->hugepage_sz = ms->hugepage_sz; param->window_size = max;
lib/librte_eal/meson.build +1 -1
@@ -21,7 +21,7 @@ else error('unsupported system type "@0@"'.format(host_machine.system())) endif -version = 8 # the version of the EAL API +version = 9 # the version of the EAL API allow_experimental_apis = true deps += 'compat' deps += 'kvargs'
lib/librte_mempool/rte_mempool.c +27 -8
@@ -99,25 +99,44 @@ static unsigned optimize_object_size(unsigned obj_size)return new_obj_size *RTE_MEMPOOL_ALIGN ; } +struct pagesz_walk_arg { +int socket_id; +size_t min; +}; +static int find_min_pagesz (const struct rte_memseg_list *msl,void *arg) { -size_t *min = arg; +struct pagesz_walk_arg *wa = arg; +bool valid; -if (msl->page_sz < *min) - *min = msl->page_sz; +/* +* we need to only look at page sizes available for a particular socket +* ID. so, we either need an exact match on socket ID (can match both +* native and external memory), or, if SOCKET_ID_ANY was specified as a +* socket ID argument, we must only look at native memory and ignore any +* page sizes associated with external memory. +*/ + valid = msl->socket_id == wa->socket_id; + valid |= wa->socket_id ==SOCKET_ID_ANY && msl->external ==0 ; + +if (valid && msl->page_sz < wa->min) + wa->min = msl->page_sz;return 0 ; }static size_t -get_min_page_size (void ) +get_min_page_size (int socket_id) { -size_t min_pagesz =SIZE_MAX ; +struct pagesz_walk_arg wa; -rte_memseg_list_walk (find_min_pagesz, &min_pagesz); + wa.min =SIZE_MAX ; + wa.socket_id = socket_id; -return min_pagesz ==SIZE_MAX ? (size_t )getpagesize () : min_pagesz; +rte_memseg_list_walk (find_min_pagesz, &wa); + +return wa.min ==SIZE_MAX ? (size_t )getpagesize () : wa.min; }
@@ -470,7 +489,7 @@ rte_mempool_populate_default(struct rte_mempool *mp) pg_sz =0 ; pg_shift =0 ; }else if (try_contig) { - pg_sz =get_min_page_size (); + pg_sz =get_min_page_size (mp->socket_id); pg_shift =rte_bsf32 (pg_sz); }else { pg_sz =getpagesize ();
test/test/test_malloc.c +3 -0
@@ -711,6 +711,9 @@ check_socket_mem(const struct rte_memseg_list *msl, void *arg) {int32_t *socket = arg; +if (msl->external) +return 0 ; +return *socket == msl->socket_id; }
test/test/test_memzone.c +3 -0
@@ -115,6 +115,9 @@ find_available_pagesz(const struct rte_memseg_list *msl, void *arg) {struct walk_arg *wa = arg; +if (msl->external) +return 0 ; +if (msl->page_sz ==RTE_PGSIZE_2M ) wa->hugepage_2MB_avail =1 ;if (msl->page_sz ==RTE_PGSIZE_1G )
Signature · 1 line
2.17.1
Re: [PATCH v6 02/21] mem: allow memseg lists to be marked as external
Quoted 20 lines · Anatoly Burakov
When we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Quoted 4 lines
Signed-off-by Anatoly Burakov <[email protected]>Acked-by Andrew Rybchenko <[email protected]>---
Specifically for bus/fslmc perspective and generically for others:
Acked-by Shreyansh Jain <[email protected]>
Re: [PATCH v6 02/21] mem: allow memseg lists to be marked as external
Quoted 21 lines · Shreyansh Jain
Quoted 20 lines · Anatoly Burakov
When we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Quoted 10 lines
Quoted 4 lines
Signed-off-by Anatoly Burakov <[email protected]>Acked-by Andrew Rybchenko <[email protected]>---
Specifically for bus/fslmc perspective and generically for others:
Acked-by Shreyansh Jain <[email protected]>
Actually, this patch may need some further adjustment, since it makes assumption about not wanting to map external memory for DMA.
Specifically - there's an fslmc dma map function that now skips external memory segments. Are you sure that's how it's supposed to be?
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v6 02/21] mem: allow memseg lists to be marked as external
Quoted 29 lines · Anatoly
Quoted 28 lines · Shreyansh Jain
Quoted 20 lines · Anatoly Burakov
When we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Quoted 4 lines
Signed-off-by Anatoly Burakov <[email protected]>Acked-by Andrew Rybchenko <[email protected]>---
Specifically for bus/fslmc perspective and generically for others:
Quoted 8 lines
Quoted 3 lines
Acked-by Shreyansh Jain <[email protected]>Actually, this patch may need some further adjustment, since it makes assumption about not wanting to map external memory for DMA.
Specifically - there's an fslmc dma map function that now skips external memory segments. Are you sure that's how it's supposed to be?
I thought over that.
For now yes. If we need to map external memory, and there is an event that would be called back, it should be handled separately. So, for example, a PMD level API to handle such requests from applications.
The point is that how the external memory is handled is use-case specific - the need to have its events reported back is definitely there, but its handling is still a grey area.
Once the patches make their way in, I can always come back and tune that.
Re: [PATCH v6 02/21] mem: allow memseg lists to be marked as external
Quoted 34 lines · Shreyansh Jain
Quoted 33 lines · Anatoly
Quoted 31 lines · Shreyansh Jain
Quoted 20 lines · Anatoly Burakov
When we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Quoted 4 lines
Signed-off-by Anatoly Burakov <[email protected]>Acked-by Andrew Rybchenko <[email protected]>---
Specifically for bus/fslmc perspective and generically for others:
Acked-by Shreyansh Jain <[email protected]>
Quoted 7 lines
Quoted 4 lines
Actually, this patch may need some further adjustment, since it makes assumption about not wanting to map external memory for DMA.
Specifically - there's an fslmc dma map function that now skips external memory segments. Are you sure that's how it's supposed to be?
I thought over that.
For now yes. If we need to map external memory, and there is an event that would be called back, it should be handled separately. So, for example, a PMD level API to handle such requests from applications.
Well, technically such an event is already available, now that external memory allocations trigger mem events :)
Quoted 5 lines
The point is that how the external memory is handled is use-case specific - the need to have its events reported back is definitely there, but its handling is still a grey area.
Once the patches make their way in, I can always come back and tune that.
OK, fair enough.
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v6 02/21] mem: allow memseg lists to be marked as external
In reply to Anatoly BurakovQuoted 153 lines · Anatoly Burakov
When we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Signed-off-by Anatoly Burakov <[email protected]>Acked-by Andrew Rybchenko <[email protected]>---
Notes:
v3:
- Add comment to explain the process of picking up minimum page sizes for mempoolv2:
- Add documentation changes and ABI breakv1:
- Adjust all calls to memseg walk functions to ignore external segments where it made sense to do sodoc/guides/rel_notes/deprecation.rst | 15 -------- doc/guides/rel_notes/release_18_11.rst | 13 ++++++- drivers/bus/fslmc/fslmc_vfio.c | 7 ++-- drivers/net/mlx4/mlx4_mr.c | 3 ++ drivers/net/mlx5/mlx5.c | 5 ++- drivers/net/mlx5/mlx5_mr.c | 3 ++ drivers/net/virtio/virtio_user/vhost_kernel.c | 5 ++- lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal.c | 3 ++ lib/librte_eal/bsdapp/eal/eal_memory.c | 7 ++-- lib/librte_eal/common/eal_common_memory.c | 3 ++ .../common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/common/include/rte_memory.h | 9 +++++ lib/librte_eal/common/malloc_elem.c | 10 ++++-- lib/librte_eal/common/malloc_heap.c | 9 +++-- lib/librte_eal/common/rte_malloc.c | 2 +- lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +++++- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 9 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 ++++++--- lib/librte_eal/meson.build | 2 +- lib/librte_mempool/rte_mempool.c | 35 ++++++++++++++----- test/test/test_malloc.c | 3 ++ test/test/test_memzone.c | 3 ++ 24 files changed, 134 insertions(+), 44 deletions(-)doc/guides/rel_notes/deprecation.rst +0
-15@@ -11,21 +11,6 @@ API and ABI deprecation notices are to be posted here. Deprecation Notices ------------------- -* eal: certain structures will change in EAL on account of upcoming external - memory support. Aside from internal changes leading to an ABI break, the - following externally visible changes will also be implemented: - - - ``rte_memseg_list`` will change to include a boolean flag indicating - whether a particular memseg list is externally allocated. This will have - implications for any users of memseg-walk-related functions, as they will - now have to skip externally allocated segments in most cases if the intent - is to only iterate over internal DPDK memory. - - ``socket_id`` parameter across the entire DPDK will gain additional meaning, - as some socket ID's will now be representing externally allocated memory. No - changes will be required for existing code as backwards compatibility will - be kept, and those who do not use this feature will not see these extra - socket ID's. - * eal: both declaring and identifying devices will be streamlined in v18.11. New functions will appear to query a specific port from buses, classes of device and device drivers. Device declaration will be made coherent with thedoc/guides/rel_notes/release_18_11.rst +12
-1@@ -91,6 +91,13 @@ API Changes flag the MAC can be properly configured in any case. This is particularly important for bonding. +* eal: The following API changes were made in 18.11: + + - ``rte_memseg_list`` structure now has an additional flag indicating whether + the memseg list is externally allocated. This will have implications for any + users of memseg-walk-related functions, as they will now have to skip + externally allocated segments in most cases if the intent is to only iterate + over internal DPDK memory. ABI Changes -----------@@ -107,6 +114,10 @@ ABI Changes ========================================================= +* eal: EAL library ABI version was changed due to previously announced work on + supporting external memory in DPDK. Structure ``rte_memseg_list`` now has + a new flag indicating whether the memseg list refers to external memory. + Removed Items -------------@@ -152,7 +163,7 @@ The libraries prepended with a plus sign were incremented in this version. librte_compressdev.so.1 librte_cryptodev.so.5 librte_distributor.so.1 - librte_eal.so.8 + + librte_eal.so.9 librte_ethdev.so.10 librte_eventdev.so.4 librte_flow_classify.so.1drivers/bus/fslmc/fslmc_vfio.c +5
-2@@ -317,12 +317,15 @@ fslmc_unmap_dma(uint64_t vaddr, uint64_t iovaddr __rte_unused, size_t len) }static int -fslmc_dmamap_seg(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +fslmc_dmamap_seg (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {int *n_segs = arg;int ret; +if (msl->external) +return 0 ; + ret =fslmc_map_dma (ms->addr_64, ms->iova, ms->len);if (ret) DPAA2_BUS_ERR("Unable to VFIO map (addr=%p, len=%zu)",drivers/net/mlx4/mlx4_mr.c +0
-0
drivers/net/mlx4/mlx4_mr.c mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl, · line 501
Quoted 6 lines
drivers/net/mlx4/mlx4_mr.c +3
-0@@ -496,6 +496,9 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl, {struct mr_find_contig_memsegs_data *data = arg; +if (msl->external) +return 0 ; +
Because memory free event for external memory is available, current design of mlx4/mlx5 memory mgmt can accommodate the new external memory support. So, please remove it so that PMD can traverse external memory as well.
Quoted 10 lines
drivers/net/mlx4/mlx4_mr.c +0
-0if (data->addr < ms->addr_64 || data->addr >= ms->addr_64 + len)return 0 ;/* Found, save it and stop walking. */ drivers/net/mlx5/mlx5.c +0
-0@@ -568,11 +568,14 @@ static struct rte_pci_driver mlx5_driver;static void *uar_base;
drivers/net/mlx5/mlx5.c static struct rte_pci_driver mlx5_driver; · line 578
Quoted 10 lines
drivers/net/mlx5/mlx5.c +4
-1static int -find_lower_va_bound(const struct rte_memseg_list *msl __rte_unused, +find_lower_va_bound (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {void **addr = arg; +if (msl->external) +return 0 ; +
This one is fine.
But can you please remove the blank line?
That's a rule by former maintainers. :-)
Quoted 8 lines
drivers/net/mlx5/mlx5.c +0
-0if (*addr ==NULL ) *addr = ms->addr;else drivers/net/mlx5/mlx5_mr.c +0
-0
drivers/net/mlx5/mlx5_mr.c mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl, · line 491
Quoted 6 lines
drivers/net/mlx5/mlx5_mr.c +3
-0@@ -486,6 +486,9 @@ mr_find_contig_memsegs_cb(const struct rte_memseg_list *msl, {struct mr_find_contig_memsegs_data *data = arg; +if (msl->external) +return 0 ; +
Like I mentioned, please remove it.
If those two changes in mlx4/5_mr.c are removed, for the whole patch,
Acked-by Yongseok Koh <[email protected]>
Thanks
Yongseok
PATCH v6 3/21 malloc: index heaps using heap ID rather than NUMA node
In reply to Anatoly BurakovSwitch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation.
Signed-off-by Anatoly Burakov <[email protected]>
--- config/common_base | 1 + config/rte_config.h | 1 + .../common/include/rte_eal_memconfig.h | 4 +- .../common/include/rte_malloc_heap.h | 1 + lib/librte_eal/common/malloc_heap.c | 98 +++++++++++++------ lib/librte_eal/common/malloc_heap.h | 3 + lib/librte_eal/common/rte_malloc.c | 41 +++++--- 7 files changed, 106 insertions(+), 43 deletions(-)
config/common_base +1 -0
@@ -61,6 +61,7 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_LIBRTE_EAL=y CONFIG_RTE_MAX_LCORE=128 CONFIG_RTE_MAX_NUMA_NODES=8 +CONFIG_RTE_MAX_HEAPS=32 CONFIG_RTE_MAX_MEMSEG_LISTS=64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever is smaller
config/rte_config.h +1 -0
@@ -24,6 +24,7 @@#define RTE_BUILD_SHARED_LIB /* EAL defines */ +#define RTE_MAX_HEAPS 32#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768
lib/librte_eal/common/include/rte_eal_memconfig.h +2 -2
@@ -72,8 +72,8 @@ struct rte_mem_config {struct rte_tailq_head tailq_head[RTE_MAX_TAILQ ];/**< Tailqs for objects */ -/* Heaps of Malloc per socket */ -struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES ]; +/* Heaps of Malloc */ +struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ];/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it.
lib/librte_eal/common/include/rte_malloc_heap.h +1 -0
@@ -27,6 +27,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size; + unsigned int socket_id; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */
lib/librte_eal/common/malloc_heap.c +69 -29
204 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -46,6 +46,9 @@ malloc_heap_get_stats(struct malloc_heap *heap,void malloc_heap_dump (struct malloc_heap *heap,FILE *f); +int +malloc_socket_to_heap_id (unsigned int socket_id); +int rte_eal_malloc_heap_init (void );
lib/librte_eal/common/rte_malloc.c +29 -12
72 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v6 03/21] malloc: index heaps using heap ID rather than NUMA node
Quoted 104 lines · Anatoly Burakov
Switch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation.
Signed-off-by Anatoly Burakov <[email protected]>--- config/common_base | 1 + config/rte_config.h | 1 + .../common/include/rte_eal_memconfig.h | 4 +- .../common/include/rte_malloc_heap.h | 1 + lib/librte_eal/common/malloc_heap.c | 98 +++++++++++++------ lib/librte_eal/common/malloc_heap.h | 3 + lib/librte_eal/common/rte_malloc.c | 41 +++++--- 7 files changed, 106 insertions(+), 43 deletions(-)config/common_base +1
-0@@ -61,6 +61,7 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_LIBRTE_EAL=y CONFIG_RTE_MAX_LCORE=128 CONFIG_RTE_MAX_NUMA_NODES=8 +CONFIG_RTE_MAX_HEAPS=32 CONFIG_RTE_MAX_MEMSEG_LISTS=64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever issmaller
config/rte_config.h +1
-0@@ -24,6 +24,7 @@#define RTE_BUILD_SHARED_LIB /* EAL defines */ +#define RTE_MAX_HEAPS 32#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768lib/librte_eal/common/include/rte_eal_memconfig.h +0
-0@@ -72,8 +72,8 @@ struct rte_mem_config {struct rte_tailq_head tailq_head[RTE_MAX_TAILQ ];/**< Tailqs for objects */
lib/librte_eal/common/include/rte_eal_memconfig.h +2
-2-/* Heaps of Malloc per socket */ -struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES ]; +/* Heaps of Malloc */ +struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ];/* address of mem_config in primary process. used to map shared config into
* exact same address the primary process maps it.lib/librte_eal/common/include/rte_malloc_heap.h +1
-0@@ -27,6 +27,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size; + unsigned int socket_id; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */ lib/librte_eal/common/malloc_heap.c +15
-0@@ -66,6 +66,21 @@ check_hugepage_sz(unsigned flags, uint64_t hugepage_sz)return check_flag & flags; } +int +malloc_socket_to_heap_id (unsigned int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +int i; + +for (i =0 ; i <RTE_MAX_HEAPS ; i++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[i]; + +if (heap->socket_id == socket_id) +return i; + } +return -1 ; +} +/* * Expand the heap with a memory area. */
lib/librte_eal/common/malloc_heap.c +0
-0@@ -93,12 +108,13 @@ malloc_add_seg(const struct rte_memseg_list *msl,struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct rte_memseg_list *found_msl;
lib/librte_eal/common/malloc_heap.c malloc_add_seg(const struct rte_memseg_list *msl, · line 118
Quoted 10 lines
lib/librte_eal/common/malloc_heap.c +2
-2struct malloc_heap *heap; -int msl_idx; +int msl_idx, heap_idx;if (msl->external)return 0 ; - heap = &mcfg->malloc_heaps[msl->socket_id]; + heap_idx =malloc_socket_to_heap_id (msl->socket_id);
malloc_socket_to_heap_id can return -1 so it requires to handle that possibility.
Quoted 315 lines
lib/librte_eal/common/malloc_heap.c +1
-0+ heap = &mcfg->malloc_heaps[heap_idx];/* msl is const, so find it */ msl_idx = msl - mcfg->memsegs;lib/librte_eal/common/malloc_heap.c +6
-3@@ -111,6 +127,7 @@ malloc_add_seg(const struct rte_memseg_list *msl,malloc_heap_add_memory (heap, found_msl, ms->addr, len); heap->total_size += len; + heap->socket_id = msl->socket_id;RTE_LOG (DEBUG ,EAL ,"Added %zuM to heap on socket %i\n" , len >>20 , msl->socket_id);@@ -561,12 +578,14 @@ alloc_more_mem_on_socket(struct malloc_heap *heap, size_t size, int socket,/* this will try lower page sizes first */ static void * -heap_alloc_on_socket(const char *type, size_t size, int socket, - unsigned int flags, size_t align, size_t bound, bool contig) +malloc_heap_alloc_on_heap_id(const char *type, size_t size, + unsigned int heap_id, unsigned int flags, size_t align, + size_t bound, bool contig) { struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; -struct malloc_heap *heap = &mcfg->malloc_heaps[socket]; +struct malloc_heap *heap = &mcfg->malloc_heaps[heap_id];unsigned int size_flags = flags & ~RTE_MEMZONE_SIZE_HINT_ONLY ; +int socket_id;void *ret; rte_spinlock_lock(&(heap->lock));lib/librte_eal/common/malloc_heap.c +4
-0@@ -584,12 +603,28 @@ heap_alloc_on_socket(const char *type, size_t size,int socket, * we may still be able to allocate memory from appropriate page sizes, * we just need to request more memory first. */ + + socket_id =rte_socket_id_by_idx (heap_id); +/* +* if socket ID is negative, we cannot find a socket ID for this heap -
+ * which means it's an external heap. those can have unexpected
pagelib/librte_eal/common/malloc_heap.c +2
-0+* sizes, so if the user asked to allocate from there - assume user +* knows what they're doing, and allow allocating from there with any
lib/librte_eal/common/malloc_heap.c +5
-0+* page size flags. +*/ +if (socket_id <0 ) + size_flags |=RTE_MEMZONE_SIZE_HINT_ONLY ; + ret = heap_alloc(heap, type, size, size_flags, align, bound,contig); if (ret != NULL) goto alloc_unlock;- if (!alloc_more_mem_on_socket(heap, size, socket, flags, align,
bound,lib/librte_eal/common/malloc_heap.c +6
-1- contig)) { +/* if socket ID is invalid, this is an external heap */ +if (socket_id <0 ) + goto alloc_unlock; + +if (!alloc_more_mem_on_socket (heap, size, socket_id, flags, align, + bound, contig)) { ret = heap_alloc(heap, type, size, flags, align, bound,contig);
/* this should have succeeded */lib/librte_eal/common/malloc_heap.c +1
-1@@ -605,7 +640,7 @@ void * malloc_heap_alloc(const char *type, size_t size, int socket_arg, unsigned int flags, size_t align, size_t bound, bool contig) { - int socket, i, cur_socket; + int socket, heap_id, i;void *ret;/* return NULL if size is 0 or alignment is not power-of-2 */lib/librte_eal/common/malloc_heap.c +11
-9@@ -620,22 +655,25 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg, else socket = socket_arg; -/* Check socket parameter */ -if (socket >=RTE_MAX_NUMA_NODES ) +/* turn socket ID into heap ID */ + heap_id =malloc_socket_to_heap_id (socket); +/* if heap id is negative, socket ID was invalid */ +if (heap_id <0 )return NULL ; - ret =heap_alloc_on_socket (type, size, socket, flags, align, bound, - contig); + ret =malloc_heap_alloc_on_heap_id (type, size, heap_id, flags, align, + bound, contig);if (ret !=NULL || socket_arg !=SOCKET_ID_ANY )return ret; -/* try other heaps */ +/* try other heaps. we are only iterating through native DPDK sockets, +* so external heaps won't be included. +*/ for (i =0 ; i < (int )rte_socket_count (); i++) { - cur_socket =rte_socket_id_by_idx (i); -if (cur_socket == socket) +if (i == heap_id)continue ; - ret =heap_alloc_on_socket (type, size, cur_socket, flags, - align, bound, contig); + ret = malloc_heap_alloc_on_heap_id(type, size, i, flags,align,
lib/librte_eal/common/malloc_heap.c +4
-3+ bound, contig);if (ret !=NULL )return ret; }@@ -643,11 +681,11 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg, }static void * -heap_alloc_biggest_on_socket (const char *type,int socket,unsigned int flags, -size_t align,bool contig) +heap_alloc_biggest_on_heap_id (const char *type,unsigned int heap_id, +unsigned int flags,size_t align,bool contig) { struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; -struct malloc_heap *heap = &mcfg->malloc_heaps[socket]; +struct malloc_heap *heap = &mcfg->malloc_heaps[heap_id];void *ret; rte_spinlock_lock(&(heap->lock));lib/librte_eal/common/malloc_heap.c +1
-1@@ -665,7 +703,7 @@ void * malloc_heap_alloc_biggest(const char *type, int socket_arg, unsigned int flags, size_t align, bool contig) { - int socket, i, cur_socket; + int socket, i, cur_socket, heap_id;void *ret;/* return NULL if align is not power-of-2 */lib/librte_eal/common/malloc_heap.c +5
-3@@ -680,11 +718,13 @@ malloc_heap_alloc_biggest(const char *type, int socket_arg, unsigned int flags, else socket = socket_arg; -/* Check socket parameter */ -if (socket >=RTE_MAX_NUMA_NODES ) +/* turn socket ID into heap ID */ + heap_id =malloc_socket_to_heap_id (socket); +/* if heap id is negative, socket ID was invalid */ +if (heap_id <0 )return NULL ; - ret =heap_alloc_biggest_on_socket (type, socket, flags, align, + ret =heap_alloc_biggest_on_heap_id (type, heap_id, flags, align, contig);if (ret !=NULL || socket_arg !=SOCKET_ID_ANY )return ret;lib/librte_eal/common/malloc_heap.c +2
-2@@ -694,8 +734,8 @@ malloc_heap_alloc_biggest(const char *type, int socket_arg, unsigned int flags, cur_socket =rte_socket_id_by_idx (i);if (cur_socket == socket)continue ; - ret =heap_alloc_biggest_on_socket (type, cur_socket, flags, - align, contig); + ret =heap_alloc_biggest_on_heap_id (type, i, flags, align, + contig);if (ret !=NULL )return ret;}lib/librte_eal/common/malloc_heap.c +0
-0@@ -760,7 +800,7 @@ malloc_heap_free(struct malloc_elem *elem)/* ...of which we can't avail if we are in legacy mode, or if this is an
* externally allocated segment.
*/lib/librte_eal/common/malloc_heap.c +2
-2-if (internal_config.legacy_mem || msl->external) +if (internal_config.legacy_mem || (msl->external >0 )) goto free_unlock;/* check if we can free any memory back to the system */ @@ -917,7 +957,7 @@ malloc_heap_resize(struct malloc_elem *elem, size_t size) }/* -* Function to retrieve data for heap on given socket +* Function to retrieve data for a given heap */ int malloc_heap_get_stats(struct
malloc_heap*heap,lib/librte_eal/common/malloc_heap.c +1
-1@@ -955,7 +995,7 @@ malloc_heap_get_stats(struct malloc_heap *heap, }/* -* Function to retrieve data for heap on given socket +* Function to retrieve data for a given heap */ void malloc_heap_dump (struct malloc_heap *heap,FILE *f)lib/librte_eal/common/malloc_heap.h +3
-0@@ -46,6 +46,9 @@ malloc_heap_get_stats(struct malloc_heap *heap,void malloc_heap_dump (struct malloc_heap *heap,FILE *f); +int +malloc_socket_to_heap_id (unsigned int socket_id); +int rte_eal_malloc_heap_init (void );lib/librte_eal/common/rte_malloc.c +12
-3@@ -152,11 +152,20 @@ rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +int heap_idx, ret =-1 ; -if (socket >=RTE_MAX_NUMA_NODES || socket <0 ) -return -1 ; +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); -return malloc_heap_get_stats (&mcfg->malloc_heaps[socket], socket_stats); + heap_idx =malloc_socket_to_heap_id (socket); +if (heap_idx <0 ) + goto unlock; + + ret =malloc_heap_get_stats (&mcfg->malloc_heaps[heap_idx], + socket_stats); +unlock: +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; }/*
lib/librte_eal/common/rte_malloc.c +6
-4@@ -168,12 +177,14 @@ rte_malloc_dump_heaps(FILE *f)struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;unsigned int idx; -for (idx =0 ; idx <rte_socket_count (); idx++) { -unsigned int socket =rte_socket_id_by_idx (idx); -fprintf (f,"Heap on socket %i:\n" , socket); -malloc_heap_dump (&mcfg->malloc_heaps[socket], f); +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); + +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +fprintf (f,"Heap id: %u\n" , idx); +malloc_heap_dump (&mcfg->malloc_heaps[idx], f); } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); }/*
lib/librte_eal/common/rte_malloc.c +10
-5@@ -182,14 +193,19 @@ rte_malloc_dump_heaps(FILE *f)void rte_malloc_dump_stats(FILE *f, __rte_unused const char *type) { -unsigned int socket; +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int heap_id;struct rte_malloc_socket_stats sock_stats; + +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +/* Iterate through all initialised heaps */ -for (socket=0 ; socket<RTE_MAX_NUMA_NODES ; socket++) { -if ((rte_malloc_get_socket_stats (socket, &sock_stats) <0 )) -continue ; +for (heap_id =0 ; heap_id <RTE_MAX_HEAPS ; heap_id++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[heap_id]; -fprintf (f,"Socket:%u\n" , socket); +malloc_heap_get_stats (heap, &sock_stats); + +fprintf (f,"Heap id:%u\n" , heap_id);fprintf (f,"\tHeap_size:%zu,\n" , sock_stats.heap_totalsz_bytes);fprintf(f, "\tFree_size:%zu,\n", sock_stats.heap_freesz_bytes); fprintf(f, "\tAlloc_size:%zu,\n",sock_stats.heap_allocsz_bytes);
lib/librte_eal/common/rte_malloc.c +1
-1@@ -198,6 +214,7 @@ rte_malloc_dump_stats(FILE *f, __rte_unused const char *type)fprintf (f,"\tAlloc_count:%u,\n" ,sock_stats.alloc_count);fprintf (f,"\tFree_count:%u,\n" , sock_stats.free_count); } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock);return ; } --2.17.1
Re: [PATCH v6 03/21] malloc: index heaps using heap ID rather than NUMA node
Quoted 114 lines · Alejandro Lucero
Quoted 111 lines · Anatoly Burakov
Switch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation.
Signed-off-by Anatoly Burakov <[email protected]>--- config/common_base | 1 + config/rte_config.h | 1 + .../common/include/rte_eal_memconfig.h | 4 +- .../common/include/rte_malloc_heap.h | 1 + lib/librte_eal/common/malloc_heap.c | 98 +++++++++++++------ lib/librte_eal/common/malloc_heap.h | 3 + lib/librte_eal/common/rte_malloc.c | 41 +++++--- 7 files changed, 106 insertions(+), 43 deletions(-)config/common_base +1
-0@@ -61,6 +61,7 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_LIBRTE_EAL=y CONFIG_RTE_MAX_LCORE=128 CONFIG_RTE_MAX_NUMA_NODES=8 +CONFIG_RTE_MAX_HEAPS=32 CONFIG_RTE_MAX_MEMSEG_LISTS=64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever issmaller
config/rte_config.h +1
-0@@ -24,6 +24,7 @@#define RTE_BUILD_SHARED_LIB /* EAL defines */ +#define RTE_MAX_HEAPS 32#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768lib/librte_eal/common/include/rte_eal_memconfig.h +0
-0@@ -72,8 +72,8 @@ struct rte_mem_config {struct rte_tailq_head tailq_head[RTE_MAX_TAILQ ];/**< Tailqs for objects */
lib/librte_eal/common/include/rte_eal_memconfig.h +2
-2-/* Heaps of Malloc per socket */ -struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES ]; +/* Heaps of Malloc */ +struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ];/* address of mem_config in primary process. used to map shared config into
* exact same address the primary process maps it.lib/librte_eal/common/include/rte_malloc_heap.h +1
-0@@ -27,6 +27,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size; + unsigned int socket_id; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */ lib/librte_eal/common/malloc_heap.c +15
-0@@ -66,6 +66,21 @@ check_hugepage_sz(unsigned flags, uint64_t hugepage_sz)return check_flag & flags; } +int +malloc_socket_to_heap_id (unsigned int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +int i; + +for (i =0 ; i <RTE_MAX_HEAPS ; i++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[i]; + +if (heap->socket_id == socket_id) +return i; + } +return -1 ; +} +/* * Expand the heap with a memory area. */lib/librte_eal/common/malloc_heap.c +1
-1@@ -93,12 +108,13 @@ malloc_add_seg(const struct rte_memseg_list *msl,struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct rte_memseg_list *found_msl;struct malloc_heap *heap; -int msl_idx; +int msl_idx, heap_idx;if (msl->external)return 0 ;
Quoted 7 lines
lib/librte_eal/common/malloc_heap.c malloc_add_seg(const struct rte_memseg_list *msl, · line 118
Quoted 3 lines
lib/librte_eal/common/malloc_heap.c +1
-1- heap = &mcfg->malloc_heaps[msl->socket_id]; + heap_idx =malloc_socket_to_heap_id (msl->socket_id);
malloc_socket_to_heap_idcan return -1 so it requires to handle that possibility.
Not really, this is called from memseg walk function - we know the msl and its socket ID are valid. Or at least something has gone *very* wrong if we got a -1 result :) However, i guess this check won't hurt.
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v6 03/21] malloc: index heaps using heap ID rather than NUMA node
Quoted 127 lines · Anatoly
Quoted 1 line · Alejandro Lucero
On Thu, Sep 27, 2018 at 11:47 AM Anatoly Burakov <
Quoted 18 lines
Quoted 16 lines · wrote:
Switch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation.
Signed-off-by Anatoly Burakov <[email protected]>--- config/common_base | 1 + config/rte_config.h | 1 + .../common/include/rte_eal_memconfig.h | 4 +- .../common/include/rte_malloc_heap.h | 1 + lib/librte_eal/common/malloc_heap.c | 98 +++++++++++++------ lib/librte_eal/common/malloc_heap.h | 3 + lib/librte_eal/common/rte_malloc.c | 41 +++++--- 7 files changed, 106 insertions(+), 43 deletions(-)Quoted 11 lines
config/common_base CONFIG_RTE_CACHE_LINE_SIZE=64 · line 66
Quoted 11 lines
config/common_base +1
-0@@ -61,6 +61,7 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_LIBRTE_EAL=y CONFIG_RTE_MAX_LCORE=128 CONFIG_RTE_MAX_NUMA_NODES=8 +CONFIG_RTE_MAX_HEAPS=32 CONFIG_RTE_MAX_MEMSEG_LISTS=64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LISTpages
Quoted 21 lines
Quoted 21 lines
config/common_base +0
-0# or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever issmaller
config/rte_config.h +1
-0@@ -24,6 +24,7 @@#define RTE_BUILD_SHARED_LIB /* EAL defines */ +#define RTE_MAX_HEAPS 32#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768lib/librte_eal/common/include/rte_eal_memconfig.h +0
-0@@ -72,8 +72,8 @@ struct rte_mem_config {Quoted 1 line
lib/librte_eal/common/include/rte_eal_memconfig.h struct rte_mem_config { · line 73
Quoted 1 line
lib/librte_eal/common/include/rte_eal_memconfig.h +0
-0struct rte_tailq_head tailq_head[RTE_MAX_TAILQ ];/**< Tailqs for
Quoted 22 lines
Quoted 22 lines
objects */
lib/librte_eal/common/include/rte_eal_memconfig.h +2
-2-/* Heaps of Malloc per socket */ -struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES ]; +/* Heaps of Malloc */ +struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ];/* address of mem_config in primary process. used to map shared config into
* exact same address the primary process maps it.lib/librte_eal/common/include/rte_malloc_heap.h +1
-0@@ -27,6 +27,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size; +unsigned int socket_id; } __rte_cache_aligned;Quoted 7 lines
lib/librte_eal/common/malloc_heap.c check_hugepage_sz(unsigned flags, uint64_t
Quoted 7 lines
lib/librte_eal/common/include/rte_malloc_heap.h +0
-0#endif /* _RTE_MALLOC_HEAP_H_ */ lib/librte_eal/common/malloc_heap.c +0
-0@@ -66,6 +66,21 @@ check_hugepage_sz(unsigned flags, uint64_t
hugepage_sz)Quoted 33 lines
Quoted 33 lines
lib/librte_eal/common/malloc_heap.c +15
-0return check_flag & flags; } +int +malloc_socket_to_heap_id (unsigned int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +int i; + +for (i =0 ; i <RTE_MAX_HEAPS ; i++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[i]; + +if (heap->socket_id == socket_id) +return i; + } +return -1 ; +} +/* * Expand the heap with a memory area. */lib/librte_eal/common/malloc_heap.c +1
-1@@ -93,12 +108,13 @@ malloc_add_seg(const struct rte_memseg_list *msl,struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct rte_memseg_list *found_msl;struct malloc_heap *heap; -int msl_idx; +int msl_idx, heap_idx;if (msl->external)return 0 ;Quoted 7 lines
lib/librte_eal/common/malloc_heap.c malloc_add_seg(const struct rte_memseg_list *msl, · line 118
Quoted 3 lines
lib/librte_eal/common/malloc_heap.c +1
-1- heap = &mcfg->malloc_heaps[msl->socket_id]; + heap_idx =malloc_socket_to_heap_id (msl->socket_id);
malloc_socket_to_heap_idcan return -1 so it requires to handle that possibility.
Quoted 5 lines
Not really, this is called from memseg walk function - we know the msl and its socket ID are valid. Or at least something has gone *very* wrong if we got a -1 result :) However, i guess this check won't hurt.
Although that error is impossible now, not doing the check could be a problem if there is another code path in the future where socket_id has not checked yet.
Quoted 4 lines
--
Thanks,
Anatoly
PATCH v6 4/21 mem: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
This changes the semantics of what we understand by "socket ID", so document the change in the release notes.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 7 +++++++ lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- lib/librte_eal/common/malloc_heap.c | 2 +- lib/librte_eal/common/rte_malloc.c | 4 ---- 4 files changed, 13 insertions(+), 8 deletions(-)
doc/guides/rel_notes/release_18_11.rst +7 -0
@@ -98,6 +98,13 @@ API Changes users of memseg-walk-related functions, as they will now have to skip externally allocated segments in most cases if the intent is to only iterate over internal DPDK memory. + - ``socket_id`` parameter across the entire DPDK has gained additional + meaning, as some socket ID's will now be representing externally allocated + memory. No changes will be required for existing code as backwards + compatibility will be kept, and those who do not use this feature will not + see these extra socket ID's. Any new API's must not check socket ID + parameters themselves, and must instead leave it to the memory subsystem to + decide whether socket ID is a valid one. ABI Changes -----------
lib/librte_eal/common/eal_common_memzone.c +5 -3
@@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,return NULL ; } -if ((socket_id !=SOCKET_ID_ANY ) && - (socket_id >=RTE_MAX_NUMA_NODES || socket_id <0 )) { +if ((socket_id !=SOCKET_ID_ANY ) && socket_id <0 ) { rte_errno =EINVAL ;return NULL ; } -if (!rte_eal_has_hugepages ()) +/* only set socket to SOCKET_ID_ANY if we aren't allocating for an +* external heap. +*/ +if (!rte_eal_has_hugepages () && socket_id <RTE_MAX_NUMA_NODES ) socket_id =SOCKET_ID_ANY ; contig = (flags &RTE_MEMZONE_IOVA_CONTIG ) !=0 ;
lib/librte_eal/common/malloc_heap.c +1 -1
@@ -647,7 +647,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,if (size ==0 || (align && !rte_is_power_of_2 (align)))return NULL ; -if (!rte_eal_has_hugepages ()) +if (!rte_eal_has_hugepages () && socket_arg <RTE_MAX_NUMA_NODES ) socket_arg =SOCKET_ID_ANY ;if (socket_arg ==SOCKET_ID_ANY )
lib/librte_eal/common/rte_malloc.c +0 -4
@@ -47,10 +47,6 @@ rte_malloc_socket(const char *type, size_t size, unsigned int align,if (!rte_eal_has_hugepages ()) socket_arg =SOCKET_ID_ANY ; -/* Check socket parameter */ -if (socket_arg >=RTE_MAX_NUMA_NODES ) -return NULL ; -return malloc_heap_alloc (type, size, socket_arg,0 , align ==0 ?1 : align,0 , false); }
Signature · 1 line
2.17.1
Re: [PATCH v6 04/21] mem: do not check for invalid socket ID
Quoted 45 lines · Anatoly Burakov
We will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
This changes the semantics of what we understand by "socket ID", so document the change in the release notes.
Signed-off-by Anatoly Burakov <[email protected]>--- doc/guides/rel_notes/release_18_11.rst | 7 +++++++ lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- lib/librte_eal/common/malloc_heap.c | 2 +- lib/librte_eal/common/rte_malloc.c | 4 ---- 4 files changed, 13 insertions(+), 8 deletions(-)doc/guides/rel_notes/release_18_11.rst +0
-0@@ -98,6 +98,13 @@ API Changes users of memseg-walk-related functions, as they will now have to skip externally allocated segments in most cases if the intent is to onlyiterate
over internal DPDK memory.doc/guides/rel_notes/release_18_11.rst +2
-0+ - ``socket_id`` parameter across the entire DPDK has gained additional + meaning, as some socket ID's will now be representing externallyallocated
doc/guides/rel_notes/release_18_11.rst +2
-0+ memory. No changes will be required for existing code as backwards + compatibility will be kept, and those who do not use this featurewill not
doc/guides/rel_notes/release_18_11.rst +2
-0+ see these extra socket ID's. Any new API's must not check socket ID + parameters themselves, and must instead leave it to the memorysubsystem to
doc/guides/rel_notes/release_18_11.rst +1
-0+ decide whether socket ID is a valid one. ABI Changes -----------lib/librte_eal/common/eal_common_memzone.c +0
-0
lib/librte_eal/common/eal_common_memzone.c memzone_reserve_aligned_thread_unsafe(const char · line 125
Quoted 8 lines
lib/librte_eal/common/eal_common_memzone.c +1
-2@@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len, return NULL; } -if ((socket_id !=SOCKET_ID_ANY ) && - (socket_id >=RTE_MAX_NUMA_NODES || socket_id <0 )) { +if ((socket_id !=SOCKET_ID_ANY ) && socket_id <0 ) {
Should not it be better to use RTE_MAX_HEAP instead of removing the check?
Quoted 33 lines
lib/librte_eal/common/eal_common_memzone.c +4
-1rte_errno =EINVAL ;return NULL ; } -if (!rte_eal_has_hugepages ()) +/* only set socket to SOCKET_ID_ANY if we aren't allocating for an +* external heap. +*/ +if (!rte_eal_has_hugepages () && socket_id <RTE_MAX_NUMA_NODES ) socket_id =SOCKET_ID_ANY ;contig = (flags & RTE_MEMZONE_IOVA_CONTIG) != 0;lib/librte_eal/common/malloc_heap.c +1
-1@@ -647,7 +647,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg, if (size == 0 || (align && !rte_is_power_of_2(align)))return NULL ; -if (!rte_eal_has_hugepages ()) +if (!rte_eal_has_hugepages () && socket_arg <RTE_MAX_NUMA_NODES ) socket_arg =SOCKET_ID_ANY ;if (socket_arg == SOCKET_ID_ANY)lib/librte_eal/common/rte_malloc.c +0
-0
lib/librte_eal/common/rte_malloc.c rte_malloc_socket(const char *type, size_t size, · line 51
Quoted 9 lines
lib/librte_eal/common/rte_malloc.c +0
-4@@ -47,10 +47,6 @@ rte_malloc_socket(const char *type, size_t size,unsigned int align, if (!rte_eal_has_hugepages()) socket_arg =SOCKET_ID_ANY ; -/* Check socket parameter */ -if (socket_arg >=RTE_MAX_NUMA_NODES ) -return NULL ; -
Sane than before. Better to keep the sanity check using RTE_MAX_HEAPS.
Quoted 6 lines
lib/librte_eal/common/rte_malloc.c +0
-0return malloc_heap_alloc(type, size, socket_arg, 0,align == 0 ? 1 : align, 0, false);}
--
2.17.1
Re: [PATCH v6 04/21] mem: do not check for invalid socket ID
Quoted 52 lines · Alejandro Lucero
Quoted 49 lines · Anatoly Burakov
We will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
This changes the semantics of what we understand by "socket ID", so document the change in the release notes.
Signed-off-by Anatoly Burakov <[email protected]>--- doc/guides/rel_notes/release_18_11.rst | 7 +++++++ lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- lib/librte_eal/common/malloc_heap.c | 2 +- lib/librte_eal/common/rte_malloc.c | 4 ---- 4 files changed, 13 insertions(+), 8 deletions(-)doc/guides/rel_notes/release_18_11.rst +0
-0@@ -98,6 +98,13 @@ API Changes users of memseg-walk-related functions, as they will now have to skip externally allocated segments in most cases if the intent is to onlyiterate
over internal DPDK memory.doc/guides/rel_notes/release_18_11.rst +2
-0+ - ``socket_id`` parameter across the entire DPDK has gained additional + meaning, as some socket ID's will now be representing externallyallocated
doc/guides/rel_notes/release_18_11.rst +2
-0+ memory. No changes will be required for existing code as backwards + compatibility will be kept, and those who do not use this featurewill not
doc/guides/rel_notes/release_18_11.rst +2
-0+ see these extra socket ID's. Any new API's must not check socket ID + parameters themselves, and must instead leave it to the memorysubsystem to
doc/guides/rel_notes/release_18_11.rst +1
-0+ decide whether socket ID is a valid one. ABI Changes -----------lib/librte_eal/common/eal_common_memzone.c +0
-0@@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len, return NULL; }
Quoted 6 lines
lib/librte_eal/common/eal_common_memzone.c memzone_reserve_aligned_thread_unsafe(const char *name, size_t len, · line 127
Quoted 4 lines
lib/librte_eal/common/eal_common_memzone.c +1
-2-if ((socket_id !=SOCKET_ID_ANY ) && - (socket_id >=RTE_MAX_NUMA_NODES || socket_id <0 )) { +if ((socket_id !=SOCKET_ID_ANY ) && socket_id <0 ) {Should not it be better to use
RTE_MAX_HEAPinstead of removing the check?
First of all, maximum number of heaps should not concern the rest of the code - this is purely internal detail of rte_malloc.
More importantly, socket ID is completely independent from number of heaps. Socket ID is incremented each time a new heap is created, and they are not reused. If you create and destroy a heap 100 times - you'll get 100 different socket ID's, even though max number of heaps is less than that.
Quoted 40 lines
Quoted 40 lines
lib/librte_eal/common/eal_common_memzone.c +4
-1rte_errno =EINVAL ;return NULL ; } -if (!rte_eal_has_hugepages ()) +/* only set socket to SOCKET_ID_ANY if we aren't allocating for an +* external heap. +*/ +if (!rte_eal_has_hugepages () && socket_id <RTE_MAX_NUMA_NODES ) socket_id =SOCKET_ID_ANY ;contig = (flags & RTE_MEMZONE_IOVA_CONTIG) != 0;lib/librte_eal/common/malloc_heap.c +1
-1@@ -647,7 +647,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg, if (size == 0 || (align && !rte_is_power_of_2(align)))return NULL ; -if (!rte_eal_has_hugepages ()) +if (!rte_eal_has_hugepages () && socket_arg <RTE_MAX_NUMA_NODES ) socket_arg =SOCKET_ID_ANY ;if (socket_arg == SOCKET_ID_ANY)lib/librte_eal/common/rte_malloc.c +0
-0@@ -47,10 +47,6 @@ rte_malloc_socket(const char *type, size_t size,unsigned int align, if (!rte_eal_has_hugepages()) socket_arg =SOCKET_ID_ANY ;
Quoted 7 lines
lib/librte_eal/common/rte_malloc.c rte_malloc_socket(const char *type, size_t size, unsigned int align,
Quoted 5 lines
lib/librte_eal/common/rte_malloc.c +0
-4-/* Check socket parameter */ -if (socket_arg >=RTE_MAX_NUMA_NODES ) -return NULL ; -Sane than before. Better to keep the sanity check using
RTE_MAX_HEAPS.
same as above :)
Signature · 2 lines
Thanks, Anatoly
Re: [PATCH v6 04/21] mem: do not check for invalid socket ID
Quoted 63 lines · Anatoly
Quoted 1 line · Alejandro Lucero
On Thu, Sep 27, 2018 at 11:41 AM Anatoly Burakov <
Quoted 25 lines
Quoted 23 lines · wrote:
We will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
This changes the semantics of what we understand by "socket ID", so document the change in the release notes.
Signed-off-by Anatoly Burakov <[email protected]>--- doc/guides/rel_notes/release_18_11.rst | 7 +++++++ lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- lib/librte_eal/common/malloc_heap.c | 2 +- lib/librte_eal/common/rte_malloc.c | 4 ---- 4 files changed, 13 insertions(+), 8 deletions(-)doc/guides/rel_notes/release_18_11.rst +0
-0@@ -98,6 +98,13 @@ API Changes users of memseg-walk-related functions, as they will now have toskip
Quoted 1 line
Quoted 1 line
externally allocated segments in most cases if the intent is to
only
Quoted 3 lines
Quoted 3 lines
iterate
over internal DPDK memory.
+ - ``socket_id`` parameter across the entire DPDK has gainedadditional
Quoted 22 lines
Quoted 22 lines
+ meaning, as some socket ID's will now be representing externally
allocateddoc/guides/rel_notes/release_18_11.rst +2
-0+ memory. No changes will be required for existing code as backwards + compatibility will be kept, and those who do not use this featurewill not
doc/guides/rel_notes/release_18_11.rst +2
-0+ see these extra socket ID's. Any new API's must not check socket ID + parameters themselves, and must instead leave it to the memorysubsystem to
doc/guides/rel_notes/release_18_11.rst +1
-0+ decide whether socket ID is a valid one. ABI Changes -----------lib/librte_eal/common/eal_common_memzone.c +0
-0@@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len, return NULL; }Quoted 6 lines
lib/librte_eal/common/eal_common_memzone.c memzone_reserve_aligned_thread_unsafe(const char *name, size_t len, · line 127
Quoted 4 lines
lib/librte_eal/common/eal_common_memzone.c +1
-2-if ((socket_id !=SOCKET_ID_ANY ) && - (socket_id >=RTE_MAX_NUMA_NODES || socket_id <0 )) { +if ((socket_id !=SOCKET_ID_ANY ) && socket_id <0 ) {Should not it be better to use
RTE_MAX_HEAPinstead of removing the
Quoted 6 lines
check?
First of all, maximum number of heaps should not concern the rest of the code - this is purely internal detail of
rte_malloc.
In a previous patch you say that:
"Switch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation."
If I understand this right, heaps linked to physical sockets get a heap ID, and then external heaps will get IDs starting from the higher socket/heap ID + 1.
So, assuming RTE_MAX_HEAPS is really the maximum number of allowed heaps (which does not seem so reading your next paragraph), it would be a good sanity check to use RTE_MAX_HEAPS for the socket id.
More importantly, socket ID is completely independent from number of
Quoted 6 lines
heaps. Socket ID is incremented each time a new heap is created, and they are not reused. If you create and destroy a heap 100 times - you'll get 100 different socket ID's, even though max number of heaps is less than that.
I do not understand this. It is true there is no check regarding RTE_MAX_HEAPS when creating new heaps, then nor sure what the limit refers to. And then there is code like dumping heaps info or getting info from the heap based on socket id that will not work.
Quoted 56 lines
Quoted 9 lines
lib/librte_eal/common/eal_common_memzone.c memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,
Quoted 9 lines
lib/librte_eal/common/eal_common_memzone.c +1
-1rte_errno =EINVAL ;return NULL ; } -if (!rte_eal_has_hugepages ()) +/* only set socket to SOCKET_ID_ANY if we aren't allocating for an
Quoted 31 lines
Quoted 31 lines
lib/librte_eal/common/eal_common_memzone.c +3
-0+* external heap. +*/ +if (!rte_eal_has_hugepages () && socket_id <RTE_MAX_NUMA_NODES ) socket_id =SOCKET_ID_ANY ;contig = (flags & RTE_MEMZONE_IOVA_CONTIG) != 0;lib/librte_eal/common/malloc_heap.c +1
-1@@ -647,7 +647,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg, if (size == 0 || (align && !rte_is_power_of_2(align)))return NULL ; -if (!rte_eal_has_hugepages ()) +if (!rte_eal_has_hugepages () && socket_arg <RTE_MAX_NUMA_NODES ) socket_arg =SOCKET_ID_ANY ;if (socket_arg == SOCKET_ID_ANY)lib/librte_eal/common/rte_malloc.c +0
-0@@ -47,10 +47,6 @@ rte_malloc_socket(const char *type, size_t size,unsigned int align, if (!rte_eal_has_hugepages()) socket_arg =SOCKET_ID_ANY ;Quoted 7 lines
lib/librte_eal/common/rte_malloc.c rte_malloc_socket(const char *type, size_t size, unsigned int align,
Quoted 5 lines
lib/librte_eal/common/rte_malloc.c +0
-4-/* Check socket parameter */ -if (socket_arg >=RTE_MAX_NUMA_NODES ) -return NULL ; -Sane than before. Better to keep the sanity check using
RTE_MAX_HEAPS.same as above :)
--
Thanks,
Anatoly
Re: [PATCH v6 04/21] mem: do not check for invalid socket ID
Quoted 80 lines · Alejandro Lucero
On Thu, Sep 27, 2018 at 2:22 PM Burakov, Anatoly <[email protected] <mailto:[email protected]>> wrote:
On 27-Sep-18 2:14 PM, Alejandro Lucero wrote: > On Thu, Sep 27, 2018 at 11:41 AM Anatoly Burakov <[email protected] <mailto:[email protected]>> > wrote: > >> We will be assigning "invalid" socket ID's to external heap, and >> malloc will now be able to verify if a supplied socket ID is in >> fact a valid one, rendering parameter checks for sockets >> obsolete. >> >> This changes the semantics of what we understand by "socket ID", >> so document the change in the release notes. >> >> Signed-off-by: Anatoly Burakov <[email protected] <mailto:[email protected]>> >> --- >> doc/guides/rel_notes/release_18_11.rst | 7 +++++++ >> lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- >> lib/librte_eal/common/malloc_heap.c | 2 +- >> lib/librte_eal/common/rte_malloc.c | 4 ---- >> 4 files changed, 13 insertions(+), 8 deletions(-) >> >> diff --git a/doc/guides/rel_notes/release_18_11.rst >> b/doc/guides/rel_notes/release_18_11.rst >> index 5fc71e208..6ee236302 100644 >> --- a/doc/guides/rel_notes/release_18_11.rst >> +++ b/doc/guides/rel_notes/release_18_11.rst >> @@ -98,6 +98,13 @@ API Changes >> users of memseg-walk-related functions, as they will now have to skip >> externally allocated segments in most cases if the intent is to only >> iterate >> over internal DPDK memory. >> + - ``socket_id`` parameter across the entire DPDK has gained additional >> + meaning, as some socket ID's will now be representing externally >> allocated >> + memory. No changes will be required for existing code as backwards >> + compatibility will be kept, and those who do not use this feature >> will not >> + see these extra socket ID's. Any new API's must not check socket ID >> + parameters themselves, and must instead leave it to the memory >> subsystem to >> + decide whether socket ID is a valid one. >> >> ABI Changes >> ----------- >> diff --git a/lib/librte_eal/common/eal_common_memzone.c >> b/lib/librte_eal/common/eal_common_memzone.c >> index 7300fe05d..b7081afbf 100644 >> --- a/lib/librte_eal/common/eal_common_memzone.c >> +++ b/lib/librte_eal/common/eal_common_memzone.c >> @@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char >> *name, size_t len, >> return NULL; >> } >> >> - if ((socket_id != SOCKET_ID_ANY) && >> - (socket_id >= RTE_MAX_NUMA_NODES || socket_id < 0)) { >> + if ((socket_id != SOCKET_ID_ANY) && socket_id < 0) { >> > > Should not it be better to use RTE_MAX_HEAP instead of removing the check? First of all, maximum number of heaps should not concern the rest of the code - this is purely internal detail of rte_malloc.
Quoted 8 lines
In a previous patch you say that:
"Switch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation."
If I understand this right, heaps linked to physical sockets get a heap ID, and then external heaps will get IDs starting from the higher socket/heap ID + 1.
Yes and no.
Socket ID is an externally visible identification of "where to allocate from" (a heap). Heap ID is used internally. Normally, there is a 1:1 correspondence of NUMA node to heap ID, but there may be cases where e.g. only NUMA nodes 0 and 7 are detected, so you'll have socket 0 and 7 as valid socket ID's. However, these socket ID's will be internally resolved into heap ID's 0 and 1, not 0 and 7.
So, in *most* cases, socket ID for an internal heap is equivalent to its heap ID, but it is by accident. Heap ID is an internal identifier used by the malloc heap, and it is not visible externally - it is only known to malloc itself. Even memzone knows nothing about heap ID's - only socket ID's.
Quoted 11 lines
So, assuming
RTE_MAX_HEAPSis really the maximum number of allowed heaps (which does not seem so reading your next paragraph), it would be a good sanity check to useRTE_MAX_HEAPSfor the socket id.More importantly, socket ID is completely independent from number of heaps. Socket ID is incremented each time a new heap is created, and they are not reused. If you create and destroy a heap 100 times - you'll
get 100 different socket ID's, even though max number of heaps is less than that.I do not understand this. It is true there is no check regarding
RTE_MAX_HEAPSwhen creating new heaps,
There is one :) RTE_MAX_HEAPS is length of malloc heaps array (shared in memory). If we cannot find a vacant spot in heaps array, the heap will not be created.
However, *socket ID* is indeed limited only to INT_MAX. Socket ID is not heap ID - socket ID is an externally visible identifier. Multiple socket ID's can resolve to the same heap ID.
For example, if you create and destroy a heap 5 times one after the other, you'll get 5 different socket ID's, but all of them would have pointed to the same heap ID (but not at the same time).
So, semantically speaking, heap ID isn't really "an ID" as such, it's an index into heap array. Unlike socket ID, it has no meaning.
Quoted 1 line
then nor sure what the limit refers to. And then there is code like dumping heaps info or getting info from the heap based on socket id that will not work.
It is probably unclear because the ordering of this patchset is not ideal (and i'm not sure how to make it any better).
The code for dumping or getting heap info's accepts socket ID, but it translates it into heap ID, because that's what malloc uses internally to differentiate between the heaps. Heap ID is there to break dependency between NUMA node ID and position in the malloc heap array.
Signature · 2 lines
Thanks, Anatoly
PATCH v6 5/21 flow_classify: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_flow_classify/rte_flow_classify.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/librte_flow_classify/rte_flow_classify.c +1 -2
@@ -247,8 +247,7 @@ rte_flow_classifier_check_params(struct rte_flow_classifier_params *params) }/* socket */ -if ((params->socket_id <0 ) || - (params->socket_id >=RTE_MAX_NUMA_NODES )) { +if (params->socket_id <0 ) {RTE_FLOW_CLASSIFY_LOG (ERR ,"%s: Incorrect value for parameter socket_id\n" , __func__);
Signature · 1 line
2.17.1
Re: [PATCH v6 05/21] flow_classify: do not check for invalid socket ID
Quoted 15 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Thursday, September 27, 2018 11:41 AM
To: [email protected]
Cc: Iremonger, Bernard <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Wiles, Keith <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v6 05/21]flow_classify: do not check for invalid socket ID
Quoted 5 lines
We will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
PATCH v6 6/21 pipeline: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_pipeline/rte_pipeline.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/librte_pipeline/rte_pipeline.c +1 -2
@@ -178,8 +178,7 @@ rte_pipeline_check_params(struct rte_pipeline_params *params) }/* socket */ -if ((params->socket_id <0 ) || - (params->socket_id >=RTE_MAX_NUMA_NODES )) { +if (params->socket_id <0 ) {RTE_LOG (ERR ,PIPELINE ,"%s: Incorrect value for parameter socket_id\n" , __func__);
Signature · 1 line
2.17.1
PATCH v6 7/21 sched: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_sched/rte_sched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_sched/rte_sched.c +1 -1
@@ -329,7 +329,7 @@ rte_sched_port_check_params(struct rte_sched_port_params *params)return -1 ;/* socket */ -if ((params->socket <0 ) || (params->socket >=RTE_MAX_NUMA_NODES )) +if (params->socket <0 )return -3 ;/* rate */
Signature · 1 line
2.17.1
PATCH v6 8/21 malloc: add name to malloc heaps
In reply to Anatoly BurakovWe will need to refer to external heaps in some way. While we use heap ID's internally, for external API use it has to be something more user-friendly. So, we will be using a string to uniquely identify a heap.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 1 + lib/librte_eal/common/include/rte_malloc_heap.h | 2 ++ lib/librte_eal/common/malloc_heap.c | 17 ++++++++++++++++- lib/librte_eal/common/rte_malloc.c | 1 + 4 files changed, 20 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_18_11.rst +1 -0
@@ -124,6 +124,7 @@ ABI Changes * eal: EAL library ABI version was changed due to previously announced work on supporting external memory in DPDK. Structure ``rte_memseg_list`` now has a new flag indicating whether the memseg list refers to external memory. + Structure ``rte_malloc_heap`` now has a ``heap_name`` string member. Removed Items -------------
lib/librte_eal/common/include/rte_malloc_heap.h +2 -0
@@ -12,6 +12,7 @@/* Number of free lists per heap, grouped by size. */ #define RTE_HEAP_NUM_FREELISTS 13 +#define RTE_HEAP_NAME_MAX_LEN 32/* dummy definition, for pointers */ struct malloc_elem ;
@@ -28,6 +29,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size;unsigned int socket_id; + char name[RTE_HEAP_NAME_MAX_LEN]; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */
lib/librte_eal/common/malloc_heap.c +16 -1
@@ -127,7 +127,6 @@ malloc_add_seg(const struct rte_memseg_list *msl,malloc_heap_add_memory (heap, found_msl, ms->addr, len); heap->total_size += len; - heap->socket_id = msl->socket_id;RTE_LOG (DEBUG ,EAL ,"Added %zuM to heap on socket %i\n" , len >>20 , msl->socket_id);
@@ -1020,6 +1019,22 @@ intrte_eal_malloc_heap_init (void) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int i; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign names to default DPDK heaps */ +for (i =0 ; i <rte_socket_count (); i++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[i]; +char heap_name[RTE_HEAP_NAME_MAX_LEN ]; +int socket_id =rte_socket_id_by_idx (i); + +snprintf (heap_name,sizeof (heap_name) -1 , +"socket_%i" , socket_id); +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); + heap->socket_id = socket_id; + } + } +if (register_mp_requests ()) {RTE_LOG (ERR ,EAL ,"Couldn't register malloc multiprocess actions\n" );
lib/librte_eal/common/rte_malloc.c +1 -0
@@ -202,6 +202,7 @@ rte_malloc_dump_stats(FILE *f, __rte_unused const char *type)malloc_heap_get_stats (heap, &sock_stats);fprintf (f,"Heap id:%u\n" , heap_id); +fprintf (f,"\tHeap name:%s\n" , heap->name);fprintf (f,"\tHeap_size:%zu,\n" , sock_stats.heap_totalsz_bytes);fprintf (f,"\tFree_size:%zu,\n" , sock_stats.heap_freesz_bytes);fprintf (f,"\tAlloc_size:%zu,\n" , sock_stats.heap_allocsz_bytes);
Signature · 1 line
2.17.1
PATCH v6 9/21 malloc: add function to query socket ID of named heap
In reply to Anatoly BurakovWhen we will be creating external heaps, they will have their own "fake" socket ID, so add a function that will map the heap name to its socket ID.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 14 ++++++++ lib/librte_eal/common/rte_malloc.c | 37 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +14 -0
@@ -263,6 +263,20 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Find socket ID corresponding to a named heap. +* +* @param name +* Heap name to find socket ID for +* @return +* Socket ID in case of success (a non-negative number) +* -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``name`` was NULL +* ENOENT - heap identified by the name ``name`` was not found +*/ +int __rte_experimental +rte_malloc_heap_get_socket (const char *name); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +37 -0
@@ -8,6 +8,7 @@#include <string.h> #include <sys/queue.h> +#include <rte_errno.h> #include <rte_memcpy.h> #include <rte_memory.h> #include <rte_eal.h>
@@ -183,6 +184,42 @@ rte_malloc_dump_heaps(FILE *f)rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); } +int +rte_malloc_heap_get_socket (const char *name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +unsigned int idx; +int ret; + +if (name ==NULL || +strnlen (name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if (!strncmp (name, tmp->name,RTE_HEAP_NAME_MAX_LEN )) { + heap = tmp; +break ; + } + } + +if (heap !=NULL ) { + ret = heap->socket_id; + }else { + rte_errno =ENOENT ; + ret =-1 ; + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_get_socket; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
Signature · 1 line
2.17.1
PATCH v6 10/21 malloc: add function to check if socket is external
In reply to Anatoly BurakovAn API is needed to check whether a particular socket ID belongs to an internal or external heap. Prime user of this would be mempool allocator, because normal assumptions of IOVA contiguousness in IOVA as VA mode do not hold in case of externally allocated memory.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 15 +++++++++++++ lib/librte_eal/common/rte_malloc.c | 25 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + lib/librte_mempool/rte_mempool.c | 22 ++++++++++++++++--- 4 files changed, 60 insertions(+), 3 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +15 -0
@@ -277,6 +277,21 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_get_socket (const char *name); +/** +* Check if a given socket ID refers to externally allocated memory. +* +* @note Passing SOCKET_ID_ANY will return 0. +* +* @param socket_id +* Socket ID to check +* @return +* 1 if socket ID refers to externally allocated memory +* 0 if socket ID refers to internal DPDK memory +* -1 if socket ID is invalid +*/ +int __rte_experimental +rte_malloc_heap_socket_is_external (int socket_id); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +25 -0
@@ -220,6 +220,31 @@ rte_malloc_heap_get_socket(const char *name)return ret; } +int +rte_malloc_heap_socket_is_external (int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int idx; +int ret =-1 ; + +if (socket_id ==SOCKET_ID_ANY ) +return 0 ; + +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if ((int )tmp->socket_id == socket_id) { +/* external memory always has large socket ID's */ + ret = tmp->socket_id >=RTE_MAX_NUMA_NODES ; +break ; + } + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_get_socket; + rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
lib/librte_mempool/rte_mempool.c +19 -3
@@ -428,12 +428,18 @@ rte_mempool_populate_default(struct rte_mempool *mp)rte_iova_t iova;unsigned mz_id, n;int ret; -bool no_contig, try_contig, no_pageshift; +bool no_contig, try_contig, no_pageshift, external; ret =mempool_ops_alloc_once (mp);if (ret !=0 )return ret; +/* check if we can retrieve a valid socket ID */ + ret =rte_malloc_heap_socket_is_external (mp->socket_id); +if (ret <0 ) +return -EINVAL ; + external = ret; +/* mempool must not be populated */ if (mp->nb_mem_chunks !=0 )return -EEXIST ;
@@ -481,9 +487,19 @@ rte_mempool_populate_default(struct rte_mempool *mp)* in one contiguous chunk as well (otherwise we might end up wasting a * 1G page on a 10MB memzone). If we fail to get enough contiguous * memory, then we'll go and reserve space page-by-page. +* +* We also have to take into account the fact that memory that we're +* going to allocate from can belong to an externally allocated memory +* area, in which case the assumption of IOVA as VA mode being +* synonymous with IOVA contiguousness will not hold. We should also try +* to go for contiguous memory even if we're in no-huge mode, because +* external memory may in fact be IOVA-contiguous. */ - no_pageshift = no_contig ||rte_eal_iova_mode () ==RTE_IOVA_VA ; - try_contig = !no_contig && !no_pageshift &&rte_eal_has_hugepages (); + external =rte_malloc_heap_socket_is_external (mp->socket_id) ==1 ; + no_pageshift = no_contig || + (!external &&rte_eal_iova_mode () ==RTE_IOVA_VA ); + try_contig = !no_contig && !no_pageshift && + (rte_eal_has_hugepages () || external);if (no_pageshift) { pg_sz =0 ;
Signature · 1 line
2.17.1
PATCH v6 11/21 malloc: allow creating malloc heaps
In reply to Anatoly BurakovAdd API to allow creating new malloc heaps. They will be created with socket ID's going above RTE_MAX_NUMA_NODES, to avoid clashing with internal heaps.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 2 + .../common/include/rte_eal_memconfig.h | 3 ++ lib/librte_eal/common/include/rte_malloc.h | 19 +++++++ lib/librte_eal/common/malloc_heap.c | 37 +++++++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 52 +++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 7 files changed, 117 insertions(+)
doc/guides/rel_notes/release_18_11.rst +2 -0
@@ -125,6 +125,8 @@ ABI Changes supporting external memory in DPDK. Structure ``rte_memseg_list`` now has a new flag indicating whether the memseg list refers to external memory. Structure ``rte_malloc_heap`` now has a ``heap_name`` string member. + Structure ``rte_eal_memconfig`` has been extended to contain next socket + ID for externally allocated memory segments. Removed Items -------------
lib/librte_eal/common/include/rte_eal_memconfig.h +3 -0
@@ -75,6 +75,9 @@ struct rte_mem_config {/* Heaps of Malloc */ struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ]; +/* next socket ID for external malloc heap */ +int next_socket_id; +/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it. */
lib/librte_eal/common/include/rte_malloc.h +19 -0
@@ -263,6 +263,25 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Creates a new empty malloc heap with a specified name. +* +* @note Heaps created via this call will automatically get assigned a unique +* socket ID, which can be found using ``rte_malloc_heap_get_socket()`` +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on successful creation +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* EEXIST - heap by name of ``heap_name`` already exists +* ENOSPC - no more space in internal config to store a new heap +*/ +int __rte_experimental +rte_malloc_heap_create (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +37 -0
@@ -29,6 +29,10 @@#include "malloc_heap.h" #include "malloc_mp.h" +/* start external socket ID's at a very high number */ +#define CONST_MAX (a, b) (a > b ? a : b)/* RTE_MAX is not a constant */ +#define EXTERNAL_HEAP_MIN_SOCKET_ID (CONST_MAX((1 << 8), RTE_MAX_NUMA_NODES)) +static unsigned check_hugepage_sz (unsigned flags,uint64_t hugepage_sz) {
@@ -1015,6 +1019,36 @@ malloc_heap_dump(struct malloc_heap *heap, FILE *f)rte_spinlock_unlock (&heap->lock); } +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +uint32_t next_socket_id = mcfg->next_socket_id; + +/* prevent overflow. did you really create 2 billion heaps??? */ +if (next_socket_id >INT32_MAX ) { +RTE_LOG (ERR ,EAL ,"Cannot assign new socket ID's\n" ); + rte_errno =ENOSPC ; +return -1 ; + } + +/* initialize empty heap */ + heap->alloc_count =0 ; + heap->first =NULL ; + heap->last =NULL ; +LIST_INIT (heap->free_head); +rte_spinlock_init (&heap->lock); + heap->total_size =0 ; + heap->socket_id = next_socket_id; + +/* we hold a global mem hotplug writelock, so it's safe to increment */ + mcfg->next_socket_id++; + +/* set up name */ +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
@@ -1022,6 +1056,9 @@ rte_eal_malloc_heap_init(void)unsigned int i;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign min socket ID to external heaps */ + mcfg->next_socket_id =EXTERNAL_HEAP_MIN_SOCKET_ID ; +/* assign names to default DPDK heaps */ for (i =0 ; i <rte_socket_count (); i++) {struct malloc_heap *heap = &mcfg->malloc_heaps[i];
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -33,6 +33,9 @@ void * malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags, size_t align, bool contig); +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +52 -0
61 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_create; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v6 12/21 malloc: allow destroying heaps
In reply to Anatoly BurakovAdd an API to destroy specified heap.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 23 +++++++++ lib/librte_eal/common/malloc_heap.c | 22 ++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 58 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 107 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +23 -0
@@ -282,6 +282,29 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_create (const char *heap_name); +/** +* Destroys a previously created malloc heap with specified name. +* +* @note This function will return a failure result if not all memory allocated +* from the heap has been freed back to the heap +* +* @note This function will return a failure result if not all memory segments +* were removed from the heap prior to its destruction +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* ENOENT - heap by the name of ``heap_name`` was not found +* EPERM - attempting to destroy reserved heap +* EBUSY - heap still contains data +*/ +int __rte_experimental +rte_malloc_heap_destroy (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +22 -0
@@ -1049,6 +1049,28 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name)return 0 ; } +int +malloc_heap_destroy (struct malloc_heap *heap) +{ +if (heap->alloc_count !=0 ) { +RTE_LOG (ERR ,EAL ,"Heap is still in use\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->first !=NULL || heap->last !=NULL ) { +RTE_LOG (ERR ,EAL ,"Heap still contains memory segments\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->total_size !=0 ) +RTE_LOG (ERR ,EAL ,"Total size not zero, heap is likely corrupt\n" ); + +/* after this, the lock will be dropped */ +memset (heap,0 ,sizeof (*heap)); + +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -36,6 +36,9 @@ malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags,int malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int +malloc_heap_destroy (struct malloc_heap *heap); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +58 -0
67 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_create; + rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v6 13/21 malloc: allow adding memory to named heaps
In reply to Anatoly BurakovAdd an API to add externally allocated memory to malloc heap. The memory will be stored in memseg lists like regular DPDK memory. Multiple segments are allowed within a heap. If IOVA table is not provided, IOVA addresses are filled in with RTE_BAD_IOVA.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 39 ++++++++++++ lib/librte_eal/common/malloc_heap.c | 74 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 51 +++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 169 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +39 -0
@@ -263,6 +263,45 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Add memory chunk to a heap with specified name. +* +* @note Multiple memory chunks can be added to the same heap +* +* @note Memory must be previously allocated for DPDK to be able to use it as a +* malloc heap. Failing to do so will result in undefined behavior, up to and +* including segmentation faults. +* +* @note Calling this function will erase any contents already present at the +* supplied memory address. +* +* @param heap_name +* Name of the heap to add memory chunk to +* @param va_addr +* Start of virtual area to add to the heap +* @param len +* Length of virtual area to add to the heap +* @param iova_addrs +* Array of page IOVA addresses corresponding to each page in this memory +* area. Can be NULL, in which case page IOVA addresses will be set to +* RTE_BAD_IOVA. +* @param n_pages +* Number of elements in the iova_addrs array. Ignored if ``iova_addrs`` +* is NULL. +* @param page_sz +* Page size of the underlying memory +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to add memory to a reserved heap +* ENOSPC - no more space in internal config to store a new memory chunk +*/ +int __rte_experimental +rte_malloc_heap_memory_add (const char *heap_name,void *va_addr,size_t len, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +74 -0
80 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -39,6 +39,10 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name);int malloc_heap_destroy (struct malloc_heap *heap); +int +malloc_heap_add_external_memory (struct malloc_heap *heap,void *va_addr, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +51 -0
57 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -321,6 +321,7 @@ EXPERIMENTAL { rte_malloc_heap_create; rte_malloc_heap_destroy; rte_malloc_heap_get_socket; + rte_malloc_heap_memory_add; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v6 14/21 malloc: allow removing memory from named heaps
In reply to Anatoly BurakovAdd an API to remove memory from specified heaps. This will first check if all elements within the region are free, and that the region is the original region that was added to the heap (by comparing its length to length of memory addressed by the underlying memseg list).
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++ lib/librte_eal/common/malloc_heap.c | 54 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 39 ++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 125 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -302,6 +302,33 @@ int __rte_experimental rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +/** +* Remove memory chunk from heap with specified name. +* +* @note Memory chunk being removed must be the same as one that was added; +* partially removing memory chunks is not supported +* +* @note Memory area must not contain any allocated elements to allow its +* removal from the heap +* +* @param heap_name +* Name of the heap to remove memory from +* @param va_addr +* Virtual address to remove from the heap +* @param len +* Length of virtual area to remove from the heap +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to remove memory from a reserved heap +* ENOENT - heap or memory chunk was not found +* EBUSY - memory chunk still contains data +*/ +int __rte_experimental +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +54 -0
66 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -43,6 +43,10 @@ int malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +int +malloc_heap_remove_external_memory (struct malloc_heap *heap,void *va_addr, +size_t len); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +39 -0
@@ -379,6 +379,45 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,return ret; } +int +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +int ret; + +if (heap_name ==NULL || va_addr ==NULL || len ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_write_lock (&mcfg->memory_hotplug_lock); +/* find our heap */ + heap =find_named_heap (heap_name); +if (heap ==NULL ) { + rte_errno =ENOENT ; + ret =-1 ; + goto unlock; + } +if (heap->socket_id <RTE_MAX_NUMA_NODES ) { +/* cannot remove memory from internal heaps */ + rte_errno =EPERM ; + ret =-1 ; + goto unlock; + } + +rte_spinlock_lock (&heap->lock); + ret =malloc_heap_remove_external_memory (heap, va_addr, len); +rte_spinlock_unlock (&heap->lock); + +unlock: +rte_rwlock_write_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v6 15/21 malloc: allow attaching to external memory chunks
In reply to Anatoly BurakovIn order to use external memory in multiple processes, we need to attach to primary process's memseg lists, so add a new API to do that. It is the responsibility of the user to ensure that memory is accessible and that it has been previously added to the malloc heap by another process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 28 ++++++++ lib/librte_eal/common/rte_malloc.c | 83 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 112 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +28 -0
@@ -268,6 +268,10 @@ rte_malloc_get_socket_stats(int socket,* * @note Multiple memory chunks can be added to the same heap * +* @note Before accessing this memory in other processes, it needs to be +* attached in each of those processes by calling +* ``rte_malloc_heap_memory_attach`` in each other process. +* * @note Memory must be previously allocated for DPDK to be able to use it as a * malloc heap. Failing to do so will result in undefined behavior, up to and * including segmentation faults.
@@ -329,6 +333,30 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, int __rte_experimentalrte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** +* Attach to an already existing chunk of external memory in another process. +* +* @note This function must be called before any attempt is made to use an +* already existing external memory chunk. This function does *not* need to +* be called if a call to ``rte_malloc_heap_memory_add`` was made in the +* current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful attach +* -1 on unsuccessful attach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to attach memory to a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +83 -0
89 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_attach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v6 16/21 malloc: allow detaching from external memory
In reply to Anatoly BurakovAdd API to detach from existing chunk of external memory in a process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++++++++++ lib/librte_eal/common/rte_malloc.c | 31 +++++++++++++++++----- lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+), 7 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -315,6 +315,9 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,* @note Memory area must not contain any allocated elements to allow its * removal from the heap * +* @note All other processes must detach from the memory chunk prior to it being +* removed from the heap. +* * @param heap_name * Name of the heap to remove memory from * @param va_addr
@@ -357,6 +360,30 @@ rte_malloc_heap_memory_remove(const char *heap_name, void *va_addr, size_t len); int __rte_experimentalrte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** +* Detach from a chunk of external memory in secondary process. +* +* @note This function must be called in before any attempt is made to remove +* external memory from the heap in another process. This function does *not* +* need to be called if a call to ``rte_malloc_heap_memory_remove`` will be +* called in current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful detach +* -1 on unsuccessful detach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to detach memory from a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +24 -7
@@ -422,10 +422,11 @@ struct sync_mem_walk_arg {void *va_addr;size_t len;int result; +bool attach; };static int -attach_mem_walk (const struct rte_memseg_list *msl,void *arg) +sync_mem_walk (const struct rte_memseg_list *msl,void *arg) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct sync_mem_walk_arg *wa = arg;
@@ -440,7 +441,10 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; - ret =rte_fbarray_attach (&found_msl->memseg_arr); +if (wa->attach) + ret =rte_fbarray_attach (&found_msl->memseg_arr); +else + ret =rte_fbarray_detach (&found_msl->memseg_arr);if (ret <0 ) wa->result = -rte_errno;
@@ -451,8 +455,8 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg)return 0 ; } -int -rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +static int +sync_memory (const char *heap_name,void *va_addr,size_t len,bool attach) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct malloc_heap *heap =NULL ;
@@ -475,20 +479,21 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len) ret =-1 ; goto unlock; } -/* we shouldn't be able to attach to internal heaps */ +/* we shouldn't be able to sync to internal heaps */ if (heap->socket_id <RTE_MAX_NUMA_NODES ) { rte_errno =EPERM ; ret =-1 ; goto unlock; } -/* find corresponding memseg list to attach to */ +/* find corresponding memseg list to sync to */ wa.va_addr = va_addr; wa.len = len; wa.result = -ENOENT ;/* fail unless explicitly told to succeed */ + wa.attach = attach;/* we're already holding a read lock */ -rte_memseg_list_walk_thread_unsafe (attach_mem_walk, &wa); +rte_memseg_list_walk_thread_unsafe (sync_mem_walk, &wa);if (wa.result <0 ) { rte_errno = -wa.result;
@@ -501,6 +506,18 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len)return ret; } +int +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, true); +} + +int +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, false); +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -323,6 +323,7 @@ EXPERIMENTAL { rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; rte_malloc_heap_memory_attach; + rte_malloc_heap_memory_detach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v6 17/21 malloc: enable event callbacks for external memory
In reply to Anatoly BurakovWhen adding or removing external memory from the memory map, there may be actions that need to be taken on account of this memory (e.g. DMA mapping). Add support for triggering callbacks when adding, removing, attaching or detaching external memory.
Some memory event callback handlers will need additional logic to handle external memory regions. For example, virtio callback has to completely ignore externally allocated memory, because there is no way to find file descriptors backing the memory address in a generic fashion. All other callbacks have also been adjusted to handle RTE_BAD_IOVA as IOVA address, as this is one of the expected use cases for external memory support.
Signed-off-by Anatoly Burakov <[email protected]>
--- drivers/bus/fslmc/fslmc_vfio.c | 7 +++++ .../net/virtio/virtio_user/virtio_user_dev.c | 8 ++++++ lib/librte_eal/common/malloc_heap.c | 7 +++++ lib/librte_eal/common/rte_malloc.c | 27 ++++++++++++++++--- lib/librte_eal/linuxapp/eal/eal_vfio.c | 10 +++++-- 5 files changed, 53 insertions(+), 6 deletions(-)
drivers/bus/fslmc/fslmc_vfio.c +7 -0
@@ -221,6 +221,13 @@ fslmc_memevent_cb(enum rte_mem_event type, const void *addr, size_t len, "alloc" : "dealloc", va, virt_addr, iova_addr, map_len); +/* iova_addr may be set to RTE_BAD_IOVA */ +if (iova_addr ==RTE_BAD_IOVA ) { +DPAA2_BUS_DEBUG ("Segment has invalid iova, skipping\n" ); + cur_len += map_len; +continue ; + } +if (type ==RTE_MEM_EVENT_ALLOC ) ret =fslmc_map_dma (virt_addr, iova_addr, map_len);else
drivers/net/virtio/virtio_user/virtio_user_dev.c +8 -0
@@ -13,6 +13,8 @@#include <sys/types.h> #include <sys/stat.h> +#include <rte_eal_memconfig.h> +#include "vhost.h" #include "virtio_user_dev.h" #include "../virtio_ethdev.h"
@@ -282,8 +284,14 @@ virtio_user_mem_event_cb(enum rte_mem_event type __rte_unused, void *arg) {struct virtio_user_dev *dev = arg; +struct rte_memseg_list *msl;uint16_t i; +/* ignore externally allocated memory */ + msl =rte_mem_virt2memseg_list (addr); +if (msl->external) +return ; +pthread_mutex_lock (&dev->mutex);if (dev->started == false)
lib/librte_eal/common/malloc_heap.c +7 -0
@@ -1027,6 +1027,9 @@ destroy_seg(struct malloc_elem *elem, size_t len) msl = elem->msl; +/* notify all subscribers that a memory area is going to be removed */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , elem, len); +/* this element can be removed */ malloc_elem_free_list_remove (elem);malloc_elem_hide_region (elem, elem, len);
@@ -1116,6 +1119,10 @@ malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr,RTE_LOG (DEBUG ,EAL ,"Added segment for heap %s starting at %p\n" , heap->name, va_addr); +/* notify all subscribers that a new memory area has been added */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, seg_len); +return 0 ; }
lib/librte_eal/common/rte_malloc.c +23 -4
@@ -25,6 +25,7 @@#include <rte_malloc.h> #include "malloc_elem.h" #include "malloc_heap.h" +#include "eal_memalloc.h" /* Free the memory space back to heap */
@@ -441,15 +442,29 @@ sync_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; -if (wa->attach) +if (wa->attach) { ret =rte_fbarray_attach (&found_msl->memseg_arr); -else + }else { +/* notify all subscribers that a memory area is about to +* be removed +*/ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , + msl->base_va, msl->len); ret =rte_fbarray_detach (&found_msl->memseg_arr); + } -if (ret <0 ) +if (ret <0 ) { wa->result = -rte_errno; -else + }else { +/* notify all subscribers that a new memory area was +* added +*/ +if (wa->attach) +eal_memalloc_mem_event_notify ( +RTE_MEM_EVENT_ALLOC , + msl->base_va, msl->len); wa->result =0 ; + }return 1 ; }return 0 ;
@@ -499,6 +514,10 @@ sync_memory(const char *heap_name, void *va_addr, size_t len, bool attach) rte_errno = -wa.result; ret =-1 ; }else { +/* notify all subscribers that a new memory area was added */ +if (attach) +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, len); ret =0 ; } unlock:
lib/librte_eal/linuxapp/eal/eal_vfio.c +8 -2
@@ -509,7 +509,7 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, msl =rte_mem_virt2memseg_list (addr);/* for IOVA as VA mode, no need to care for IOVA addresses */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) { +if (rte_eal_iova_mode () ==RTE_IOVA_VA && msl->external ==0 ) {uint64_t vfio_va = (uint64_t )(uintptr_t )addr;if (type ==RTE_MEM_EVENT_ALLOC ) vfio_dma_mem_map(default_vfio_cfg, vfio_va, vfio_va,
@@ -523,13 +523,19 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,/* memsegs are contiguous in memory */ ms =rte_mem_virt2memseg (addr, msl);while (cur_len < len) { +/* some memory segments may have invalid IOVA */ +if (ms->iova ==RTE_BAD_IOVA ) { +RTE_LOG (DEBUG ,EAL ,"Memory segment at %p has bad IOVA, skipping\n" , + ms->addr); + goto next; + }if (type ==RTE_MEM_EVENT_ALLOC )vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,1 );else vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,0 ); - +next: cur_len += ms->len; ++ms; }
Signature · 1 line
2.17.1
PATCH v6 18/21 test: add unit tests for external memory support
In reply to Anatoly BurakovAdd simple unit tests to test external memory support. The tests are pretty basic and mostly consist of checking if invalid API calls are handled correctly, plus a simple allocation/deallocation test for malloc and memzone.
Signed-off-by Anatoly Burakov <[email protected]>
--- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++++++++++++++++++++ 4 files changed, 401 insertions(+), 4 deletions(-) create mode 100644 test/test/test_external_mem.c
test/test/Makefile +1 -0
@@ -71,6 +71,7 @@ SRCS-y += test_bitmap.c SRCS-y += test_reciprocal_division.c SRCS-y += test_reciprocal_division_perf.c SRCS-y += test_fbarray.c +SRCS-y += test_external_mem.c SRCS-y += test_ring.c SRCS-y += test_ring_perf.c
test/test/autotest_data.py +10 -4
@@ -477,10 +477,16 @@ "Report": None, }, { - "Name": "Fbarray autotest", - "Command": "fbarray_autotest", - "Func": default_autotest, - "Report": None, + "Name": "Fbarray autotest", + "Command": "fbarray_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "External memory autotest", + "Command": "external_mem_autotest", + "Func": default_autotest, + "Report": None, }, # #Please always keep all dump tests at the end and together!
test/test/meson.build +1 -0
@@ -155,6 +155,7 @@ test_names = [ 'eventdev_common_autotest', 'eventdev_octeontx_autotest', 'eventdev_sw_autotest', + 'external_mem_autotest', 'func_reentrancy_autotest', 'flow_classify_autotest', 'hash_scaling_autotest',
test/test/test_external_mem.c new file +389 -0
389 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 19/21 app/testpmd: add support for external memory
In reply to Anatoly BurakovCurrently, mempools can only be allocated either using native DPDK memory, or anonymous memory. This patch will add two new methods to allocate mempool using external memory (regular or hugepage memory), and add documentation about it to testpmd user guide.
It adds a new flag "--mp-alloc", with four possible values:
native (use regular DPDK allocator), anon (use anonymous mempool), xmem (use externally allocated memory area), and xmemhuge (use externally allocated hugepage memory area). Old flag "--mp-anon" is kept for compatibility.
All external memory is allocated using the same external heap, but each will allocate and add a new memory area.
Signed-off-by Anatoly Burakov <[email protected]>
Suggested-by Konstantin Ananyev <[email protected]>
--- app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 305 ++++++++++++++++++++++++-- app/test-pmd/testpmd.h | 13 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + 5 files changed, 349 insertions(+), 25 deletions(-)
app/test-pmd/config.c +19 -2
@@ -2413,6 +2413,23 @@ fwd_config_setup(void)simple_fwd_config_setup (); } +static const char * +mp_alloc_to_str (uint8_t mode) +{ +switch (mode) { +case MP_ALLOC_NATIVE : +return "native" ; +case MP_ALLOC_ANON : +return "anon" ; +case MP_ALLOC_XMEM : +return "xmem" ; +case MP_ALLOC_XMEM_HUGE : +return "xmemhuge" ; +default : +return "invalid" ; + } +} +void pkt_fwd_config_display (struct fwd_config *cfg) {
@@ -2421,12 +2438,12 @@ pkt_fwd_config_display(struct fwd_config *cfg)streamid_t sm_id;printf ("%s packet forwarding%s - ports=%d - cores=%d - streams=%d - " -"NUMA support %s, MP over anonymous pages %s\n" , +"NUMA support %s, MP allocation mode: %s\n" , cfg->fwd_eng->fwd_mode_name, retry_enabled ==0 ?"" :" with retry" , cfg->nb_fwd_ports, cfg->nb_fwd_lcores, cfg->nb_fwd_streams, numa_support ==1 ?"enabled" :"disabled" , - mp_anon !=0 ?"enabled" :"disabled" ); +mp_alloc_to_str (mp_alloc_type));if (retry_enabled) printf("TX retry num: %u, delay between TX retries: %uus\n",
app/test-pmd/parameters.c +22 -1
@@ -190,6 +190,11 @@ usage(char* progname)printf (" --vxlan-gpe-port=N: UPD port of tunnel VXLAN-GPE\n" );printf (" --mlockall: lock all memory\n" );printf (" --no-mlockall: do not lock all memory\n" ); +printf (" --mp-alloc <native|anon|xmem|xmemhuge>: mempool allocation method.\n" +" native: use regular DPDK memory to create and populate mempool\n" +" anon: use regular DPDK memory to create and anonymous memory to populate mempool\n" +" xmem: use anonymous memory to create and populate mempool\n" +" xmemhuge: use anonymous hugepage memory to create and populate mempool\n" ); }#ifdef RTE_LIBRTE_CMDLINE
@@ -625,6 +630,7 @@ launch_args_parse(int argc, char** argv) { "vxlan-gpe-port", 1, 0, 0 }, { "mlockall", 0, 0, 0 }, { "no-mlockall", 0, 0, 0 }, + { "mp-alloc", 1, 0, 0 }, { 0, 0, 0, 0 }, };
@@ -743,7 +749,22 @@ launch_args_parse(int argc, char** argv)if (!strcmp (lgopts[opt_idx].name,"numa" )) numa_support =1 ;if (!strcmp (lgopts[opt_idx].name,"mp-anon" )) { - mp_anon =1 ; + mp_alloc_type =MP_ALLOC_ANON ; + } +if (!strcmp (lgopts[opt_idx].name,"mp-alloc" )) { +if (!strcmp (optarg,"native" )) + mp_alloc_type =MP_ALLOC_NATIVE ; +else if (!strcmp (optarg,"anon" )) + mp_alloc_type =MP_ALLOC_ANON ; +else if (!strcmp (optarg,"xmem" )) + mp_alloc_type =MP_ALLOC_XMEM ; +else if (!strcmp (optarg,"xmemhuge" )) + mp_alloc_type =MP_ALLOC_XMEM_HUGE ; +else +rte_exit (EXIT_FAILURE , +"mp-alloc %s invalid - must be: " +"native, anon or xmem\n" , + optarg); }if (!strcmp (lgopts[opt_idx].name,"port-numa-config" )) {if (parse_portnuma_config (optarg))
app/test-pmd/testpmd.c +284 -21
338 lines not on this page · raw message
app/test-pmd/testpmd.h +12 -1
@@ -69,6 +69,16 @@ enum { PORT_TOPOLOGY_LOOP, }; +enum { +MP_ALLOC_NATIVE ,/**< allocate and populate mempool natively */ +MP_ALLOC_ANON , +/**< allocate mempool natively, but populate using anonymous memory */ +MP_ALLOC_XMEM , +/**< allocate and populate mempool using anonymous memory */ +MP_ALLOC_XMEM_HUGE +/**< allocate and populate mempool using anonymous hugepage memory */ +}; +#ifdef RTE_TEST_PMD_RECORD_BURST_STATS /** * The data structure associated with RX and TX packet burst statistics
@@ -304,7 +314,8 @@ extern uint8_t numa_support; /**< set by "--numa" parameter */extern uint16_t port_topology;/**< set by "--port-topology" parameter */ extern uint8_t no_flush_rx;/**<set by "--no-flush-rx" parameter */ extern uint8_t flow_isolate_all;/**< set by "--flow-isolate-all */ -extern uint8_t mp_anon;/**< set by "--mp-anon" parameter */ +extern uint8_t mp_alloc_type; +/**< set by "--mp-anon" or "--mp-alloc" parameter */ extern uint8_t no_link_check;/**<set by "--disable-link-check" parameter */ extern volatile int test_done;/* stop packet forwarding when set to 1. */ extern uint8_t lsc_interrupt;/**< disabled by "--no-lsc-interrupt" parameter */
doc/guides/testpmd_app_ug/run_app.rst +12 -0
@@ -498,3 +498,15 @@ The commandline options are: * ``--no-mlockall`` Disable locking all memory. + +* ``--mp-alloc <native|anon|xmem|xmemhuge>`` + + Select mempool allocation mode: + + * native: create and populate mempool using native DPDK memory + * anon: create mempool using native DPDK memory, but populate using + anonymous memory + * xmem: create and populate mempool using externally and anonymously + allocated area + * xmemhuge: create and populate mempool using externally and anonymously + allocated hugepage area
Signature · 1 line
2.17.1
PATCH v6 20/21 doc: add external memory feature to the release notes
In reply to Anatoly BurakovDocument the addition of external memory support to DPDK.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 5 +++++ 1 file changed, 5 insertions(+)
doc/guides/rel_notes/release_18_11.rst +5 -0
@@ -67,6 +67,11 @@ New Features SR-IOV option in Hyper-V and Azure. This is an alternative to the previous vdev_netvsc, tap, and failsafe drivers combination. +* **Added support for using externally allocated memory in DPDK.** + + DPDK has gained support for creating new ``rte_malloc`` heaps referencing + memory that was created outside of DPDK's own page allocator, and using that + memory natively with any other DPDK library or data structure. API Changes -----------
Signature · 1 line
2.17.1
PATCH v6 21/21 doc: add external memory feature to programmer's guide
In reply to Anatoly BurakovAdd a short chapter on usage of external memory in DPDK to the Programmer's Guide.
Signed-off-by Anatoly Burakov <[email protected]>
--- .../prog_guide/env_abstraction_layer.rst | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +37 -0
@@ -213,6 +213,43 @@ Normally, these options do not need to be changed. can later be mapped into that preallocated VA space (if dynamic memory mode is enabled), and can optionally be mapped into it at startup. +Support for Externally Allocated Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is possible to use externally allocated memory in DPDK, using a set of malloc +heap API's. Support for externally allocated memory is implemented through +overloading the socket ID - externally allocated heaps will have socket ID's +that would be considered invalid under normal circumstances. Requesting an +allocation to take place from a specified externally allocated memory is a +matter of supplying the correct socket ID to DPDK allocator, either directly +(e.g. through a call to ``rte_malloc``) or indirectly (through data +structure-specific allocation API's such as ``rte_ring_create``). + +Since there is no way DPDK can verify whether memory are is available or valid, +this responsibility falls on the shoulders of the user. All multiprocess +synchronization is also user's responsibility, as well as ensuring that all +calls to add/attach/detach/remove memory are done in the correct order. It is +not required to attach to a memory area in all processes - only attach to memory +areas as needed. + +The expected workflow is as follows: + +* Get a pointer to memory area +* Create a named heap +* Add memory area(s) to the heap + - If IOVA table is not specified, IOVA addresses will be assumed to be + unavailable, and DMA mappings will not be performed + - Other processes must attach to the memory area before they can use it +* Get socket ID used for the heap +* Use normal DPDK allocation procedures, using supplied socket ID +* If memory area is no longer needed, it can be removed from the heap + - Other processes must detach from this memory area before it can be removed +* If heap is no longer needed, remove it + - Socket ID will become invalid and will not be reused + +For more information, please refer to ``rte_malloc`` API documentation, +specifically the ``rte_malloc_heap_*`` family of function calls. + PCI Access ~~~~~~~~~~
Signature · 1 line
2.17.1
v7 cover Support externally allocated memory in DPDK
In reply to Anatoly BurakovThis is a proposal to enable using externally allocated memory in DPDK.
In a nutshell, here is what is being done here:
- Index internal malloc heaps by NUMA node index, rather than NUMA node itself (external heaps will have ID's in order of creation)
- Add identifier string to malloc heap, to uniquely identify it
- Each new heap will receive a unique socket ID that will be used by allocator to decide from which heap (internal or external) to allocate requested amount of memory
- Allow creating named heaps and add/remove memory to/from those heaps
- Allocate memseg lists at runtime, to keep track of IOVA addresses of externally allocated memory
- If IOVA addresses aren't provided, use RTE_BAD_IOVA
- Allow malloc and memzones to allocate from external heaps
- Allow other data structures to allocate from externall heaps
The responsibility to ensure memory is accessible before using it is on the shoulders of the user - there is no checking done with regards to validity of the memory (nor could there be...).
The general approach is to create heap and add memory into it. For any other process wishing to use the same memory, said memory must first be attached (otherwise some things will not work).
A design decision was made to make multiprocess synchronization a manual process. Due to underlying issues with attaching to fbarrays in secondary processes, this design was deemed to be better because we don't want to fail to create external heap in the primary because something in the secondary has failed when in fact we may not eve have wanted this memory to be accessible in the secondary in the first place.
Using external memory in multiprocess is *hard*, because not only memory space needs to be preallocated, but it also needs to be attached in each process to allow other processes to access the page table. The attach API call may or may not succeed, depending on memory layout, for reasons similar to other multiprocess failures. This is treated as a "known issue" for this release.
v7 -> v6 changes:
- Fixed missing IOVA address setup in testpmd
- Fixed MLX drivers as per Yongseok's comments
- Added a check for invalid heap idx on adding memory to heap
v6 -> v5 changes:
- Fixed documentation formatting as per Marko's comments
v5 -> v4 changes:
- All processes are now able to create and destroy malloc heaps
- Memory is automatically mapped for DMA on adding it to heap
- Mem event callbacks are triggered on adding/removing memory
- Fixed compile issues on FreeBSD
- Better documentation on API/ABI changes
v4 -> v3 changes:
- Dropped sample application in favor of new testpmd flag
- Added new flag to testpmd, with four options of mempool allocation
- Added new API to check if a socket ID belongs to an external heap
- Adjusted malloc and mempool code to not make any assumptions about IOVA-contiguousness when dealing with externally allocated memory
v3 -> v2 changes:
- Rebase on top of latest master
- Clarifications added to mempool code as per Andrew Rynchenko's comments
v2 -> v1 changes:
- Fixed NULL dereference on heap socket ID lookup
- Fixed memseg offset calculation on adding memory to heap
- Improved unit test to test for above bugfixes
- Restricted heap creation to primary processes only
- Added sample application
- Added documentation
RFC -> v1 changes:
- Removed the "named heaps" API, allocate using fake socket ID instead
- Added multiprocess support
- Everything is now thread-safe
- Numerous bugfixes and API improvements
Anatoly Burakov (21):
mem: add length to memseg list
mem: allow memseg lists to be marked as external
malloc: index heaps using heap ID rather than NUMA node
mem: do not check for invalid socket ID
flow_classify: do not check for invalid socket ID
pipeline: do not check for invalid socket ID
sched: do not check for invalid socket ID
malloc: add name to malloc heaps
malloc: add function to query socket ID of named heap
malloc: add function to check if socket is external
malloc: allow creating malloc heaps
malloc: allow destroying heaps
malloc: allow adding memory to named heaps
malloc: allow removing memory from named heaps
malloc: allow attaching to external memory chunks
malloc: allow detaching from external memory
malloc: enable event callbacks for external memory
test: add unit tests for external memory support app/testpmd: add support for external memory
doc: add external memory feature to the release notes
doc: add external memory feature to programmer's guide
app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 318 ++++++++++++- app/test-pmd/testpmd.h | 13 +- config/common_base | 1 + config/rte_config.h | 1 + .../prog_guide/env_abstraction_layer.rst | 37 ++ doc/guides/rel_notes/deprecation.rst | 15 - doc/guides/rel_notes/release_18_11.rst | 28 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + drivers/bus/fslmc/fslmc_vfio.c | 13 +- drivers/bus/pci/linux/pci.c | 2 +- drivers/net/mlx5/mlx5.c | 4 +- drivers/net/virtio/virtio_user/vhost_kernel.c | 5 +- .../net/virtio/virtio_user/virtio_user_dev.c | 8 + lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal.c | 3 + lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +- lib/librte_eal/common/eal_common_memory.c | 8 +- lib/librte_eal/common/eal_common_memzone.c | 8 +- .../common/include/rte_eal_memconfig.h | 9 +- lib/librte_eal/common/include/rte_malloc.h | 192 ++++++++ .../common/include/rte_malloc_heap.h | 3 + lib/librte_eal/common/include/rte_memory.h | 9 + lib/librte_eal/common/malloc_elem.c | 10 +- lib/librte_eal/common/malloc_heap.c | 320 +++++++++++-- lib/librte_eal/common/malloc_heap.h | 17 + lib/librte_eal/common/rte_malloc.c | 429 +++++++++++++++++- lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +- lib/librte_eal/linuxapp/eal/eal_vfio.c | 27 +- lib/librte_eal/meson.build | 2 +- lib/librte_eal/rte_eal_version.map | 8 + lib/librte_flow_classify/rte_flow_classify.c | 3 +- lib/librte_mempool/rte_mempool.c | 57 ++- lib/librte_pipeline/rte_pipeline.c | 3 +- lib/librte_sched/rte_sched.c | 2 +- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++ test/test/test_malloc.c | 3 + test/test/test_memzone.c | 3 + 45 files changed, 1923 insertions(+), 138 deletions(-) create mode 100644 test/test/test_external_mem.c
Signature · 1 line
2.17.1
PATCH v7 1/21 mem: add length to memseg list
Previously, to calculate length of memory area covered by a memseg list, we would've needed to multiply page size by length of fbarray backing that memseg list. This is not obvious and unnecessarily low level, so store length in the memseg list itself.
Signed-off-by Anatoly Burakov <[email protected]>
Acked-by Shreyansh Jain <[email protected]>
--- drivers/bus/pci/linux/pci.c | 2 +- lib/librte_eal/bsdapp/eal/eal_memory.c | 2 ++ lib/librte_eal/common/eal_common_memory.c | 5 ++--- lib/librte_eal/common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/linuxapp/eal/eal_memalloc.c | 3 ++- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +++- 6 files changed, 11 insertions(+), 6 deletions(-)
drivers/bus/pci/linux/pci.c +1 -1
@@ -119,7 +119,7 @@ rte_pci_unmap_device(struct rte_pci_device *dev)static int find_max_end_va (const struct rte_memseg_list *msl,void *arg) { -size_t sz = msl->memseg_arr.len * msl->page_sz; +size_t sz = msl->len;void *end_va =RTE_PTR_ADD (msl->base_va, sz);void **max_va = arg;
lib/librte_eal/bsdapp/eal/eal_memory.c +2 -0
@@ -79,6 +79,7 @@ rte_eal_hugepage_init(void) } msl->base_va = addr; msl->page_sz = page_sz; + msl->len = internal_config.memory; msl->socket_id =0 ;/* populate memsegs. each memseg is 1 page long */
@@ -370,6 +371,7 @@ alloc_va_space(struct rte_memseg_list *msl)return -1 ; } msl->base_va = addr; + msl->len = mem_sz;return 0 ; }
lib/librte_eal/common/eal_common_memory.c +2 -3
@@ -171,7 +171,7 @@ virt2memseg(const void *addr, const struct rte_memseg_list *msl)/* a memseg list was specified, check if it's the right one */ start = msl->base_va; - end =RTE_PTR_ADD (start, (size_t )msl->page_sz * msl->memseg_arr.len); + end =RTE_PTR_ADD (start, msl->len);if (addr < start || addr >= end)return NULL ;
@@ -194,8 +194,7 @@ virt2memseg_list(const void *addr) msl = &mcfg->memsegs[msl_idx]; start = msl->base_va; - end =RTE_PTR_ADD (start, - (size_t )msl->page_sz * msl->memseg_arr.len); + end =RTE_PTR_ADD (start, msl->len);if (addr >= start && addr < end)break ; }
lib/librte_eal/common/include/rte_eal_memconfig.h +1 -0
@@ -30,6 +30,7 @@ struct rte_memseg_list {uint64_t addr_64;/**< Makes sure addr is always 64-bits */ }; +size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ volatile uint32_t version;/**< version number for multiprocess sync. */
lib/librte_eal/linuxapp/eal/eal_memalloc.c +2 -1
@@ -986,7 +986,7 @@ free_seg_walk(const struct rte_memseg_list *msl, void *arg)int msl_idx, seg_idx, ret, dir_fd =-1 ; start_addr = (uintptr_t ) msl->base_va; - end_addr = start_addr + msl->memseg_arr.len * (size_t )msl->page_sz; + end_addr = start_addr + msl->len;if ((uintptr_t )wa->ms->addr < start_addr || (uintptr_t )wa->ms->addr >= end_addr)
@@ -1472,6 +1472,7 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,return -1 ; } local_msl->base_va = primary_msl->base_va; + local_msl->len = primary_msl->len;return 0 ; }
lib/librte_eal/linuxapp/eal/eal_memory.c +3 -1
@@ -861,6 +861,7 @@ alloc_va_space(struct rte_memseg_list *msl)return -1 ; } msl->base_va = addr; + msl->len = mem_sz;return 0 ; }
@@ -1369,6 +1370,7 @@ eal_legacy_hugepage_init(void) msl->base_va = addr; msl->page_sz = page_sz; msl->socket_id =0 ; + msl->len = internal_config.memory;/* populate memsegs. each memseg is one page long */ for (cur_seg =0 ; cur_seg < n_segs; cur_seg++) {
@@ -1615,7 +1617,7 @@ eal_legacy_hugepage_init(void)if (msl->memseg_arr.count >0 )continue ;/* this is an unused list, deallocate it */ - mem_sz = (size_t )msl->page_sz * msl->memseg_arr.len; + mem_sz = msl->len;munmap (msl->base_va, mem_sz); msl->base_va =NULL ;
Signature · 1 line
2.17.1
PATCH v7 2/21 mem: allow memseg lists to be marked as external
In reply to Anatoly BurakovWhen we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so bump the EAL library ABI version and document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Signed-off-by Anatoly Burakov <[email protected]>
Acked-by Andrew Rybchenko <[email protected]>
Acked-by Yongseok Koh <[email protected]>
---
Notes:
v3:
- Add comment to explain the process of picking up minimum page sizes for mempool
v2:
- Add documentation changes and ABI break
v1:
- Adjust all calls to memseg walk functions to ignore external segments where it made sense to do so
doc/guides/rel_notes/deprecation.rst | 15 -------- doc/guides/rel_notes/release_18_11.rst | 13 ++++++- drivers/bus/fslmc/fslmc_vfio.c | 6 +++- drivers/net/mlx5/mlx5.c | 4 ++- drivers/net/virtio/virtio_user/vhost_kernel.c | 5 ++- lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal.c | 3 ++ lib/librte_eal/bsdapp/eal/eal_memory.c | 7 ++-- lib/librte_eal/common/eal_common_memory.c | 3 ++ .../common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/common/include/rte_memory.h | 9 +++++ lib/librte_eal/common/malloc_elem.c | 10 ++++-- lib/librte_eal/common/malloc_heap.c | 9 +++-- lib/librte_eal/common/rte_malloc.c | 2 +- lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +++++- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 9 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 ++++++--- lib/librte_eal/meson.build | 2 +- lib/librte_mempool/rte_mempool.c | 35 ++++++++++++++----- test/test/test_malloc.c | 3 ++ test/test/test_memzone.c | 3 ++ 22 files changed, 127 insertions(+), 43 deletions(-)
doc/guides/rel_notes/deprecation.rst +0 -15
@@ -11,21 +11,6 @@ API and ABI deprecation notices are to be posted here. Deprecation Notices ------------------- -* eal: certain structures will change in EAL on account of upcoming external - memory support. Aside from internal changes leading to an ABI break, the - following externally visible changes will also be implemented: - - - ``rte_memseg_list`` will change to include a boolean flag indicating - whether a particular memseg list is externally allocated. This will have - implications for any users of memseg-walk-related functions, as they will - now have to skip externally allocated segments in most cases if the intent - is to only iterate over internal DPDK memory. - - ``socket_id`` parameter across the entire DPDK will gain additional meaning, - as some socket ID's will now be representing externally allocated memory. No - changes will be required for existing code as backwards compatibility will - be kept, and those who do not use this feature will not see these extra - socket ID's. - * eal: both declaring and identifying devices will be streamlined in v18.11. New functions will appear to query a specific port from buses, classes of device and device drivers. Device declaration will be made coherent with the
doc/guides/rel_notes/release_18_11.rst +12 -1
@@ -91,6 +91,13 @@ API Changes flag the MAC can be properly configured in any case. This is particularly important for bonding. +* eal: The following API changes were made in 18.11: + + - ``rte_memseg_list`` structure now has an additional flag indicating whether + the memseg list is externally allocated. This will have implications for any + users of memseg-walk-related functions, as they will now have to skip + externally allocated segments in most cases if the intent is to only iterate + over internal DPDK memory. ABI Changes -----------
@@ -107,6 +114,10 @@ ABI Changes ========================================================= +* eal: EAL library ABI version was changed due to previously announced work on + supporting external memory in DPDK. Structure ``rte_memseg_list`` now has + a new flag indicating whether the memseg list refers to external memory. + Removed Items -------------
@@ -152,7 +163,7 @@ The libraries prepended with a plus sign were incremented in this version. librte_compressdev.so.1 librte_cryptodev.so.5 librte_distributor.so.1 - librte_eal.so.8 + + librte_eal.so.9 librte_ethdev.so.10 librte_eventdev.so.4 librte_flow_classify.so.1
drivers/bus/fslmc/fslmc_vfio.c +5 -1
@@ -318,11 +318,15 @@ fslmc_unmap_dma(uint64_t vaddr, uint64_t iovaddr __rte_unused, size_t len)static int fslmc_dmamap_seg(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +const struct rte_memseg *ms,void *arg) {int *n_segs = arg;int ret; +/* if IOVA address is invalid, skip */ +if (ms->iova ==RTE_BAD_IOVA ) +return 0 ; + ret =fslmc_map_dma (ms->addr_64, ms->iova, ms->len);if (ret) DPAA2_BUS_ERR("Unable to VFIO map (addr=%p, len=%zu)",
drivers/net/mlx5/mlx5.c +3 -1
@@ -568,11 +568,13 @@ static struct rte_pci_driver mlx5_driver;static void *uar_base;static int -find_lower_va_bound(const struct rte_memseg_list *msl __rte_unused, +find_lower_va_bound (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {void **addr = arg; +if (msl->external) +return 0 ;if (*addr ==NULL ) *addr = ms->addr;else
drivers/net/virtio/virtio_user/vhost_kernel.c +4 -1
@@ -75,13 +75,16 @@ struct walk_arg {uint32_t region_nr; };static int -add_memory_region(const struct rte_memseg_list *msl __rte_unused, +add_memory_region (const struct rte_memseg_list *msl,const struct rte_memseg *ms,size_t len,void *arg) {struct walk_arg *wa = arg;struct vhost_memory_region *mr;void *start_addr; +if (msl->external) +return 0 ; +if (wa->region_nr >= max_regions)return -1 ;
lib/librte_eal/bsdapp/eal/Makefile +1 -1
@@ -22,7 +22,7 @@ LDLIBS += -lrte_kvargs EXPORT_MAP := ../../rte_eal_version.map -LIBABIVER := 8 +LIBABIVER := 9 # specific to bsdapp exec-env SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) := eal.c
lib/librte_eal/bsdapp/eal/eal.c +3 -0
@@ -502,6 +502,9 @@ check_socket(const struct rte_memseg_list *msl, void *arg) {int *socket_id = arg; +if (msl->external) +return 0 ; +if (msl->socket_id == *socket_id && msl->memseg_arr.count !=0 )return 1 ;
lib/librte_eal/bsdapp/eal/eal_memory.c +5 -2
@@ -236,12 +236,15 @@ struct attach_walk_args {int seg_idx; };static int -attach_segment(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +attach_segment (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {struct attach_walk_args *wa = arg;void *addr; +if (msl->external) +return 0 ; + addr =mmap (ms->addr, ms->len,PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_FIXED , wa->fd_hugepage, wa->seg_idx *EAL_PAGE_SIZE );
lib/librte_eal/common/eal_common_memory.c +3 -0
@@ -272,6 +272,9 @@ physmem_size(const struct rte_memseg_list *msl, void *arg) {uint64_t *total_len = arg; +if (msl->external) +return 0 ; + *total_len += msl->memseg_arr.count * msl->page_sz;return 0 ;
lib/librte_eal/common/include/rte_eal_memconfig.h +1 -0
@@ -33,6 +33,7 @@ struct rte_memseg_list {size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ +unsigned int external;/**< 1 if this list points to external memory */ volatile uint32_t version;/**< version number for multiprocess sync. */ struct rte_fbarray memseg_arr; };
lib/librte_eal/common/include/rte_memory.h +9 -0
@@ -215,6 +215,9 @@ typedef int (*rte_memseg_list_walk_t)(const struct rte_memseg_list *msl,* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
@@ -233,6 +236,9 @@ rte_memseg_walk(rte_memseg_walk_t func, void *arg);* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
@@ -251,6 +257,9 @@ rte_memseg_contig_walk(rte_memseg_contig_walk_t func, void *arg);* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
lib/librte_eal/common/malloc_elem.c +7 -3
@@ -39,10 +39,14 @@ malloc_elem_find_max_iova_contig(struct malloc_elem *elem, size_t align) contig_seg_start =RTE_PTR_ALIGN_CEIL (data_start, align);/* if we're in IOVA as VA mode, or if we're in legacy mode with -* hugepages, all elements are IOVA-contiguous. +* hugepages, all elements are IOVA-contiguous. however, we can only +* make these assumptions about internal memory - externally allocated +* segments have to be checked. */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA || - (internal_config.legacy_mem &&rte_eal_has_hugepages ())) +if (!elem->msl->external && + (rte_eal_iova_mode () ==RTE_IOVA_VA || + (internal_config.legacy_mem && +rte_eal_has_hugepages ())))return RTE_PTR_DIFF (data_end, contig_seg_start); cur_page =RTE_PTR_ALIGN_FLOOR (contig_seg_start, page_sz);
lib/librte_eal/common/malloc_heap.c +7 -2
@@ -95,6 +95,9 @@ malloc_add_seg(const struct rte_memseg_list *msl,struct malloc_heap *heap;int msl_idx; +if (msl->external) +return 0 ; + heap = &mcfg->malloc_heaps[msl->socket_id];/* msl is const, so find it */
@@ -754,8 +757,10 @@ malloc_heap_free(struct malloc_elem *elem)/* anything after this is a bonus */ ret =0 ; -/* ...of which we can't avail if we are in legacy mode */ -if (internal_config.legacy_mem) +/* ...of which we can't avail if we are in legacy mode, or if this is an +* externally allocated segment. +*/ +if (internal_config.legacy_mem || msl->external) goto free_unlock;/* check if we can free any memory back to the system */
lib/librte_eal/common/rte_malloc.c +1 -1
@@ -223,7 +223,7 @@ rte_malloc_virt2iova(const void *addr)if (elem ==NULL )return RTE_BAD_IOVA ; -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) +if (!elem->msl->external &&rte_eal_iova_mode () ==RTE_IOVA_VA )return (uintptr_t ) addr; ms =rte_mem_virt2memseg (addr, elem->msl);
lib/librte_eal/linuxapp/eal/Makefile +1 -1
@@ -10,7 +10,7 @@ ARCH_DIR ?= $(RTE_ARCH) EXPORT_MAP := ../../rte_eal_version.map VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR) -LIBABIVER := 8 +LIBABIVER := 9 VPATH += $(RTE_SDK)/lib/librte_eal/common
lib/librte_eal/linuxapp/eal/eal.c +9 -1
@@ -725,6 +725,9 @@ check_socket(const struct rte_memseg_list *msl, void *arg) {int *socket_id = arg; +if (msl->external) +return 0 ; +return *socket_id == msl->socket_id; }
@@ -1059,7 +1062,12 @@ mark_freeable(const struct rte_memseg_list *msl, const struct rte_memseg *ms, void *arg __rte_unused) {/* ms is const, so find this memseg */ -struct rte_memseg *found =rte_mem_virt2memseg (ms->addr, msl); +struct rte_memseg *found; + +if (msl->external) +return 0 ; + + found =rte_mem_virt2memseg (ms->addr, msl); found->flags &= ~RTE_MEMSEG_FLAG_DO_NOT_FREE ;
lib/librte_eal/linuxapp/eal/eal_memalloc.c +9 -0
@@ -1408,6 +1408,9 @@ sync_walk(const struct rte_memseg_list *msl, void *arg __rte_unused)unsigned int i;int msl_idx; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; primary_msl = &mcfg->memsegs[msl_idx]; local_msl = &local_memsegs[msl_idx];
@@ -1456,6 +1459,9 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,char name[PATH_MAX ];int msl_idx, ret; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; primary_msl = &mcfg->memsegs[msl_idx]; local_msl = &local_memsegs[msl_idx];
@@ -1509,6 +1515,9 @@ fd_list_create_walk(const struct rte_memseg_list *msl,unsigned int len;int msl_idx; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; len = msl->memseg_arr.len;
lib/librte_eal/linuxapp/eal/eal_vfio.c +13 -4
@@ -1082,11 +1082,14 @@ rte_vfio_get_group_num(const char *sysfs_base, }static int -type1_map(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +type1_map (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {int *vfio_container_fd = arg; +if (msl->external) +return 0 ; +return vfio_type1_dma_mem_map (*vfio_container_fd, ms->addr_64, ms->iova, ms->len,1 ); }
@@ -1196,11 +1199,14 @@ vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova, }static int -vfio_spapr_map_walk(const struct rte_memseg_list *msl __rte_unused, +vfio_spapr_map_walk (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {int *vfio_container_fd = arg; +if (msl->external) +return 0 ; +return vfio_spapr_dma_mem_map (*vfio_container_fd, ms->addr_64, ms->iova, ms->len,1 ); }
@@ -1210,12 +1216,15 @@ struct spapr_walk_param {uint64_t hugepage_sz; };static int -vfio_spapr_window_size_walk(const struct rte_memseg_list *msl __rte_unused, +vfio_spapr_window_size_walk (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {struct spapr_walk_param *param = arg;uint64_t max = ms->iova + ms->len; +if (msl->external) +return 0 ; +if (max > param->window_size) { param->hugepage_sz = ms->hugepage_sz; param->window_size = max;
lib/librte_eal/meson.build +1 -1
@@ -21,7 +21,7 @@ else error('unsupported system type "@0@"'.format(host_machine.system())) endif -version = 8 # the version of the EAL API +version = 9 # the version of the EAL API allow_experimental_apis = true deps += 'compat' deps += 'kvargs'
lib/librte_mempool/rte_mempool.c +27 -8
@@ -99,25 +99,44 @@ static unsigned optimize_object_size(unsigned obj_size)return new_obj_size *RTE_MEMPOOL_ALIGN ; } +struct pagesz_walk_arg { +int socket_id; +size_t min; +}; +static int find_min_pagesz (const struct rte_memseg_list *msl,void *arg) { -size_t *min = arg; +struct pagesz_walk_arg *wa = arg; +bool valid; -if (msl->page_sz < *min) - *min = msl->page_sz; +/* +* we need to only look at page sizes available for a particular socket +* ID. so, we either need an exact match on socket ID (can match both +* native and external memory), or, if SOCKET_ID_ANY was specified as a +* socket ID argument, we must only look at native memory and ignore any +* page sizes associated with external memory. +*/ + valid = msl->socket_id == wa->socket_id; + valid |= wa->socket_id ==SOCKET_ID_ANY && msl->external ==0 ; + +if (valid && msl->page_sz < wa->min) + wa->min = msl->page_sz;return 0 ; }static size_t -get_min_page_size (void ) +get_min_page_size (int socket_id) { -size_t min_pagesz =SIZE_MAX ; +struct pagesz_walk_arg wa; -rte_memseg_list_walk (find_min_pagesz, &min_pagesz); + wa.min =SIZE_MAX ; + wa.socket_id = socket_id; -return min_pagesz ==SIZE_MAX ? (size_t )getpagesize () : min_pagesz; +rte_memseg_list_walk (find_min_pagesz, &wa); + +return wa.min ==SIZE_MAX ? (size_t )getpagesize () : wa.min; }
@@ -470,7 +489,7 @@ rte_mempool_populate_default(struct rte_mempool *mp) pg_sz =0 ; pg_shift =0 ; }else if (try_contig) { - pg_sz =get_min_page_size (); + pg_sz =get_min_page_size (mp->socket_id); pg_shift =rte_bsf32 (pg_sz); }else { pg_sz =getpagesize ();
test/test/test_malloc.c +3 -0
@@ -711,6 +711,9 @@ check_socket_mem(const struct rte_memseg_list *msl, void *arg) {int32_t *socket = arg; +if (msl->external) +return 0 ; +return *socket == msl->socket_id; }
test/test/test_memzone.c +3 -0
@@ -115,6 +115,9 @@ find_available_pagesz(const struct rte_memseg_list *msl, void *arg) {struct walk_arg *wa = arg; +if (msl->external) +return 0 ; +if (msl->page_sz ==RTE_PGSIZE_2M ) wa->hugepage_2MB_avail =1 ;if (msl->page_sz ==RTE_PGSIZE_1G )
Signature · 1 line
2.17.1
PATCH v7 3/21 malloc: index heaps using heap ID rather than NUMA node
In reply to Anatoly BurakovSwitch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation.
Signed-off-by Anatoly Burakov <[email protected]>
--- config/common_base | 1 + config/rte_config.h | 1 + .../common/include/rte_eal_memconfig.h | 4 +- .../common/include/rte_malloc_heap.h | 1 + lib/librte_eal/common/malloc_heap.c | 102 +++++++++++++----- lib/librte_eal/common/malloc_heap.h | 3 + lib/librte_eal/common/rte_malloc.c | 41 ++++--- 7 files changed, 110 insertions(+), 43 deletions(-)
config/common_base +1 -0
@@ -61,6 +61,7 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_LIBRTE_EAL=y CONFIG_RTE_MAX_LCORE=128 CONFIG_RTE_MAX_NUMA_NODES=8 +CONFIG_RTE_MAX_HEAPS=32 CONFIG_RTE_MAX_MEMSEG_LISTS=64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever is smaller
config/rte_config.h +1 -0
@@ -24,6 +24,7 @@#define RTE_BUILD_SHARED_LIB /* EAL defines */ +#define RTE_MAX_HEAPS 32#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768
lib/librte_eal/common/include/rte_eal_memconfig.h +2 -2
@@ -72,8 +72,8 @@ struct rte_mem_config {struct rte_tailq_head tailq_head[RTE_MAX_TAILQ ];/**< Tailqs for objects */ -/* Heaps of Malloc per socket */ -struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES ]; +/* Heaps of Malloc */ +struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ];/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it.
lib/librte_eal/common/include/rte_malloc_heap.h +1 -0
@@ -27,6 +27,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size; + unsigned int socket_id; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */
lib/librte_eal/common/malloc_heap.c +73 -29
208 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -46,6 +46,9 @@ malloc_heap_get_stats(struct malloc_heap *heap,void malloc_heap_dump (struct malloc_heap *heap,FILE *f); +int +malloc_socket_to_heap_id (unsigned int socket_id); +int rte_eal_malloc_heap_init (void );
lib/librte_eal/common/rte_malloc.c +29 -12
72 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v7 4/21 mem: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
This changes the semantics of what we understand by "socket ID", so document the change in the release notes.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 7 +++++++ lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- lib/librte_eal/common/malloc_heap.c | 2 +- lib/librte_eal/common/rte_malloc.c | 4 ---- 4 files changed, 13 insertions(+), 8 deletions(-)
doc/guides/rel_notes/release_18_11.rst +7 -0
@@ -98,6 +98,13 @@ API Changes users of memseg-walk-related functions, as they will now have to skip externally allocated segments in most cases if the intent is to only iterate over internal DPDK memory. + - ``socket_id`` parameter across the entire DPDK has gained additional + meaning, as some socket ID's will now be representing externally allocated + memory. No changes will be required for existing code as backwards + compatibility will be kept, and those who do not use this feature will not + see these extra socket ID's. Any new API's must not check socket ID + parameters themselves, and must instead leave it to the memory subsystem to + decide whether socket ID is a valid one. ABI Changes -----------
lib/librte_eal/common/eal_common_memzone.c +5 -3
@@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,return NULL ; } -if ((socket_id !=SOCKET_ID_ANY ) && - (socket_id >=RTE_MAX_NUMA_NODES || socket_id <0 )) { +if ((socket_id !=SOCKET_ID_ANY ) && socket_id <0 ) { rte_errno =EINVAL ;return NULL ; } -if (!rte_eal_has_hugepages ()) +/* only set socket to SOCKET_ID_ANY if we aren't allocating for an +* external heap. +*/ +if (!rte_eal_has_hugepages () && socket_id <RTE_MAX_NUMA_NODES ) socket_id =SOCKET_ID_ANY ; contig = (flags &RTE_MEMZONE_IOVA_CONTIG ) !=0 ;
lib/librte_eal/common/malloc_heap.c +1 -1
@@ -651,7 +651,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,if (size ==0 || (align && !rte_is_power_of_2 (align)))return NULL ; -if (!rte_eal_has_hugepages ()) +if (!rte_eal_has_hugepages () && socket_arg <RTE_MAX_NUMA_NODES ) socket_arg =SOCKET_ID_ANY ;if (socket_arg ==SOCKET_ID_ANY )
lib/librte_eal/common/rte_malloc.c +0 -4
@@ -47,10 +47,6 @@ rte_malloc_socket(const char *type, size_t size, unsigned int align,if (!rte_eal_has_hugepages ()) socket_arg =SOCKET_ID_ANY ; -/* Check socket parameter */ -if (socket_arg >=RTE_MAX_NUMA_NODES ) -return NULL ; -return malloc_heap_alloc (type, size, socket_arg,0 , align ==0 ?1 : align,0 , false); }
Signature · 1 line
2.17.1
PATCH v7 5/21 flow_classify: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_flow_classify/rte_flow_classify.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/librte_flow_classify/rte_flow_classify.c +1 -2
@@ -247,8 +247,7 @@ rte_flow_classifier_check_params(struct rte_flow_classifier_params *params) }/* socket */ -if ((params->socket_id <0 ) || - (params->socket_id >=RTE_MAX_NUMA_NODES )) { +if (params->socket_id <0 ) {RTE_FLOW_CLASSIFY_LOG (ERR ,"%s: Incorrect value for parameter socket_id\n" , __func__);
Signature · 1 line
2.17.1
PATCH v7 6/21 pipeline: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_pipeline/rte_pipeline.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/librte_pipeline/rte_pipeline.c +1 -2
@@ -178,8 +178,7 @@ rte_pipeline_check_params(struct rte_pipeline_params *params) }/* socket */ -if ((params->socket_id <0 ) || - (params->socket_id >=RTE_MAX_NUMA_NODES )) { +if (params->socket_id <0 ) {RTE_LOG (ERR ,PIPELINE ,"%s: Incorrect value for parameter socket_id\n" , __func__);
Signature · 1 line
2.17.1
PATCH v7 7/21 sched: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_sched/rte_sched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_sched/rte_sched.c +1 -1
@@ -329,7 +329,7 @@ rte_sched_port_check_params(struct rte_sched_port_params *params)return -1 ;/* socket */ -if ((params->socket <0 ) || (params->socket >=RTE_MAX_NUMA_NODES )) +if (params->socket <0 )return -3 ;/* rate */
Signature · 1 line
2.17.1
PATCH v7 8/21 malloc: add name to malloc heaps
In reply to Anatoly BurakovWe will need to refer to external heaps in some way. While we use heap ID's internally, for external API use it has to be something more user-friendly. So, we will be using a string to uniquely identify a heap.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 1 + lib/librte_eal/common/include/rte_malloc_heap.h | 2 ++ lib/librte_eal/common/malloc_heap.c | 17 ++++++++++++++++- lib/librte_eal/common/rte_malloc.c | 1 + 4 files changed, 20 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_18_11.rst +1 -0
@@ -124,6 +124,7 @@ ABI Changes * eal: EAL library ABI version was changed due to previously announced work on supporting external memory in DPDK. Structure ``rte_memseg_list`` now has a new flag indicating whether the memseg list refers to external memory. + Structure ``rte_malloc_heap`` now has a ``heap_name`` string member. Removed Items -------------
lib/librte_eal/common/include/rte_malloc_heap.h +2 -0
@@ -12,6 +12,7 @@/* Number of free lists per heap, grouped by size. */ #define RTE_HEAP_NUM_FREELISTS 13 +#define RTE_HEAP_NAME_MAX_LEN 32/* dummy definition, for pointers */ struct malloc_elem ;
@@ -28,6 +29,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size;unsigned int socket_id; + char name[RTE_HEAP_NAME_MAX_LEN]; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */
lib/librte_eal/common/malloc_heap.c +16 -1
@@ -131,7 +131,6 @@ malloc_add_seg(const struct rte_memseg_list *msl,malloc_heap_add_memory (heap, found_msl, ms->addr, len); heap->total_size += len; - heap->socket_id = msl->socket_id;RTE_LOG (DEBUG ,EAL ,"Added %zuM to heap on socket %i\n" , len >>20 , msl->socket_id);
@@ -1024,6 +1023,22 @@ intrte_eal_malloc_heap_init (void) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int i; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign names to default DPDK heaps */ +for (i =0 ; i <rte_socket_count (); i++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[i]; +char heap_name[RTE_HEAP_NAME_MAX_LEN ]; +int socket_id =rte_socket_id_by_idx (i); + +snprintf (heap_name,sizeof (heap_name) -1 , +"socket_%i" , socket_id); +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); + heap->socket_id = socket_id; + } + } +if (register_mp_requests ()) {RTE_LOG (ERR ,EAL ,"Couldn't register malloc multiprocess actions\n" );
lib/librte_eal/common/rte_malloc.c +1 -0
@@ -202,6 +202,7 @@ rte_malloc_dump_stats(FILE *f, __rte_unused const char *type)malloc_heap_get_stats (heap, &sock_stats);fprintf (f,"Heap id:%u\n" , heap_id); +fprintf (f,"\tHeap name:%s\n" , heap->name);fprintf (f,"\tHeap_size:%zu,\n" , sock_stats.heap_totalsz_bytes);fprintf (f,"\tFree_size:%zu,\n" , sock_stats.heap_freesz_bytes);fprintf (f,"\tAlloc_size:%zu,\n" , sock_stats.heap_allocsz_bytes);
Signature · 1 line
2.17.1
PATCH v7 9/21 malloc: add function to query socket ID of named heap
In reply to Anatoly BurakovWhen we will be creating external heaps, they will have their own "fake" socket ID, so add a function that will map the heap name to its socket ID.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 14 ++++++++ lib/librte_eal/common/rte_malloc.c | 37 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +14 -0
@@ -263,6 +263,20 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Find socket ID corresponding to a named heap. +* +* @param name +* Heap name to find socket ID for +* @return +* Socket ID in case of success (a non-negative number) +* -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``name`` was NULL +* ENOENT - heap identified by the name ``name`` was not found +*/ +int __rte_experimental +rte_malloc_heap_get_socket (const char *name); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +37 -0
@@ -8,6 +8,7 @@#include <string.h> #include <sys/queue.h> +#include <rte_errno.h> #include <rte_memcpy.h> #include <rte_memory.h> #include <rte_eal.h>
@@ -183,6 +184,42 @@ rte_malloc_dump_heaps(FILE *f)rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); } +int +rte_malloc_heap_get_socket (const char *name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +unsigned int idx; +int ret; + +if (name ==NULL || +strnlen (name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if (!strncmp (name, tmp->name,RTE_HEAP_NAME_MAX_LEN )) { + heap = tmp; +break ; + } + } + +if (heap !=NULL ) { + ret = heap->socket_id; + }else { + rte_errno =ENOENT ; + ret =-1 ; + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_get_socket; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
Signature · 1 line
2.17.1
PATCH v7 10/21 malloc: add function to check if socket is external
In reply to Anatoly BurakovAn API is needed to check whether a particular socket ID belongs to an internal or external heap. Prime user of this would be mempool allocator, because normal assumptions of IOVA contiguousness in IOVA as VA mode do not hold in case of externally allocated memory.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 15 +++++++++++++ lib/librte_eal/common/rte_malloc.c | 25 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + lib/librte_mempool/rte_mempool.c | 22 ++++++++++++++++--- 4 files changed, 60 insertions(+), 3 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +15 -0
@@ -277,6 +277,21 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_get_socket (const char *name); +/** +* Check if a given socket ID refers to externally allocated memory. +* +* @note Passing SOCKET_ID_ANY will return 0. +* +* @param socket_id +* Socket ID to check +* @return +* 1 if socket ID refers to externally allocated memory +* 0 if socket ID refers to internal DPDK memory +* -1 if socket ID is invalid +*/ +int __rte_experimental +rte_malloc_heap_socket_is_external (int socket_id); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +25 -0
@@ -220,6 +220,31 @@ rte_malloc_heap_get_socket(const char *name)return ret; } +int +rte_malloc_heap_socket_is_external (int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int idx; +int ret =-1 ; + +if (socket_id ==SOCKET_ID_ANY ) +return 0 ; + +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if ((int )tmp->socket_id == socket_id) { +/* external memory always has large socket ID's */ + ret = tmp->socket_id >=RTE_MAX_NUMA_NODES ; +break ; + } + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_get_socket; + rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
lib/librte_mempool/rte_mempool.c +19 -3
@@ -428,12 +428,18 @@ rte_mempool_populate_default(struct rte_mempool *mp)rte_iova_t iova;unsigned mz_id, n;int ret; -bool no_contig, try_contig, no_pageshift; +bool no_contig, try_contig, no_pageshift, external; ret =mempool_ops_alloc_once (mp);if (ret !=0 )return ret; +/* check if we can retrieve a valid socket ID */ + ret =rte_malloc_heap_socket_is_external (mp->socket_id); +if (ret <0 ) +return -EINVAL ; + external = ret; +/* mempool must not be populated */ if (mp->nb_mem_chunks !=0 )return -EEXIST ;
@@ -481,9 +487,19 @@ rte_mempool_populate_default(struct rte_mempool *mp)* in one contiguous chunk as well (otherwise we might end up wasting a * 1G page on a 10MB memzone). If we fail to get enough contiguous * memory, then we'll go and reserve space page-by-page. +* +* We also have to take into account the fact that memory that we're +* going to allocate from can belong to an externally allocated memory +* area, in which case the assumption of IOVA as VA mode being +* synonymous with IOVA contiguousness will not hold. We should also try +* to go for contiguous memory even if we're in no-huge mode, because +* external memory may in fact be IOVA-contiguous. */ - no_pageshift = no_contig ||rte_eal_iova_mode () ==RTE_IOVA_VA ; - try_contig = !no_contig && !no_pageshift &&rte_eal_has_hugepages (); + external =rte_malloc_heap_socket_is_external (mp->socket_id) ==1 ; + no_pageshift = no_contig || + (!external &&rte_eal_iova_mode () ==RTE_IOVA_VA ); + try_contig = !no_contig && !no_pageshift && + (rte_eal_has_hugepages () || external);if (no_pageshift) { pg_sz =0 ;
Signature · 1 line
2.17.1
PATCH v7 11/21 malloc: allow creating malloc heaps
In reply to Anatoly BurakovAdd API to allow creating new malloc heaps. They will be created with socket ID's going above RTE_MAX_NUMA_NODES, to avoid clashing with internal heaps.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 2 + .../common/include/rte_eal_memconfig.h | 3 ++ lib/librte_eal/common/include/rte_malloc.h | 19 +++++++ lib/librte_eal/common/malloc_heap.c | 37 +++++++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 52 +++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 7 files changed, 117 insertions(+)
doc/guides/rel_notes/release_18_11.rst +2 -0
@@ -125,6 +125,8 @@ ABI Changes supporting external memory in DPDK. Structure ``rte_memseg_list`` now has a new flag indicating whether the memseg list refers to external memory. Structure ``rte_malloc_heap`` now has a ``heap_name`` string member. + Structure ``rte_eal_memconfig`` has been extended to contain next socket + ID for externally allocated memory segments. Removed Items -------------
lib/librte_eal/common/include/rte_eal_memconfig.h +3 -0
@@ -75,6 +75,9 @@ struct rte_mem_config {/* Heaps of Malloc */ struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ]; +/* next socket ID for external malloc heap */ +int next_socket_id; +/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it. */
lib/librte_eal/common/include/rte_malloc.h +19 -0
@@ -263,6 +263,25 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Creates a new empty malloc heap with a specified name. +* +* @note Heaps created via this call will automatically get assigned a unique +* socket ID, which can be found using ``rte_malloc_heap_get_socket()`` +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on successful creation +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* EEXIST - heap by name of ``heap_name`` already exists +* ENOSPC - no more space in internal config to store a new heap +*/ +int __rte_experimental +rte_malloc_heap_create (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +37 -0
@@ -29,6 +29,10 @@#include "malloc_heap.h" #include "malloc_mp.h" +/* start external socket ID's at a very high number */ +#define CONST_MAX (a, b) (a > b ? a : b)/* RTE_MAX is not a constant */ +#define EXTERNAL_HEAP_MIN_SOCKET_ID (CONST_MAX((1 << 8), RTE_MAX_NUMA_NODES)) +static unsigned check_hugepage_sz (unsigned flags,uint64_t hugepage_sz) {
@@ -1019,6 +1023,36 @@ malloc_heap_dump(struct malloc_heap *heap, FILE *f)rte_spinlock_unlock (&heap->lock); } +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +uint32_t next_socket_id = mcfg->next_socket_id; + +/* prevent overflow. did you really create 2 billion heaps??? */ +if (next_socket_id >INT32_MAX ) { +RTE_LOG (ERR ,EAL ,"Cannot assign new socket ID's\n" ); + rte_errno =ENOSPC ; +return -1 ; + } + +/* initialize empty heap */ + heap->alloc_count =0 ; + heap->first =NULL ; + heap->last =NULL ; +LIST_INIT (heap->free_head); +rte_spinlock_init (&heap->lock); + heap->total_size =0 ; + heap->socket_id = next_socket_id; + +/* we hold a global mem hotplug writelock, so it's safe to increment */ + mcfg->next_socket_id++; + +/* set up name */ +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
@@ -1026,6 +1060,9 @@ rte_eal_malloc_heap_init(void)unsigned int i;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign min socket ID to external heaps */ + mcfg->next_socket_id =EXTERNAL_HEAP_MIN_SOCKET_ID ; +/* assign names to default DPDK heaps */ for (i =0 ; i <rte_socket_count (); i++) {struct malloc_heap *heap = &mcfg->malloc_heaps[i];
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -33,6 +33,9 @@ void * malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags, size_t align, bool contig); +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +52 -0
61 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_create; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v7 12/21 malloc: allow destroying heaps
In reply to Anatoly BurakovAdd an API to destroy specified heap.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 23 +++++++++ lib/librte_eal/common/malloc_heap.c | 22 ++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 58 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 107 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +23 -0
@@ -282,6 +282,29 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_create (const char *heap_name); +/** +* Destroys a previously created malloc heap with specified name. +* +* @note This function will return a failure result if not all memory allocated +* from the heap has been freed back to the heap +* +* @note This function will return a failure result if not all memory segments +* were removed from the heap prior to its destruction +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* ENOENT - heap by the name of ``heap_name`` was not found +* EPERM - attempting to destroy reserved heap +* EBUSY - heap still contains data +*/ +int __rte_experimental +rte_malloc_heap_destroy (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +22 -0
@@ -1053,6 +1053,28 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name)return 0 ; } +int +malloc_heap_destroy (struct malloc_heap *heap) +{ +if (heap->alloc_count !=0 ) { +RTE_LOG (ERR ,EAL ,"Heap is still in use\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->first !=NULL || heap->last !=NULL ) { +RTE_LOG (ERR ,EAL ,"Heap still contains memory segments\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->total_size !=0 ) +RTE_LOG (ERR ,EAL ,"Total size not zero, heap is likely corrupt\n" ); + +/* after this, the lock will be dropped */ +memset (heap,0 ,sizeof (*heap)); + +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -36,6 +36,9 @@ malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags,int malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int +malloc_heap_destroy (struct malloc_heap *heap); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +58 -0
67 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_create; + rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v7 13/21 malloc: allow adding memory to named heaps
In reply to Anatoly BurakovAdd an API to add externally allocated memory to malloc heap. The memory will be stored in memseg lists like regular DPDK memory. Multiple segments are allowed within a heap. If IOVA table is not provided, IOVA addresses are filled in with RTE_BAD_IOVA.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 39 ++++++++++++ lib/librte_eal/common/malloc_heap.c | 74 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 51 +++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 169 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +39 -0
@@ -263,6 +263,45 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Add memory chunk to a heap with specified name. +* +* @note Multiple memory chunks can be added to the same heap +* +* @note Memory must be previously allocated for DPDK to be able to use it as a +* malloc heap. Failing to do so will result in undefined behavior, up to and +* including segmentation faults. +* +* @note Calling this function will erase any contents already present at the +* supplied memory address. +* +* @param heap_name +* Name of the heap to add memory chunk to +* @param va_addr +* Start of virtual area to add to the heap +* @param len +* Length of virtual area to add to the heap +* @param iova_addrs +* Array of page IOVA addresses corresponding to each page in this memory +* area. Can be NULL, in which case page IOVA addresses will be set to +* RTE_BAD_IOVA. +* @param n_pages +* Number of elements in the iova_addrs array. Ignored if ``iova_addrs`` +* is NULL. +* @param page_sz +* Page size of the underlying memory +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to add memory to a reserved heap +* ENOSPC - no more space in internal config to store a new memory chunk +*/ +int __rte_experimental +rte_malloc_heap_memory_add (const char *heap_name,void *va_addr,size_t len, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +74 -0
80 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -39,6 +39,10 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name);int malloc_heap_destroy (struct malloc_heap *heap); +int +malloc_heap_add_external_memory (struct malloc_heap *heap,void *va_addr, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +51 -0
57 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -321,6 +321,7 @@ EXPERIMENTAL { rte_malloc_heap_create; rte_malloc_heap_destroy; rte_malloc_heap_get_socket; + rte_malloc_heap_memory_add; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v7 14/21 malloc: allow removing memory from named heaps
In reply to Anatoly BurakovAdd an API to remove memory from specified heaps. This will first check if all elements within the region are free, and that the region is the original region that was added to the heap (by comparing its length to length of memory addressed by the underlying memseg list).
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++ lib/librte_eal/common/malloc_heap.c | 54 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 39 ++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 125 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -302,6 +302,33 @@ int __rte_experimental rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +/** +* Remove memory chunk from heap with specified name. +* +* @note Memory chunk being removed must be the same as one that was added; +* partially removing memory chunks is not supported +* +* @note Memory area must not contain any allocated elements to allow its +* removal from the heap +* +* @param heap_name +* Name of the heap to remove memory from +* @param va_addr +* Virtual address to remove from the heap +* @param len +* Length of virtual area to remove from the heap +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to remove memory from a reserved heap +* ENOENT - heap or memory chunk was not found +* EBUSY - memory chunk still contains data +*/ +int __rte_experimental +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +54 -0
66 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -43,6 +43,10 @@ int malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +int +malloc_heap_remove_external_memory (struct malloc_heap *heap,void *va_addr, +size_t len); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +39 -0
@@ -379,6 +379,45 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,return ret; } +int +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +int ret; + +if (heap_name ==NULL || va_addr ==NULL || len ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_write_lock (&mcfg->memory_hotplug_lock); +/* find our heap */ + heap =find_named_heap (heap_name); +if (heap ==NULL ) { + rte_errno =ENOENT ; + ret =-1 ; + goto unlock; + } +if (heap->socket_id <RTE_MAX_NUMA_NODES ) { +/* cannot remove memory from internal heaps */ + rte_errno =EPERM ; + ret =-1 ; + goto unlock; + } + +rte_spinlock_lock (&heap->lock); + ret =malloc_heap_remove_external_memory (heap, va_addr, len); +rte_spinlock_unlock (&heap->lock); + +unlock: +rte_rwlock_write_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v7 15/21 malloc: allow attaching to external memory chunks
In reply to Anatoly BurakovIn order to use external memory in multiple processes, we need to attach to primary process's memseg lists, so add a new API to do that. It is the responsibility of the user to ensure that memory is accessible and that it has been previously added to the malloc heap by another process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 28 ++++++++ lib/librte_eal/common/rte_malloc.c | 83 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 112 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +28 -0
@@ -268,6 +268,10 @@ rte_malloc_get_socket_stats(int socket,* * @note Multiple memory chunks can be added to the same heap * +* @note Before accessing this memory in other processes, it needs to be +* attached in each of those processes by calling +* ``rte_malloc_heap_memory_attach`` in each other process. +* * @note Memory must be previously allocated for DPDK to be able to use it as a * malloc heap. Failing to do so will result in undefined behavior, up to and * including segmentation faults.
@@ -329,6 +333,30 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, int __rte_experimentalrte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** +* Attach to an already existing chunk of external memory in another process. +* +* @note This function must be called before any attempt is made to use an +* already existing external memory chunk. This function does *not* need to +* be called if a call to ``rte_malloc_heap_memory_add`` was made in the +* current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful attach +* -1 on unsuccessful attach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to attach memory to a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +83 -0
89 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_attach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v7 16/21 malloc: allow detaching from external memory
In reply to Anatoly BurakovAdd API to detach from existing chunk of external memory in a process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++++++++++ lib/librte_eal/common/rte_malloc.c | 31 +++++++++++++++++----- lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+), 7 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -315,6 +315,9 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,* @note Memory area must not contain any allocated elements to allow its * removal from the heap * +* @note All other processes must detach from the memory chunk prior to it being +* removed from the heap. +* * @param heap_name * Name of the heap to remove memory from * @param va_addr
@@ -357,6 +360,30 @@ rte_malloc_heap_memory_remove(const char *heap_name, void *va_addr, size_t len); int __rte_experimentalrte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** +* Detach from a chunk of external memory in secondary process. +* +* @note This function must be called in before any attempt is made to remove +* external memory from the heap in another process. This function does *not* +* need to be called if a call to ``rte_malloc_heap_memory_remove`` will be +* called in current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful detach +* -1 on unsuccessful detach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to detach memory from a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +24 -7
@@ -422,10 +422,11 @@ struct sync_mem_walk_arg {void *va_addr;size_t len;int result; +bool attach; };static int -attach_mem_walk (const struct rte_memseg_list *msl,void *arg) +sync_mem_walk (const struct rte_memseg_list *msl,void *arg) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct sync_mem_walk_arg *wa = arg;
@@ -440,7 +441,10 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; - ret =rte_fbarray_attach (&found_msl->memseg_arr); +if (wa->attach) + ret =rte_fbarray_attach (&found_msl->memseg_arr); +else + ret =rte_fbarray_detach (&found_msl->memseg_arr);if (ret <0 ) wa->result = -rte_errno;
@@ -451,8 +455,8 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg)return 0 ; } -int -rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +static int +sync_memory (const char *heap_name,void *va_addr,size_t len,bool attach) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct malloc_heap *heap =NULL ;
@@ -475,20 +479,21 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len) ret =-1 ; goto unlock; } -/* we shouldn't be able to attach to internal heaps */ +/* we shouldn't be able to sync to internal heaps */ if (heap->socket_id <RTE_MAX_NUMA_NODES ) { rte_errno =EPERM ; ret =-1 ; goto unlock; } -/* find corresponding memseg list to attach to */ +/* find corresponding memseg list to sync to */ wa.va_addr = va_addr; wa.len = len; wa.result = -ENOENT ;/* fail unless explicitly told to succeed */ + wa.attach = attach;/* we're already holding a read lock */ -rte_memseg_list_walk_thread_unsafe (attach_mem_walk, &wa); +rte_memseg_list_walk_thread_unsafe (sync_mem_walk, &wa);if (wa.result <0 ) { rte_errno = -wa.result;
@@ -501,6 +506,18 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len)return ret; } +int +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, true); +} + +int +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, false); +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -323,6 +323,7 @@ EXPERIMENTAL { rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; rte_malloc_heap_memory_attach; + rte_malloc_heap_memory_detach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v7 17/21 malloc: enable event callbacks for external memory
In reply to Anatoly BurakovWhen adding or removing external memory from the memory map, there may be actions that need to be taken on account of this memory (e.g. DMA mapping). Add support for triggering callbacks when adding, removing, attaching or detaching external memory.
Some memory event callback handlers will need additional logic to handle external memory regions. For example, virtio callback has to completely ignore externally allocated memory, because there is no way to find file descriptors backing the memory address in a generic fashion. All other callbacks have also been adjusted to handle RTE_BAD_IOVA as IOVA address, as this is one of the expected use cases for external memory support.
Signed-off-by Anatoly Burakov <[email protected]>
--- drivers/bus/fslmc/fslmc_vfio.c | 7 +++++ .../net/virtio/virtio_user/virtio_user_dev.c | 8 ++++++ lib/librte_eal/common/malloc_heap.c | 7 +++++ lib/librte_eal/common/rte_malloc.c | 27 ++++++++++++++++--- lib/librte_eal/linuxapp/eal/eal_vfio.c | 10 +++++-- 5 files changed, 53 insertions(+), 6 deletions(-)
drivers/bus/fslmc/fslmc_vfio.c +7 -0
@@ -221,6 +221,13 @@ fslmc_memevent_cb(enum rte_mem_event type, const void *addr, size_t len, "alloc" : "dealloc", va, virt_addr, iova_addr, map_len); +/* iova_addr may be set to RTE_BAD_IOVA */ +if (iova_addr ==RTE_BAD_IOVA ) { +DPAA2_BUS_DEBUG ("Segment has invalid iova, skipping\n" ); + cur_len += map_len; +continue ; + } +if (type ==RTE_MEM_EVENT_ALLOC ) ret =fslmc_map_dma (virt_addr, iova_addr, map_len);else
drivers/net/virtio/virtio_user/virtio_user_dev.c +8 -0
@@ -13,6 +13,8 @@#include <sys/types.h> #include <sys/stat.h> +#include <rte_eal_memconfig.h> +#include "vhost.h" #include "virtio_user_dev.h" #include "../virtio_ethdev.h"
@@ -282,8 +284,14 @@ virtio_user_mem_event_cb(enum rte_mem_event type __rte_unused, void *arg) {struct virtio_user_dev *dev = arg; +struct rte_memseg_list *msl;uint16_t i; +/* ignore externally allocated memory */ + msl =rte_mem_virt2memseg_list (addr); +if (msl->external) +return ; +pthread_mutex_lock (&dev->mutex);if (dev->started == false)
lib/librte_eal/common/malloc_heap.c +7 -0
@@ -1031,6 +1031,9 @@ destroy_seg(struct malloc_elem *elem, size_t len) msl = elem->msl; +/* notify all subscribers that a memory area is going to be removed */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , elem, len); +/* this element can be removed */ malloc_elem_free_list_remove (elem);malloc_elem_hide_region (elem, elem, len);
@@ -1120,6 +1123,10 @@ malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr,RTE_LOG (DEBUG ,EAL ,"Added segment for heap %s starting at %p\n" , heap->name, va_addr); +/* notify all subscribers that a new memory area has been added */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, seg_len); +return 0 ; }
lib/librte_eal/common/rte_malloc.c +23 -4
@@ -25,6 +25,7 @@#include <rte_malloc.h> #include "malloc_elem.h" #include "malloc_heap.h" +#include "eal_memalloc.h" /* Free the memory space back to heap */
@@ -441,15 +442,29 @@ sync_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; -if (wa->attach) +if (wa->attach) { ret =rte_fbarray_attach (&found_msl->memseg_arr); -else + }else { +/* notify all subscribers that a memory area is about to +* be removed +*/ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , + msl->base_va, msl->len); ret =rte_fbarray_detach (&found_msl->memseg_arr); + } -if (ret <0 ) +if (ret <0 ) { wa->result = -rte_errno; -else + }else { +/* notify all subscribers that a new memory area was +* added +*/ +if (wa->attach) +eal_memalloc_mem_event_notify ( +RTE_MEM_EVENT_ALLOC , + msl->base_va, msl->len); wa->result =0 ; + }return 1 ; }return 0 ;
@@ -499,6 +514,10 @@ sync_memory(const char *heap_name, void *va_addr, size_t len, bool attach) rte_errno = -wa.result; ret =-1 ; }else { +/* notify all subscribers that a new memory area was added */ +if (attach) +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, len); ret =0 ; } unlock:
lib/librte_eal/linuxapp/eal/eal_vfio.c +8 -2
@@ -509,7 +509,7 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, msl =rte_mem_virt2memseg_list (addr);/* for IOVA as VA mode, no need to care for IOVA addresses */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) { +if (rte_eal_iova_mode () ==RTE_IOVA_VA && msl->external ==0 ) {uint64_t vfio_va = (uint64_t )(uintptr_t )addr;if (type ==RTE_MEM_EVENT_ALLOC ) vfio_dma_mem_map(default_vfio_cfg, vfio_va, vfio_va,
@@ -523,13 +523,19 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,/* memsegs are contiguous in memory */ ms =rte_mem_virt2memseg (addr, msl);while (cur_len < len) { +/* some memory segments may have invalid IOVA */ +if (ms->iova ==RTE_BAD_IOVA ) { +RTE_LOG (DEBUG ,EAL ,"Memory segment at %p has bad IOVA, skipping\n" , + ms->addr); + goto next; + }if (type ==RTE_MEM_EVENT_ALLOC )vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,1 );else vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,0 ); - +next: cur_len += ms->len; ++ms; }
Signature · 1 line
2.17.1
PATCH v7 18/21 test: add unit tests for external memory support
In reply to Anatoly BurakovAdd simple unit tests to test external memory support. The tests are pretty basic and mostly consist of checking if invalid API calls are handled correctly, plus a simple allocation/deallocation test for malloc and memzone.
Signed-off-by Anatoly Burakov <[email protected]>
--- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++++++++++++++++++++ 4 files changed, 401 insertions(+), 4 deletions(-) create mode 100644 test/test/test_external_mem.c
test/test/Makefile +1 -0
@@ -71,6 +71,7 @@ SRCS-y += test_bitmap.c SRCS-y += test_reciprocal_division.c SRCS-y += test_reciprocal_division_perf.c SRCS-y += test_fbarray.c +SRCS-y += test_external_mem.c SRCS-y += test_ring.c SRCS-y += test_ring_perf.c
test/test/autotest_data.py +10 -4
@@ -477,10 +477,16 @@ "Report": None, }, { - "Name": "Fbarray autotest", - "Command": "fbarray_autotest", - "Func": default_autotest, - "Report": None, + "Name": "Fbarray autotest", + "Command": "fbarray_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "External memory autotest", + "Command": "external_mem_autotest", + "Func": default_autotest, + "Report": None, }, # #Please always keep all dump tests at the end and together!
test/test/meson.build +1 -0
@@ -155,6 +155,7 @@ test_names = [ 'eventdev_common_autotest', 'eventdev_octeontx_autotest', 'eventdev_sw_autotest', + 'external_mem_autotest', 'func_reentrancy_autotest', 'flow_classify_autotest', 'hash_scaling_autotest',
test/test/test_external_mem.c new file +389 -0
389 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v7 19/21 app/testpmd: add support for external memory
In reply to Anatoly BurakovCurrently, mempools can only be allocated either using native DPDK memory, or anonymous memory. This patch will add two new methods to allocate mempool using external memory (regular or hugepage memory), and add documentation about it to testpmd user guide.
It adds a new flag "--mp-alloc", with four possible values:
native (use regular DPDK allocator), anon (use anonymous mempool), xmem (use externally allocated memory area), and xmemhuge (use externally allocated hugepage memory area). Old flag "--mp-anon" is kept for compatibility.
All external memory is allocated using the same external heap, but each will allocate and add a new memory area.
Signed-off-by Anatoly Burakov <[email protected]>
Suggested-by Konstantin Ananyev <[email protected]>
--- app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 318 ++++++++++++++++++++++++-- app/test-pmd/testpmd.h | 13 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + 5 files changed, 362 insertions(+), 25 deletions(-)
app/test-pmd/config.c +19 -2
@@ -2413,6 +2413,23 @@ fwd_config_setup(void)simple_fwd_config_setup (); } +static const char * +mp_alloc_to_str (uint8_t mode) +{ +switch (mode) { +case MP_ALLOC_NATIVE : +return "native" ; +case MP_ALLOC_ANON : +return "anon" ; +case MP_ALLOC_XMEM : +return "xmem" ; +case MP_ALLOC_XMEM_HUGE : +return "xmemhuge" ; +default : +return "invalid" ; + } +} +void pkt_fwd_config_display (struct fwd_config *cfg) {
@@ -2421,12 +2438,12 @@ pkt_fwd_config_display(struct fwd_config *cfg)streamid_t sm_id;printf ("%s packet forwarding%s - ports=%d - cores=%d - streams=%d - " -"NUMA support %s, MP over anonymous pages %s\n" , +"NUMA support %s, MP allocation mode: %s\n" , cfg->fwd_eng->fwd_mode_name, retry_enabled ==0 ?"" :" with retry" , cfg->nb_fwd_ports, cfg->nb_fwd_lcores, cfg->nb_fwd_streams, numa_support ==1 ?"enabled" :"disabled" , - mp_anon !=0 ?"enabled" :"disabled" ); +mp_alloc_to_str (mp_alloc_type));if (retry_enabled) printf("TX retry num: %u, delay between TX retries: %uus\n",
app/test-pmd/parameters.c +22 -1
@@ -190,6 +190,11 @@ usage(char* progname)printf (" --vxlan-gpe-port=N: UPD port of tunnel VXLAN-GPE\n" );printf (" --mlockall: lock all memory\n" );printf (" --no-mlockall: do not lock all memory\n" ); +printf (" --mp-alloc <native|anon|xmem|xmemhuge>: mempool allocation method.\n" +" native: use regular DPDK memory to create and populate mempool\n" +" anon: use regular DPDK memory to create and anonymous memory to populate mempool\n" +" xmem: use anonymous memory to create and populate mempool\n" +" xmemhuge: use anonymous hugepage memory to create and populate mempool\n" ); }#ifdef RTE_LIBRTE_CMDLINE
@@ -625,6 +630,7 @@ launch_args_parse(int argc, char** argv) { "vxlan-gpe-port", 1, 0, 0 }, { "mlockall", 0, 0, 0 }, { "no-mlockall", 0, 0, 0 }, + { "mp-alloc", 1, 0, 0 }, { 0, 0, 0, 0 }, };
@@ -743,7 +749,22 @@ launch_args_parse(int argc, char** argv)if (!strcmp (lgopts[opt_idx].name,"numa" )) numa_support =1 ;if (!strcmp (lgopts[opt_idx].name,"mp-anon" )) { - mp_anon =1 ; + mp_alloc_type =MP_ALLOC_ANON ; + } +if (!strcmp (lgopts[opt_idx].name,"mp-alloc" )) { +if (!strcmp (optarg,"native" )) + mp_alloc_type =MP_ALLOC_NATIVE ; +else if (!strcmp (optarg,"anon" )) + mp_alloc_type =MP_ALLOC_ANON ; +else if (!strcmp (optarg,"xmem" )) + mp_alloc_type =MP_ALLOC_XMEM ; +else if (!strcmp (optarg,"xmemhuge" )) + mp_alloc_type =MP_ALLOC_XMEM_HUGE ; +else +rte_exit (EXIT_FAILURE , +"mp-alloc %s invalid - must be: " +"native, anon or xmem\n" , + optarg); }if (!strcmp (lgopts[opt_idx].name,"port-numa-config" )) {if (parse_portnuma_config (optarg))
app/test-pmd/testpmd.c +297 -21
351 lines not on this page · raw message
app/test-pmd/testpmd.h +12 -1
@@ -69,6 +69,16 @@ enum { PORT_TOPOLOGY_LOOP, }; +enum { +MP_ALLOC_NATIVE ,/**< allocate and populate mempool natively */ +MP_ALLOC_ANON , +/**< allocate mempool natively, but populate using anonymous memory */ +MP_ALLOC_XMEM , +/**< allocate and populate mempool using anonymous memory */ +MP_ALLOC_XMEM_HUGE +/**< allocate and populate mempool using anonymous hugepage memory */ +}; +#ifdef RTE_TEST_PMD_RECORD_BURST_STATS /** * The data structure associated with RX and TX packet burst statistics
@@ -304,7 +314,8 @@ extern uint8_t numa_support; /**< set by "--numa" parameter */extern uint16_t port_topology;/**< set by "--port-topology" parameter */ extern uint8_t no_flush_rx;/**<set by "--no-flush-rx" parameter */ extern uint8_t flow_isolate_all;/**< set by "--flow-isolate-all */ -extern uint8_t mp_anon;/**< set by "--mp-anon" parameter */ +extern uint8_t mp_alloc_type; +/**< set by "--mp-anon" or "--mp-alloc" parameter */ extern uint8_t no_link_check;/**<set by "--disable-link-check" parameter */ extern volatile int test_done;/* stop packet forwarding when set to 1. */ extern uint8_t lsc_interrupt;/**< disabled by "--no-lsc-interrupt" parameter */
doc/guides/testpmd_app_ug/run_app.rst +12 -0
@@ -498,3 +498,15 @@ The commandline options are: * ``--no-mlockall`` Disable locking all memory. + +* ``--mp-alloc <native|anon|xmem|xmemhuge>`` + + Select mempool allocation mode: + + * native: create and populate mempool using native DPDK memory + * anon: create mempool using native DPDK memory, but populate using + anonymous memory + * xmem: create and populate mempool using externally and anonymously + allocated area + * xmemhuge: create and populate mempool using externally and anonymously + allocated hugepage area
Signature · 1 line
2.17.1
PATCH v7 20/21 doc: add external memory feature to the release notes
In reply to Anatoly BurakovDocument the addition of external memory support to DPDK.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 5 +++++ 1 file changed, 5 insertions(+)
doc/guides/rel_notes/release_18_11.rst +5 -0
@@ -67,6 +67,11 @@ New Features SR-IOV option in Hyper-V and Azure. This is an alternative to the previous vdev_netvsc, tap, and failsafe drivers combination. +* **Added support for using externally allocated memory in DPDK.** + + DPDK has gained support for creating new ``rte_malloc`` heaps referencing + memory that was created outside of DPDK's own page allocator, and using that + memory natively with any other DPDK library or data structure. API Changes -----------
Signature · 1 line
2.17.1
PATCH v7 21/21 doc: add external memory feature to programmer's guide
In reply to Anatoly BurakovAdd a short chapter on usage of external memory in DPDK to the Programmer's Guide.
Signed-off-by Anatoly Burakov <[email protected]>
--- .../prog_guide/env_abstraction_layer.rst | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +37 -0
@@ -213,6 +213,43 @@ Normally, these options do not need to be changed. can later be mapped into that preallocated VA space (if dynamic memory mode is enabled), and can optionally be mapped into it at startup. +Support for Externally Allocated Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is possible to use externally allocated memory in DPDK, using a set of malloc +heap API's. Support for externally allocated memory is implemented through +overloading the socket ID - externally allocated heaps will have socket ID's +that would be considered invalid under normal circumstances. Requesting an +allocation to take place from a specified externally allocated memory is a +matter of supplying the correct socket ID to DPDK allocator, either directly +(e.g. through a call to ``rte_malloc``) or indirectly (through data +structure-specific allocation API's such as ``rte_ring_create``). + +Since there is no way DPDK can verify whether memory are is available or valid, +this responsibility falls on the shoulders of the user. All multiprocess +synchronization is also user's responsibility, as well as ensuring that all +calls to add/attach/detach/remove memory are done in the correct order. It is +not required to attach to a memory area in all processes - only attach to memory +areas as needed. + +The expected workflow is as follows: + +* Get a pointer to memory area +* Create a named heap +* Add memory area(s) to the heap + - If IOVA table is not specified, IOVA addresses will be assumed to be + unavailable, and DMA mappings will not be performed + - Other processes must attach to the memory area before they can use it +* Get socket ID used for the heap +* Use normal DPDK allocation procedures, using supplied socket ID +* If memory area is no longer needed, it can be removed from the heap + - Other processes must detach from this memory area before it can be removed +* If heap is no longer needed, remove it + - Socket ID will become invalid and will not be reused + +For more information, please refer to ``rte_malloc`` API documentation, +specifically the ``rte_malloc_heap_*`` family of function calls. + PCI Access ~~~~~~~~~~
Signature · 1 line
2.17.1
v8 cover Support externally allocated memory in DPDK
In reply to Anatoly BurakovThis is a proposal to enable using externally allocated memory in DPDK.
In a nutshell, here is what is being done here:
- Index internal malloc heaps by NUMA node index, rather than NUMA node itself (external heaps will have ID's in order of creation)
- Add identifier string to malloc heap, to uniquely identify it
- Each new heap will receive a unique socket ID that will be used by allocator to decide from which heap (internal or external) to allocate requested amount of memory
- Allow creating named heaps and add/remove memory to/from those heaps
- Allocate memseg lists at runtime, to keep track of IOVA addresses of externally allocated memory
- If IOVA addresses aren't provided, use RTE_BAD_IOVA
- Allow malloc and memzones to allocate from external heaps
- Allow other data structures to allocate from externall heaps
The responsibility to ensure memory is accessible before using it is on the shoulders of the user - there is no checking done with regards to validity of the memory (nor could there be...).
The general approach is to create heap and add memory into it. For any other process wishing to use the same memory, said memory must first be attached (otherwise some things will not work).
A design decision was made to make multiprocess synchronization a manual process. Due to underlying issues with attaching to fbarrays in secondary processes, this design was deemed to be better because we don't want to fail to create external heap in the primary because something in the secondary has failed when in fact we may not eve have wanted this memory to be accessible in the secondary in the first place.
Using external memory in multiprocess is *hard*, because not only memory space needs to be preallocated, but it also needs to be attached in each process to allow other processes to access the page table. The attach API call may or may not succeed, depending on memory layout, for reasons similar to other multiprocess failures. This is treated as a "known issue" for this release.
v8 -> v7 changes:
- Rebase on latest master
- More documentation on ABI changes
v7 -> v6 changes:
- Fixed missing IOVA address setup in testpmd
- Fixed MLX drivers as per Yongseok's comments
- Added a check for invalid heap idx on adding memory to heap
v6 -> v5 changes:
- Fixed documentation formatting as per Marko's comments
v5 -> v4 changes:
- All processes are now able to create and destroy malloc heaps
- Memory is automatically mapped for DMA on adding it to heap
- Mem event callbacks are triggered on adding/removing memory
- Fixed compile issues on FreeBSD
- Better documentation on API/ABI changes
v4 -> v3 changes:
- Dropped sample application in favor of new testpmd flag
- Added new flag to testpmd, with four options of mempool allocation
- Added new API to check if a socket ID belongs to an external heap
- Adjusted malloc and mempool code to not make any assumptions about IOVA-contiguousness when dealing with externally allocated memory
v3 -> v2 changes:
- Rebase on top of latest master
- Clarifications added to mempool code as per Andrew Rynchenko's comments
v2 -> v1 changes:
- Fixed NULL dereference on heap socket ID lookup
- Fixed memseg offset calculation on adding memory to heap
- Improved unit test to test for above bugfixes
- Restricted heap creation to primary processes only
- Added sample application
- Added documentation
RFC -> v1 changes:
- Removed the "named heaps" API, allocate using fake socket ID instead
- Added multiprocess support
- Everything is now thread-safe
- Numerous bugfixes and API improvements
Anatoly Burakov (21):
mem: add length to memseg list
mem: allow memseg lists to be marked as external
malloc: index heaps using heap ID rather than NUMA node
mem: do not check for invalid socket ID
flow_classify: do not check for invalid socket ID
pipeline: do not check for invalid socket ID
sched: do not check for invalid socket ID
malloc: add name to malloc heaps
malloc: add function to query socket ID of named heap
malloc: add function to check if socket is external
malloc: allow creating malloc heaps
malloc: allow destroying heaps
malloc: allow adding memory to named heaps
malloc: allow removing memory from named heaps
malloc: allow attaching to external memory chunks
malloc: allow detaching from external memory
malloc: enable event callbacks for external memory
test: add unit tests for external memory support app/testpmd: add support for external memory
doc: add external memory feature to the release notes
doc: add external memory feature to programmer's guide
app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 320 ++++++++++++- app/test-pmd/testpmd.h | 13 +- config/common_base | 1 + config/rte_config.h | 1 + .../prog_guide/env_abstraction_layer.rst | 37 ++ doc/guides/rel_notes/deprecation.rst | 15 - doc/guides/rel_notes/release_18_11.rst | 37 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + drivers/bus/fslmc/fslmc_vfio.c | 13 +- drivers/bus/pci/linux/pci.c | 2 +- drivers/net/mlx5/mlx5.c | 4 +- drivers/net/virtio/virtio_user/vhost_kernel.c | 3 + .../net/virtio/virtio_user/virtio_user_dev.c | 6 + lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal.c | 3 + lib/librte_eal/bsdapp/eal/eal_memory.c | 9 +- lib/librte_eal/common/eal_common_memory.c | 8 +- lib/librte_eal/common/eal_common_memzone.c | 8 +- .../common/include/rte_eal_memconfig.h | 9 +- lib/librte_eal/common/include/rte_malloc.h | 192 ++++++++ .../common/include/rte_malloc_heap.h | 3 + lib/librte_eal/common/include/rte_memory.h | 9 + lib/librte_eal/common/malloc_elem.c | 10 +- lib/librte_eal/common/malloc_heap.c | 320 +++++++++++-- lib/librte_eal/common/malloc_heap.h | 17 + lib/librte_eal/common/rte_malloc.c | 429 +++++++++++++++++- lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 12 +- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +- lib/librte_eal/linuxapp/eal/eal_vfio.c | 27 +- lib/librte_eal/meson.build | 2 +- lib/librte_eal/rte_eal_version.map | 8 + lib/librte_flow_classify/rte_flow_classify.c | 3 +- lib/librte_mempool/rte_mempool.c | 57 ++- lib/librte_pipeline/rte_pipeline.c | 3 +- lib/librte_sched/rte_sched.c | 2 +- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++ test/test/test_malloc.c | 3 + test/test/test_memzone.c | 3 + 45 files changed, 1930 insertions(+), 138 deletions(-) create mode 100644 test/test/test_external_mem.c
Signature · 1 line
2.17.1
PATCH v8 1/21 mem: add length to memseg list
Previously, to calculate length of memory area covered by a memseg list, we would've needed to multiply page size by length of fbarray backing that memseg list. This is not obvious and unnecessarily low level, so store length in the memseg list itself.
This breaks ABI, so bump the EAL ABI version and document the change.
Signed-off-by Anatoly Burakov <[email protected]>
Acked-by Shreyansh Jain <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 8 +++++++- drivers/bus/pci/linux/pci.c | 2 +- lib/librte_eal/bsdapp/eal/Makefile | 2 +- lib/librte_eal/bsdapp/eal/eal_memory.c | 2 ++ lib/librte_eal/common/eal_common_memory.c | 5 ++--- lib/librte_eal/common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/linuxapp/eal/Makefile | 2 +- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 3 ++- lib/librte_eal/linuxapp/eal/eal_memory.c | 4 +++- lib/librte_eal/meson.build | 2 +- 10 files changed, 21 insertions(+), 10 deletions(-)
doc/guides/rel_notes/release_18_11.rst +7 -1
@@ -134,6 +134,12 @@ ABI Changes ========================================================= +* eal: EAL library ABI version was changed due to previously announced work on + supporting external memory in DPDK: + - structure ``rte_memseg_list`` now has a new field indicating length + of memory addressed by the segment list + + Removed Items -------------
@@ -179,7 +185,7 @@ The libraries prepended with a plus sign were incremented in this version. librte_compressdev.so.1 librte_cryptodev.so.5 librte_distributor.so.1 - librte_eal.so.8 + + librte_eal.so.9 librte_ethdev.so.10 librte_eventdev.so.4 librte_flow_classify.so.1
drivers/bus/pci/linux/pci.c +1 -1
@@ -119,7 +119,7 @@ rte_pci_unmap_device(struct rte_pci_device *dev)static int find_max_end_va (const struct rte_memseg_list *msl,void *arg) { -size_t sz = msl->memseg_arr.len * msl->page_sz; +size_t sz = msl->len;void *end_va =RTE_PTR_ADD (msl->base_va, sz);void **max_va = arg;
lib/librte_eal/bsdapp/eal/Makefile +1 -1
@@ -22,7 +22,7 @@ LDLIBS += -lrte_kvargs EXPORT_MAP := ../../rte_eal_version.map -LIBABIVER := 8 +LIBABIVER := 9 # specific to bsdapp exec-env SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) := eal.c
lib/librte_eal/bsdapp/eal/eal_memory.c +2 -0
@@ -79,6 +79,7 @@ rte_eal_hugepage_init(void) } msl->base_va = addr; msl->page_sz = page_sz; + msl->len = internal_config.memory; msl->socket_id =0 ;/* populate memsegs. each memseg is 1 page long */
@@ -370,6 +371,7 @@ alloc_va_space(struct rte_memseg_list *msl)return -1 ; } msl->base_va = addr; + msl->len = mem_sz;return 0 ; }
lib/librte_eal/common/eal_common_memory.c +2 -3
@@ -171,7 +171,7 @@ virt2memseg(const void *addr, const struct rte_memseg_list *msl)/* a memseg list was specified, check if it's the right one */ start = msl->base_va; - end =RTE_PTR_ADD (start, (size_t )msl->page_sz * msl->memseg_arr.len); + end =RTE_PTR_ADD (start, msl->len);if (addr < start || addr >= end)return NULL ;
@@ -194,8 +194,7 @@ virt2memseg_list(const void *addr) msl = &mcfg->memsegs[msl_idx]; start = msl->base_va; - end =RTE_PTR_ADD (start, - (size_t )msl->page_sz * msl->memseg_arr.len); + end =RTE_PTR_ADD (start, msl->len);if (addr >= start && addr < end)break ; }
lib/librte_eal/common/include/rte_eal_memconfig.h +1 -0
@@ -30,6 +30,7 @@ struct rte_memseg_list {uint64_t addr_64;/**< Makes sure addr is always 64-bits */ }; +size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ volatile uint32_t version;/**< version number for multiprocess sync. */
lib/librte_eal/linuxapp/eal/Makefile +1 -1
@@ -10,7 +10,7 @@ ARCH_DIR ?= $(RTE_ARCH) EXPORT_MAP := ../../rte_eal_version.map VPATH += $(RTE_SDK)/lib/librte_eal/common/arch/$(ARCH_DIR) -LIBABIVER := 8 +LIBABIVER := 9 VPATH += $(RTE_SDK)/lib/librte_eal/common
lib/librte_eal/linuxapp/eal/eal_memalloc.c +2 -1
@@ -986,7 +986,7 @@ free_seg_walk(const struct rte_memseg_list *msl, void *arg)int msl_idx, seg_idx, ret, dir_fd =-1 ; start_addr = (uintptr_t ) msl->base_va; - end_addr = start_addr + msl->memseg_arr.len * (size_t )msl->page_sz; + end_addr = start_addr + msl->len;if ((uintptr_t )wa->ms->addr < start_addr || (uintptr_t )wa->ms->addr >= end_addr)
@@ -1472,6 +1472,7 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,return -1 ; } local_msl->base_va = primary_msl->base_va; + local_msl->len = primary_msl->len;return 0 ; }
lib/librte_eal/linuxapp/eal/eal_memory.c +3 -1
@@ -861,6 +861,7 @@ alloc_va_space(struct rte_memseg_list *msl)return -1 ; } msl->base_va = addr; + msl->len = mem_sz;return 0 ; }
@@ -1369,6 +1370,7 @@ eal_legacy_hugepage_init(void) msl->base_va = addr; msl->page_sz = page_sz; msl->socket_id =0 ; + msl->len = internal_config.memory;/* populate memsegs. each memseg is one page long */ for (cur_seg =0 ; cur_seg < n_segs; cur_seg++) {
@@ -1615,7 +1617,7 @@ eal_legacy_hugepage_init(void)if (msl->memseg_arr.count >0 )continue ;/* this is an unused list, deallocate it */ - mem_sz = (size_t )msl->page_sz * msl->memseg_arr.len; + mem_sz = msl->len;munmap (msl->base_va, mem_sz); msl->base_va =NULL ;
lib/librte_eal/meson.build +1 -1
@@ -21,7 +21,7 @@ else error('unsupported system type "@0@"'.format(host_machine.system())) endif -version = 8 # the version of the EAL API +version = 9 # the version of the EAL API allow_experimental_apis = true deps += 'compat' deps += 'kvargs'
Signature · 1 line
2.17.1
Re: [PATCH v8 01/21] mem: add length to memseg list
lib/librte_eal/common/include/rte_eal_memconfig.h struct rte_memseg_list { · line 36
Quoted 12 lines · On Mon, 1 Oct 2018 13:56:09 +0100 Anatoly Burakov <[email protected]>…
lib/librte_eal/common/include/rte_eal_memconfig.h +1
-0@@ -30,6 +30,7 @@ struct rte_memseg_list {uint64_t addr_64;/**< Makes sure addr is always 64-bits */ }; +size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ volatile uint32_t version;/**< version number for multiprocess sync. */
If you are going to break ABI, why not try and rearrange to eliminate holes:
Output of pahole (on x86 64 bit):
struct rte_memseg_list {
union {
void * base_va; /* 0 8 */
uint64_t addr_64; /* 0 8 */
}; /* 0 8 */
size_t len; /* 8 8 */
int socket_id; /* 16 4 */
/* XXX 4 bytes hole, try to pack */
uint64_t page_sz; /* 24 8 */
volatile uint32_t version; /* 32 4 */
/* XXX 4 bytes hole, try to pack */
struct rte_fbarray memseg_arr; /* 40 96 */
/* XXX last struct has 4 bytes of padding */
/* size: 136, cachelines: 3, members: 6 */
/* sum members: 128, holes: 2, sum holes: 8 */
/* paddings: 1, sum paddings: 4 */
/* last cacheline: 8 bytes */
};Re: [PATCH v8 01/21] mem: add length to memseg list
Quoted 37 lines · Stephen Hemminger
lib/librte_eal/common/include/rte_eal_memconfig.h struct rte_memseg_list { · line 36
Quoted 12 lines · On Mon, 1 Oct 2018 13:56:09 +0100 Anatoly Burakov <[email protected]>…
lib/librte_eal/common/include/rte_eal_memconfig.h +1
-0@@ -30,6 +30,7 @@ struct rte_memseg_list {uint64_t addr_64;/**< Makes sure addr is always 64-bits */ }; +size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ volatile uint32_t version;/**< version number for multiprocess sync. */ If you are going to break ABI, why not try and rearrange to eliminate holes:
Output of pahole (on x86 64 bit):
struct rte_memseg_list { union { void * base_va; /* 0 8 */ uint64_t addr_64; /* 0 8 */ }; /* 0 8 */ size_t len; /* 8 8 */ int socket_id; /* 16 4 */ /* XXX 4 bytes hole, try to pack */ uint64_t page_sz; /* 24 8 */ volatile uint32_t version; /* 32 4 */ /* XXX 4 bytes hole, try to pack */ struct rte_fbarray memseg_arr; /* 40 96 */
Quoted 8 lines
/* XXX last struct has 4 bytes of padding */ /* size: 136, cachelines: 3, members: 6 */ /* sum members: 128, holes: 2, sum holes: 8 */ /* paddings: 1, sum paddings: 4 */ /* last cacheline: 8 bytes */ };
Hi Stephen,
This data structure isn't performance-critical in any remote sense, but sure, I can do that.
Signature · 2 lines
Thanks, Anatoly
PATCH v8 2/21 mem: allow memseg lists to be marked as external
In reply to Anatoly BurakovWhen we allocate and use DPDK memory, we need to be able to differentiate between DPDK hugepage segments and segments that were made part of DPDK but are externally allocated. Add such a property to memseg lists.
This breaks the ABI, so document the change in release notes. This also breaks a few internal assumptions about memory contiguousness, so adjust malloc code in a few places.
All current calls for memseg walk functions were adjusted to ignore external segments where it made sense.
Mempools is a special case, because we may be asked to allocate a mempool on a specific socket, and we need to ignore all page sizes on other heaps or other sockets. Previously, this assumption of knowing all page sizes was not a problem, but it will be now, so we have to match socket ID with page size when calculating minimum page size for a mempool.
Signed-off-by Anatoly Burakov <[email protected]>
Acked-by Andrew Rybchenko <[email protected]>
Acked-by Yongseok Koh <[email protected]>
---
Notes:
v3:
- Add comment to explain the process of picking up minimum page sizes for mempool
v2:
- Add documentation changes and ABI break
v1:
- Adjust all calls to memseg walk functions to ignore external segments where it made sense to do so
doc/guides/rel_notes/deprecation.rst | 15 -------- doc/guides/rel_notes/release_18_11.rst | 9 ++++- drivers/bus/fslmc/fslmc_vfio.c | 6 +++- drivers/net/mlx5/mlx5.c | 4 ++- drivers/net/virtio/virtio_user/vhost_kernel.c | 3 ++ lib/librte_eal/bsdapp/eal/eal.c | 3 ++ lib/librte_eal/bsdapp/eal/eal_memory.c | 7 ++-- lib/librte_eal/common/eal_common_memory.c | 3 ++ .../common/include/rte_eal_memconfig.h | 1 + lib/librte_eal/common/include/rte_memory.h | 9 +++++ lib/librte_eal/common/malloc_elem.c | 10 ++++-- lib/librte_eal/common/malloc_heap.c | 9 +++-- lib/librte_eal/common/rte_malloc.c | 2 +- lib/librte_eal/linuxapp/eal/eal.c | 10 +++++- lib/librte_eal/linuxapp/eal/eal_memalloc.c | 9 +++++ lib/librte_eal/linuxapp/eal/eal_vfio.c | 17 ++++++--- lib/librte_mempool/rte_mempool.c | 35 ++++++++++++++----- test/test/test_malloc.c | 3 ++ test/test/test_memzone.c | 3 ++ 19 files changed, 119 insertions(+), 39 deletions(-)
doc/guides/rel_notes/deprecation.rst +0 -15
@@ -11,21 +11,6 @@ API and ABI deprecation notices are to be posted here. Deprecation Notices ------------------- -* eal: certain structures will change in EAL on account of upcoming external - memory support. Aside from internal changes leading to an ABI break, the - following externally visible changes will also be implemented: - - - ``rte_memseg_list`` will change to include a boolean flag indicating - whether a particular memseg list is externally allocated. This will have - implications for any users of memseg-walk-related functions, as they will - now have to skip externally allocated segments in most cases if the intent - is to only iterate over internal DPDK memory. - - ``socket_id`` parameter across the entire DPDK will gain additional meaning, - as some socket ID's will now be representing externally allocated memory. No - changes will be required for existing code as backwards compatibility will - be kept, and those who do not use this feature will not see these extra - socket ID's. - * eal: both declaring and identifying devices will be streamlined in v18.11. New functions will appear to query a specific port from buses, classes of device and device drivers. Device declaration will be made coherent with the
doc/guides/rel_notes/release_18_11.rst +8 -1
@@ -102,6 +102,12 @@ API Changes Also, make sure to start the actual text at the margin. ========================================================= +* eal: ``rte_memseg_list`` structure now has an additional flag indicating + whether the memseg list is externally allocated. This will have implications + for any users of memseg-walk-related functions, as they will now have to skip + externally allocated segments in most cases if the intent is to only iterate + over internal DPDK memory. + * mbuf: The ``__rte_mbuf_raw_free()`` and ``__rte_pktmbuf_prefree_seg()`` functions were deprecated since 17.05 and are replaced by ``rte_mbuf_raw_free()`` and ``rte_pktmbuf_prefree_seg()``.
@@ -118,7 +124,6 @@ API Changes To request keeping CRC, application should set ``DEV_RX_OFFLOAD_KEEP_CRC`` Rx offload. - ABI Changes -----------
@@ -138,6 +143,8 @@ ABI Changes supporting external memory in DPDK: - structure ``rte_memseg_list`` now has a new field indicating length of memory addressed by the segment list + - structure ``rte_memseg_list`` now has a new flag indicating whether + the memseg list refers to external memory Removed Items
drivers/bus/fslmc/fslmc_vfio.c +5 -1
@@ -318,11 +318,15 @@ fslmc_unmap_dma(uint64_t vaddr, uint64_t iovaddr __rte_unused, size_t len)static int fslmc_dmamap_seg(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +const struct rte_memseg *ms,void *arg) {int *n_segs = arg;int ret; +/* if IOVA address is invalid, skip */ +if (ms->iova ==RTE_BAD_IOVA ) +return 0 ; + ret =fslmc_map_dma (ms->addr_64, ms->iova, ms->len);if (ret) DPAA2_BUS_ERR("Unable to VFIO map (addr=%p, len=%zu)",
drivers/net/mlx5/mlx5.c +3 -1
@@ -568,11 +568,13 @@ static struct rte_pci_driver mlx5_driver;static void *uar_base;static int -find_lower_va_bound(const struct rte_memseg_list *msl __rte_unused, +find_lower_va_bound (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {void **addr = arg; +if (msl->external) +return 0 ;if (*addr ==NULL ) *addr = ms->addr;else
drivers/net/virtio/virtio_user/vhost_kernel.c +3 -0
@@ -78,6 +78,9 @@ add_memseg_list(const struct rte_memseg_list *msl, void *arg)void *start_addr;uint64_t len; +if (msl->external) +return 0 ; +if (vm->nregions >= max_regions)return -1 ;
lib/librte_eal/bsdapp/eal/eal.c +3 -0
@@ -502,6 +502,9 @@ check_socket(const struct rte_memseg_list *msl, void *arg) {int *socket_id = arg; +if (msl->external) +return 0 ; +if (msl->socket_id == *socket_id && msl->memseg_arr.count !=0 )return 1 ;
lib/librte_eal/bsdapp/eal/eal_memory.c +5 -2
@@ -236,12 +236,15 @@ struct attach_walk_args {int seg_idx; };static int -attach_segment(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +attach_segment (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {struct attach_walk_args *wa = arg;void *addr; +if (msl->external) +return 0 ; + addr =mmap (ms->addr, ms->len,PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_FIXED , wa->fd_hugepage, wa->seg_idx *EAL_PAGE_SIZE );
lib/librte_eal/common/eal_common_memory.c +3 -0
@@ -272,6 +272,9 @@ physmem_size(const struct rte_memseg_list *msl, void *arg) {uint64_t *total_len = arg; +if (msl->external) +return 0 ; + *total_len += msl->memseg_arr.count * msl->page_sz;return 0 ;
lib/librte_eal/common/include/rte_eal_memconfig.h +1 -0
@@ -33,6 +33,7 @@ struct rte_memseg_list {size_t len;/**< Length of memory area covered by this memseg list. */ int socket_id;/**< Socket ID for all memsegs in this list. */ uint64_t page_sz;/**< Page size for all memsegs in this list. */ +unsigned int external;/**< 1 if this list points to external memory */ volatile uint32_t version;/**< version number for multiprocess sync. */ struct rte_fbarray memseg_arr; };
lib/librte_eal/common/include/rte_memory.h +9 -0
@@ -215,6 +215,9 @@ typedef int (*rte_memseg_list_walk_t)(const struct rte_memseg_list *msl,* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
@@ -233,6 +236,9 @@ rte_memseg_walk(rte_memseg_walk_t func, void *arg);* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
@@ -251,6 +257,9 @@ rte_memseg_contig_walk(rte_memseg_contig_walk_t func, void *arg);* @note This function read-locks the memory hotplug subsystem, and thus cannot * be used within memory-related callback functions. * +* @note This function will also walk through externally allocated segments. It +* is up to the user to decide whether to skip through these segments. +* * @param func * Iterator function * @param arg
lib/librte_eal/common/malloc_elem.c +7 -3
@@ -39,10 +39,14 @@ malloc_elem_find_max_iova_contig(struct malloc_elem *elem, size_t align) contig_seg_start =RTE_PTR_ALIGN_CEIL (data_start, align);/* if we're in IOVA as VA mode, or if we're in legacy mode with -* hugepages, all elements are IOVA-contiguous. +* hugepages, all elements are IOVA-contiguous. however, we can only +* make these assumptions about internal memory - externally allocated +* segments have to be checked. */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA || - (internal_config.legacy_mem &&rte_eal_has_hugepages ())) +if (!elem->msl->external && + (rte_eal_iova_mode () ==RTE_IOVA_VA || + (internal_config.legacy_mem && +rte_eal_has_hugepages ())))return RTE_PTR_DIFF (data_end, contig_seg_start); cur_page =RTE_PTR_ALIGN_FLOOR (contig_seg_start, page_sz);
lib/librte_eal/common/malloc_heap.c +7 -2
@@ -95,6 +95,9 @@ malloc_add_seg(const struct rte_memseg_list *msl,struct malloc_heap *heap;int msl_idx; +if (msl->external) +return 0 ; + heap = &mcfg->malloc_heaps[msl->socket_id];/* msl is const, so find it */
@@ -754,8 +757,10 @@ malloc_heap_free(struct malloc_elem *elem)/* anything after this is a bonus */ ret =0 ; -/* ...of which we can't avail if we are in legacy mode */ -if (internal_config.legacy_mem) +/* ...of which we can't avail if we are in legacy mode, or if this is an +* externally allocated segment. +*/ +if (internal_config.legacy_mem || msl->external) goto free_unlock;/* check if we can free any memory back to the system */
lib/librte_eal/common/rte_malloc.c +1 -1
@@ -223,7 +223,7 @@ rte_malloc_virt2iova(const void *addr)if (elem ==NULL )return RTE_BAD_IOVA ; -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) +if (!elem->msl->external &&rte_eal_iova_mode () ==RTE_IOVA_VA )return (uintptr_t ) addr; ms =rte_mem_virt2memseg (addr, elem->msl);
lib/librte_eal/linuxapp/eal/eal.c +9 -1
@@ -725,6 +725,9 @@ check_socket(const struct rte_memseg_list *msl, void *arg) {int *socket_id = arg; +if (msl->external) +return 0 ; +return *socket_id == msl->socket_id; }
@@ -1059,7 +1062,12 @@ mark_freeable(const struct rte_memseg_list *msl, const struct rte_memseg *ms, void *arg __rte_unused) {/* ms is const, so find this memseg */ -struct rte_memseg *found =rte_mem_virt2memseg (ms->addr, msl); +struct rte_memseg *found; + +if (msl->external) +return 0 ; + + found =rte_mem_virt2memseg (ms->addr, msl); found->flags &= ~RTE_MEMSEG_FLAG_DO_NOT_FREE ;
lib/librte_eal/linuxapp/eal/eal_memalloc.c +9 -0
@@ -1408,6 +1408,9 @@ sync_walk(const struct rte_memseg_list *msl, void *arg __rte_unused)unsigned int i;int msl_idx; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; primary_msl = &mcfg->memsegs[msl_idx]; local_msl = &local_memsegs[msl_idx];
@@ -1456,6 +1459,9 @@ secondary_msl_create_walk(const struct rte_memseg_list *msl,char name[PATH_MAX ];int msl_idx, ret; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; primary_msl = &mcfg->memsegs[msl_idx]; local_msl = &local_memsegs[msl_idx];
@@ -1509,6 +1515,9 @@ fd_list_create_walk(const struct rte_memseg_list *msl,unsigned int len;int msl_idx; +if (msl->external) +return 0 ; + msl_idx = msl - mcfg->memsegs; len = msl->memseg_arr.len;
lib/librte_eal/linuxapp/eal/eal_vfio.c +13 -4
@@ -1082,11 +1082,14 @@ rte_vfio_get_group_num(const char *sysfs_base, }static int -type1_map(const struct rte_memseg_list *msl __rte_unused, -const struct rte_memseg *ms,void *arg) +type1_map (const struct rte_memseg_list *msl,const struct rte_memseg *ms, +void *arg) {int *vfio_container_fd = arg; +if (msl->external) +return 0 ; +return vfio_type1_dma_mem_map (*vfio_container_fd, ms->addr_64, ms->iova, ms->len,1 ); }
@@ -1196,11 +1199,14 @@ vfio_spapr_dma_do_map(int vfio_container_fd, uint64_t vaddr, uint64_t iova, }static int -vfio_spapr_map_walk(const struct rte_memseg_list *msl __rte_unused, +vfio_spapr_map_walk (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {int *vfio_container_fd = arg; +if (msl->external) +return 0 ; +return vfio_spapr_dma_mem_map (*vfio_container_fd, ms->addr_64, ms->iova, ms->len,1 ); }
@@ -1210,12 +1216,15 @@ struct spapr_walk_param {uint64_t hugepage_sz; };static int -vfio_spapr_window_size_walk(const struct rte_memseg_list *msl __rte_unused, +vfio_spapr_window_size_walk (const struct rte_memseg_list *msl,const struct rte_memseg *ms,void *arg) {struct spapr_walk_param *param = arg;uint64_t max = ms->iova + ms->len; +if (msl->external) +return 0 ; +if (max > param->window_size) { param->hugepage_sz = ms->hugepage_sz; param->window_size = max;
lib/librte_mempool/rte_mempool.c +27 -8
@@ -99,25 +99,44 @@ static unsigned optimize_object_size(unsigned obj_size)return new_obj_size *RTE_MEMPOOL_ALIGN ; } +struct pagesz_walk_arg { +int socket_id; +size_t min; +}; +static int find_min_pagesz (const struct rte_memseg_list *msl,void *arg) { -size_t *min = arg; +struct pagesz_walk_arg *wa = arg; +bool valid; -if (msl->page_sz < *min) - *min = msl->page_sz; +/* +* we need to only look at page sizes available for a particular socket +* ID. so, we either need an exact match on socket ID (can match both +* native and external memory), or, if SOCKET_ID_ANY was specified as a +* socket ID argument, we must only look at native memory and ignore any +* page sizes associated with external memory. +*/ + valid = msl->socket_id == wa->socket_id; + valid |= wa->socket_id ==SOCKET_ID_ANY && msl->external ==0 ; + +if (valid && msl->page_sz < wa->min) + wa->min = msl->page_sz;return 0 ; }static size_t -get_min_page_size (void ) +get_min_page_size (int socket_id) { -size_t min_pagesz =SIZE_MAX ; +struct pagesz_walk_arg wa; -rte_memseg_list_walk (find_min_pagesz, &min_pagesz); + wa.min =SIZE_MAX ; + wa.socket_id = socket_id; -return min_pagesz ==SIZE_MAX ? (size_t )getpagesize () : min_pagesz; +rte_memseg_list_walk (find_min_pagesz, &wa); + +return wa.min ==SIZE_MAX ? (size_t )getpagesize () : wa.min; }
@@ -470,7 +489,7 @@ rte_mempool_populate_default(struct rte_mempool *mp) pg_sz =0 ; pg_shift =0 ; }else if (try_contig) { - pg_sz =get_min_page_size (); + pg_sz =get_min_page_size (mp->socket_id); pg_shift =rte_bsf32 (pg_sz); }else { pg_sz =getpagesize ();
test/test/test_malloc.c +3 -0
@@ -711,6 +711,9 @@ check_socket_mem(const struct rte_memseg_list *msl, void *arg) {int32_t *socket = arg; +if (msl->external) +return 0 ; +return *socket == msl->socket_id; }
test/test/test_memzone.c +3 -0
@@ -115,6 +115,9 @@ find_available_pagesz(const struct rte_memseg_list *msl, void *arg) {struct walk_arg *wa = arg; +if (msl->external) +return 0 ; +if (msl->page_sz ==RTE_PGSIZE_2M ) wa->hugepage_2MB_avail =1 ;if (msl->page_sz ==RTE_PGSIZE_1G )
Signature · 1 line
2.17.1
PATCH v8 3/21 malloc: index heaps using heap ID rather than NUMA node
In reply to Anatoly BurakovSwitch over all parts of EAL to use heap ID instead of NUMA node ID to identify heaps. Heap ID for DPDK-internal heaps is NUMA node's index within the detected NUMA node list. Heap ID for external heaps will be order of their creation.
This breaks the ABI, so document the changes.
Signed-off-by Anatoly Burakov <[email protected]>
--- config/common_base | 1 + config/rte_config.h | 1 + doc/guides/rel_notes/release_18_11.rst | 5 +- .../common/include/rte_eal_memconfig.h | 4 +- .../common/include/rte_malloc_heap.h | 1 + lib/librte_eal/common/malloc_heap.c | 102 +++++++++++++----- lib/librte_eal/common/malloc_heap.h | 3 + lib/librte_eal/common/rte_malloc.c | 41 ++++--- 8 files changed, 114 insertions(+), 44 deletions(-)
config/common_base +1 -0
@@ -61,6 +61,7 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_LIBRTE_EAL=y CONFIG_RTE_MAX_LCORE=128 CONFIG_RTE_MAX_NUMA_NODES=8 +CONFIG_RTE_MAX_HEAPS=32 CONFIG_RTE_MAX_MEMSEG_LISTS=64 # each memseg list will be limited to either RTE_MAX_MEMSEG_PER_LIST pages # or RTE_MAX_MEM_MB_PER_LIST megabytes worth of memory, whichever is smaller
config/rte_config.h +1 -0
@@ -24,6 +24,7 @@#define RTE_BUILD_SHARED_LIB /* EAL defines */ +#define RTE_MAX_HEAPS 32#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768
doc/guides/rel_notes/release_18_11.rst +4 -1
@@ -145,7 +145,10 @@ ABI Changes of memory addressed by the segment list - structure ``rte_memseg_list`` now has a new flag indicating whether the memseg list refers to external memory - + - structure ``rte_malloc_heap`` now has a new field indicating socket + ID the malloc heap belongs to + - structure ``rte_mem_config`` has had its ``malloc_heaps`` array + resized from ``RTE_MAX_NUMA_NODES`` to ``RTE_MAX_HEAPS`` value Removed Items -------------
lib/librte_eal/common/include/rte_eal_memconfig.h +2 -2
@@ -72,8 +72,8 @@ struct rte_mem_config {struct rte_tailq_head tailq_head[RTE_MAX_TAILQ ];/**< Tailqs for objects */ -/* Heaps of Malloc per socket */ -struct malloc_heap malloc_heaps[RTE_MAX_NUMA_NODES ]; +/* Heaps of Malloc */ +struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ];/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it.
lib/librte_eal/common/include/rte_malloc_heap.h +1 -0
@@ -27,6 +27,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size; + unsigned int socket_id; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */
lib/librte_eal/common/malloc_heap.c +73 -29
208 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -46,6 +46,9 @@ malloc_heap_get_stats(struct malloc_heap *heap,void malloc_heap_dump (struct malloc_heap *heap,FILE *f); +int +malloc_socket_to_heap_id (unsigned int socket_id); +int rte_eal_malloc_heap_init (void );
lib/librte_eal/common/rte_malloc.c +29 -12
72 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v8 4/21 mem: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
This changes the semantics of what we understand by "socket ID", so document the change in the release notes.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 7 +++++++ lib/librte_eal/common/eal_common_memzone.c | 8 +++++--- lib/librte_eal/common/malloc_heap.c | 2 +- lib/librte_eal/common/rte_malloc.c | 4 ---- 4 files changed, 13 insertions(+), 8 deletions(-)
doc/guides/rel_notes/release_18_11.rst +7 -0
@@ -107,6 +107,13 @@ API Changes for any users of memseg-walk-related functions, as they will now have to skip externally allocated segments in most cases if the intent is to only iterate over internal DPDK memory. + ``socket_id`` parameter across the entire DPDK has gained additional meaning, + as some socket ID's will now be representing externally allocated memory. No + changes will be required for existing code as backwards compatibility will be + kept, and those who do not use this feature will not see these extra socket + ID's. Any new API's must not check socket ID parameters themselves, and must + instead leave it to the memory subsystem to decide whether socket ID is a + valid one. * mbuf: The ``__rte_mbuf_raw_free()`` and ``__rte_pktmbuf_prefree_seg()`` functions were deprecated since 17.05 and are replaced by
lib/librte_eal/common/eal_common_memzone.c +5 -3
@@ -120,13 +120,15 @@ memzone_reserve_aligned_thread_unsafe(const char *name, size_t len,return NULL ; } -if ((socket_id !=SOCKET_ID_ANY ) && - (socket_id >=RTE_MAX_NUMA_NODES || socket_id <0 )) { +if ((socket_id !=SOCKET_ID_ANY ) && socket_id <0 ) { rte_errno =EINVAL ;return NULL ; } -if (!rte_eal_has_hugepages ()) +/* only set socket to SOCKET_ID_ANY if we aren't allocating for an +* external heap. +*/ +if (!rte_eal_has_hugepages () && socket_id <RTE_MAX_NUMA_NODES ) socket_id =SOCKET_ID_ANY ; contig = (flags &RTE_MEMZONE_IOVA_CONTIG ) !=0 ;
lib/librte_eal/common/malloc_heap.c +1 -1
@@ -651,7 +651,7 @@ malloc_heap_alloc(const char *type, size_t size, int socket_arg,if (size ==0 || (align && !rte_is_power_of_2 (align)))return NULL ; -if (!rte_eal_has_hugepages ()) +if (!rte_eal_has_hugepages () && socket_arg <RTE_MAX_NUMA_NODES ) socket_arg =SOCKET_ID_ANY ;if (socket_arg ==SOCKET_ID_ANY )
lib/librte_eal/common/rte_malloc.c +0 -4
@@ -47,10 +47,6 @@ rte_malloc_socket(const char *type, size_t size, unsigned int align,if (!rte_eal_has_hugepages ()) socket_arg =SOCKET_ID_ANY ; -/* Check socket parameter */ -if (socket_arg >=RTE_MAX_NUMA_NODES ) -return NULL ; -return malloc_heap_alloc (type, size, socket_arg,0 , align ==0 ?1 : align,0 , false); }
Signature · 1 line
2.17.1
PATCH v8 5/21 flow_classify: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_flow_classify/rte_flow_classify.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/librte_flow_classify/rte_flow_classify.c +1 -2
@@ -247,8 +247,7 @@ rte_flow_classifier_check_params(struct rte_flow_classifier_params *params) }/* socket */ -if ((params->socket_id <0 ) || - (params->socket_id >=RTE_MAX_NUMA_NODES )) { +if (params->socket_id <0 ) {RTE_FLOW_CLASSIFY_LOG (ERR ,"%s: Incorrect value for parameter socket_id\n" , __func__);
Signature · 1 line
2.17.1
PATCH v8 6/21 pipeline: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_pipeline/rte_pipeline.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
lib/librte_pipeline/rte_pipeline.c +1 -2
@@ -178,8 +178,7 @@ rte_pipeline_check_params(struct rte_pipeline_params *params) }/* socket */ -if ((params->socket_id <0 ) || - (params->socket_id >=RTE_MAX_NUMA_NODES )) { +if (params->socket_id <0 ) {RTE_LOG (ERR ,PIPELINE ,"%s: Incorrect value for parameter socket_id\n" , __func__);
Signature · 1 line
2.17.1
PATCH v8 7/21 sched: do not check for invalid socket ID
In reply to Anatoly BurakovWe will be assigning "invalid" socket ID's to external heap, and malloc will now be able to verify if a supplied socket ID is in fact a valid one, rendering parameter checks for sockets obsolete.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_sched/rte_sched.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/librte_sched/rte_sched.c +1 -1
@@ -329,7 +329,7 @@ rte_sched_port_check_params(struct rte_sched_port_params *params)return -1 ;/* socket */ -if ((params->socket <0 ) || (params->socket >=RTE_MAX_NUMA_NODES )) +if (params->socket <0 )return -3 ;/* rate */
Signature · 1 line
2.17.1
PATCH v8 8/21 malloc: add name to malloc heaps
In reply to Anatoly BurakovWe will need to refer to external heaps in some way. While we use heap ID's internally, for external API use it has to be something more user-friendly. So, we will be using a string to uniquely identify a heap.
This breaks the ABI, so document the change.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 2 ++ lib/librte_eal/common/include/rte_malloc_heap.h | 2 ++ lib/librte_eal/common/malloc_heap.c | 17 ++++++++++++++++- lib/librte_eal/common/rte_malloc.c | 1 + 4 files changed, 21 insertions(+), 1 deletion(-)
doc/guides/rel_notes/release_18_11.rst +2 -0
@@ -156,6 +156,8 @@ ABI Changes ID the malloc heap belongs to - structure ``rte_mem_config`` has had its ``malloc_heaps`` array resized from ``RTE_MAX_NUMA_NODES`` to ``RTE_MAX_HEAPS`` value + - structure ``rte_malloc_heap`` now has a ``heap_name`` member + Removed Items -------------
lib/librte_eal/common/include/rte_malloc_heap.h +2 -0
@@ -12,6 +12,7 @@/* Number of free lists per heap, grouped by size. */ #define RTE_HEAP_NUM_FREELISTS 13 +#define RTE_HEAP_NAME_MAX_LEN 32/* dummy definition, for pointers */ struct malloc_elem ;
@@ -28,6 +29,7 @@ struct malloc_heap {unsigned alloc_count;size_t total_size;unsigned int socket_id; + char name[RTE_HEAP_NAME_MAX_LEN]; } __rte_cache_aligned;#endif /* _RTE_MALLOC_HEAP_H_ */
lib/librte_eal/common/malloc_heap.c +16 -1
@@ -131,7 +131,6 @@ malloc_add_seg(const struct rte_memseg_list *msl,malloc_heap_add_memory (heap, found_msl, ms->addr, len); heap->total_size += len; - heap->socket_id = msl->socket_id;RTE_LOG (DEBUG ,EAL ,"Added %zuM to heap on socket %i\n" , len >>20 , msl->socket_id);
@@ -1024,6 +1023,22 @@ intrte_eal_malloc_heap_init (void) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int i; + +if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign names to default DPDK heaps */ +for (i =0 ; i <rte_socket_count (); i++) { +struct malloc_heap *heap = &mcfg->malloc_heaps[i]; +char heap_name[RTE_HEAP_NAME_MAX_LEN ]; +int socket_id =rte_socket_id_by_idx (i); + +snprintf (heap_name,sizeof (heap_name) -1 , +"socket_%i" , socket_id); +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); + heap->socket_id = socket_id; + } + } +if (register_mp_requests ()) {RTE_LOG (ERR ,EAL ,"Couldn't register malloc multiprocess actions\n" );
lib/librte_eal/common/rte_malloc.c +1 -0
@@ -202,6 +202,7 @@ rte_malloc_dump_stats(FILE *f, __rte_unused const char *type)malloc_heap_get_stats (heap, &sock_stats);fprintf (f,"Heap id:%u\n" , heap_id); +fprintf (f,"\tHeap name:%s\n" , heap->name);fprintf (f,"\tHeap_size:%zu,\n" , sock_stats.heap_totalsz_bytes);fprintf (f,"\tFree_size:%zu,\n" , sock_stats.heap_freesz_bytes);fprintf (f,"\tAlloc_size:%zu,\n" , sock_stats.heap_allocsz_bytes);
Signature · 1 line
2.17.1
PATCH v8 9/21 malloc: add function to query socket ID of named heap
In reply to Anatoly BurakovWhen we will be creating external heaps, they will have their own "fake" socket ID, so add a function that will map the heap name to its socket ID.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 14 ++++++++ lib/librte_eal/common/rte_malloc.c | 37 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +14 -0
@@ -263,6 +263,20 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Find socket ID corresponding to a named heap. +* +* @param name +* Heap name to find socket ID for +* @return +* Socket ID in case of success (a non-negative number) +* -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``name`` was NULL +* ENOENT - heap identified by the name ``name`` was not found +*/ +int __rte_experimental +rte_malloc_heap_get_socket (const char *name); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +37 -0
@@ -8,6 +8,7 @@#include <string.h> #include <sys/queue.h> +#include <rte_errno.h> #include <rte_memcpy.h> #include <rte_memory.h> #include <rte_eal.h>
@@ -183,6 +184,42 @@ rte_malloc_dump_heaps(FILE *f)rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); } +int +rte_malloc_heap_get_socket (const char *name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +unsigned int idx; +int ret; + +if (name ==NULL || +strnlen (name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if (!strncmp (name, tmp->name,RTE_HEAP_NAME_MAX_LEN )) { + heap = tmp; +break ; + } + } + +if (heap !=NULL ) { + ret = heap->socket_id; + }else { + rte_errno =ENOENT ; + ret =-1 ; + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_get_socket; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
Signature · 1 line
2.17.1
PATCH v8 10/21 malloc: add function to check if socket is external
In reply to Anatoly BurakovAn API is needed to check whether a particular socket ID belongs to an internal or external heap. Prime user of this would be mempool allocator, because normal assumptions of IOVA contiguousness in IOVA as VA mode do not hold in case of externally allocated memory.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 15 +++++++++++++ lib/librte_eal/common/rte_malloc.c | 25 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + lib/librte_mempool/rte_mempool.c | 22 ++++++++++++++++--- 4 files changed, 60 insertions(+), 3 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +15 -0
@@ -277,6 +277,21 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_get_socket (const char *name); +/** +* Check if a given socket ID refers to externally allocated memory. +* +* @note Passing SOCKET_ID_ANY will return 0. +* +* @param socket_id +* Socket ID to check +* @return +* 1 if socket ID refers to externally allocated memory +* 0 if socket ID refers to internal DPDK memory +* -1 if socket ID is invalid +*/ +int __rte_experimental +rte_malloc_heap_socket_is_external (int socket_id); +/** * Dump statistics. *
lib/librte_eal/common/rte_malloc.c +25 -0
@@ -220,6 +220,31 @@ rte_malloc_heap_get_socket(const char *name)return ret; } +int +rte_malloc_heap_socket_is_external (int socket_id) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +unsigned int idx; +int ret =-1 ; + +if (socket_id ==SOCKET_ID_ANY ) +return 0 ; + +rte_rwlock_read_lock (&mcfg->memory_hotplug_lock); +for (idx =0 ; idx <RTE_MAX_HEAPS ; idx++) { +struct malloc_heap *tmp = &mcfg->malloc_heaps[idx]; + +if ((int )tmp->socket_id == socket_id) { +/* external memory always has large socket ID's */ + ret = tmp->socket_id >=RTE_MAX_NUMA_NODES ; +break ; + } + } +rte_rwlock_read_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +/* * Print stats on memory type. If type is NULL, info on all types is printed */
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_get_socket; + rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister; rte_mem_event_callback_register;
lib/librte_mempool/rte_mempool.c +19 -3
@@ -428,12 +428,18 @@ rte_mempool_populate_default(struct rte_mempool *mp)rte_iova_t iova;unsigned mz_id, n;int ret; -bool no_contig, try_contig, no_pageshift; +bool no_contig, try_contig, no_pageshift, external; ret =mempool_ops_alloc_once (mp);if (ret !=0 )return ret; +/* check if we can retrieve a valid socket ID */ + ret =rte_malloc_heap_socket_is_external (mp->socket_id); +if (ret <0 ) +return -EINVAL ; + external = ret; +/* mempool must not be populated */ if (mp->nb_mem_chunks !=0 )return -EEXIST ;
@@ -481,9 +487,19 @@ rte_mempool_populate_default(struct rte_mempool *mp)* in one contiguous chunk as well (otherwise we might end up wasting a * 1G page on a 10MB memzone). If we fail to get enough contiguous * memory, then we'll go and reserve space page-by-page. +* +* We also have to take into account the fact that memory that we're +* going to allocate from can belong to an externally allocated memory +* area, in which case the assumption of IOVA as VA mode being +* synonymous with IOVA contiguousness will not hold. We should also try +* to go for contiguous memory even if we're in no-huge mode, because +* external memory may in fact be IOVA-contiguous. */ - no_pageshift = no_contig ||rte_eal_iova_mode () ==RTE_IOVA_VA ; - try_contig = !no_contig && !no_pageshift &&rte_eal_has_hugepages (); + external =rte_malloc_heap_socket_is_external (mp->socket_id) ==1 ; + no_pageshift = no_contig || + (!external &&rte_eal_iova_mode () ==RTE_IOVA_VA ); + try_contig = !no_contig && !no_pageshift && + (rte_eal_has_hugepages () || external);if (no_pageshift) { pg_sz =0 ;
Signature · 1 line
2.17.1
PATCH v8 11/21 malloc: allow creating malloc heaps
In reply to Anatoly BurakovAdd API to allow creating new malloc heaps. They will be created with socket ID's going above RTE_MAX_NUMA_NODES, to avoid clashing with internal heaps.
This breaks the ABI, so document the change.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 2 + .../common/include/rte_eal_memconfig.h | 3 ++ lib/librte_eal/common/include/rte_malloc.h | 19 +++++++ lib/librte_eal/common/malloc_heap.c | 37 +++++++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 52 +++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 7 files changed, 117 insertions(+)
doc/guides/rel_notes/release_18_11.rst +2 -0
@@ -157,6 +157,8 @@ ABI Changes - structure ``rte_mem_config`` has had its ``malloc_heaps`` array resized from ``RTE_MAX_NUMA_NODES`` to ``RTE_MAX_HEAPS`` value - structure ``rte_malloc_heap`` now has a ``heap_name`` member + - structure ``rte_eal_memconfig`` has been extended to contain next + socket ID for externally allocated segments Removed Items
lib/librte_eal/common/include/rte_eal_memconfig.h +3 -0
@@ -75,6 +75,9 @@ struct rte_mem_config {/* Heaps of Malloc */ struct malloc_heap malloc_heaps[RTE_MAX_HEAPS ]; +/* next socket ID for external malloc heap */ +int next_socket_id; +/* address of mem_config in primary process. used to map shared config into * exact same address the primary process maps it. */
lib/librte_eal/common/include/rte_malloc.h +19 -0
@@ -263,6 +263,25 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Creates a new empty malloc heap with a specified name. +* +* @note Heaps created via this call will automatically get assigned a unique +* socket ID, which can be found using ``rte_malloc_heap_get_socket()`` +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on successful creation +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* EEXIST - heap by name of ``heap_name`` already exists +* ENOSPC - no more space in internal config to store a new heap +*/ +int __rte_experimental +rte_malloc_heap_create (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +37 -0
@@ -29,6 +29,10 @@#include "malloc_heap.h" #include "malloc_mp.h" +/* start external socket ID's at a very high number */ +#define CONST_MAX (a, b) (a > b ? a : b)/* RTE_MAX is not a constant */ +#define EXTERNAL_HEAP_MIN_SOCKET_ID (CONST_MAX((1 << 8), RTE_MAX_NUMA_NODES)) +static unsigned check_hugepage_sz (unsigned flags,uint64_t hugepage_sz) {
@@ -1019,6 +1023,36 @@ malloc_heap_dump(struct malloc_heap *heap, FILE *f)rte_spinlock_unlock (&heap->lock); } +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +uint32_t next_socket_id = mcfg->next_socket_id; + +/* prevent overflow. did you really create 2 billion heaps??? */ +if (next_socket_id >INT32_MAX ) { +RTE_LOG (ERR ,EAL ,"Cannot assign new socket ID's\n" ); + rte_errno =ENOSPC ; +return -1 ; + } + +/* initialize empty heap */ + heap->alloc_count =0 ; + heap->first =NULL ; + heap->last =NULL ; +LIST_INIT (heap->free_head); +rte_spinlock_init (&heap->lock); + heap->total_size =0 ; + heap->socket_id = next_socket_id; + +/* we hold a global mem hotplug writelock, so it's safe to increment */ + mcfg->next_socket_id++; + +/* set up name */ +strlcpy (heap->name, heap_name,RTE_HEAP_NAME_MAX_LEN ); +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
@@ -1026,6 +1060,9 @@ rte_eal_malloc_heap_init(void)unsigned int i;if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) { +/* assign min socket ID to external heaps */ + mcfg->next_socket_id =EXTERNAL_HEAP_MIN_SOCKET_ID ; +/* assign names to default DPDK heaps */ for (i =0 ; i <rte_socket_count (); i++) {struct malloc_heap *heap = &mcfg->malloc_heaps[i];
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -33,6 +33,9 @@ void * malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags, size_t align, bool contig); +int +malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +52 -0
61 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -318,6 +318,7 @@ EXPERIMENTAL { rte_fbarray_set_used; rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; + rte_malloc_heap_create; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v8 12/21 malloc: allow destroying heaps
In reply to Anatoly BurakovAdd an API to destroy specified heap.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 23 +++++++++ lib/librte_eal/common/malloc_heap.c | 22 ++++++++ lib/librte_eal/common/malloc_heap.h | 3 ++ lib/librte_eal/common/rte_malloc.c | 58 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 107 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +23 -0
@@ -282,6 +282,29 @@ rte_malloc_get_socket_stats(int socket, int __rte_experimentalrte_malloc_heap_create (const char *heap_name); +/** +* Destroys a previously created malloc heap with specified name. +* +* @note This function will return a failure result if not all memory allocated +* from the heap has been freed back to the heap +* +* @note This function will return a failure result if not all memory segments +* were removed from the heap prior to its destruction +* +* @param heap_name +* Name of the heap to create. +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - ``heap_name`` was NULL, empty or too long +* ENOENT - heap by the name of ``heap_name`` was not found +* EPERM - attempting to destroy reserved heap +* EBUSY - heap still contains data +*/ +int __rte_experimental +rte_malloc_heap_destroy (const char *heap_name); +/** * Find socket ID corresponding to a named heap. *
lib/librte_eal/common/malloc_heap.c +22 -0
@@ -1053,6 +1053,28 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name)return 0 ; } +int +malloc_heap_destroy (struct malloc_heap *heap) +{ +if (heap->alloc_count !=0 ) { +RTE_LOG (ERR ,EAL ,"Heap is still in use\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->first !=NULL || heap->last !=NULL ) { +RTE_LOG (ERR ,EAL ,"Heap still contains memory segments\n" ); + rte_errno =EBUSY ; +return -1 ; + } +if (heap->total_size !=0 ) +RTE_LOG (ERR ,EAL ,"Total size not zero, heap is likely corrupt\n" ); + +/* after this, the lock will be dropped */ +memset (heap,0 ,sizeof (*heap)); + +return 0 ; +} +int rte_eal_malloc_heap_init (void ) {
lib/librte_eal/common/malloc_heap.h +3 -0
@@ -36,6 +36,9 @@ malloc_heap_alloc_biggest(const char *type, int socket, unsigned int flags,int malloc_heap_create (struct malloc_heap *heap,const char *heap_name); +int +malloc_heap_destroy (struct malloc_heap *heap); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +58 -0
67 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -319,6 +319,7 @@ EXPERIMENTAL { rte_log_register_type_and_pick_level; rte_malloc_dump_heaps; rte_malloc_heap_create; + rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v8 13/21 malloc: allow adding memory to named heaps
In reply to Anatoly BurakovAdd an API to add externally allocated memory to malloc heap. The memory will be stored in memseg lists like regular DPDK memory. Multiple segments are allowed within a heap. If IOVA table is not provided, IOVA addresses are filled in with RTE_BAD_IOVA.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 39 ++++++++++++ lib/librte_eal/common/malloc_heap.c | 74 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 51 +++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 169 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +39 -0
@@ -263,6 +263,45 @@ int rte_malloc_get_socket_stats(int socket, struct rte_malloc_socket_stats *socket_stats); +/** +* Add memory chunk to a heap with specified name. +* +* @note Multiple memory chunks can be added to the same heap +* +* @note Memory must be previously allocated for DPDK to be able to use it as a +* malloc heap. Failing to do so will result in undefined behavior, up to and +* including segmentation faults. +* +* @note Calling this function will erase any contents already present at the +* supplied memory address. +* +* @param heap_name +* Name of the heap to add memory chunk to +* @param va_addr +* Start of virtual area to add to the heap +* @param len +* Length of virtual area to add to the heap +* @param iova_addrs +* Array of page IOVA addresses corresponding to each page in this memory +* area. Can be NULL, in which case page IOVA addresses will be set to +* RTE_BAD_IOVA. +* @param n_pages +* Number of elements in the iova_addrs array. Ignored if ``iova_addrs`` +* is NULL. +* @param page_sz +* Page size of the underlying memory +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to add memory to a reserved heap +* ENOSPC - no more space in internal config to store a new memory chunk +*/ +int __rte_experimental +rte_malloc_heap_memory_add (const char *heap_name,void *va_addr,size_t len, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +74 -0
80 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -39,6 +39,10 @@ malloc_heap_create(struct malloc_heap *heap, const char *heap_name);int malloc_heap_destroy (struct malloc_heap *heap); +int +malloc_heap_add_external_memory (struct malloc_heap *heap,void *va_addr, +rte_iova_t iova_addrs[],unsigned int n_pages,size_t page_sz); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +51 -0
57 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -321,6 +321,7 @@ EXPERIMENTAL { rte_malloc_heap_create; rte_malloc_heap_destroy; rte_malloc_heap_get_socket; + rte_malloc_heap_memory_add; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v8 14/21 malloc: allow removing memory from named heaps
In reply to Anatoly BurakovAdd an API to remove memory from specified heaps. This will first check if all elements within the region are free, and that the region is the original region that was added to the heap (by comparing its length to length of memory addressed by the underlying memseg list).
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++ lib/librte_eal/common/malloc_heap.c | 54 ++++++++++++++++++++++ lib/librte_eal/common/malloc_heap.h | 4 ++ lib/librte_eal/common/rte_malloc.c | 39 ++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 5 files changed, 125 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -302,6 +302,33 @@ int __rte_experimental rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +/** +* Remove memory chunk from heap with specified name. +* +* @note Memory chunk being removed must be the same as one that was added; +* partially removing memory chunks is not supported +* +* @note Memory area must not contain any allocated elements to allow its +* removal from the heap +* +* @param heap_name +* Name of the heap to remove memory from +* @param va_addr +* Virtual address to remove from the heap +* @param len +* Length of virtual area to remove from the heap +* +* @return +* - 0 on success +* - -1 in case of error, with rte_errno set to one of the following: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to remove memory from a reserved heap +* ENOENT - heap or memory chunk was not found +* EBUSY - memory chunk still contains data +*/ +int __rte_experimental +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/malloc_heap.c +54 -0
66 lines not on this page · raw message
lib/librte_eal/common/malloc_heap.h +4 -0
@@ -43,6 +43,10 @@ int malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr, rte_iova_t iova_addrs[], unsigned int n_pages, size_t page_sz); +int +malloc_heap_remove_external_memory (struct malloc_heap *heap,void *va_addr, +size_t len); +int malloc_heap_free (struct malloc_elem *elem);
lib/librte_eal/common/rte_malloc.c +39 -0
@@ -379,6 +379,45 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,return ret; } +int +rte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len) +{ +struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config; +struct malloc_heap *heap =NULL ; +int ret; + +if (heap_name ==NULL || va_addr ==NULL || len ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) ==0 || +strnlen (heap_name,RTE_HEAP_NAME_MAX_LEN ) == +RTE_HEAP_NAME_MAX_LEN ) { + rte_errno =EINVAL ; +return -1 ; + } +rte_rwlock_write_lock (&mcfg->memory_hotplug_lock); +/* find our heap */ + heap =find_named_heap (heap_name); +if (heap ==NULL ) { + rte_errno =ENOENT ; + ret =-1 ; + goto unlock; + } +if (heap->socket_id <RTE_MAX_NUMA_NODES ) { +/* cannot remove memory from internal heaps */ + rte_errno =EPERM ; + ret =-1 ; + goto unlock; + } + +rte_spinlock_lock (&heap->lock); + ret =malloc_heap_remove_external_memory (heap, va_addr, len); +rte_spinlock_unlock (&heap->lock); + +unlock: +rte_rwlock_write_unlock (&mcfg->memory_hotplug_lock); + +return ret; +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register; rte_mem_alloc_validator_unregister;
Signature · 1 line
2.17.1
PATCH v8 15/21 malloc: allow attaching to external memory chunks
In reply to Anatoly BurakovIn order to use external memory in multiple processes, we need to attach to primary process's memseg lists, so add a new API to do that. It is the responsibility of the user to ensure that memory is accessible and that it has been previously added to the malloc heap by another process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 28 ++++++++ lib/librte_eal/common/rte_malloc.c | 83 ++++++++++++++++++++++ lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 112 insertions(+)
lib/librte_eal/common/include/rte_malloc.h +28 -0
@@ -268,6 +268,10 @@ rte_malloc_get_socket_stats(int socket,* * @note Multiple memory chunks can be added to the same heap * +* @note Before accessing this memory in other processes, it needs to be +* attached in each of those processes by calling +* ``rte_malloc_heap_memory_attach`` in each other process. +* * @note Memory must be previously allocated for DPDK to be able to use it as a * malloc heap. Failing to do so will result in undefined behavior, up to and * including segmentation faults.
@@ -329,6 +333,30 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len, int __rte_experimentalrte_malloc_heap_memory_remove (const char *heap_name,void *va_addr,size_t len); +/** +* Attach to an already existing chunk of external memory in another process. +* +* @note This function must be called before any attempt is made to use an +* already existing external memory chunk. This function does *not* need to +* be called if a call to ``rte_malloc_heap_memory_add`` was made in the +* current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful attach +* -1 on unsuccessful attach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to attach memory to a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +83 -0
89 lines not on this page · raw message
lib/librte_eal/rte_eal_version.map +1 -0
@@ -322,6 +322,7 @@ EXPERIMENTAL { rte_malloc_heap_destroy; rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; + rte_malloc_heap_memory_attach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v8 16/21 malloc: allow detaching from external memory
In reply to Anatoly BurakovAdd API to detach from existing chunk of external memory in a process.
Signed-off-by Anatoly Burakov <[email protected]>
--- lib/librte_eal/common/include/rte_malloc.h | 27 +++++++++++++++++++ lib/librte_eal/common/rte_malloc.c | 31 +++++++++++++++++----- lib/librte_eal/rte_eal_version.map | 1 + 3 files changed, 52 insertions(+), 7 deletions(-)
lib/librte_eal/common/include/rte_malloc.h +27 -0
@@ -315,6 +315,9 @@ rte_malloc_heap_memory_add(const char *heap_name, void *va_addr, size_t len,* @note Memory area must not contain any allocated elements to allow its * removal from the heap * +* @note All other processes must detach from the memory chunk prior to it being +* removed from the heap. +* * @param heap_name * Name of the heap to remove memory from * @param va_addr
@@ -357,6 +360,30 @@ rte_malloc_heap_memory_remove(const char *heap_name, void *va_addr, size_t len); int __rte_experimentalrte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len); +/** +* Detach from a chunk of external memory in secondary process. +* +* @note This function must be called in before any attempt is made to remove +* external memory from the heap in another process. This function does *not* +* need to be called if a call to ``rte_malloc_heap_memory_remove`` will be +* called in current process. +* +* @param heap_name +* Heap name to which this chunk of memory belongs +* @param va_addr +* Start address of memory chunk to attach to +* @param len +* Length of memory chunk to attach to +* @return +* 0 on successful detach +* -1 on unsuccessful detach, with rte_errno set to indicate cause for error: +* EINVAL - one of the parameters was invalid +* EPERM - attempted to detach memory from a reserved heap +* ENOENT - heap or memory chunk was not found +*/ +int __rte_experimental +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len); +/** * Creates a new empty malloc heap with a specified name. *
lib/librte_eal/common/rte_malloc.c +24 -7
@@ -422,10 +422,11 @@ struct sync_mem_walk_arg {void *va_addr;size_t len;int result; +bool attach; };static int -attach_mem_walk (const struct rte_memseg_list *msl,void *arg) +sync_mem_walk (const struct rte_memseg_list *msl,void *arg) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct sync_mem_walk_arg *wa = arg;
@@ -440,7 +441,10 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; - ret =rte_fbarray_attach (&found_msl->memseg_arr); +if (wa->attach) + ret =rte_fbarray_attach (&found_msl->memseg_arr); +else + ret =rte_fbarray_detach (&found_msl->memseg_arr);if (ret <0 ) wa->result = -rte_errno;
@@ -451,8 +455,8 @@ attach_mem_walk(const struct rte_memseg_list *msl, void *arg)return 0 ; } -int -rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +static int +sync_memory (const char *heap_name,void *va_addr,size_t len,bool attach) {struct rte_mem_config *mcfg =rte_eal_get_configuration ()->mem_config;struct malloc_heap *heap =NULL ;
@@ -475,20 +479,21 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len) ret =-1 ; goto unlock; } -/* we shouldn't be able to attach to internal heaps */ +/* we shouldn't be able to sync to internal heaps */ if (heap->socket_id <RTE_MAX_NUMA_NODES ) { rte_errno =EPERM ; ret =-1 ; goto unlock; } -/* find corresponding memseg list to attach to */ +/* find corresponding memseg list to sync to */ wa.va_addr = va_addr; wa.len = len; wa.result = -ENOENT ;/* fail unless explicitly told to succeed */ + wa.attach = attach;/* we're already holding a read lock */ -rte_memseg_list_walk_thread_unsafe (attach_mem_walk, &wa); +rte_memseg_list_walk_thread_unsafe (sync_mem_walk, &wa);if (wa.result <0 ) { rte_errno = -wa.result;
@@ -501,6 +506,18 @@ rte_malloc_heap_memory_attach(const char *heap_name, void *va_addr, size_t len)return ret; } +int +rte_malloc_heap_memory_attach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, true); +} + +int +rte_malloc_heap_memory_detach (const char *heap_name,void *va_addr,size_t len) +{ +return sync_memory (heap_name, va_addr, len, false); +} +int rte_malloc_heap_create (const char *heap_name) {
lib/librte_eal/rte_eal_version.map +1 -0
@@ -323,6 +323,7 @@ EXPERIMENTAL { rte_malloc_heap_get_socket; rte_malloc_heap_memory_add; rte_malloc_heap_memory_attach; + rte_malloc_heap_memory_detach; rte_malloc_heap_memory_remove; rte_malloc_heap_socket_is_external; rte_mem_alloc_validator_register;
Signature · 1 line
2.17.1
PATCH v8 17/21 malloc: enable event callbacks for external memory
In reply to Anatoly BurakovWhen adding or removing external memory from the memory map, there may be actions that need to be taken on account of this memory (e.g. DMA mapping). Add support for triggering callbacks when adding, removing, attaching or detaching external memory.
Some memory event callback handlers will need additional logic to handle external memory regions. For example, virtio callback has to completely ignore externally allocated memory, because there is no way to find file descriptors backing the memory address in a generic fashion. All other callbacks have also been adjusted to handle RTE_BAD_IOVA as IOVA address, as this is one of the expected use cases for external memory support.
Signed-off-by Anatoly Burakov <[email protected]>
--- drivers/bus/fslmc/fslmc_vfio.c | 7 +++++ .../net/virtio/virtio_user/virtio_user_dev.c | 6 +++++ lib/librte_eal/common/malloc_heap.c | 7 +++++ lib/librte_eal/common/rte_malloc.c | 27 ++++++++++++++++--- lib/librte_eal/linuxapp/eal/eal_vfio.c | 10 +++++-- 5 files changed, 51 insertions(+), 6 deletions(-)
drivers/bus/fslmc/fslmc_vfio.c +7 -0
@@ -221,6 +221,13 @@ fslmc_memevent_cb(enum rte_mem_event type, const void *addr, size_t len, "alloc" : "dealloc", va, virt_addr, iova_addr, map_len); +/* iova_addr may be set to RTE_BAD_IOVA */ +if (iova_addr ==RTE_BAD_IOVA ) { +DPAA2_BUS_DEBUG ("Segment has invalid iova, skipping\n" ); + cur_len += map_len; +continue ; + } +if (type ==RTE_MEM_EVENT_ALLOC ) ret =fslmc_map_dma (virt_addr, iova_addr, map_len);else
drivers/net/virtio/virtio_user/virtio_user_dev.c +6 -0
@@ -301,8 +301,14 @@ virtio_user_mem_event_cb(enum rte_mem_event type __rte_unused, void *arg) {struct virtio_user_dev *dev = arg; +struct rte_memseg_list *msl;uint16_t i; +/* ignore externally allocated memory */ + msl =rte_mem_virt2memseg_list (addr); +if (msl->external) +return ; +pthread_mutex_lock (&dev->mutex);if (dev->started == false)
lib/librte_eal/common/malloc_heap.c +7 -0
@@ -1031,6 +1031,9 @@ destroy_seg(struct malloc_elem *elem, size_t len) msl = elem->msl; +/* notify all subscribers that a memory area is going to be removed */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , elem, len); +/* this element can be removed */ malloc_elem_free_list_remove (elem);malloc_elem_hide_region (elem, elem, len);
@@ -1120,6 +1123,10 @@ malloc_heap_add_external_memory(struct malloc_heap *heap, void *va_addr,RTE_LOG (DEBUG ,EAL ,"Added segment for heap %s starting at %p\n" , heap->name, va_addr); +/* notify all subscribers that a new memory area has been added */ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, seg_len); +return 0 ; }
lib/librte_eal/common/rte_malloc.c +23 -4
@@ -25,6 +25,7 @@#include <rte_malloc.h> #include "malloc_elem.h" #include "malloc_heap.h" +#include "eal_memalloc.h" /* Free the memory space back to heap */
@@ -441,15 +442,29 @@ sync_mem_walk(const struct rte_memseg_list *msl, void *arg) msl_idx = msl - mcfg->memsegs; found_msl = &mcfg->memsegs[msl_idx]; -if (wa->attach) +if (wa->attach) { ret =rte_fbarray_attach (&found_msl->memseg_arr); -else + }else { +/* notify all subscribers that a memory area is about to +* be removed +*/ +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_FREE , + msl->base_va, msl->len); ret =rte_fbarray_detach (&found_msl->memseg_arr); + } -if (ret <0 ) +if (ret <0 ) { wa->result = -rte_errno; -else + }else { +/* notify all subscribers that a new memory area was +* added +*/ +if (wa->attach) +eal_memalloc_mem_event_notify ( +RTE_MEM_EVENT_ALLOC , + msl->base_va, msl->len); wa->result =0 ; + }return 1 ; }return 0 ;
@@ -499,6 +514,10 @@ sync_memory(const char *heap_name, void *va_addr, size_t len, bool attach) rte_errno = -wa.result; ret =-1 ; }else { +/* notify all subscribers that a new memory area was added */ +if (attach) +eal_memalloc_mem_event_notify (RTE_MEM_EVENT_ALLOC , + va_addr, len); ret =0 ; } unlock:
lib/librte_eal/linuxapp/eal/eal_vfio.c +8 -2
@@ -509,7 +509,7 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len, msl =rte_mem_virt2memseg_list (addr);/* for IOVA as VA mode, no need to care for IOVA addresses */ -if (rte_eal_iova_mode () ==RTE_IOVA_VA ) { +if (rte_eal_iova_mode () ==RTE_IOVA_VA && msl->external ==0 ) {uint64_t vfio_va = (uint64_t )(uintptr_t )addr;if (type ==RTE_MEM_EVENT_ALLOC ) vfio_dma_mem_map(default_vfio_cfg, vfio_va, vfio_va,
@@ -523,13 +523,19 @@ vfio_mem_event_callback(enum rte_mem_event type, const void *addr, size_t len,/* memsegs are contiguous in memory */ ms =rte_mem_virt2memseg (addr, msl);while (cur_len < len) { +/* some memory segments may have invalid IOVA */ +if (ms->iova ==RTE_BAD_IOVA ) { +RTE_LOG (DEBUG ,EAL ,"Memory segment at %p has bad IOVA, skipping\n" , + ms->addr); + goto next; + }if (type ==RTE_MEM_EVENT_ALLOC )vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,1 );else vfio_dma_mem_map (default_vfio_cfg, ms->addr_64, ms->iova, ms->len,0 ); - +next: cur_len += ms->len; ++ms; }
Signature · 1 line
2.17.1
PATCH v8 18/21 test: add unit tests for external memory support
In reply to Anatoly BurakovAdd simple unit tests to test external memory support. The tests are pretty basic and mostly consist of checking if invalid API calls are handled correctly, plus a simple allocation/deallocation test for malloc and memzone.
Signed-off-by Anatoly Burakov <[email protected]>
--- test/test/Makefile | 1 + test/test/autotest_data.py | 14 +- test/test/meson.build | 1 + test/test/test_external_mem.c | 389 ++++++++++++++++++++++++++++++++++ 4 files changed, 401 insertions(+), 4 deletions(-) create mode 100644 test/test/test_external_mem.c
test/test/Makefile +1 -0
@@ -71,6 +71,7 @@ SRCS-y += test_bitmap.c SRCS-y += test_reciprocal_division.c SRCS-y += test_reciprocal_division_perf.c SRCS-y += test_fbarray.c +SRCS-y += test_external_mem.c SRCS-y += test_ring.c SRCS-y += test_ring_perf.c
test/test/autotest_data.py +10 -4
@@ -477,10 +477,16 @@ "Report": None, }, { - "Name": "Fbarray autotest", - "Command": "fbarray_autotest", - "Func": default_autotest, - "Report": None, + "Name": "Fbarray autotest", + "Command": "fbarray_autotest", + "Func": default_autotest, + "Report": None, + }, + { + "Name": "External memory autotest", + "Command": "external_mem_autotest", + "Func": default_autotest, + "Report": None, }, # #Please always keep all dump tests at the end and together!
test/test/meson.build +1 -0
@@ -155,6 +155,7 @@ test_names = [ 'eventdev_common_autotest', 'eventdev_octeontx_autotest', 'eventdev_sw_autotest', + 'external_mem_autotest', 'func_reentrancy_autotest', 'flow_classify_autotest', 'hash_scaling_autotest',
test/test/test_external_mem.c new file +389 -0
389 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v8 19/21 app/testpmd: add support for external memory
In reply to Anatoly BurakovCurrently, mempools can only be allocated either using native DPDK memory, or anonymous memory. This patch will add two new methods to allocate mempool using external memory (regular or hugepage memory), and add documentation about it to testpmd user guide.
It adds a new flag "--mp-alloc", with four possible values:
native (use regular DPDK allocator), anon (use anonymous mempool), xmem (use externally allocated memory area), and xmemhuge (use externally allocated hugepage memory area). Old flag "--mp-anon" is kept for compatibility.
All external memory is allocated using the same external heap, but each will allocate and add a new memory area.
Signed-off-by Anatoly Burakov <[email protected]>
Suggested-by Konstantin Ananyev <[email protected]>
--- app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 320 ++++++++++++++++++++++++-- app/test-pmd/testpmd.h | 13 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + 5 files changed, 364 insertions(+), 25 deletions(-)
app/test-pmd/config.c +19 -2
@@ -2423,6 +2423,23 @@ fwd_config_setup(void)simple_fwd_config_setup (); } +static const char * +mp_alloc_to_str (uint8_t mode) +{ +switch (mode) { +case MP_ALLOC_NATIVE : +return "native" ; +case MP_ALLOC_ANON : +return "anon" ; +case MP_ALLOC_XMEM : +return "xmem" ; +case MP_ALLOC_XMEM_HUGE : +return "xmemhuge" ; +default : +return "invalid" ; + } +} +void pkt_fwd_config_display (struct fwd_config *cfg) {
@@ -2431,12 +2448,12 @@ pkt_fwd_config_display(struct fwd_config *cfg)streamid_t sm_id;printf ("%s packet forwarding%s - ports=%d - cores=%d - streams=%d - " -"NUMA support %s, MP over anonymous pages %s\n" , +"NUMA support %s, MP allocation mode: %s\n" , cfg->fwd_eng->fwd_mode_name, retry_enabled ==0 ?"" :" with retry" , cfg->nb_fwd_ports, cfg->nb_fwd_lcores, cfg->nb_fwd_streams, numa_support ==1 ?"enabled" :"disabled" , - mp_anon !=0 ?"enabled" :"disabled" ); +mp_alloc_to_str (mp_alloc_type));if (retry_enabled) printf("TX retry num: %u, delay between TX retries: %uus\n",
app/test-pmd/parameters.c +22 -1
@@ -190,6 +190,11 @@ usage(char* progname)printf (" --vxlan-gpe-port=N: UPD port of tunnel VXLAN-GPE\n" );printf (" --mlockall: lock all memory\n" );printf (" --no-mlockall: do not lock all memory\n" ); +printf (" --mp-alloc <native|anon|xmem|xmemhuge>: mempool allocation method.\n" +" native: use regular DPDK memory to create and populate mempool\n" +" anon: use regular DPDK memory to create and anonymous memory to populate mempool\n" +" xmem: use anonymous memory to create and populate mempool\n" +" xmemhuge: use anonymous hugepage memory to create and populate mempool\n" ); }#ifdef RTE_LIBRTE_CMDLINE
@@ -625,6 +630,7 @@ launch_args_parse(int argc, char** argv) { "vxlan-gpe-port", 1, 0, 0 }, { "mlockall", 0, 0, 0 }, { "no-mlockall", 0, 0, 0 }, + { "mp-alloc", 1, 0, 0 }, { 0, 0, 0, 0 }, };
@@ -743,7 +749,22 @@ launch_args_parse(int argc, char** argv)if (!strcmp (lgopts[opt_idx].name,"numa" )) numa_support =1 ;if (!strcmp (lgopts[opt_idx].name,"mp-anon" )) { - mp_anon =1 ; + mp_alloc_type =MP_ALLOC_ANON ; + } +if (!strcmp (lgopts[opt_idx].name,"mp-alloc" )) { +if (!strcmp (optarg,"native" )) + mp_alloc_type =MP_ALLOC_NATIVE ; +else if (!strcmp (optarg,"anon" )) + mp_alloc_type =MP_ALLOC_ANON ; +else if (!strcmp (optarg,"xmem" )) + mp_alloc_type =MP_ALLOC_XMEM ; +else if (!strcmp (optarg,"xmemhuge" )) + mp_alloc_type =MP_ALLOC_XMEM_HUGE ; +else +rte_exit (EXIT_FAILURE , +"mp-alloc %s invalid - must be: " +"native, anon or xmem\n" , + optarg); }if (!strcmp (lgopts[opt_idx].name,"port-numa-config" )) {if (parse_portnuma_config (optarg))
app/test-pmd/testpmd.c +299 -21
353 lines not on this page · raw message
app/test-pmd/testpmd.h +12 -1
@@ -69,6 +69,16 @@ enum { PORT_TOPOLOGY_LOOP, }; +enum { +MP_ALLOC_NATIVE ,/**< allocate and populate mempool natively */ +MP_ALLOC_ANON , +/**< allocate mempool natively, but populate using anonymous memory */ +MP_ALLOC_XMEM , +/**< allocate and populate mempool using anonymous memory */ +MP_ALLOC_XMEM_HUGE +/**< allocate and populate mempool using anonymous hugepage memory */ +}; +#ifdef RTE_TEST_PMD_RECORD_BURST_STATS /** * The data structure associated with RX and TX packet burst statistics
@@ -304,7 +314,8 @@ extern uint8_t numa_support; /**< set by "--numa" parameter */extern uint16_t port_topology;/**< set by "--port-topology" parameter */ extern uint8_t no_flush_rx;/**<set by "--no-flush-rx" parameter */ extern uint8_t flow_isolate_all;/**< set by "--flow-isolate-all */ -extern uint8_t mp_anon;/**< set by "--mp-anon" parameter */ +extern uint8_t mp_alloc_type; +/**< set by "--mp-anon" or "--mp-alloc" parameter */ extern uint8_t no_link_check;/**<set by "--disable-link-check" parameter */ extern volatile int test_done;/* stop packet forwarding when set to 1. */ extern uint8_t lsc_interrupt;/**< disabled by "--no-lsc-interrupt" parameter */
doc/guides/testpmd_app_ug/run_app.rst +12 -0
@@ -498,3 +498,15 @@ The commandline options are: * ``--no-mlockall`` Disable locking all memory. + +* ``--mp-alloc <native|anon|xmem|xmemhuge>`` + + Select mempool allocation mode: + + * native: create and populate mempool using native DPDK memory + * anon: create mempool using native DPDK memory, but populate using + anonymous memory + * xmem: create and populate mempool using externally and anonymously + allocated area + * xmemhuge: create and populate mempool using externally and anonymously + allocated hugepage area
Signature · 1 line
2.17.1
Re: [PATCH v8 19/21] app/testpmd: add support for external memory
Hi Anatoly,
Quoted 134 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, October 1, 2018 1:56 PM
To: [email protected]
Cc: Lu, Wenzhuo <[email protected]>; Wu, Jingjing <[email protected]>; Iremonger, Bernard <[email protected]>; Mcnamara, John <[email protected]>; Kovacevic, Marko <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Wiles, Keith <[email protected]>; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v8 19/21] app/testpmd: add support for external memoryCurrently, mempools can only be allocated either using native DPDK memory, or anonymous memory. This patch will add two new methods to allocate mempool using external memory (regular or hugepage memory), and add documentation about it to testpmd user guide.
It adds a new flag "--mp-alloc", with four possible values:
native (use regular DPDK allocator), anon (use anonymous mempool), xmem (use externally allocated memory area), and xmemhuge (use externally allocated hugepage memory area). Old flag "--mp-anon" is kept for compatibility.All external memory is allocated using the same external heap, but each will allocate and add a new memory area.
Signed-off-by Anatoly Burakov <[email protected]>Suggested-by Konstantin Ananyev <[email protected]>--- app/test-pmd/config.c | 21 +- app/test-pmd/parameters.c | 23 +- app/test-pmd/testpmd.c | 320 ++++++++++++++++++++++++-- app/test-pmd/testpmd.h | 13 +- doc/guides/testpmd_app_ug/run_app.rst | 12 + 5 files changed, 364 insertions(+), 25 deletions(-)app/test-pmd/config.c index renamed from app/test-pmd/config.c +0
-0
794aa5268..3b921cfc6100644app/test-pmd/config.c +17
-0@@ -2423,6 +2423,23 @@ fwd_config_setup(void)simple_fwd_config_setup (); } +static const char * +mp_alloc_to_str (uint8_t mode) +{ +switch (mode) { +case MP_ALLOC_NATIVE : +return "native" ; +case MP_ALLOC_ANON : +return "anon" ; +case MP_ALLOC_XMEM : +return "xmem" ; +case MP_ALLOC_XMEM_HUGE : +return "xmemhuge" ; +default : +return "invalid" ; + } +} +void pkt_fwd_config_display(struct fwd_config *cfg) { @@ -2431,12 +2448,12 @@ pkt_fwd_config_display(struct fwd_config *cfg)app/test-pmd/config.c +1
-1streamid_t sm_id; printf("%s packet forwarding%s - ports=%d - cores=%d - streams=%d - " - "NUMA support %s, MP over anonymous pages %s\n", + "NUMA support %s, MP allocation mode: %s\n", cfg->fwd_eng->fwd_mode_name, retry_enabled == 0 ? "" : " with retry", cfg->nb_fwd_ports, cfg->nb_fwd_lcores, cfg-Quoted 1 line
nb_fwd_streams,app/test-pmd/config.c +1
-1numa_support ==1 ?"enabled" :"disabled" , - mp_anon != 0 ? "enabled" : "disabled"); + mp_alloc_to_str(mp_alloc_type));if (retry_enabled) printf("TX retry num: %u, delay between TX retries: %uus\n",app/test-pmd/parameters.c index renamed from app/test-pmd/parameters.c +0
-0
9220e1c1b..b4016668c100644app/test-pmd/parameters.c +1
-0@@ -190,6 +190,11 @@ usage(char* progname)printf (" --vxlan-gpe-port=N: UPD port of tunnel VXLAN-GPE\n" );printf (" --mlockall: lock all memory\n" );printf (" --no-mlockall: do not lock all memory\n" ); + printf(" --mp-alloc <native|anon|xmem|xmemhuge>: mempoolallocation method.\n"
app/test-pmd/parameters.c +1
-0+ " native: use regular DPDK memory to create and populatemempool\n"
app/test-pmd/parameters.c +1
-0+ " anon: use regular DPDK memory to create and anonymousmemory to populate mempool\n"
app/test-pmd/parameters.c +1
-0+ " xmem: use anonymous memory to create and populatemempool\n"
app/test-pmd/parameters.c +1
-0+ " xmemhuge: use anonymous hugepage memory to create andpopulate mempool\n");
}#ifdef
RTE_LIBRTE_CMDLINEapp/test-pmd/parameters.c +15
-1@@ -625,6 +630,7 @@ launch_args_parse(int argc, char** argv) { "vxlan-gpe-port", 1, 0, 0 }, { "mlockall", 0, 0, 0 }, { "no-mlockall", 0, 0, 0 }, + { "mp-alloc", 1, 0, 0 }, { 0, 0, 0, 0 }, };@@ -743,7 +749,22 @@ launch_args_parse(int argc, char** argv)if (!strcmp (lgopts[opt_idx].name,"numa" )) numa_support =1 ;if (!strcmp (lgopts[opt_idx].name,"mp-anon" )) { - mp_anon =1 ; + mp_alloc_type =MP_ALLOC_ANON ; + } +if (!strcmp (lgopts[opt_idx].name,"mp-alloc" )) { +if (!strcmp (optarg,"native" )) + mp_alloc_type =MP_ALLOC_NATIVE ; +else if (!strcmp (optarg,"anon" )) + mp_alloc_type =MP_ALLOC_ANON ; +else if (!strcmp (optarg,"xmem" )) + mp_alloc_type =MP_ALLOC_XMEM ; +else if (!strcmp (optarg,"xmemhuge" )) + mp_alloc_type =MP_ALLOC_XMEM_HUGE ; +else + rte_exit(EXIT_FAILURE, + "mp-alloc %s invalid - must be:"
app/test-pmd/parameters.c launch_args_parse(int argc, char** argv) · line 766
Quoted 1 line
app/test-pmd/parameters.c +1
-0+ "native, anon or xmem\n",
Should xmemhuge be added to above line?
Quoted 105 lines
app/test-pmd/parameters.c +1
-0+ optarg); }if (!strcmp (lgopts[opt_idx].name,"port-numa-config" )){
app/test-pmd/parameters.c +0
-0if (parse_portnuma_config (optarg))app/test-pmd/testpmd.c index renamed from app/test-pmd/testpmd.c +0
-0
001f0e552..255a9c664100644app/test-pmd/testpmd.c +11
-0@@ -27,6 +27,7 @@#include <rte_log.h> #include <rte_debug.h> #include <rte_cycles.h> +#include <rte_malloc_heap.h> #include <rte_memory.h> #include <rte_memcpy.h> #include <rte_launch.h> @@ -63,6 +64,22 @@#include "testpmd.h" +#ifndef MAP_HUGETLB +/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */ +#define HUGE_FLAG (0x40000) #else #define HUGE_FLAG MAP_HUGETLB#endif + +#ifndef MAP_HUGE_SHIFT +/* older kernels (or FreeBSD) will not have this define */ #define +HUGE_SHIFT (26) #else #define HUGE_SHIFT MAP_HUGE_SHIFT #endif + +#define EXTMEM_HEAP_NAME "extmem" +uint16_t verbose_level =0 ;/**< Silent by default. */ int testpmd_logtype;/**< Log type for testpmd logs */
app/test-pmd/testpmd.c +0
-1@@ -88,9 +105,13 @@ uint8_t numa_support = 1; /**< numa enabled bydefault */ uint8_t socket_num =UMA_NO_CONFIG ;/* -* Use ANONYMOUS mapped memory (might be not physically continuous) for mbufs.
+ * Select mempool allocation type:
+ * - native: use regular DPDK memory
+ * - anon: use regular DPDK memory to create mempool, but populate using
+ * anonymous memory (may not be IOVA-contiguous)
+ * - xmem: use externally allocated hugepage memory
*/app/test-pmd/testpmd.c +1
-1-uint8_t mp_anon =0 ; +uint8_t mp_alloc_type =MP_ALLOC_NATIVE ;/* * Store specified sockets on which memory pool to be used by ports @@ - 527,6 +548,231 @@
set_def_fwd_config(void)app/test-pmd/testpmd.c +49
-0set_default_fwd_ports_config (); } +/* extremely pessimistic estimation of memory required to create a +mempool */ static int calc_mem_size (uint32_t nb_mbufs,uint32_t +mbuf_sz,size_t pgsz,size_t *out) { +unsigned int n_pages, mbuf_per_pg, leftover; +uint64_t total_mem, mbuf_mem, obj_sz; + +/* there is no good way to predict how much space the mempool will +* occupy because it will allocate chunks on the fly, and some of those +* will come from default DPDK memory while some will come from our +* external memory, so just assume 128MB will be enough for everyone. +*/ +uint64_t hdr_mem =128 <<20 ; + +/* account for possible non-contiguousness */ + obj_sz =rte_mempool_calc_obj_size (mbuf_sz,0 ,NULL ); +if (obj_sz > pgsz) { +TESTPMD_LOG (ERR ,"Object size is bigger than page size\n" ); +return -1 ; + } + + mbuf_per_pg = pgsz / obj_sz; + leftover = (nb_mbufs % mbuf_per_pg) >0 ; + n_pages = (nb_mbufs / mbuf_per_pg) + leftover; + + mbuf_mem = n_pages * pgsz; + + total_mem =RTE_ALIGN (hdr_mem + mbuf_mem, pgsz); + +if (total_mem >SIZE_MAX ) { +TESTPMD_LOG (ERR ,"Memory size too big\n" ); +return -1 ; + } + *out = (size_t )total_mem; + +return 0 ; +} + +static inline uint32_t +bsf64 (uint64_t v) +{ +return (uint32_t )__builtin_ctzll (v); +} + +static inline uint32_t +log2_u64 (uint64_t v) +{ +if (v ==0 ) +return 0 ; + v =rte_align64pow2 (v);
app/test-pmd/testpmd.c set_def_fwd_config(void) · line 610
Quoted 10 lines
app/test-pmd/testpmd.c +10
-0+return bsf64 (v); +} + +static int +pagesz_flags (uint64_t page_sz) +{ +/* as per mmap() manpage, all page sizes are log2 of page size +* shifted by MAP_HUGE_SHIFT +*/ +int log2 =log2_u64 (page_sz);
Missing blank line after declarations.
Quoted 318 lines
app/test-pmd/testpmd.c +30
-0+return (log2 <<HUGE_SHIFT ); +} + +static void * +alloc_mem (size_t memsz,size_t pgsz,bool huge) { +void *addr; +int flags; + +/* allocate anonymous hugepages */ + flags =MAP_ANONYMOUS |MAP_PRIVATE ; +if (huge) + flags |=HUGE_FLAG |pagesz_flags (pgsz); + + addr =mmap (NULL , memsz,PROT_READ |PROT_WRITE , flags,-1 ,0 ); +if (addr ==MAP_FAILED ) +return NULL ; + +return addr; +} + +struct extmem_param { +void *addr; +size_t len; +size_t pgsz; +rte_iova_t *iova_table; +unsigned int iova_table_len; +}; + +static int +create_extmem(uint32_t nb_mbufs, uint32_t mbuf_sz, struct extmem_param*param,
app/test-pmd/testpmd.c +3
-0+ bool huge) +{ +uint64_t pgsizes[] = {RTE_PGSIZE_2M ,RTE_PGSIZE_1G ,/* x86_64, ARM */
app/test-pmd/testpmd.c +21
-0+ RTE_PGSIZE_16M, RTE_PGSIZE_16G}; /* POWER */ +unsigned int cur_page, n_pages, pgsz_idx; +size_t mem_sz, cur_pgsz; +rte_iova_t *iovas =NULL ; +void *addr; +int ret; + +for (pgsz_idx =0 ; pgsz_idx <RTE_DIM (pgsizes); pgsz_idx++) { +/* skip anything that is too big */ +if (pgsizes[pgsz_idx] >SIZE_MAX ) +continue ; + + cur_pgsz = pgsizes[pgsz_idx]; + +/* if we were told not to allocate hugepages, override */ +if (!huge) + cur_pgsz =sysconf (_SC_PAGESIZE); + + ret =calc_mem_size (nb_mbufs, mbuf_sz, cur_pgsz, &mem_sz); +if (ret <0 ) { + TESTPMD_LOG(ERR, "Cannot calculate memorysize\n");
app/test-pmd/testpmd.c +20
-0+return -1 ; + } + +/* allocate our memory */ + addr =alloc_mem (mem_sz, cur_pgsz, huge); + +/* if we couldn't allocate memory with a specified page size, +* that doesn't mean we can't do it with other page sizes, so +* try another one. +*/ +if (addr ==NULL ) +continue ; + +/* store IOVA addresses for every page in this memory area */ + n_pages = mem_sz / cur_pgsz; + + iovas =malloc (sizeof (*iovas) * n_pages); + +if (iovas ==NULL ) { + TESTPMD_LOG(ERR, "Cannot allocate memory for iovaaddresses\n");
app/test-pmd/testpmd.c +71
-0+ goto fail; + } +/* lock memory if it's not huge pages */ +if (!huge) +mlock (addr, mem_sz); + +/* populate IOVA addresses */ +for (cur_page =0 ; cur_page < n_pages; cur_page++) { +rte_iova_t iova; +size_t offset; +void *cur; + + offset = cur_pgsz * cur_page; + cur =RTE_PTR_ADD (addr, offset); + iova =rte_mem_virt2iova (cur); + + iovas[cur_page] = iova; + } + +break ; + } +/* if we couldn't allocate anything */ +if (iovas ==NULL ) +return -1 ; + + param->addr = addr; + param->len = mem_sz; + param->pgsz = cur_pgsz; + param->iova_table = iovas; + param->iova_table_len = n_pages; + +return 0 ; +fail: +if (iovas) +free (iovas); +if (addr) +munmap (addr, mem_sz); + +return -1 ; +} + +static int +setup_extmem (uint32_t nb_mbufs,uint32_t mbuf_sz,bool huge) { +struct extmem_param param; +int socket_id, ret; + +memset (¶m,0 ,sizeof (param)); + +/* check if our heap exists */ + socket_id =rte_malloc_heap_get_socket (EXTMEM_HEAP_NAME ); +if (socket_id <0 ) { +/* create our heap */ + ret =rte_malloc_heap_create (EXTMEM_HEAP_NAME ); +if (ret <0 ) { +TESTPMD_LOG (ERR ,"Cannot create heap\n" ); +return -1 ; + } + } + + ret =create_extmem (nb_mbufs, mbuf_sz, ¶m, huge); +if (ret <0 ) { +TESTPMD_LOG (ERR ,"Cannot create memory area\n" ); +return -1 ; + } + +/* we now have a valid memory area, so add it to heap */ + ret =rte_malloc_heap_memory_add (EXTMEM_HEAP_NAME , + param.addr, param.len, param.iova_table, + param.iova_table_len, param.pgsz); + +/* when using VFIO, memory is automatically mapped for DMA by EAL */
app/test-pmd/testpmd.c +58
-11+ +/* not needed any more */ +free (param.iova_table); + +if (ret <0 ) { +TESTPMD_LOG (ERR ,"Cannot add memory to heap\n" ); +munmap (param.addr, param.len); +return -1 ; + } + +/* success */ + +TESTPMD_LOG (DEBUG ,"Allocated %zuMB of external memory\n" , + param.len >>20 ); + +return 0 ; +} +/* * Configuration initialisation done once at init time. */ @@ -545,27 +791,59 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n" , pool_name, nb_mbuf, mbuf_seg_size, socket_id); -if (mp_anon !=0 ) { - rte_mp =rte_mempool_create_empty (pool_name, nb_mbuf, - mb_size, (unsigned ) mb_mempool_cache, -sizeof (struct rte_pktmbuf_pool_private ), - socket_id,0 ); -if (rte_mp ==NULL ) - goto err; +switch (mp_alloc_type) { +case MP_ALLOC_NATIVE : + { +/* wrapper to rte_mempool_create() */ +TESTPMD_LOG (INFO ,"preferred mempool ops selected: %s\n", +rte_mbuf_best_mempool_ops ()); + rte_mp =rte_pktmbuf_pool_create (pool_name, nb_mbuf, + mb_mempool_cache,0 , mbuf_seg_size, socket_id); +break ; + } +case MP_ALLOC_ANON : + { + rte_mp =rte_mempool_create_empty (pool_name, nb_mbuf, + mb_size, (unsigned int ) mb_mempool_cache, +sizeof (struct rte_pktmbuf_pool_private ), + socket_id,0 ); +if (rte_mp ==NULL ) + goto err; + +if (rte_mempool_populate_anon (rte_mp) ==0 ) { +rte_mempool_free (rte_mp); + rte_mp =NULL ; + goto err; + } +rte_pktmbuf_pool_init (rte_mp,NULL ); +rte_mempool_obj_iter (rte_mp, rte_pktmbuf_init,NULL ); +break ; + } +case MP_ALLOC_XMEM : +case MP_ALLOC_XMEM_HUGE : + { +int heap_socket; +bool huge = mp_alloc_type ==MP_ALLOC_XMEM_HUGE ; -if (rte_mempool_populate_anon (rte_mp) ==0 ) { -rte_mempool_free (rte_mp); - rte_mp =NULL ; - goto err; +if (setup_extmem (nb_mbuf, mbuf_seg_size, huge) <0 ) + rte_exit(EXIT_FAILURE, "Could not create external memory\n"); + + heap_socket = + rte_malloc_heap_get_socket(EXTMEM_HEAP_NAME); +if (heap_socket <0 ) + rte_exit(EXIT_FAILURE, "Could not get externalmemory socket
app/test-pmd/testpmd.c +3
-0+ID\n"); + + TESTPMD_LOG(INFO, "preferred mempool opsselected: %s\n",
app/test-pmd/testpmd.c +2
-0+ rte_mbuf_best_mempool_ops()); + rte_mp = rte_pktmbuf_pool_create(pool_name,
nb_mbuf,app/test-pmd/testpmd.c +1
-0+ mb_mempool_cache, 0,
mbuf_seg_size,app/test-pmd/testpmd.c +6
-0+ heap_socket); +break ; + } +default : + { + rte_exit(EXIT_FAILURE, "Invalid mempool creationmode\n");
app/test-pmd/testpmd.c +0
-8} -rte_pktmbuf_pool_init (rte_mp,NULL ); -rte_mempool_obj_iter (rte_mp, rte_pktmbuf_init,NULL ); - }else { -/* wrapper to rte_mempool_create() */ -TESTPMD_LOG (INFO ,"preferred mempool ops selected: %s\n" , -rte_mbuf_best_mempool_ops ()); - rte_mp =rte_pktmbuf_pool_create (pool_name, nb_mbuf, - mb_mempool_cache,0 , mbuf_seg_size, socket_id); } err:app/test-pmd/testpmd.h index renamed from app/test-pmd/testpmd.h +0
-0
a1f661472..65e0cec90100644app/test-pmd/testpmd.h +4
-0@@ -69,6 +69,16 @@ enum { PORT_TOPOLOGY_LOOP, }; +enum { +MP_ALLOC_NATIVE ,/**< allocate and populate mempool natively */ +MP_ALLOC_ANON , +/**< allocate mempool natively, but populate using anonymous memory */
app/test-pmd/testpmd.h +4
-0+MP_ALLOC_XMEM , +/**< allocate and populate mempool using anonymous memory */ +MP_ALLOC_XMEM_HUGE +/**< allocate and populate mempool using anonymous hugepage memory */
app/test-pmd/testpmd.h +2
-0+}; +#ifdef RTE_TEST_PMD_RECORD_BURST_STATS /** * The data structure associated with RX and TX packet burst statistics @@ - 304,7 +314,8 @@ extern
uint8_tnuma_support; /**< set by "--numa" parameter */ externuint16_tport_topology; /**< set by "--port-topology" parameter */ externuint8_tno_flush_rx; /**<set by "--no-flush-rx" parameter */ externuint8_tflow_isolate_all; /**< set by "--flow-isolate-all */ -externuint8_tmp_anon; /**< set by "--mp-anon" parameter */app/test-pmd/testpmd.h +2
-0+extern uint8_t mp_alloc_type; +/**< set by "--mp-anon" or "--mp-alloc" parameter */ extern uint8_t no_link_check;/**<set by "--disable-link-check" parameter */ extern volatile int
test_done; /* stop packet forwarding when set to 1. */ externuint8_tlsc_interrupt; /**< disabled by "--no-lsc-interrupt" parameter */ diff --gita/doc/guides/testpmd_app_ug/run_app.rstb/doc/guides/testpmd_app_ug/run_app.rst
indexf301c2b6f..67a8532a4100644doc/guides/testpmd_app_ug/run_app.rst +11
-0@@ -498,3 +498,15 @@ The commandline options are: * ``--no-mlockall`` Disable locking all memory. + +* ``--mp-alloc <native|anon|xmem|xmemhuge>`` + + Select mempool allocation mode: + + * native: create and populate mempool using native DPDK memory + * anon: create mempool using native DPDK memory, but populate using + anonymous memory + * xmem: create and populate mempool using externally and anonymously + allocated area + * xmemhuge: create and populate mempool using externally and
Quoted 4 lines
anonymously
+ allocated hugepage area
--
2.17.1
The following checkpatch warnings is testpmd.c should probably be fixed.
WARNING: line over 80 characters
#332: FILE: app/test-pmd/testpmd.c:685:
+ TESTPMD_LOG(ERR, "Cannot allocate memory for iova addresses\n");
WARNING: line over 80 characters
#441: FILE: app/test-pmd/testpmd.c:798:
+ TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
WARNING: line over 80 characters
#476: FILE: app/test-pmd/testpmd.c:829:
+ rte_exit(EXIT_FAILURE, "Could not create external memory\n");
WARNING: line over 80 characters
#481: FILE: app/test-pmd/testpmd.c:834:
+ rte_exit(EXIT_FAILURE, "Could not get external memory socket ID\n");
WARNING: line over 80 characters
#483: FILE: app/test-pmd/testpmd.c:836:
+ TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
WARNING: line over 80 characters
#492: FILE: app/test-pmd/testpmd.c:845:
+ rte_exit(EXIT_FAILURE, "Invalid mempool creation mode\n");
Regards,
Bernard.
Re: [PATCH v8 19/21] app/testpmd: add support for external memory
Hi Bernard,
Thanks for your review! Comments inline.
Quoted 7 lines
app/test-pmd/parameters.c launch_args_parse(int argc, char** argv) · line 766
Quoted 4 lines
(quoted diff) +2
-0+ rte_exit(EXIT_FAILURE, + "mp-alloc %s invalid - must be:"
app/test-pmd/parameters.c +1
-0+ "native, anon or xmem\n",Should xmemhuge be added to above line?
Yes :)
Quoted 12 lines
Quoted 12 lines
app/test-pmd/parameters.c +1
-0+ optarg); }if (!strcmp (lgopts[opt_idx].name,"port-numa-config" )){
if (parse_portnuma_config(optarg))app/test-pmd/testpmd.c index renamed from app/test-pmd/testpmd.c +0
-0
001f0e552..255a9c664100644app/test-pmd/testpmd.c +0
-0@@ -27,6 +27,7 @@#include <rte_log.h> #include <rte_debug.h>
<snip>
Quoted 10 lines
app/test-pmd/testpmd.c set_def_fwd_config(void) · line 610
Quoted 7 lines
app/test-pmd/testpmd.c +7
-0+static int +pagesz_flags (uint64_t page_sz) +{ +/* as per mmap() manpage, all page sizes are log2 of page size +* shifted by MAP_HUGE_SHIFT +*/ +int log2 =log2_u64 (page_sz);Missing blank line after declarations.
Thanks, will fix.
Quoted 9 lines
Quoted 9 lines
app/test-pmd/testpmd.c +9
-0+return (log2 <<HUGE_SHIFT ); +} + +static void * +alloc_mem (size_t memsz,size_t pgsz,bool huge) { +void *addr; +int flags; + +/* allocate anon
<snip>
Quoted 25 lines
Quoted 2 lines
--
2.17.1The following checkpatch warnings is
testpmd.cshould probably be fixed.WARNING: line over 80 characters
#332: FILE:app/test-pmd/testpmd.c:685:
+TESTPMD_LOG(ERR, "Cannot allocate memory for iova addresses\n");WARNING: line over 80 characters
#441: FILE:app/test-pmd/testpmd.c:798:
+ TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",WARNING: line over 80 characters
#476: FILE:app/test-pmd/testpmd.c:829:
+rte_exit(EXIT_FAILURE, "Could not create external memory\n");WARNING: line over 80 characters
#481: FILE:app/test-pmd/testpmd.c:834:
+rte_exit(EXIT_FAILURE, "Could not get external memory socket ID\n");WARNING: line over 80 characters
#483: FILE:app/test-pmd/testpmd.c:836:
+ TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
Quoted 9 lines
WARNING: line over 80 characters
#492: FILE:app/test-pmd/testpmd.c:845:
+rte_exit(EXIT_FAILURE, "Invalid mempool creation mode\n");Regards,
Bernard.
These should be ignored (and they indeed are ignored by DPDK's own checkpatch script). Strings are allowed to go over 80 characters so as to not make it hard to grep for them.
Signature · 2 lines
Thanks, Anatoly
PATCH v8 20/21 doc: add external memory feature to the release notes
In reply to Anatoly BurakovDocument the addition of external memory support to DPDK.
Signed-off-by Anatoly Burakov <[email protected]>
--- doc/guides/rel_notes/release_18_11.rst | 6 ++++++ 1 file changed, 6 insertions(+)
doc/guides/rel_notes/release_18_11.rst +6 -0
@@ -54,6 +54,12 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= +* **Added support for using externally allocated memory in DPDK.** + + DPDK has gained support for creating new ``rte_malloc`` heaps referencing + memory that was created outside of DPDK's own page allocator, and using that + memory natively with any other DPDK library or data structure. + * **Add support to offload more flow match and actions for CXGBE PMD** Flow API support has been enhanced for CXGBE Poll Mode Driver to offload:
Signature · 1 line
2.17.1