Skip to contentSkip to conversation

config: customize max memzones configuration

RFC In review Ophir Munk 3 participants 4 messages Jan 30, 2023 – Feb 12, 2023

lore
Share & tools
lore

Ophir Munk

author
#0 lore raw

RFC 1/1 config: customize max memzones configuration

In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.

An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.

I would like to suggest three options.

Option 1
========
Add a Meson option in meson options.txt and remove the RTE_MAX_MEMZONE definition from config/rte_config.h

For example,

config/meson.build

 # set other values pulled from the build options
 dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
 +dpdk_conf.set('RTE_MAX_MEMZONE', get_option('max_memzones'))
 dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))

meson_options.txt

option('max_lcores', type: 'integer', value: 128,
description: 'maximum number of cores/threads supported by EAL')
+option('max_memzones', type: 'integer', value: 2560,
+ description: 'maximum number of memory zones supported by EAL')
option('max_numa_nodes', type: 'integer', value: 32,
description: 'maximum number of NUMA nodes supported by EAL')

config/rte_config.h

#define RTE_MAX_MEM_MB_PER_TYPE 65536
-#define RTE_MAX_MEMZONE 2560
#define RTE_MAX_TAILQ 32

Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and make RTE_MAX_MEMZONE conditional in config/rte_config.h

For example, see the code of this commit.

Option 3
========
Add a function which must be called before rte_eal_init():
void rte_memzone_set_max(int max) {memzone_max = max;} If not called, the default memzone (RTE_MAX_MEMZONE) is used.

With this option there is no need to recompile DPDK and it allows using an in-box packaged DPDK.

---
 config/rte_config.h | 2 ++
 1 file changed, 2 insertions(+)
config/rte_config.h +2 -0
index fed174a..ca653cc 100644
@@ -35,7 +35,9 @@
 #define RTE_MAX_MEM_MB_PER_LIST 32768
 #define RTE_MAX_MEMSEG_PER_TYPE 32768
 #define RTE_MAX_MEM_MB_PER_TYPE 65536
+#ifndef RTE_MAX_MEMZONE
 #define RTE_MAX_MEMZONE 2560
+#endif
 #define RTE_MAX_TAILQ 32
 #define RTE_LOG_DP_LEVEL RTE_LOG_INFO
 #define RTE_BACKTRACE 1
Signature · 1 line
2.8.4
#1 lore raw
Quoted 39 lines · Ophir Munk

In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.

An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.

I would like to suggest three options.

Option 1
========
Add a Meson option in meson options.txt and remove the RTE_MAX_MEMZONE definition from config/rte_config.h

For example,

config/meson.build

 # set other values pulled from the build options
 dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
 +dpdk_conf.set('RTE_MAX_MEMZONE', get_option('max_memzones'))
 dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))

meson_options.txt

option('max_lcores', type: 'integer', value: 128,
description: 'maximum number of cores/threads supported by EAL')
+option('max_memzones', type: 'integer', value: 2560,
+ description: 'maximum number of memory zones supported by EAL')
option('max_numa_nodes', type: 'integer', value: 32,
description: 'maximum number of NUMA nodes supported by EAL')

Quoted 6 lines

config/rte_config.h

#define RTE_MAX_MEM_MB_PER_TYPE 65536
-#define RTE_MAX_MEMZONE 2560
#define RTE_MAX_TAILQ 32

Of the 3 options, I think this first option would probably be my preferred one.

/Bruce

#2 lore raw
In reply to Ophir Munk

2023-01-30 11:23 (UTC+0200), Ophir Munk:

Quoted 21 lines

In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.

An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.

I would like to suggest three options.

Option 1
========
Add a Meson option in meson options.txt and remove the RTE_MAX_MEMZONE definition from config/rte_config.h

[...]

Quoted 8 lines

Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and make RTE_MAX_MEMZONE conditional in config/rte_config.h

For example, see the code of this commit.

Quoted 8 lines

Option 3
========
Add a function which must be called before rte_eal_init():
void rte_memzone_set_max(int max) {memzone_max = max;} If not called, the default memzone (RTE_MAX_MEMZONE) is used.

With this option there is no need to recompile DPDK and it allows using an in-box packaged DPDK.

[...]

Ideally, there should be no limitation at all.
I vote for some compile-time solution for now
with a plan to remove the restriction inside EAL later.
Option 2 does not expose a user-facing option that would be obsoleted anyway, so it seems preferable to me, but Option 1 is also OK and consistent.

RTE_MAX_MEMZONE is needed currently, because struct rte_memzone are stored in rte_fbarray (mapped file-backed array) with a fixed capacity. Unlike e.g. rte_eth_devices, this is not used for efficient access by index and is not even exposed to PMDs, so the storage can be changed painlessly. In DPDK, only net/qede uses this constant and also for slow-path bookkeeping.

author
#3 lore raw

Thanks Dmitry and Bruce for your inputs. Option 1 is my favorite. We all agree on Option 1. I will send a patch based on it.

Thank you.

Quoted 64 lines

-----Original Message-----
From: Dmitry Kozlyuk <[email protected]>
Sent: Monday, 30 January 2023 12:01
To: Ophir Munk <[email protected]>
Cc: [email protected]; Ophir Munk <[email protected]>; Matan Azrad <[email protected]>; Thomas Monjalon <[email protected]>; Bruce Richardson <[email protected]>; Lior Margalit <[email protected]>
Subject: Re: [RFC] config: customize max memzones configuration

2023-01-30 11:23 (UTC+0200), Ophir Munk:

Quoted 14 lines

In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.

An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.

Quoted 5 lines

I would like to suggest three options.

Option 1
========
Add a Meson option in meson options.txt and remove the

RTE_MAX_MEMZONE

Quoted 2 lines

definition from config/rte_config.h

[...]

Quoted 8 lines

Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and make RTE_MAX_MEMZONE conditional in config/rte_config.h

For example, see the code of this commit.

Quoted 8 lines

Option 3
========
Add a function which must be called before rte_eal_init():
void rte_memzone_set_max(int max) {memzone_max = max;} If not called, the default memzone (RTE_MAX_MEMZONE) is used.

With this option there is no need to recompile DPDK and it allows using an in-box packaged DPDK.

[...]

Ideally, there should be no limitation at all.
I vote for some compile-time solution for now with a plan to remove the restriction inside EAL later.
Option 2 does not expose a user-facing option that would be obsoleted anyway, so it seems preferable to me, but Option 1 is also OK and consistent.

RTE_MAX_MEMZONE is needed currently, because struct rte_memzone are stored in rte_fbarray (mapped file-backed array) with a fixed capacity. Unlike e.g. rte_eth_devices, this is not used for efficient access by index and is not even exposed to PMDs, so the storage can be changed painlessly. In DPDK, only net/qede uses this constant and also for slow-path bookkeeping.