config: customize max memzones configuration
RFC 1/1 config: customize max memzones configuration
In current DPDK the RTE_MAX_MEMZONE definition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.
An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.
I would like to suggest three options.
Option 1
========
Add a Meson option in meson options.txt and remove the RTE_MAX_MEMZONE definition from config/rte_config.h
For example,
config/meson.build
# set other values pulled from the build options
dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores'))
+dpdk_conf.set('RTE_MAX_MEMZONE', get_option('max_memzones'))
dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))meson_options.txt
option('max_lcores', type: 'integer', value: 128,
description: 'maximum number of cores/threads supported by EAL')
+option('max_memzones', type: 'integer', value: 2560,
+ description: 'maximum number of memory zones supported by EAL')
option('max_numa_nodes', type: 'integer', value: 32,
description: 'maximum number of NUMA nodes supported by EAL')
config/rte_config.h
#define RTE_MAX_MEM_MB_PER_TYPE 65536
-#define RTE_MAX_MEMZONE 2560
#define RTE_MAX_TAILQ 32
Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and make RTE_MAX_MEMZONE conditional in config/rte_config.h
For example, see the code of this commit.
Option 3
========
Add a function which must be called before rte_eal_init():
void rte_memzone_set_max(int max) {memzone_max = max;} If not called, the default memzone (RTE_MAX_MEMZONE) is used.
With this option there is no need to recompile DPDK and it allows using an in-box packaged DPDK.
--- config/rte_config.h | 2 ++ 1 file changed, 2 insertions(+)
config/rte_config.h +2 -0
@@ -35,7 +35,9 @@#define RTE_MAX_MEM_MB_PER_LIST 32768#define RTE_MAX_MEMSEG_PER_TYPE 32768#define RTE_MAX_MEM_MB_PER_TYPE 65536 +#ifndef RTE_MAX_MEMZONE #define RTE_MAX_MEMZONE 2560 +#endif #define RTE_MAX_TAILQ 32#define RTE_LOG_DP_LEVEL RTE_LOG_INFO#define RTE_BACKTRACE 1
Signature · 1 line
2.8.4
Quoted 39 lines · Ophir Munk
In current DPDK the
RTE_MAX_MEMZONEdefinition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.
I would like to suggest three options.
Option 1
========
Add a Meson option in meson options.txt and remove theRTE_MAX_MEMZONEdefinition fromconfig/rte_config.hFor example,
config/meson.build
# set other values pulled from the build options dpdk_conf.set('RTE_MAX_LCORE', get_option('max_lcores')) +dpdk_conf.set('RTE_MAX_MEMZONE', get_option('max_memzones')) dpdk_conf.set('RTE_MAX_NUMA_NODES', get_option('max_numa_nodes'))meson_options.txt
option('max_lcores', type: 'integer', value: 128,
description: 'maximum number of cores/threads supported by EAL')
+option('max_memzones', type: 'integer', value: 2560,
+ description: 'maximum number of memory zones supported by EAL')
option('max_numa_nodes', type: 'integer', value: 32,
description: 'maximum number of NUMA nodes supported by EAL')
Quoted 6 lines
config/rte_config.h#define
RTE_MAX_MEM_MB_PER_TYPE65536
-#defineRTE_MAX_MEMZONE2560
#defineRTE_MAX_TAILQ32
Of the 3 options, I think this first option would probably be my preferred one.
/Bruce
2023-01-30 11:23 (UTC+0200), Ophir Munk:
Quoted 21 lines
In current DPDK the
RTE_MAX_MEMZONEdefinition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.
I would like to suggest three options.
Option 1
========
Add a Meson option in meson options.txt and remove theRTE_MAX_MEMZONEdefinition fromconfig/rte_config.h
[...]
Quoted 8 lines
Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and makeRTE_MAX_MEMZONEconditional inconfig/rte_config.hFor example, see the code of this commit.
Quoted 8 lines
Option 3
========
Add a function which must be called beforerte_eal_init():
voidrte_memzone_set_max(int max){memzone_max= max;} If not called, the default memzone (RTE_MAX_MEMZONE) is used.With this option there is no need to recompile DPDK and it allows using an in-box packaged DPDK.
[...]
Ideally, there should be no limitation at all.
I vote for some compile-time solution for now
with a plan to remove the restriction inside EAL later.
Option 2 does not expose a user-facing option that would be obsoleted anyway, so it seems preferable to me, but Option 1 is also OK and consistent.
RTE_MAX_MEMZONE is needed currently, because struct rte_memzone are stored in rte_fbarray (mapped file-backed array) with a fixed capacity. Unlike e.g. rte_eth_devices, this is not used for efficient access by index and is not even exposed to PMDs, so the storage can be changed painlessly. In DPDK, only net/qede uses this constant and also for slow-path bookkeeping.
Thanks Dmitry and Bruce for your inputs. Option 1 is my favorite. We all agree on Option 1. I will send a patch based on it.
Thank you.
Quoted 64 lines
-----Original Message-----
From: Dmitry Kozlyuk <[email protected]>
Sent: Monday, 30 January 2023 12:01
To: Ophir Munk <[email protected]>
Cc: [email protected]; Ophir Munk <[email protected]>; Matan Azrad <[email protected]>; Thomas Monjalon <[email protected]>; Bruce Richardson <[email protected]>; Lior Margalit <[email protected]>
Subject: Re: [RFC] config: customize max memzones configuration2023-01-30 11:23 (UTC+0200), Ophir Munk:
Quoted 14 lines
In current DPDK the
RTE_MAX_MEMZONEdefinition is unconditionally hard coded as 2560. For applications requiring different values of this parameter – it is more convenient to set its value as part of the meson command line or to set the max value via an rte API - rather than changing the dpdk source code per application.An example would be of an application that uses the DPDK mempool library which is based on DPDK memzone library. The application may need to create a number of steering tables, each of which will require its own mempool allocation. This RFC is not about how to optimize the application usage of mempool nor about how to improve the mempool implementation based on memzone. It is about how to make the max memzone definition - build-time or run-time customized.
Quoted 5 lines
I would like to suggest three options.
Option 1
========
Add a Meson option in meson options.txt and remove the
RTE_MAX_MEMZONEQuoted 2 lines
definition from
config/rte_config.h[...]
Quoted 8 lines
Option 2
========
Use Meson setup -Dc_args="-DRTE_MAX_MEMZONE=XXX" and makeRTE_MAX_MEMZONEconditional inconfig/rte_config.hFor example, see the code of this commit.
Quoted 8 lines
Option 3
========
Add a function which must be called beforerte_eal_init():
voidrte_memzone_set_max(int max){memzone_max= max;} If not called, the default memzone (RTE_MAX_MEMZONE) is used.With this option there is no need to recompile DPDK and it allows using an in-box packaged DPDK.
[...]
Ideally, there should be no limitation at all.
I vote for some compile-time solution for now with a plan to remove the restriction inside EAL later.
Option 2 does not expose a user-facing option that would be obsoleted anyway, so it seems preferable to me, but Option 1 is also OK and consistent.
RTE_MAX_MEMZONEis needed currently, becausestruct rte_memzoneare stored inrte_fbarray(mapped file-backed array) with a fixed capacity. Unlike e.g.rte_eth_devices, this is not used for efficient access by index and is not even exposed to PMDs, so the storage can be changed painlessly. In DPDK, only net/qede uses this constant and also for slow-path bookkeeping.