mempool: fix mempool obj alignment for non x86
PATCH 1/1 mempool: fix mempool obj alignment for non x86
From: Jerin Jacob <[email protected]>
The exiting optimize_object_size() function address the memory object alignment constraint on x86 for better performance.
Different (Mirco) architecture may have different memory alignment constraint for better performance and it not same as the existing optimize_object_size() function. Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing arch_mem_object_align() function to abstract the differences in different (mirco) architectures and avoid wasting memory for mempool object alignment for the architecture the existing optimize_object_size() is not valid.
Additional details:
https://www.mail-archive.com/[email protected]/msg149157.html
--- doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-)
doc/guides/prog_guide/mempool_lib.rst +3 -3
@@ -27,10 +27,10 @@ In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), statistics about get from/put in the pool are stored in the mempool structure. Statistics are per-lcore to avoid concurrent access to statistics counters. -Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on X86 architecture +------------------------------------------------ -Depending on hardware memory configuration, performance can be greatly improved by adding a specific padding between objects. +Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects. The objective is to ensure that the beginning of each object starts on a different channel and rank in memory so that all channels are equally loaded. This is particularly true for packet buffers when doing L3 forwarding or flow classification.
lib/librte_mempool/rte_mempool.c +13 -4
@@ -45,6 +45,7 @@ EAL_REGISTER_TAILQ(rte_mempool_tailq)#define CALC_CACHE_FLUSHTHRESH (c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) +#if defined(RTE_ARCH_X86 )/* * return the greatest common divisor between a and b (fast algorithm) *
@@ -74,12 +75,13 @@ static unsigned get_gcd(unsigned a, unsigned b) }/* -* Depending on memory configuration, objects addresses are spread +* Depending on memory configuration on x86 arch, objects addresses are spread * between channels and ranks in RAM: the pool allocator will add * padding between objects. This function return the new size of the * object. */ -static unsigned optimize_object_size (unsigned obj_size) +static unsigned +arch_mem_object_align (unsigned obj_size) {unsigned nrank, nchan;unsigned new_obj_size;
@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++;return new_obj_size *RTE_MEMPOOL_ALIGN ; } +#else +static unsigned +arch_mem_object_align (unsigned obj_size) +{ +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;
@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,*/ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size =optimize_object_size (sz->header_size + sz->elt_size + - sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; }
Signature · 1 line
2.24.1
Hi Jerin,
It got two coding style warnings, otherwise,
WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of 'unsigned' #144: FILE: lib/librte_mempool/rte_mempool.c:84:
+arch_mem_object_align(unsigned obj_size)
WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of 'unsigned' #154: FILE: lib/librte_mempool/rte_mempool.c:106:
+arch_mem_object_align(unsigned obj_size)
Quoted 5 lines · Gavin Hu
Hi Jerin,
It got two coding style warnings, otherwise,
Reviewed-by Gavin Hu <[email protected]>
Thanks Gavin for review. Since the existing code is using "unsigned" in that file, I thought of not change by this patch.
If someone thinks, It is better to change then I can send v2 by fixing "unsigned" to "unsigned int" across the file as a first patch in the series.
Quoted 8 lines
WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of 'unsigned' #144: FILE:
lib/librte_mempool/rte_mempool.c:84:
+arch_mem_object_align(unsignedobj_size)WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of 'unsigned' #154: FILE:
lib/librte_mempool/rte_mempool.c:106:
+arch_mem_object_align(unsignedobj_size)
Re: [dpdk-dev] [PATCH] mempool: fix mempool obj alignment for nonx86
Quoted 8 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Jerin Jacob
Sent: Friday, December 20, 2019 4:45 AMQuoted 3 lines · Gavin Hu
Hi Jerin,
Quoted 9 lines
Quoted 2 lines
It got two coding style warnings, otherwise,
Reviewed-by Gavin Hu <[email protected]>Thanks Gavin for review. Since the existing code is using "unsigned" in that file, I thought of not change by this patch.
If someone thinks, It is better to change then I can send v2 by fixing "unsigned" to "unsigned int" across the file as a first patch in the series.
The use of the type "unsigned" is a general issue with older DPDK libraries. Anyone touching any of these libraries will get these warnings. They should all be fixed. I'm not saying you should do it; I'm only suggesting that someone should create a dedicated patch to fix them all.
Quoted 10 lines
Quoted 2 lines
WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of
'unsigned'
Quoted 4 lines
#144: FILE:
lib/librte_mempool/rte_mempool.c:84:
+arch_mem_object_align(unsignedobj_size)WARNING:UNSPECIFIED_INT: Prefer 'unsigned int' to bare use of
'unsigned'
Quoted 2 lines
#154: FILE:
lib/librte_mempool/rte_mempool.c:106:
+arch_mem_object_align(unsignedobj_size)
<snip>
Quoted 11 lines
From: Jerin Jacob <[email protected]>
The exiting
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (Mirco) architecture may have different memory alignment constraint for better performance and it not same as the existing
optimize_object_size()function. Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
If I understand correctly, address interleaving is the characteristic of the memory controller and not the CPU.
For ex: different SoCs using the same Arm architecture might have different memory controllers. So, the solution should not be architecture specific, but SoC specific.
Quoted 73 lines
Introducing
arch_mem_object_align()function to abstract the differences in different (mirco) architectures and avoid wasting memory for mempool object alignment for the architecture the existingoptimize_object_size()is not valid.Additional details:
https://www.mail-archive.com/[email protected]/msg149157.htmlFixes af75078fece3 ("first public release")Signed-off-by Jerin Jacob <[email protected]>--- doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-)doc/guides/prog_guide/mempool_lib.rst +0
-0@@ -27,10 +27,10 @@ In debug mode(
CONFIG_RTE_LIBRTE_MEMPOOL_DEBUGis enabled), statistics about get from/put in the pool are stored in the mempool structure.
Statistics are per-lcore to avoid concurrent access to statistics counters.doc/guides/prog_guide/mempool_lib.rst +2
-3-Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on X86 architecture +------------------------------------------------ -Depending on hardware memory configuration, performance can be greatlyimproved by adding a specific padding between objects.
+Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects.
The objective is to ensure that the beginning of each object starts on a
different channel and rank in memory so that all channels are equally loaded.This is particularly true for packet buffers when doing L3 forwarding or flow
classification.lib/librte_mempool/rte_mempool.c +5
-2@@ -45,6 +45,7 @@ EAL_REGISTER_TAILQ(rte_mempool_tailq)#define CALC_CACHE_FLUSHTHRESH (c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) +#if defined(RTE_ARCH_X86 )/* * return the greatest common divisor between a and b (fast algorithm) * @@ -74,12 +75,13 @@ static unsigned get_gcd(unsigned a, unsigned b) }/* -* Depending on memory configuration, objects addresses are spread +* Depending on memory configuration on x86 arch, objects addresses are +spread * between channels and ranks in RAM: the pool allocator will add * padding between objects. This function return the new size of the * object. */ -static unsigned optimize_object_size (unsigned obj_size) +static unsigned +arch_mem_object_align (unsigned obj_size) {unsigned nrank, nchan;unsigned new_obj_size;
lib/librte_mempool/rte_mempool.c static unsigned optimize_object_size(unsigned · line 104
Quoted 4 lines
lib/librte_mempool/rte_mempool.c +1
-0@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++; return new_obj_size * RTE_MEMPOOL_ALIGN; } +#else
This applies to add Arm (PPC as well) SoCs which might have different schemes depending on the memory controller. IMO, this should not be architecture specific.
Quoted 23 lines
lib/librte_mempool/rte_mempool.c +5
-1+static unsigned +arch_mem_object_align (unsigned obj_size) { +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size,uint32_t flags, */ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size = optimize_object_size(sz->header_size + sz-Quoted 1 line
elt_size+lib/librte_mempool/rte_mempool.c +2
-2- sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; } --2.24.1
Quoted 9 lines · Honnappa Nagarahalli
<snip>
Quoted 6 lines
From: Jerin Jacob <[email protected]>
The exiting
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.
Quoted 7 lines
Quoted 5 lines
Different (Mirco) architecture may have different memory alignment constraint for better performance and it not same as the existing
optimize_object_size()function. Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.If I understand correctly, address interleaving is the characteristic of the memory controller and not the CPU.
For ex: different SoCs using the same Arm architecture might have different memory controllers. So, the solution should not be architecture specific, but SoC specific.
Yes. See below.
Quoted 4 lines
Quoted 4 lines
lib/librte_mempool/rte_mempool.c +2
-1-static unsigned optimize_object_size (unsigned obj_size) +static unsigned +arch_mem_object_align (unsigned obj_size) {
Quoted 8 lines
lib/librte_mempool/rte_mempool.c static unsigned optimize_object_size(unsigned · line 104
Quoted 7 lines
unsigned nrank, nchan; unsigned new_obj_size;lib/librte_mempool/rte_mempool.c +1
-0@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++; return new_obj_size * RTE_MEMPOOL_ALIGN; } +#else This applies to add Arm (PPC as well) SoCs which might have different schemes depending on the memory controller. IMO, this should not be architecture specific.
I agree in principle.
I will summarize the
https://www.mail-archive.com/[email protected]/msg149157.html feedback:
1) For x86 arch, it is architecture-specific
2) For power PC arch, It is architecture-specific
3) For the ARM case, it will be the memory controller specific.
4) For the ARM case, The memory controller is not using the existing x86 arch formula.
5) If it is memory/arch-specific, Can userspace code find the optimal alignment? In the case of octeontx2/arm64,
the memory controller does XOR on PA address which userspace code doesn't have much control.
This patch address the known case of (1), (2), (4) and (5). (2) can be added to this framework when POWER9 folks want it.
We can extend this patch to address (3) if there is a case. Without the actual requirement(If some can share the formula of alignment which is the memory
controller specific and it does not come under (4))) then we can create extra layer for the memory controller and abstraction to probe it.
Again there is no standard way of probing the memory controller in userspace and we need platform #define, which won't work for distribution build.
So solution needs to be arch-specific and then fine-tune to memory controller if possible.
I can work on creating an extra layer of code if some can provide the details of the memory controller and probing mechanism or this patch be extended to support such case if it arises in future.
Thoughts?
Quoted 25 lines
Quoted 24 lines
lib/librte_mempool/rte_mempool.c +5
-1+static unsigned +arch_mem_object_align (unsigned obj_size) { +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size,uint32_t flags, */ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size = optimize_object_size(sz->header_size + sz-Quoted 1 line
elt_size+lib/librte_mempool/rte_mempool.c +2
-2- sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; } --2.24.1
<snip>
Quoted 59 lines
Quoted 11 lines
Quoted 10 lines
From: Jerin Jacob <[email protected]>
The exiting
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (Mirco) architecture may have different memory alignment constraint for better performance and it not same as the existing
optimize_object_size()function. Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.If I understand correctly, address interleaving is the characteristic of the
memory controller and not the CPU.
Quoted 1 line
For ex: different SoCs using the same Arm architecture might have different
memory controllers. So, the solution should not be architecture specific, but SoC specific.
Yes. See below.
Quoted 4 lines
Quoted 4 lines
lib/librte_mempool/rte_mempool.c +2
-1-static unsigned optimize_object_size (unsigned obj_size) +static unsigned +arch_mem_object_align (unsigned obj_size) {Quoted 8 lines
lib/librte_mempool/rte_mempool.c static unsigned optimize_object_size(unsigned · line 104
Quoted 7 lines
unsigned nrank, nchan; unsigned new_obj_size;lib/librte_mempool/rte_mempool.c +1
-0@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++; return new_obj_size * RTE_MEMPOOL_ALIGN; } +#else This applies to add Arm (PPC as well) SoCs which might have different
schemes depending on the memory controller. IMO, this should not be architecture specific.
I agree in principle.
I will summarize the
https://www.mail-archive.com/[email protected]/msg149157.html feedback:1) For x86 arch, it is architecture-specific
2) For power PC arch, It is architecture-specific
3) For the ARM case, it will be the memory controller specific.
4) For the ARM case, The memory controller is not using the existing x86 arch formula.
5) If it is memory/arch-specific, Can userspace code find the optimal alignment? In the case of octeontx2/arm64, the memory controller does XOR on PA address which userspace code doesn't have much control.This patch address the known case of (1), (2), (4) and (5). (2) can be added to this framework when POWER9 folks want it.
We can extend this patch to address (3) if there is a case. Without the actual requirement(If some can share the formula of alignment which is the memory controller specific and it does not come under (4))) then we can create extra layer for the memory controller and abstraction to probe it. Again there is no standard way of probing the memory controller in userspace and we need platform #define, which won't work for distribution build.
So solution needs to be arch-specific and then fine-tune to memory controller if possible.
Quoted 2 lines
I can work on creating an extra layer of code if some can provide the details of the memory controller and probing mechanism or this patch be extended
Inputs for BlueField, DPAAx, ThunderX2 would be helpful.
Quoted 3 lines
to support such case if it arises in future.
Thoughts?
How much memory will this save for your platform? Is it affecting performance?
Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
lib/librte_mempool/rte_mempool.c +5
-1+static unsigned +arch_mem_object_align (unsigned obj_size) { +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size,uint32_t flags, */ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size = optimize_object_size(sz->header_size + sz-Quoted 1 line
elt_size+lib/librte_mempool/rte_mempool.c +3
-2- sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; } --2.24.1
Quoted 61 lines · Honnappa Nagarahalli
<snip>
Quoted 59 lines
Quoted 11 lines
Quoted 10 lines
From: Jerin Jacob <[email protected]>
The exiting
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (Mirco) architecture may have different memory alignment constraint for better performance and it not same as the existing
optimize_object_size()function. Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.If I understand correctly, address interleaving is the characteristic of the
memory controller and not the CPU.
Quoted 1 line
For ex: different SoCs using the same Arm architecture might have different
memory controllers. So, the solution should not be architecture specific, but SoC specific.
Yes. See below.
Quoted 4 lines
Quoted 4 lines
lib/librte_mempool/rte_mempool.c +2
-1-static unsigned optimize_object_size (unsigned obj_size) +static unsigned +arch_mem_object_align (unsigned obj_size) {Quoted 8 lines
lib/librte_mempool/rte_mempool.c static unsigned optimize_object_size(unsigned · line 104
Quoted 7 lines
unsigned nrank, nchan; unsigned new_obj_size;lib/librte_mempool/rte_mempool.c +1
-0@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++; return new_obj_size * RTE_MEMPOOL_ALIGN; } +#else This applies to add Arm (PPC as well) SoCs which might have different
schemes depending on the memory controller. IMO, this should not be architecture specific.
I agree in principle.
I will summarize the
https://www.mail-archive.com/[email protected]/msg149157.html feedback:1) For x86 arch, it is architecture-specific
2) For power PC arch, It is architecture-specific
3) For the ARM case, it will be the memory controller specific.
4) For the ARM case, The memory controller is not using the existing x86 arch formula.
5) If it is memory/arch-specific, Can userspace code find the optimal alignment? In the case of octeontx2/arm64, the memory controller does XOR on PA address which userspace code doesn't have much control.This patch address the known case of (1), (2), (4) and (5). (2) can be added to this framework when POWER9 folks want it.
We can extend this patch to address (3) if there is a case. Without the actual requirement(If some can share the formula of alignment which is the memory controller specific and it does not come under (4))) then we can create extra layer for the memory controller and abstraction to probe it. Again there is no standard way of probing the memory controller in userspace and we need platform #define, which won't work for distribution build.
So solution needs to be arch-specific and then fine-tune to memory controller if possible.
Quoted 3 lines
Quoted 2 lines
I can work on creating an extra layer of code if some can provide the details of the memory controller and probing mechanism or this patch be extended
Inputs for BlueField, DPAAx, ThunderX2 would be helpful.
Yes. Probably memory controller used in n1sdp SoC also.
Quoted 5 lines
Quoted 4 lines
to support such case if it arises in future.
Thoughts?
How much memory will this save for your platform? Is it affecting performance?
No performance difference.
The existing code adding the tailer for each objs. Additional space/Trailer space will be function of number of objects in mempool and its obj_size, its alignment and selected rte_memory_get_nchannel() and rte_memory_get_nrank()
I will wait for inputs from Bluefield, DPAAx, ThunderX2 and n1sdp(if any) for any rework on the patch.
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
lib/librte_mempool/rte_mempool.c +5
-1+static unsigned +arch_mem_object_align (unsigned obj_size) { +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size,uint32_t flags, */ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size = optimize_object_size(sz->header_size + sz-Quoted 1 line
elt_size+lib/librte_mempool/rte_mempool.c +3
-2- sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; } --2.24.1
Hi,
Quoted 6 lines · Jerin Jacob
Quoted 4 lines · Honnappa Nagarahalli
<snip>
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
From: Jerin Jacob <[email protected]>
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
The exiting
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (Mirco) architecture may have different memory alignment constraint for better performance and it not same as the existing
optimize_object_size()function. Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
typo: Mirco -> Micro
Maybe the whole sentence can be reworded a bit (I think a word is missing).
Quoted 49 lines
Quoted 49 lines
Quoted 49 lines
Quoted 1 line
If I understand correctly, address interleaving is the characteristic of the
memory controller and not the CPU.
Quoted 1 line
For ex: different SoCs using the same Arm architecture might have different
memory controllers. So, the solution should not be architecture specific, but SoC specific.
Yes. See below.
Quoted 4 lines
Quoted 4 lines
lib/librte_mempool/rte_mempool.c +2
-1-static unsigned optimize_object_size (unsigned obj_size) +static unsigned +arch_mem_object_align (unsigned obj_size) {Quoted 8 lines
lib/librte_mempool/rte_mempool.c static unsigned optimize_object_size(unsigned · line 104
Quoted 7 lines
unsigned nrank, nchan; unsigned new_obj_size;lib/librte_mempool/rte_mempool.c +1
-0@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++; return new_obj_size * RTE_MEMPOOL_ALIGN; } +#else This applies to add Arm (PPC as well) SoCs which might have different
schemes depending on the memory controller. IMO, this should not be architecture specific.
I agree in principle.
I will summarize the
https://www.mail-archive.com/[email protected]/msg149157.html feedback:1) For x86 arch, it is architecture-specific
2) For power PC arch, It is architecture-specific
3) For the ARM case, it will be the memory controller specific.
4) For the ARM case, The memory controller is not using the existing x86 arch formula.
5) If it is memory/arch-specific, Can userspace code find the optimal alignment? In the case of octeontx2/arm64, the memory controller does XOR on PA address which userspace code doesn't have much control.This patch address the known case of (1), (2), (4) and (5). (2) can be added to this framework when POWER9 folks want it.
We can extend this patch to address (3) if there is a case. Without the actual requirement(If some can share the formula of alignment which is the memory controller specific and it does not come under (4))) then we can create extra layer for the memory controller and abstraction to probe it. Again there is no standard way of probing the memory controller in userspace and we need platform #define, which won't work for distribution build.
So solution needs to be arch-specific and then fine-tune to memory controller if possible.
Quoted 11 lines
Quoted 3 lines
Quoted 2 lines
I can work on creating an extra layer of code if some can provide the details of the memory controller and probing mechanism or this patch be extended
Inputs for BlueField, DPAAx, ThunderX2 would be helpful.
Yes. Probably memory controller used in n1sdp SoC also.
Quoted 5 lines
Quoted 4 lines
to support such case if it arises in future.
Thoughts?
How much memory will this save for your platform? Is it affecting performance?
Currently, I think Arm-based architectures use the default (nchan=4, nrank=1). The worst case is for object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized
64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%
...
2304 -> 2368 +2.7% (~mbuf size)
Quoted 9 lines
No performance difference.
The existing code adding the tailer for each objs. Additional space/Trailer space will be function of number of objects in mempool and its
obj_size, its alignment and selectedrte_memory_get_nchannel()andrte_memory_get_nrank()I will wait for inputs from Bluefield, DPAAx, ThunderX2 and n1sdp(if any) for any rework on the patch.
If there is no performance impact on other supporter Arm-based architectures, I think it is a step in a right direction.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_mempool/rte_mempool.c static unsigned optimize_object_size(unsigned obj_size) · line 110
Quoted 5 lines
lib/librte_mempool/rte_mempool.c +5
-0+static unsigned +arch_mem_object_align (unsigned obj_size) { +return obj_size; +} +#endif
I'd prefer "unsigned int" for new code.
Also, the opening brace should be on a separate line.
The documentation of the MEMPOOL_F_NO_SPREAD flag in the .h could be slightly modified, as you did for the comment above arch_mem_object_align().
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
lib/librte_mempool/rte_mempool.c +0
-1struct pagesz_walk_arg {int socket_id;@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size,uint32_t flags, */ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size = optimize_object_size(sz->header_size + sz-Quoted 1 line
elt_size+lib/librte_mempool/rte_mempool.c +3
-2- sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; } --2.24.1
PATCH v2 1/1 mempool: fix mempool obj alignment for non x86
In reply to [email protected]From: Jerin Jacob <[email protected]>
The existing optimize_object_size() function address the memory object alignment constraint on x86 for better performance.
Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing optimize_object_size().
Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing arch_mem_object_align() function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.
Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized
64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%
...
2304 -> 2368 +2.7% (~mbuf size)
Additional details:
https://www.mail-archive.com/[email protected]/msg149157.html
---
v2:
- Changed the return type of arch_mem_object_align() to "unsigned int" from "unsigned" to fix the checkpatch issues (Olivier Matz)
- Updated the comments for MEMPOOL_F_NO_SPREAD (Olivier Matz)
- Update the git comments to share the memory saving details.
doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- lib/librte_mempool/rte_mempool.h | 6 +++++- 3 files changed, 21 insertions(+), 8 deletions(-)
doc/guides/prog_guide/mempool_lib.rst +3 -3
@@ -27,10 +27,10 @@ In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), statistics about get from/put in the pool are stored in the mempool structure. Statistics are per-lcore to avoid concurrent access to statistics counters. -Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on X86 architecture +------------------------------------------------ -Depending on hardware memory configuration, performance can be greatly improved by adding a specific padding between objects. +Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects. The objective is to ensure that the beginning of each object starts on a different channel and rank in memory so that all channels are equally loaded. This is particularly true for packet buffers when doing L3 forwarding or flow classification.
lib/librte_mempool/rte_mempool.c +13 -4
@@ -45,6 +45,7 @@ EAL_REGISTER_TAILQ(rte_mempool_tailq)#define CALC_CACHE_FLUSHTHRESH (c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) +#if defined(RTE_ARCH_X86 )/* * return the greatest common divisor between a and b (fast algorithm) *
@@ -74,12 +75,13 @@ static unsigned get_gcd(unsigned a, unsigned b) }/* -* Depending on memory configuration, objects addresses are spread +* Depending on memory configuration on x86 arch, objects addresses are spread * between channels and ranks in RAM: the pool allocator will add * padding between objects. This function return the new size of the * object. */ -static unsigned optimize_object_size (unsigned obj_size) +static unsigned int +arch_mem_object_align (unsigned int obj_size) {unsigned nrank, nchan;unsigned new_obj_size;
@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++;return new_obj_size *RTE_MEMPOOL_ALIGN ; } +#else +static unsigned int +arch_mem_object_align (unsigned int obj_size) +{ +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;
@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,*/ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size =optimize_object_size (sz->header_size + sz->elt_size + - sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; }
lib/librte_mempool/rte_mempool.h +5 -1
@@ -260,7 +260,11 @@ struct rte_mempool {#endif } __rte_cache_aligned; -#define MEMPOOL_F_NO_SPREAD 0x0001/**< Do not spread among memory channels. */ +#define MEMPOOL_F_NO_SPREAD 0x0001 +/**< Do not spread among memory channels. It is a hint to the library, +* library honor this hint only when, if it is required by the +* (micro) architecture. +*/ #define MEMPOOL_F_NO_CACHE_ALIGN 0x0002/**< Do not align objs on cache lines.*/ #define MEMPOOL_F_SP_PUT 0x0004/**< Default put is "single-producer".*/ #define MEMPOOL_F_SC_GET 0x0008/**< Default get is "single-consumer".*/
Signature · 1 line
2.24.1
lib/librte_mempool/rte_mempool.h struct rte_mempool { · line 267
Quoted 6 lines · On Sat, 11 Jan 2020 19:04:10 +0530 <[email protected]> wrote:
lib/librte_mempool/rte_mempool.h +5
-1-#define MEMPOOL_F_NO_SPREAD 0x0001/**< Do not spread among memory channels. */ +#define MEMPOOL_F_NO_SPREAD 0x0001 +/**< Do not spread among memory channels. It is a hint to the library, +* library honor this hint only when, if it is required by the +* (micro) architecture. +*/
That text is awkward for me to read.
There maybe other reasons in future for mempool to ignore the flag.
I prefer the minor change original comment as:
Quoted 1 line
-#define
MEMPOOL_F_NO_SPREAD0x0001/**< Spreading among memory channels not required. */
PATCH v3 1/1 mempool: fix mempool obj alignment for non x86
In reply to [email protected]From: Jerin Jacob <[email protected]>
The existing optimize_object_size() function address the memory object alignment constraint on x86 for better performance.
Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing optimize_object_size().
Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing arch_mem_object_align() function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.
Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized
64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%
...
2304 -> 2368 +2.7% (~mbuf size)
Additional details:
https://www.mail-archive.com/[email protected]/msg149157.html
---
v3:
- Change comment for MEMPOOL_F_NO_SPREAD flag as
" Spreading among memory channels not required." (Stephen Hemminger)
v2:
- Changed the return type of arch_mem_object_align() to "unsigned int" from "unsigned" to fix the checkpatch issues (Olivier Matz)
- Updated the comments for MEMPOOL_F_NO_SPREAD (Olivier Matz)
- Update the git comments to share the memory saving details.
doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- lib/librte_mempool/rte_mempool.h | 3 ++- 3 files changed, 18 insertions(+), 8 deletions(-)
doc/guides/prog_guide/mempool_lib.rst +3 -3
@@ -27,10 +27,10 @@ In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), statistics about get from/put in the pool are stored in the mempool structure. Statistics are per-lcore to avoid concurrent access to statistics counters. -Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on X86 architecture +------------------------------------------------ -Depending on hardware memory configuration, performance can be greatly improved by adding a specific padding between objects. +Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects. The objective is to ensure that the beginning of each object starts on a different channel and rank in memory so that all channels are equally loaded. This is particularly true for packet buffers when doing L3 forwarding or flow classification.
lib/librte_mempool/rte_mempool.c +13 -4
@@ -45,6 +45,7 @@ EAL_REGISTER_TAILQ(rte_mempool_tailq)#define CALC_CACHE_FLUSHTHRESH (c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) +#if defined(RTE_ARCH_X86 )/* * return the greatest common divisor between a and b (fast algorithm) *
@@ -74,12 +75,13 @@ static unsigned get_gcd(unsigned a, unsigned b) }/* -* Depending on memory configuration, objects addresses are spread +* Depending on memory configuration on x86 arch, objects addresses are spread * between channels and ranks in RAM: the pool allocator will add * padding between objects. This function return the new size of the * object. */ -static unsigned optimize_object_size (unsigned obj_size) +static unsigned int +arch_mem_object_align (unsigned int obj_size) {unsigned nrank, nchan;unsigned new_obj_size;
@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++;return new_obj_size *RTE_MEMPOOL_ALIGN ; } +#else +static unsigned int +arch_mem_object_align (unsigned int obj_size) +{ +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;
@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,*/ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size =optimize_object_size (sz->header_size + sz->elt_size + - sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; }
lib/librte_mempool/rte_mempool.h +2 -1
@@ -260,7 +260,8 @@ struct rte_mempool {#endif } __rte_cache_aligned; -#define MEMPOOL_F_NO_SPREAD 0x0001/**< Do not spread among memory channels. */ +#define MEMPOOL_F_NO_SPREAD 0x0001 +/**< Spreading among memory channels not required. */ #define MEMPOOL_F_NO_CACHE_ALIGN 0x0002/**< Do not align objs on cache lines.*/ #define MEMPOOL_F_SP_PUT 0x0004/**< Default put is "single-producer".*/ #define MEMPOOL_F_SC_GET 0x0008/**< Default get is "single-consumer".*/
Signature · 1 line
2.24.1
Quoted 32 lines · On Mon, Jan 13, 2020 at 7:49 AM <[email protected]> wrote:
From: Jerin Jacob <[email protected]>
The existing
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing
optimize_object_size().Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing
arch_mem_object_align()function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%
Quoted 7 lines
...
2304 -> 2368 +2.7% (~mbuf size)Additional details:
https://www.mail-archive.com/[email protected]/msg149157.htmlFixes af75078fece3 ("first public release")
Weird to flag this as a problem in this sha1.
x86 was the only architecture supported at the time. Either we mark the introduction of new architectures as the point of backport, or we remove this tag and just let Cc: [email protected]
Quoted 1 line
It seems more an optimisation than a fix to me, but in any case, the stable maintainers will be the judges.
Quoted 24 lines
Signed-off-by Jerin Jacob <[email protected]>Reviewed-by Gavin Hu <[email protected]>---
v3:
- Change comment forMEMPOOL_F_NO_SPREADflag as
" Spreading among memory channels not required." (Stephen Hemminger)v2:
- Changed the return type ofarch_mem_object_align()to "unsigned int" from "unsigned" to fix the checkpatch issues (Olivier Matz)
- Updated the comments forMEMPOOL_F_NO_SPREAD(Olivier Matz)
- Update the git comments to share the memory saving details.doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- lib/librte_mempool/rte_mempool.h | 3 ++- 3 files changed, 18 insertions(+), 8 deletions(-)doc/guides/prog_guide/mempool_lib.rst +0
-0
doc/guides/prog_guide/mempool_lib.rst In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), · line 31
Quoted 7 lines
doc/guides/prog_guide/mempool_lib.rst +2
-2@@ -27,10 +27,10 @@ In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), statistics about get from/put in the pool are stored in the mempool structure. Statistics are per-lcore to avoid concurrent access to statistics counters. -Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on X86 architecture +------------------------------------------------
Nit: afaics in the docs, x86 is preferred to X86.
Quoted 6 lines
doc/guides/prog_guide/mempool_lib.rst +1
-1-Depending on hardware memory configuration, performance can be greatly improved by adding a specific padding between objects. +Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects. The objective is to ensure that the beginning of each object starts on a different channel and rank in memory so that all channels are equally loaded. This is particularly true for packet buffers when doing L3 forwarding or flow classification.
Signature · 1 line
David Marchand
Quoted 66 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, January 13, 2020 3:17 PM
To: Jerin Jacob Kollanukkaran <[email protected]>
Cc: dev <[email protected]>; Thomas Monjalon <[email protected]>; Olivier Matz <[email protected]>; Andrew Rybchenko <[email protected]>; Bruce Richardson <[email protected]>; Ananyev, Konstantin <[email protected]>; Hemant Agrawal <[email protected]>; Shahaf Shuler <[email protected]>; Honnappa Nagarahalli <[email protected]>; Gavin Hu <[email protected]>; [email protected]; David Christensen <[email protected]>; Burakov, Anatoly <[email protected]>; dpdk stable <[email protected]>; Kevin Traynor <[email protected]>; Luca Boccassi <[email protected]>
Subject: [EXT] Re: [dpdk-stable] [dpdk-dev] [PATCH v3] mempool: fix mempool obj alignment for non x86External Email
----------------------------------------------------------------------
Quoted 33 lines · On Mon, Jan 13, 2020 at 7:49 AM <[email protected]> wrote:
From: Jerin Jacob <[email protected]>
The existing
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing
optimize_object_size().Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing
arch_mem_object_align()function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool
header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).Examples for cache lines size = 64:
orig optimized64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%Quoted 6 lines
...
2304 -> 2368 +2.7% (~mbuf size)Additional details:
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mail-2Darchiv
e.com_dev-40dpdk.org_msg149157.html&d=DwIFaQ&c=nKjWec2b6R0mOyPaz7xtfQ&
Quoted 1 line
r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=VKkiHhyflsqwipCoE MtdUR
Quoted 1 line
SXuHSq2neWGqTRmxVfjr8&s=y-LYGZ-
Quoted 8 lines
2MsAfrGo3r5aADQnr2mUcsP7LxXT5XEmTuwE&e=
Quoted 2 lines
Fixes af75078fece3 ("first public release")Weird to flag this as a problem in this sha1.
x86 was the only architecture supported at the time.
Either we mark the introduction of new architectures as the point of backport, or we remove this tag and just let Cc: [email protected]
While committing the maintainer can take either one of the decision. No issues/opinion on this from my side.
Quoted 5 lines
Quoted 2 lines
It seems more an optimisation than a fix to me, but in any case, the stable maintainers will be the judges.
OK. No issues.
Quoted 55 lines
doc/guides/prog_guide/mempool_lib.rst In debug mode
Quoted 27 lines
Signed-off-by Jerin Jacob <[email protected]>Reviewed-by Gavin Hu <[email protected]>---
v3:
- Change comment forMEMPOOL_F_NO_SPREADflag as " Spreading among memory channels not required." (Stephen Hemminger)v2:
- Changed the return type ofarch_mem_object_align()to "unsigned int" from "unsigned" to fix the checkpatch issues (Olivier Matz)
- Updated the comments forMEMPOOL_F_NO_SPREAD(Olivier Matz)
- Update the git comments to share the memory saving details.doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- lib/librte_mempool/rte_mempool.h | 3 ++- 3 files changed, 18 insertions(+), 8 deletions(-)doc/guides/prog_guide/mempool_lib.rst +0
-0@@ -27,10 +27,10 @@ In debug mode(
CONFIG_RTE_LIBRTE_MEMPOOL_DEBUGisQuoted 1 line
enabled), statistics about get from/put in the pool are stored in the mempool
structure.
doc/guides/prog_guide/mempool_lib.rst In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), · line 31
Quoted 6 lines
Statistics are per-lcore to avoid concurrent access to statistics counters.
doc/guides/prog_guide/mempool_lib.rst +2
-2-Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on X86 architecture +------------------------------------------------Nit: afaics in the docs, x86 is preferred to X86.
Quoted 2 lines
-Depending on hardware memory configuration, performance can be greatly
improved by adding a specific padding between objects.
Quoted 1 line
+Depending on hardware memory configuration on X86 architecture,
performance can be greatly improved by adding a specific padding between objects.
Quoted 1 line
The objective is to ensure that the beginning of each object starts on a
different channel and rank in memory so that all channels are equally loaded.
Quoted 2 lines
This is particularly true for packet buffers when doing L3 forwarding or flow
classification.
--
David Marchand
PATCH v4 1/1 mempool: remove memory wastage on non x86
In reply to [email protected]From: Jerin Jacob <[email protected]>
The existing optimize_object_size() function address the memory object alignment constraint on x86 for better performance.
Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing optimize_object_size().
Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing arch_mem_object_align() function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.
Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized
64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%
...
2304 -> 2368 +2.7% (~mbuf size)
Additional details:
https://www.mail-archive.com/[email protected]/msg149157.html
---
v4:
- s/X86/x86 in the doc/guides/prog_guide/mempool_lib.rst file (David)
- Remove Fixes and CC: [email protected] as well, as this is just a improvement.(David)
- Udpate the patch heading to remove checkpatch warning due to above change.
v3:
- Change comment for MEMPOOL_F_NO_SPREAD flag as
" Spreading among memory channels not required." (Stephen Hemminger)
v2:
- Changed the return type of arch_mem_object_align() to "unsigned int" from "unsigned" to fix the checkpatch issues (Olivier Matz)
- Updated the comments for MEMPOOL_F_NO_SPREAD (Olivier Matz)
- Update the git comments to share the memory saving details.
doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- lib/librte_mempool/rte_mempool.h | 3 ++- 3 files changed, 18 insertions(+), 8 deletions(-)
doc/guides/prog_guide/mempool_lib.rst +3 -3
@@ -27,10 +27,10 @@ In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), statistics about get from/put in the pool are stored in the mempool structure. Statistics are per-lcore to avoid concurrent access to statistics counters. -Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on x86 architecture +------------------------------------------------ -Depending on hardware memory configuration, performance can be greatly improved by adding a specific padding between objects. +Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects. The objective is to ensure that the beginning of each object starts on a different channel and rank in memory so that all channels are equally loaded. This is particularly true for packet buffers when doing L3 forwarding or flow classification.
lib/librte_mempool/rte_mempool.c +13 -4
@@ -45,6 +45,7 @@ EAL_REGISTER_TAILQ(rte_mempool_tailq)#define CALC_CACHE_FLUSHTHRESH (c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) +#if defined(RTE_ARCH_X86 )/* * return the greatest common divisor between a and b (fast algorithm) *
@@ -74,12 +75,13 @@ static unsigned get_gcd(unsigned a, unsigned b) }/* -* Depending on memory configuration, objects addresses are spread +* Depending on memory configuration on x86 arch, objects addresses are spread * between channels and ranks in RAM: the pool allocator will add * padding between objects. This function return the new size of the * object. */ -static unsigned optimize_object_size (unsigned obj_size) +static unsigned int +arch_mem_object_align (unsigned int obj_size) {unsigned nrank, nchan;unsigned new_obj_size;
@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++;return new_obj_size *RTE_MEMPOOL_ALIGN ; } +#else +static unsigned int +arch_mem_object_align (unsigned int obj_size) +{ +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;
@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,*/ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size =optimize_object_size (sz->header_size + sz->elt_size + - sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; }
lib/librte_mempool/rte_mempool.h +2 -1
@@ -260,7 +260,8 @@ struct rte_mempool {#endif } __rte_cache_aligned; -#define MEMPOOL_F_NO_SPREAD 0x0001/**< Do not spread among memory channels. */ +#define MEMPOOL_F_NO_SPREAD 0x0001 +/**< Spreading among memory channels not required. */ #define MEMPOOL_F_NO_CACHE_ALIGN 0x0002/**< Do not align objs on cache lines.*/ #define MEMPOOL_F_SP_PUT 0x0004/**< Default put is "single-producer".*/ #define MEMPOOL_F_SC_GET 0x0008/**< Default get is "single-consumer".*/
Signature · 1 line
2.24.1
Re: [dpdk-dev] [PATCH v4] mempool: remove memory wastage on non x86
Quoted 32 lines · On Wed, Jan 15, 2020 at 2:35 AM <[email protected]> wrote:
From: Jerin Jacob <[email protected]>
The existing
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing
optimize_object_size().Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing
arch_mem_object_align()function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%
Quoted 8 lines
...
2304 -> 2368 +2.7% (~mbuf size)Additional details:
https://www.mail-archive.com/[email protected]/msg149157.htmlSigned-off-by Jerin Jacob <[email protected]>Reviewed-by Gavin Hu <[email protected]>
Ping for merge.
Quoted 112 lines
---
v4:- s/X86/x86 in the
doc/guides/prog_guide/mempool_lib.rstfile (David)
- Remove Fixes and CC: [email protected] as well, as this is just a improvement.(David)
- Udpate the patch heading to remove checkpatch warning due to above change.v3:
- Change comment forMEMPOOL_F_NO_SPREADflag as
" Spreading among memory channels not required." (Stephen Hemminger)v2:
- Changed the return type ofarch_mem_object_align()to "unsigned int" from "unsigned" to fix the checkpatch issues (Olivier Matz)
- Updated the comments forMEMPOOL_F_NO_SPREAD(Olivier Matz)
- Update the git comments to share the memory saving details.doc/guides/prog_guide/mempool_lib.rst | 6 +++--- lib/librte_mempool/rte_mempool.c | 17 +++++++++++++---- lib/librte_mempool/rte_mempool.h | 3 ++- 3 files changed, 18 insertions(+), 8 deletions(-)doc/guides/prog_guide/mempool_lib.rst +3
-3@@ -27,10 +27,10 @@ In debug mode (CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG is enabled), statistics about get from/put in the pool are stored in the mempool structure. Statistics are per-lcore to avoid concurrent access to statistics counters. -Memory Alignment Constraints ----------------------------- +Memory Alignment Constraints on x86 architecture +------------------------------------------------ -Depending on hardware memory configuration, performance can be greatly improved by adding a specific padding between objects. +Depending on hardware memory configuration on X86 architecture, performance can be greatly improved by adding a specific padding between objects. The objective is to ensure that the beginning of each object starts on a different channel and rank in memory so that all channels are equally loaded. This is particularly true for packet buffers when doing L3 forwarding or flow classification.lib/librte_mempool/rte_mempool.c +13
-4@@ -45,6 +45,7 @@ EAL_REGISTER_TAILQ(rte_mempool_tailq)#define CALC_CACHE_FLUSHTHRESH (c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) +#if defined(RTE_ARCH_X86 )/* * return the greatest common divisor between a and b (fast algorithm) * @@ -74,12 +75,13 @@ static unsigned get_gcd(unsigned a, unsigned b) }/* -* Depending on memory configuration, objects addresses are spread +* Depending on memory configuration on x86 arch, objects addresses are spread * between channels and ranks in RAM: the pool allocator will add * padding between objects. This function return the new size of the * object. */ -static unsigned optimize_object_size (unsigned obj_size) +static unsigned int +arch_mem_object_align (unsigned int obj_size) {unsigned nrank, nchan;unsigned new_obj_size;@@ -99,6 +101,13 @@ static unsigned optimize_object_size(unsigned obj_size) new_obj_size++;return new_obj_size *RTE_MEMPOOL_ALIGN ; } +#else +static unsigned int +arch_mem_object_align (unsigned int obj_size) +{ +return obj_size; +} +#endif struct pagesz_walk_arg {int socket_id;@@ -234,8 +243,8 @@ rte_mempool_calc_obj_size(uint32_t elt_size, uint32_t flags,*/ if ((flags &MEMPOOL_F_NO_SPREAD ) ==0 ) {unsigned new_size; - new_size =optimize_object_size (sz->header_size + sz->elt_size + - sz->trailer_size); + new_size =arch_mem_object_align + (sz->header_size + sz->elt_size + sz->trailer_size); sz->trailer_size = new_size - sz->header_size - sz->elt_size; }lib/librte_mempool/rte_mempool.h +2
-2@@ -260,7 +260,8 @@ struct rte_mempool {#endif } __rte_cache_aligned; -#define MEMPOOL_F_NO_SPREAD 0x0001/**< Do not spread among memory channels. */ +#define MEMPOOL_F_NO_SPREAD 0x0001 +/**< Spreading among memory channels not required. */ #define MEMPOOL_F_NO_CACHE_ALIGN 0x0002/**< Do not align objs on cache lines.*/ #define MEMPOOL_F_SP_PUT 0x0004/**< Default put is "single-producer".*/ #define MEMPOOL_F_SC_GET 0x0008/**< Default get is "single-consumer".*/ --2.24.1
Olivier Matz
Acked-byRe: [dpdk-dev] [PATCH v4] mempool: remove memory wastage on non x86
Quoted 36 lines · Jerin Jacob
Quoted 35 lines · On Wed, Jan 15, 2020 at 2:35 AM <[email protected]> wrote:
From: Jerin Jacob <[email protected]>
The existing
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing
optimize_object_size().Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing
arch_mem_object_align()function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%...
2304 -> 2368 +2.7% (~mbuf size)
Quoted 7 lines
Quoted 5 lines
Additional details:
https://www.mail-archive.com/[email protected]/msg149157.htmlSigned-off-by Jerin Jacob <[email protected]>Reviewed-by Gavin Hu <[email protected]>Ping for merge.
Thomas Monjalon
AppliedRe: [dpdk-dev] [PATCH v4] mempool: remove memory wastage on non x86
20/01/2020 13:24, Olivier Matz:
Quoted 40 lines · Jerin Jacob
Quoted 39 lines · On Wed, Jan 15, 2020 at 2:35 AM <[email protected]> wrote:
Quoted 38 lines
From: Jerin Jacob <[email protected]>
The existing
optimize_object_size()function address the memory object alignment constraint on x86 for better performance.Different (micro) architecture may have different memory alignment constraint for better performance and it not the same as the existing
optimize_object_size().Some use, XOR(kind of CRC) scheme to enable DRAM channel distribution based on the address and some may have a different formula.
Introducing
arch_mem_object_align()function to abstract the difference between different (micro) architectures to avoid wasting memory for mempool object alignment for the architecture that it is not required to do so.Details on the amount of memory saving:
Currently, arm64 based architectures use the default (nchan=4, nrank=1). The worst case is for an object whose size (including mempool header) is 2 cache lines, where it is optimized to 3 cache lines (+50%).
Examples for cache lines size = 64:
orig optimized64 -> 64 +0% 128 -> 192 +50% 192 -> 192 +0% 256 -> 320 +25% 320 -> 320 +0% 384 -> 448 +16%...
2304 -> 2368 +2.7% (~mbuf size)Additional details:
https://www.mail-archive.com/[email protected]/msg149157.html
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by Jerin Jacob <[email protected]>Reviewed-by Gavin Hu <[email protected]>Ping for merge.
Acked-by Olivier Matz <[email protected]>
Applied, thanks