mempool: optimize get objects with constant n
PATCH 1/1 mempool: optimize get objects with constant n
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at build time remains essentially unchanged.
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +21 -3
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + +if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; + +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 ); +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_objs, n); + +return 0 ; + } + +/* Use the cache as much as we have to return hot objects first */ + len = __extension__(__builtin_constant_p (n)) ? cache->len : +RTE_MIN (remaining, cache->len); cache->len -= len; remaining -= len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; -if (remaining ==0 ) { +if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 );
Signature · 1 line
2.17.1
Quoted 7 lines · Morten Brørup
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
Quoted 8 lines
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an average of 17 %, minimum 9.5 %, maximum 24 %.The code path where the number of objects to get is unknown at build time remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>
Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:
Quoted 16 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)lib/mempool/rte_mempool.h +1
-2@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; +
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +
This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];
alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:
memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523
Quoted 9 lines
lib/mempool/rte_mempool.h +9
-0+RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 ); +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_objs, n); + +return 0 ; + } + +/* Use the cache as much as we have to return hot objects first */ + len = __extension__(__builtin_constant_p (n)) ? cache->len : +RTE_MIN (remaining, cache->len);
This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be "> cache->len" since the previous block was untaken, therefore we don't need to call RTE_MIN.
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532
Quoted 10 lines
lib/mempool/rte_mempool.h +1
-1cache->len -= len; remaining -= len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; -if (remaining ==0 ) { +if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 );
Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.
I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.
Quoted 3 lines
Signature · 1 line
2.17.1
Quoted 58 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07Quoted 7 lines · Morten Brørup
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
Quoted 8 lines
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an average of 17 %, minimum 9.5 %, maximum 24 %.The code path where the number of objects to get is unknown at build time remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:Acked-by Bruce Richardson <[email protected]>lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp,
Quoted 9 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,void **obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +1
-2if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; +lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
Quoted 4 lines
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];
The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.
Regardless, your point here is still valid! I expected that any unrolling capable compiler can unroll *dst++ = *--src; but I can experiment with different compilers on godbolt.org to see if dst[index] = src[-index] is better.
A future version could be hand coded with vector instructions here, like in the Ethdev drivers.
Quoted 5 lines
alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:
memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);
Just for reference: It was previously discussed to ignore the stack ordering and simply copy the objects, but the idea was rejected due to the potential performance impact of not returning the hottest objects, i.e. the objects at the top of the stack, as the first in the returned array.
Quoted 17 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523
Quoted 10 lines
lib/mempool/rte_mempool.h +9
-0+RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 ); +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_objs, n); + +return 0 ; + } + +/* Use the cache as much as we have to return hot objects first */ + len = __extension__(__builtin_constant_p (n)) ? cache->len : +RTE_MIN (remaining, cache->len);This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be ">cache->len" since the previous block was untaken, therefore we don't need to callRTE_MIN.
I agree. This is almost like perl... write-only source code.
I will add a few explanatory comments.
Quoted 16 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532
Quoted 11 lines
lib/mempool/rte_mempool.h +1
-1cache->len -= len; remaining -= len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; -if (remaining ==0 ) { +if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 );Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.
Quoted 3 lines
I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.
I get your point, but the code is still short enough to grasp (after comments have been added). So I prefer to avoid code duplication.
Quoted 4 lines
Quoted 4 lines
--
2.17.1
Quoted 70 lines · Morten Brørup
Quoted 58 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07Quoted 7 lines · Morten Brørup
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
Quoted 8 lines
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an average of 17 %, minimum 9.5 %, maximum 24 %.The code path where the number of objects to get is unknown at build time remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:Acked-by Bruce Richardson <[email protected]>lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp,
Quoted 9 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,void **obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +1
-2if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; +lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
Quoted 4 lines
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.
Regardless, your point here is still valid! I expected that any unrolling capable compiler can unroll *dst++ = *--src; but I can experiment with different compilers on godbolt.org to see if dst[index] = src[-index] is better.
A future version could be hand coded with vector instructions here, like in the Ethdev drivers.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:
memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);Just for reference: It was previously discussed to ignore the stack ordering and simply copy the objects, but the idea was rejected due to the potential performance impact of not returning the hottest objects, i.e. the objects at the top of the stack, as the first in the returned array.
Ah, yes, I had forgotten that we want the order reversed, so most recent objects first. It would be interesting to see if ignoring order within a burst can lead to some performance improvements, as I expect that in most cases, the buffers to be used in a burst are to be used pretty much simultaneously. However, based on your feedback, feel free to ignore this comment and keep code as-is.
Quoted 18 lines
Quoted 17 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523
Quoted 10 lines
lib/mempool/rte_mempool.h +9
-0+RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 ); +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_objs, n); + +return 0 ; + } + +/* Use the cache as much as we have to return hot objects first */ + len = __extension__(__builtin_constant_p (n)) ? cache->len : +RTE_MIN (remaining, cache->len);This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be ">cache->len" since the previous block was untaken, therefore we don't need to callRTE_MIN.
Quoted 4 lines
I agree. This is almost like perl... write-only source code.
I will add a few explanatory comments.
Thanks.
Quoted 16 lines
Quoted 16 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532
Quoted 11 lines
lib/mempool/rte_mempool.h +1
-1cache->len -= len; remaining -= len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; -if (remaining ==0 ) { +if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 );Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.
Quoted 6 lines
Quoted 3 lines
I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.
I get your point, but the code is still short enough to grasp (after comments have been added). So I prefer to avoid code duplication.
Sure.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
--
2.17.1
Quoted 58 lines · Morten Brørup
Quoted 58 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07Quoted 7 lines · Morten Brørup
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
Quoted 8 lines
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an average of 17 %, minimum 9.5 %, maximum 24 %.The code path where the number of objects to get is unknown at build time remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:Acked-by Bruce Richardson <[email protected]>lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp,
Quoted 9 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,void **obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +1
-2if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; +lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
Quoted 7 lines
Quoted 4 lines
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.
BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!
/Bruce
18/04/2023 14:55, Bruce Richardson:
Quoted 22 lines · Morten Brørup
Quoted 21 lines
From: Bruce Richardson [mailto:[email protected]]
Quoted 19 lines · Morten Brørup
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];
Quoted 5 lines
Quoted 2 lines
The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.
BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!
Looks like this request was missed.
Quoted 27 lines
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 09.5218/04/2023 14:55, Bruce Richardson:
Quoted 17 lines · Morten Brørup
Quoted 16 lines
From: Bruce Richardson [mailto:[email protected]]
Quoted 15 lines · Morten Brørup
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];The mempool cache is a stack, so the copy loop needs get the objects in
Quoted 8 lines
decrementing order. I.e. the source index decrements and the destination index increments.
Quoted 4 lines
Quoted 1 line
BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!
Looks like this request was missed.
I intentionally omitted it, because I disagree with the suggestion.
Otherwise, reminders that the mempool cache is a stack should be plastered all over the source code, not just here. For reference, this copy loop (without such a reminder) also exists elsewhere in the same file.
Apologies for not responding to Bruce's request earlier.
07/06/2023 10:03, Morten Brørup:
Quoted 34 lines
Quoted 34 lines
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 09.5218/04/2023 14:55, Bruce Richardson:
Quoted 17 lines · Morten Brørup
Quoted 16 lines
From: Bruce Richardson [mailto:[email protected]]
Quoted 15 lines · Morten Brørup
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 10 lines
lib/mempool/rte_mempool.h +10
-0+if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];The mempool cache is a stack, so the copy loop needs get the objects in
decrementing order. I.e. the source index decrements and the destination index increments.
Quoted 4 lines
Quoted 1 line
BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!
Quoted 7 lines
Quoted 1 line
Looks like this request was missed.
I intentionally omitted it, because I disagree with the suggestion.
Otherwise, reminders that the mempool cache is a stack should be plastered all over the source code, not just here. For reference, this copy loop (without such a reminder) also exists elsewhere in the same file.
Apologies for not responding to Bruce's request earlier.
What about doing a general comment at the top of the function, with the assignment of the pointer at the end of the array:
/* The cache is a stack, so copy will be in reverse order. */
cache_objs = &cache->objs[cache->len];I could do it on apply if there is an agreement.
Quoted 52 lines
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 10.1007/06/2023 10:03, Morten Brørup:
Quoted 7 lines
Quoted 7 lines
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 09.5218/04/2023 14:55, Bruce Richardson:
Quoted 3 lines · Morten Brørup
Quoted 2 lines
From: Bruce Richardson [mailto:[email protected]]
Quoted 1 line · Morten Brørup
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4
-0+ if (__extension__(__builtin_constant_p(n)) && n <= cache->len) { + /* + * The request size is known at build time, and + * the entire request can be satisfied from the cache,Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ * so let the compiler unroll the fixed length copy loop. + */ + cache->len -= n; + for (index = 0; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and
perhaps
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
easier for compilers to unroll efficiently if it was rewritten as:
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];The mempool cache is a stack, so the copy loop needs get the objects
in
Quoted 1 line
Quoted 1 line
decrementing order. I.e. the source index decrements and the destination
index
Quoted 4 lines
Quoted 4 lines
increments.
Quoted 3 lines
Quoted 1 line
BTW: Please add this as a comment in the code too, above the loop to
avoid
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
future developers (or even future me), asking this question again!
Looks like this request was missed.
I intentionally omitted it, because I disagree with the suggestion.
Otherwise, reminders that the mempool cache is a stack should be plastered
all over the source code, not just here. For reference, this copy loop (without such a reminder) also exists elsewhere in the same file.
Quoted 2 lines
Apologies for not responding to Bruce's request earlier.
Quoted 8 lines
What about doing a general comment at the top of the function, with the assignment of the pointer at the end of the array:
/* The cache is a stack, so copy will be in reverse order. */ cache_objs = &cache->objs[cache->len];I could do it on apply if there is an agreement.
ACK from me.
For consistency, please also add the reminder to the two existing reverse order copy loops in rte_mempool_do_generic_get().
Quoted 6 lines
From: Morten Brørup
Sent: Wednesday, 7 June 2023 10.34Quoted 3 lines
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 10.10
[...]
Quoted 6 lines
Quoted 6 lines
What about doing a general comment at the top of the function, with the assignment of the pointer at the end of the array:
/* The cache is a stack, so copy will be in reverse order. */ cache_objs = &cache->objs[cache->len];
Quoted 6 lines
Quoted 2 lines
I could do it on apply if there is an agreement.
ACK from me.
For consistency, please also add the reminder to the two existing reverse order copy loops in
rte_mempool_do_generic_get().
Rubbish! I was looking at the wrong version of the code.
So, plain ACK from me. No further comments!
Sorry. :-!
Quoted 36 lines · Bruce Richardson
Quoted 7 lines · Morten Brørup
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
Quoted 8 lines
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an average of 17 %, minimum 9.5 %, maximum 24 %.The code path where the number of objects to get is unknown at build time remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:Acked-by Bruce Richardson <[email protected]>Quoted 12 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,if (unlikely (cache ==NULL )) goto driver_dequeue;
Quoted 5 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1505
Quoted 5 lines
lib/mempool/rte_mempool.h +2
-2-/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + +if (__extension__(__builtin_constant_p (n )) && n <= cache->len) {
don't take direct dependency on compiler builtins. define a macro so we don't have to play shotgun surgery later.
also what is the purpose of using __extension__ here? are you annotating the use of __builtin_constant_p() or is there more? because if that's the only reason i see no need to use __extension__ when already using a compiler specific builtin like this, that it is not standard is implied and enforced by a compile break.
Quoted 62 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 9 lines
lib/mempool/rte_mempool.h +9
-0+/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:
memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523
Quoted 9 lines
lib/mempool/rte_mempool.h +9
-0+RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 ); +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_objs, n); + +return 0 ; + } + +/* Use the cache as much as we have to return hot objects first */ + len = __extension__(__builtin_constant_p (n)) ? cache->len : +RTE_MIN (remaining, cache->len);This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be ">cache->len" since the previous block was untaken, therefore we don't need to callRTE_MIN.lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532
Quoted 10 lines
lib/mempool/rte_mempool.h +1
-1cache->len -= len; remaining -= len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; -if (remaining ==0 ) { +if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 );Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.
I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.
Quoted 3 lines
Signature · 1 line
2.17.1
Quoted 54 lines
From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup
When getting objects from the mempool, the number of objects to get
is
Quoted 8 lines
Quoted 8 lines
often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an
Quoted 3 lines
Quoted 3 lines
average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at build
time
Quoted 5 lines
Quoted 3 lines
remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you may
want to
Quoted 10 lines
take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:Acked-by Bruce Richardson <[email protected]>Quoted 4 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)Quoted 5 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct
Quoted 5 lines
lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct
rte_mempool*mp, void **obj_table,Quoted 8 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1505
Quoted 8 lines
lib/mempool/rte_mempool.h +2
-2if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + +if (__extension__(__builtin_constant_p (n )) && n <= cache->len) {don't take direct dependency on compiler builtins. define a macro so we don't have to play shotgun surgery later.
Quoted 5 lines
also what is the purpose of using
__extension__here? are you annotating the use of__builtin_constant_p()or is there more? because if that's the only reason i see no need to use__extension__when already using a compiler specific builtin like this, that it is not standard is implied and enforced by a compile break.
ARM 32-bit memcpy() [1] does it this way, so I did the same.
[1]: https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_memcpy_32.h#L122
While I agree that a macro for __builtin_constant_p() would be good, it belongs in a patch to fix portability, not in this patch.
Quoted 60 lines · Morten Brørup
Quoted 54 lines
From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup
When getting objects from the mempool, the number of objects to get
is
Quoted 8 lines
Quoted 8 lines
often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an
Quoted 3 lines
Quoted 3 lines
average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at build
time
Quoted 5 lines
Quoted 3 lines
remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you may
want to
Quoted 10 lines
take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:Acked-by Bruce Richardson <[email protected]>Quoted 4 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)Quoted 5 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct
Quoted 5 lines
lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct
rte_mempool*mp, void **obj_table,Quoted 8 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1505
Quoted 8 lines
lib/mempool/rte_mempool.h +2
-2if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + +if (__extension__(__builtin_constant_p (n )) && n <= cache->len) {don't take direct dependency on compiler builtins. define a macro so we don't have to play shotgun surgery later.
Quoted 5 lines
also what is the purpose of using
__extension__here? are you annotating the use of__builtin_constant_p()or is there more? because if that's the only reason i see no need to use__extension__when already using a compiler specific builtin like this, that it is not standard is implied and enforced by a compile break.
Quoted 3 lines
ARM 32-bit
memcpy()[1] does it this way, so I did the same.[1]: https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_memcpy_32.h#L122
i see thanks.
Quoted 2 lines
While I agree that a macro for
__builtin_constant_p()would be good, it belongs in a patch to fix portability, not in this patch.
i agree it isn't composite of this change.
would you mind introducing it as a separate patch and depending on it or do you feel that would delay this patch too much? i wouldn't mind doing it myself but there is a long merge time on my patches which means i end up having to carry the adaptations locally for weeks at a time.
thanks
Quoted 90 lines
From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.45Quoted 6 lines · Morten Brørup
Quoted 6 lines
From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup
When getting objects from the mempool, the number of objects to
get
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
often constant at build time.
This patch adds another code path for this case, so the compiler
can
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
optimize more, e.g. unroll the copy loop when the entire request
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc
9.4.0,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mempool_perf_testwith constant n shows an increase in
rate_persecQuoted 4 lines
Quoted 4 lines
by an
Quoted 3 lines
Quoted 3 lines
average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at
build
Quoted 6 lines
Quoted 6 lines
time
Quoted 5 lines
Quoted 3 lines
remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you
may
Quoted 2 lines
Quoted 2 lines
want to
Quoted 1 line
take on board for any future version. I'd strongly suggest adding
some
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
extra clarifying code comments, as I suggest below. With those exta code comments:
Acked-by Bruce Richardson <[email protected]>Review
Quoted 5 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)(quoted diff) +0
-0
b/lib/mempool/rte_mempool.hQuoted 9 lines
Quoted 9 lines
Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct
Quoted 4 lines
index
9f530db24b..ade0100ec7100644lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct
rte_mempool*mp, void **obj_table,Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
Quoted 4 lines
lib/mempool/rte_mempool.h +0
-1if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
Quoted 4 lines
lib/mempool/rte_mempool.h +2
-1- len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + + if (__extension__(__builtin_constant_p(n)) && n <= cache-len) {
Quoted 2 lines
don't take direct dependency on compiler builtins. define a macro so
we
Quoted 3 lines
Quoted 3 lines
don't have to play shotgun surgery later.
also what is the purpose of using
__extension__here? are youannotating
Quoted 1 line
Quoted 1 line
the use of
__builtin_constant_p()or is there more? because ifthat's
Quoted 1 line
Quoted 1 line
the only reason i see no need to use
__extension__when alreadyusing a
Quoted 1 line
Quoted 1 line
compiler specific builtin like this, that it is not standard is
implied
Quoted 5 lines
Quoted 1 line
and enforced by a compile break.
ARM 32-bit
memcpy()[1] does it this way, so I did the same.[1]:
https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_me
mcpy_32.h#L122i see thanks.
Quoted 2 lines
While I agree that a macro for
__builtin_constant_p()would be good,
Quoted 8 lines
it belongs in a patch to fix portability, not in this patch.
i agree it isn't composite of this change.
would you mind introducing it as a separate patch and depending on it or do you feel that would delay this patch too much? i wouldn't mind doing it myself but there is a long merge time on my patches which means i end up having to carry the adaptations locally for weeks at a time.
I would rather not.
Introducing global macros in rte_common.h usually triggers a lot of discussion and pushback, and I don't want it to hold back this patch.
Quoted 94 lines · Morten Brørup
Quoted 94 lines
From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.45Quoted 6 lines · Morten Brørup
Quoted 6 lines
From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup
When getting objects from the mempool, the number of objects to
get
Quoted 4 lines
Quoted 4 lines
is
Quoted 3 lines
Quoted 3 lines
often constant at build time.
This patch adds another code path for this case, so the compiler
can
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
optimize more, e.g. unroll the copy loop when the entire request
is
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc
9.4.0,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
mempool_perf_testwith constant n shows an increase in
rate_persecQuoted 4 lines
Quoted 4 lines
by an
Quoted 3 lines
Quoted 3 lines
average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at
build
Quoted 6 lines
Quoted 6 lines
time
Quoted 5 lines
Quoted 3 lines
remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Change looks a good idea. Some suggestions inline below, which you
may
Quoted 2 lines
Quoted 2 lines
want to
Quoted 1 line
take on board for any future version. I'd strongly suggest adding
some
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
extra clarifying code comments, as I suggest below. With those exta code comments:
Acked-by Bruce Richardson <[email protected]>Review
Quoted 5 lines
--- lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)(quoted diff) +0
-0
b/lib/mempool/rte_mempool.hQuoted 9 lines
Quoted 9 lines
Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct
Quoted 4 lines
index
9f530db24b..ade0100ec7100644lib/mempool/rte_mempool.h +0
-0@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct
rte_mempool*mp, void **obj_table,Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
Quoted 4 lines
lib/mempool/rte_mempool.h +0
-1if (unlikely (cache ==NULL )) goto driver_dequeue; -/* Use the cache as much as we have to return hot objects first */
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
Quoted 4 lines
lib/mempool/rte_mempool.h +2
-1- len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + + if (__extension__(__builtin_constant_p(n)) && n <= cache-len) {
Quoted 2 lines
don't take direct dependency on compiler builtins. define a macro so
we
Quoted 3 lines
Quoted 3 lines
don't have to play shotgun surgery later.
also what is the purpose of using
__extension__here? are youannotating
Quoted 1 line
Quoted 1 line
the use of
__builtin_constant_p()or is there more? because ifthat's
Quoted 1 line
Quoted 1 line
the only reason i see no need to use
__extension__when alreadyusing a
Quoted 1 line
Quoted 1 line
compiler specific builtin like this, that it is not standard is
implied
Quoted 5 lines
Quoted 1 line
and enforced by a compile break.
ARM 32-bit
memcpy()[1] does it this way, so I did the same.[1]:
https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_me
mcpy_32.h#L122i see thanks.
Quoted 2 lines
While I agree that a macro for
__builtin_constant_p()would be good,it belongs in a patch to fix portability, not in this patch.
i agree it isn't composite of this change.
Quoted 8 lines
Quoted 4 lines
would you mind introducing it as a separate patch and depending on it or do you feel that would delay this patch too much? i wouldn't mind doing it myself but there is a long merge time on my patches which means i end up having to carry the adaptations locally for weeks at a time.
I would rather not.
Introducing global macros in
rte_common.husually triggers a lot of discussion and pushback, and I don't want it to hold back this patch.
yeah, no kidding. i wish the process was a bit more friendly being on the receiving end. it's unfortunate because it is discouraging improvements.
i'll bring a patch for it then.
Quoted 7 lines
From: Morten Brørup
Sent: Tuesday, 18 April 2023 13.30Quoted 4 lines
From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07On Tue, Apr 11, 2023 at 08:48:45AM +0200, Morten Brørup wrote:
[...]
Quoted 19 lines
Quoted 11 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514
Quoted 9 lines
lib/mempool/rte_mempool.h +9
-0+/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; +This loop looks a little awkward to me. Would it be clearer (and
perhaps
Quoted 6 lines
easier for compilers to unroll efficiently if it was rewritten as:
cache->len -= n; cache_objs = &cache->objs[cache->len]; for (index = 0; index < n; index++) obj_table[index] = cache_objs[index];
Quoted 8 lines
The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.
Regardless, your point here is still valid! I expected that any unrolling capable compiler can unroll *dst++ = *--src; but I can experiment with different compilers on godbolt.org to see if dst[index] = src[-index] is better.
Just for the record... I have now tried experimenting with the alternative, and it makes no difference.
PATCH v3 1/1 mempool: optimize get objects with constant n
In reply to Morten BrørupWhen getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at build time remains essentially unchanged.
---
v3:
* Rebase to main.
v2:
* Added comments describing why some code is omitted when 'n' is known at buuild time.
* Improved source code readability by explicitly setting 'remaining' where relevant, instead of at initialization.
--- lib/mempool/rte_mempool.h | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
lib/mempool/rte_mempool.h +16 -5
@@ -1492,13 +1492,15 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, unsigned int n, struct rte_mempool_cache *cache) { int ret; -unsigned int remaining = n; +unsigned int remaining;uint32_t index, len;void **cache_objs;/* No cache provided */ -if (unlikely (cache ==NULL )) +if (unlikely (cache ==NULL )) { + remaining = n; goto driver_dequeue; + } cache_objs = &cache->objs[cache->len];
@@ -1518,14 +1520,23 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,return 0 ; } -/* Use the cache as much as we have to return hot objects first */ +/* +* Use the cache as much as we have to return hot objects first. +* If the request size 'n' is known at build time, the above comparison +* ensures that n > cache->len here, so omit RTE_MIN(). +*/ len = __extension__(__builtin_constant_p (n)) ? cache->len : -RTE_MIN (remaining, cache->len); +RTE_MIN (n, cache->len); cache->len -= len; - remaining -= len; + remaining = n - len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; +/* +* If the request size 'n' is known at build time, the case +* where the entire request can be satisfied from the cache +* has already been handled above, so omit handling it here. +*/ if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */
Signature · 1 line
2.17.1
PATCH v4 1/1 mempool: optimize get objects with constant n
In reply to Morten BrørupWhen getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.
The code path where the number of objects to get is unknown at build time remains essentially unchanged.
---
v4:
* Rebased to main.
v3:
* Tried to rebase to main, but failed to include all my changes. v2:
* Added comments describing why some code is omitted when 'n' is known at buuild time.
* Improved source code readability by explicitly setting 'remaining' where relevant, instead of at initialization.
--- lib/mempool/rte_mempool.h | 41 +++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-)
lib/mempool/rte_mempool.h +35 -6
@@ -1492,23 +1492,52 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, unsigned int n, struct rte_mempool_cache *cache) { int ret; -unsigned int remaining = n; +unsigned int remaining;uint32_t index, len;void **cache_objs;/* No cache provided */ -if (unlikely (cache ==NULL )) +if (unlikely (cache ==NULL )) { + remaining = n; goto driver_dequeue; + } -/* Use the cache as much as we have to return hot objects first */ - len =RTE_MIN (remaining, cache->len); cache_objs = &cache->objs[cache->len]; + +if (__extension__(__builtin_constant_p (n )) && n <= cache->len) { +/* +* The request size is known at build time, and +* the entire request can be satisfied from the cache, +* so let the compiler unroll the fixed length copy loop. +*/ + cache->len -= n; +for (index =0 ; index < n; index++) + *obj_table++ = *--cache_objs; + +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 ); +RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_objs, n); + +return 0 ; + } + +/* +* Use the cache as much as we have to return hot objects first. +* If the request size 'n' is known at build time, the above comparison +* ensures that n > cache->len here, so omit RTE_MIN(). +*/ + len = __extension__(__builtin_constant_p (n)) ? cache->len : +RTE_MIN (n, cache->len); cache->len -= len; - remaining -= len; + remaining = n - len;for (index =0 ; index < len; index++) *obj_table++ = *--cache_objs; -if (remaining ==0 ) { +/* +* If the request size 'n' is known at build time, the case +* where the entire request can be satisfied from the cache +* has already been handled above, so omit handling it here. +*/ +if (!__extension__(__builtin_constant_p (n )) && remaining ==0 ) {/* The entire request is satisfied from the cache. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, get_success_bulk,1 );
Signature · 1 line
2.17.1
18/04/2023 22:09, Morten Brørup:
Quoted 11 lines
When getting objects from the mempool, the number of objects to get is often constant at build time.
This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.
On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0,
mempool_perf_testwith constant n shows an increase inrate_persecby an average of 17 %, minimum 9.5 %, maximum 24 %.
Quoted 5 lines
The code path where the number of objects to get is unknown at build time remains essentially unchanged.
Signed-off-by Morten Brørup <[email protected]>Acked-by Bruce Richardson <[email protected]>
Applied with suggested added comment, thanks.