Skip to contentSkip to conversation

mempool: optimize get objects with constant n

PATCH Superseded Morten Brørup 4 participants 19 messages Apr 11, 2023 – Jun 7, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mempool: optimize get objects with constant n

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +21 -3
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
+
+		return 0;
+	}
+
+	/* Use the cache as much as we have to return hot objects first */
+	len = __extension__(__builtin_constant_p(n)) ? cache->len :
+			RTE_MIN(remaining, cache->len);
 	cache->len -= len;
 	remaining -= len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
-	if (remaining == 0) {
+	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
 		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#1 lore raw
Quoted 7 lines · Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

Quoted 8 lines

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

Quoted 16 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +1 -2
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:

	memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523

Quoted 9 lines
lib/mempool/rte_mempool.h +9 -0
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
+
+		return 0;
+	}
+
+	/* Use the cache as much as we have to return hot objects first */
+	len = __extension__(__builtin_constant_p(n)) ? cache->len :
+			RTE_MIN(remaining, cache->len);

This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be "> cache->len" since the previous block was untaken, therefore we don't need to call RTE_MIN.

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532

Quoted 10 lines
lib/mempool/rte_mempool.h +1 -1
 	cache->len -= len;
 	remaining -= len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
-	if (remaining == 0) {
+	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
 		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);

Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.

I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.

Quoted 3 lines
Signature · 1 line
2.17.1
#2 lore raw
Quoted 58 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07

Quoted 7 lines · Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

Quoted 8 lines

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp,

Quoted 9 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,

void **obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +1 -2
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

Quoted 4 lines
	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.

Regardless, your point here is still valid! I expected that any unrolling capable compiler can unroll *dst++ = *--src; but I can experiment with different compilers on godbolt.org to see if dst[index] = src[-index] is better.

A future version could be hand coded with vector instructions here, like in the Ethdev drivers.

Quoted 5 lines

alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:

	memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);

Just for reference: It was previously discussed to ignore the stack ordering and simply copy the objects, but the idea was rejected due to the potential performance impact of not returning the hottest objects, i.e. the objects at the top of the stack, as the first in the returned array.

Quoted 17 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523

Quoted 10 lines
lib/mempool/rte_mempool.h +9 -0
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
+
+		return 0;
+	}
+
+	/* Use the cache as much as we have to return hot objects first */
+	len = __extension__(__builtin_constant_p(n)) ? cache->len :
+			RTE_MIN(remaining, cache->len);

This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be "> cache->len" since the previous block was untaken, therefore we don't need to call RTE_MIN.

I agree. This is almost like perl... write-only source code.

I will add a few explanatory comments.

Quoted 16 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532

Quoted 11 lines
lib/mempool/rte_mempool.h +1 -1
 	cache->len -= len;
 	remaining -= len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
-	if (remaining == 0) {
+	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
 		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);

Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.

Quoted 3 lines

I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.

I get your point, but the code is still short enough to grasp (after comments have been added). So I prefer to avoid code duplication.

Quoted 4 lines
Quoted 4 lines

--
2.17.1

#3 lore raw
Quoted 70 lines · Morten Brørup
Quoted 58 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07

Quoted 7 lines · Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

Quoted 8 lines

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp,

Quoted 9 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,

void **obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +1 -2
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

Quoted 4 lines
	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.

Regardless, your point here is still valid! I expected that any unrolling capable compiler can unroll *dst++ = *--src; but I can experiment with different compilers on godbolt.org to see if dst[index] = src[-index] is better.

A future version could be hand coded with vector instructions here, like in the Ethdev drivers.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:

	memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);

Just for reference: It was previously discussed to ignore the stack ordering and simply copy the objects, but the idea was rejected due to the potential performance impact of not returning the hottest objects, i.e. the objects at the top of the stack, as the first in the returned array.

Ah, yes, I had forgotten that we want the order reversed, so most recent objects first. It would be interesting to see if ignoring order within a burst can lead to some performance improvements, as I expect that in most cases, the buffers to be used in a burst are to be used pretty much simultaneously. However, based on your feedback, feel free to ignore this comment and keep code as-is.

Quoted 18 lines
Quoted 17 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523

Quoted 10 lines
lib/mempool/rte_mempool.h +9 -0
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
+
+		return 0;
+	}
+
+	/* Use the cache as much as we have to return hot objects first */
+	len = __extension__(__builtin_constant_p(n)) ? cache->len :
+			RTE_MIN(remaining, cache->len);

This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be "> cache->len" since the previous block was untaken, therefore we don't need to call RTE_MIN.

Quoted 4 lines

I agree. This is almost like perl... write-only source code.

I will add a few explanatory comments.

Thanks.

Quoted 16 lines
Quoted 16 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532

Quoted 11 lines
lib/mempool/rte_mempool.h +1 -1
 	cache->len -= len;
 	remaining -= len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
-	if (remaining == 0) {
+	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
 		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);

Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.

Quoted 6 lines
Quoted 3 lines

I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.

I get your point, but the code is still short enough to grasp (after comments have been added). So I prefer to avoid code duplication.

Sure.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
2.17.1

#4 lore raw
In reply to Morten Brørup
Quoted 58 lines · Morten Brørup
Quoted 58 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07

Quoted 7 lines · Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

Quoted 8 lines

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp,

Quoted 9 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp,

void **obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +1 -2
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

Quoted 7 lines
Quoted 4 lines
	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.

BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!

/Bruce

#5 lore raw

18/04/2023 14:55, Bruce Richardson:

Quoted 22 lines · Morten Brørup
Quoted 21 lines

From: Bruce Richardson [mailto:[email protected]]

Quoted 19 lines · Morten Brørup

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];
Quoted 5 lines
Quoted 2 lines

The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.

BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!

Looks like this request was missed.

#6 lore raw
Quoted 27 lines

From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 09.52

18/04/2023 14:55, Bruce Richardson:

Quoted 17 lines · Morten Brørup
Quoted 16 lines

From: Bruce Richardson [mailto:[email protected]]

Quoted 15 lines · Morten Brørup

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

The mempool cache is a stack, so the copy loop needs get the objects in

Quoted 8 lines

decrementing order. I.e. the source index decrements and the destination index increments.

Quoted 4 lines
Quoted 1 line

BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!

Looks like this request was missed.

I intentionally omitted it, because I disagree with the suggestion.

Otherwise, reminders that the mempool cache is a stack should be plastered all over the source code, not just here. For reference, this copy loop (without such a reminder) also exists elsewhere in the same file.

Apologies for not responding to Bruce's request earlier.

#7 lore raw

07/06/2023 10:03, Morten Brørup:

Quoted 34 lines
Quoted 34 lines

From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 09.52

18/04/2023 14:55, Bruce Richardson:

Quoted 17 lines · Morten Brørup
Quoted 16 lines

From: Bruce Richardson [mailto:[email protected]]

Quoted 15 lines · Morten Brørup

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 10 lines
lib/mempool/rte_mempool.h +10 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

The mempool cache is a stack, so the copy loop needs get the objects in

decrementing order. I.e. the source index decrements and the destination index increments.

Quoted 4 lines
Quoted 1 line

BTW: Please add this as a comment in the code too, above the loop to avoid future developers (or even future me), asking this question again!

Quoted 7 lines
Quoted 1 line

Looks like this request was missed.

I intentionally omitted it, because I disagree with the suggestion.

Otherwise, reminders that the mempool cache is a stack should be plastered all over the source code, not just here. For reference, this copy loop (without such a reminder) also exists elsewhere in the same file.

Apologies for not responding to Bruce's request earlier.

What about doing a general comment at the top of the function, with the assignment of the pointer at the end of the array:

    /* The cache is a stack, so copy will be in reverse order. */
    cache_objs = &cache->objs[cache->len];

I could do it on apply if there is an agreement.

#8 lore raw
Quoted 52 lines

From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 10.10

07/06/2023 10:03, Morten Brørup:

Quoted 7 lines
Quoted 7 lines

From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 09.52

18/04/2023 14:55, Bruce Richardson:

Quoted 3 lines · Morten Brørup
Quoted 2 lines

From: Bruce Richardson [mailto:[email protected]]

Quoted 1 line · Morten Brørup
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +4 -0
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and

perhaps

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

easier for compilers to unroll efficiently if it was rewritten as:

	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

The mempool cache is a stack, so the copy loop needs get the objects

in

Quoted 1 line
Quoted 1 line

decrementing order. I.e. the source index decrements and the destination

index

Quoted 4 lines
Quoted 4 lines

increments.

Quoted 3 lines
Quoted 1 line

BTW: Please add this as a comment in the code too, above the loop to

avoid

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

future developers (or even future me), asking this question again!

Looks like this request was missed.

I intentionally omitted it, because I disagree with the suggestion.

Otherwise, reminders that the mempool cache is a stack should be plastered

all over the source code, not just here. For reference, this copy loop (without such a reminder) also exists elsewhere in the same file.

Quoted 2 lines

Apologies for not responding to Bruce's request earlier.

Quoted 8 lines

What about doing a general comment at the top of the function, with the assignment of the pointer at the end of the array:

    /* The cache is a stack, so copy will be in reverse order. */
    cache_objs = &cache->objs[cache->len];

I could do it on apply if there is an agreement.

ACK from me.

For consistency, please also add the reminder to the two existing reverse order copy loops in rte_mempool_do_generic_get().

#9 lore raw
In reply to Thomas Monjalon
Quoted 6 lines

From: Morten Brørup
Sent: Wednesday, 7 June 2023 10.34

Quoted 3 lines

From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, 7 June 2023 10.10

[...]

Quoted 6 lines
Quoted 6 lines

What about doing a general comment at the top of the function, with the assignment of the pointer at the end of the array:

    /* The cache is a stack, so copy will be in reverse order. */
    cache_objs = &cache->objs[cache->len];
Quoted 6 lines
Quoted 2 lines

I could do it on apply if there is an agreement.

ACK from me.

For consistency, please also add the reminder to the two existing reverse order copy loops in rte_mempool_do_generic_get().

Rubbish! I was looking at the wrong version of the code.

So, plain ACK from me. No further comments!

Sorry. :-!

#10 lore raw
In reply to Bruce Richardson
Quoted 36 lines · Bruce Richardson
Quoted 7 lines · Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

Quoted 8 lines

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may want to take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

Quoted 12 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
Quoted 5 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1505

Quoted 5 lines
lib/mempool/rte_mempool.h +2 -2
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {

don't take direct dependency on compiler builtins. define a macro so we don't have to play shotgun surgery later.

also what is the purpose of using __extension__ here? are you annotating the use of __builtin_constant_p() or is there more? because if that's the only reason i see no need to use __extension__ when already using a compiler specific builtin like this, that it is not standard is implied and enforced by a compile break.

Quoted 62 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 9 lines
lib/mempool/rte_mempool.h +9 -0
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and perhaps easier for compilers to unroll efficiently if it was rewritten as:

	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];

alternatively those last two lines can be replaced by a memcpy, which the compiler should nicely optimize itself, for constant size copy:

	memcpy(obj_table, cache_objs, sizeof(obj_table[0]) * n);

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1523

Quoted 9 lines
lib/mempool/rte_mempool.h +9 -0
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
+
+		return 0;
+	}
+
+	/* Use the cache as much as we have to return hot objects first */
+	len = __extension__(__builtin_constant_p(n)) ? cache->len :
+			RTE_MIN(remaining, cache->len);

This line confused me a lot, until I applied the patch and stared at it a lot :-). Why would the length value depend upon whether or not n is a compile-time constant?
I think it would really help here to add in a comment stating that when n is a compile-time constant, at this point it much be "> cache->len" since the previous block was untaken, therefore we don't need to call RTE_MIN.

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1532

Quoted 10 lines
lib/mempool/rte_mempool.h +1 -1
 	cache->len -= len;
 	remaining -= len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
-	if (remaining == 0) {
+	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
 		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);

Once I'd worked out the logic for the above conditional check, then this conditional adjustment was clear. I just wonder if a further comment might help here.

I am also wondering if having larger blocks for the constant and non-constant cases may help. It would lead to some duplication but may clarify the code.

Quoted 3 lines
Signature · 1 line
2.17.1
#11 lore raw
Quoted 54 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15

Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup

When getting objects from the mempool, the number of objects to get

is

Quoted 8 lines
Quoted 8 lines

often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec

by an

Quoted 3 lines
Quoted 3 lines

average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build

time

Quoted 5 lines
Quoted 3 lines

remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may

want to

Quoted 10 lines

take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

Quoted 4 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
Quoted 5 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct

Quoted 5 lines
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct

rte_mempool *mp, void **obj_table,

Quoted 8 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1505

Quoted 8 lines
lib/mempool/rte_mempool.h +2 -2
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {

don't take direct dependency on compiler builtins. define a macro so we don't have to play shotgun surgery later.

Quoted 5 lines

also what is the purpose of using __extension__ here? are you annotating the use of __builtin_constant_p() or is there more? because if that's the only reason i see no need to use __extension__ when already using a compiler specific builtin like this, that it is not standard is implied and enforced by a compile break.

ARM 32-bit memcpy() [1] does it this way, so I did the same.

[1]: https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_memcpy_32.h#L122

While I agree that a macro for __builtin_constant_p() would be good, it belongs in a patch to fix portability, not in this patch.

#12 lore raw
Quoted 60 lines · Morten Brørup
Quoted 54 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15

Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup

When getting objects from the mempool, the number of objects to get

is

Quoted 8 lines
Quoted 8 lines

often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec

by an

Quoted 3 lines
Quoted 3 lines

average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build

time

Quoted 5 lines
Quoted 3 lines

remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you may

want to

Quoted 10 lines

take on board for any future version. I'd strongly suggest adding some extra clarifying code comments, as I suggest below.
With those exta code comments:

Quoted 4 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
Quoted 5 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct

Quoted 5 lines
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..ade0100ec7 100644
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct

rte_mempool *mp, void **obj_table,

Quoted 8 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1505

Quoted 8 lines
lib/mempool/rte_mempool.h +2 -2
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {

don't take direct dependency on compiler builtins. define a macro so we don't have to play shotgun surgery later.

Quoted 5 lines

also what is the purpose of using __extension__ here? are you annotating the use of __builtin_constant_p() or is there more? because if that's the only reason i see no need to use __extension__ when already using a compiler specific builtin like this, that it is not standard is implied and enforced by a compile break.

Quoted 3 lines

ARM 32-bit memcpy() [1] does it this way, so I did the same.

[1]: https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_memcpy_32.h#L122

i see thanks.

Quoted 2 lines

While I agree that a macro for __builtin_constant_p() would be good, it belongs in a patch to fix portability, not in this patch.

i agree it isn't composite of this change.

would you mind introducing it as a separate patch and depending on it or do you feel that would delay this patch too much? i wouldn't mind doing it myself but there is a long merge time on my patches which means i end up having to carry the adaptations locally for weeks at a time.

thanks

#13 lore raw
Quoted 90 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.45

Quoted 6 lines · Morten Brørup
Quoted 6 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15

Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup

When getting objects from the mempool, the number of objects to

get

Quoted 4 lines
Quoted 4 lines

is

Quoted 3 lines
Quoted 3 lines

often constant at build time.

This patch adds another code path for this case, so the compiler

can

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

optimize more, e.g. unroll the copy loop when the entire request

is

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc

9.4.0,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

mempool_perf_test with constant n shows an increase in

rate_persec

Quoted 4 lines
Quoted 4 lines

by an

Quoted 3 lines
Quoted 3 lines

average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at

build

Quoted 6 lines
Quoted 6 lines

time

Quoted 5 lines
Quoted 3 lines

remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you

may

Quoted 2 lines
Quoted 2 lines

want to

Quoted 1 line

take on board for any future version. I'd strongly suggest adding

some

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

extra clarifying code comments, as I suggest below. With those exta code comments:

Review

Quoted 5 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/lib/mempool/rte_mempool.h

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct

Quoted 4 lines

index 9f530db24b..ade0100ec7 100644

lib/mempool/rte_mempool.h +0 -0
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct

rte_mempool *mp, void **obj_table,

Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,

Quoted 4 lines
lib/mempool/rte_mempool.h +0 -1
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects

first */

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,

Quoted 4 lines
lib/mempool/rte_mempool.h +2 -1
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache-

len) {

Quoted 2 lines

don't take direct dependency on compiler builtins. define a macro so

we

Quoted 3 lines
Quoted 3 lines

don't have to play shotgun surgery later.

also what is the purpose of using __extension__ here? are you

annotating

Quoted 1 line
Quoted 1 line

the use of __builtin_constant_p() or is there more? because if

that's

Quoted 1 line
Quoted 1 line

the only reason i see no need to use __extension__ when already

using a

Quoted 1 line
Quoted 1 line

compiler specific builtin like this, that it is not standard is

implied

Quoted 5 lines
Quoted 1 line

and enforced by a compile break.

ARM 32-bit memcpy() [1] does it this way, so I did the same.

[1]:

https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_me
mcpy_32.h#L122

i see thanks.

Quoted 2 lines

While I agree that a macro for __builtin_constant_p() would be good,

Quoted 8 lines

it belongs in a patch to fix portability, not in this patch.

i agree it isn't composite of this change.

would you mind introducing it as a separate patch and depending on it or do you feel that would delay this patch too much? i wouldn't mind doing it myself but there is a long merge time on my patches which means i end up having to carry the adaptations locally for weeks at a time.

I would rather not.

Introducing global macros in rte_common.h usually triggers a lot of discussion and pushback, and I don't want it to hold back this patch.

#14 lore raw
Quoted 94 lines · Morten Brørup
Quoted 94 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.45

Quoted 6 lines · Morten Brørup
Quoted 6 lines

From: Tyler Retzlaff [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 17.15

Quoted 2 lines · Bruce Richardson
Quoted 1 line · Morten Brørup

When getting objects from the mempool, the number of objects to

get

Quoted 4 lines
Quoted 4 lines

is

Quoted 3 lines
Quoted 3 lines

often constant at build time.

This patch adds another code path for this case, so the compiler

can

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

optimize more, e.g. unroll the copy loop when the entire request

is

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc

9.4.0,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

mempool_perf_test with constant n shows an increase in

rate_persec

Quoted 4 lines
Quoted 4 lines

by an

Quoted 3 lines
Quoted 3 lines

average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at

build

Quoted 6 lines
Quoted 6 lines

time

Quoted 5 lines
Quoted 3 lines

remains essentially unchanged.

Change looks a good idea. Some suggestions inline below, which you

may

Quoted 2 lines
Quoted 2 lines

want to

Quoted 1 line

take on board for any future version. I'd strongly suggest adding

some

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

extra clarifying code comments, as I suggest below. With those exta code comments:

Review

Quoted 5 lines
---
 lib/mempool/rte_mempool.h | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/lib/mempool/rte_mempool.h

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct

Quoted 4 lines

index 9f530db24b..ade0100ec7 100644

lib/mempool/rte_mempool.h +0 -0
@@ -1500,15 +1500,33 @@ rte_mempool_do_generic_get(struct

rte_mempool *mp, void **obj_table,

Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,

Quoted 4 lines
lib/mempool/rte_mempool.h +0 -1
 	if (unlikely(cache == NULL))
 		goto driver_dequeue;
 
-	/* Use the cache as much as we have to return hot objects

first */

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,

Quoted 4 lines
lib/mempool/rte_mempool.h +2 -1
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache-

len) {

Quoted 2 lines

don't take direct dependency on compiler builtins. define a macro so

we

Quoted 3 lines
Quoted 3 lines

don't have to play shotgun surgery later.

also what is the purpose of using __extension__ here? are you

annotating

Quoted 1 line
Quoted 1 line

the use of __builtin_constant_p() or is there more? because if

that's

Quoted 1 line
Quoted 1 line

the only reason i see no need to use __extension__ when already

using a

Quoted 1 line
Quoted 1 line

compiler specific builtin like this, that it is not standard is

implied

Quoted 5 lines
Quoted 1 line

and enforced by a compile break.

ARM 32-bit memcpy() [1] does it this way, so I did the same.

[1]:

https://elixir.bootlin.com/dpdk/v23.03/source/lib/eal/arm/include/rte_me
mcpy_32.h#L122

i see thanks.

Quoted 2 lines

While I agree that a macro for __builtin_constant_p() would be good,

it belongs in a patch to fix portability, not in this patch.

i agree it isn't composite of this change.

Quoted 8 lines
Quoted 4 lines

would you mind introducing it as a separate patch and depending on it or do you feel that would delay this patch too much? i wouldn't mind doing it myself but there is a long merge time on my patches which means i end up having to carry the adaptations locally for weeks at a time.

I would rather not.

Introducing global macros in rte_common.h usually triggers a lot of discussion and pushback, and I don't want it to hold back this patch.

yeah, no kidding. i wish the process was a bit more friendly being on the receiving end. it's unfortunate because it is discouraging improvements.

i'll bring a patch for it then.

#15 lore raw
In reply to Bruce Richardson
Quoted 7 lines

From: Morten Brørup
Sent: Tuesday, 18 April 2023 13.30

Quoted 4 lines

From: Bruce Richardson [mailto:[email protected]]
Sent: Tuesday, 18 April 2023 13.07

On Tue, Apr 11, 2023 at 08:48:45AM +0200, Morten Brørup wrote:

[...]

Quoted 19 lines
Quoted 11 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table, · line 1514

Quoted 9 lines
lib/mempool/rte_mempool.h +9 -0
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+

This loop looks a little awkward to me. Would it be clearer (and

perhaps

Quoted 6 lines

easier for compilers to unroll efficiently if it was rewritten as:

	cache->len -= n;
	cache_objs = &cache->objs[cache->len];
	for (index = 0; index < n; index++)
		obj_table[index] = cache_objs[index];
Quoted 8 lines

The mempool cache is a stack, so the copy loop needs get the objects in decrementing order. I.e. the source index decrements and the destination index increments.

Regardless, your point here is still valid! I expected that any unrolling capable compiler can unroll *dst++ = *--src; but I can experiment with different compilers on godbolt.org to see if dst[index] = src[-index] is better.

Just for the record... I have now tried experimenting with the alternative, and it makes no difference.

#16 lore raw

PATCH v3 1/1 mempool: optimize get objects with constant n

In reply to Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

---
v3:
* Rebase to main.
v2:
* Added comments describing why some code is omitted when 'n' is known at buuild time.
* Improved source code readability by explicitly setting 'remaining' where relevant, instead of at initialization.

---
 lib/mempool/rte_mempool.h | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
lib/mempool/rte_mempool.h +16 -5
index ade0100ec7..7d14dea5c4 100644
@@ -1492,13 +1492,15 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
 			   unsigned int n, struct rte_mempool_cache *cache)
 {
 	int ret;
-	unsigned int remaining = n;
+	unsigned int remaining;
 	uint32_t index, len;
 	void **cache_objs;
 
 	/* No cache provided */
-	if (unlikely(cache == NULL))
+	if (unlikely(cache == NULL)) {
+		remaining = n;
 		goto driver_dequeue;
+	}
 
 	cache_objs = &cache->objs[cache->len];
 
@@ -1518,14 +1520,23 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
 		return 0;
 	}
 
-	/* Use the cache as much as we have to return hot objects first */
+	/*
+	 * Use the cache as much as we have to return hot objects first.
+	 * If the request size 'n' is known at build time, the above comparison
+	 * ensures that n > cache->len here, so omit RTE_MIN().
+	 */
 	len = __extension__(__builtin_constant_p(n)) ? cache->len :
-			RTE_MIN(remaining, cache->len);
+			RTE_MIN(n, cache->len);
 	cache->len -= len;
-	remaining -= len;
+	remaining = n - len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
+	/*
+	 * If the request size 'n' is known at build time, the case
+	 * where the entire request can be satisfied from the cache
+	 * has already been handled above, so omit handling it here.
+	 */
 	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
Signature · 1 line
2.17.1
#17 lore raw

PATCH v4 1/1 mempool: optimize get objects with constant n

In reply to Morten Brørup

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

---
v4:
* Rebased to main.
v3:
* Tried to rebase to main, but failed to include all my changes. v2:
* Added comments describing why some code is omitted when 'n' is known at buuild time.
* Improved source code readability by explicitly setting 'remaining' where relevant, instead of at initialization.

---
 lib/mempool/rte_mempool.h | 41 +++++++++++++++++++++++++++++++++------
 1 file changed, 35 insertions(+), 6 deletions(-)
lib/mempool/rte_mempool.h +35 -6
index 9f530db24b..7d14dea5c4 100644
@@ -1492,23 +1492,52 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
 			   unsigned int n, struct rte_mempool_cache *cache)
 {
 	int ret;
-	unsigned int remaining = n;
+	unsigned int remaining;
 	uint32_t index, len;
 	void **cache_objs;
 
 	/* No cache provided */
-	if (unlikely(cache == NULL))
+	if (unlikely(cache == NULL)) {
+		remaining = n;
 		goto driver_dequeue;
+	}
 
-	/* Use the cache as much as we have to return hot objects first */
-	len = RTE_MIN(remaining, cache->len);
 	cache_objs = &cache->objs[cache->len];
+
+	if (__extension__(__builtin_constant_p(n)) && n <= cache->len) {
+		/*
+		 * The request size is known at build time, and
+		 * the entire request can be satisfied from the cache,
+		 * so let the compiler unroll the fixed length copy loop.
+		 */
+		cache->len -= n;
+		for (index = 0; index < n; index++)
+			*obj_table++ = *--cache_objs;
+
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
+		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
+
+		return 0;
+	}
+
+	/*
+	 * Use the cache as much as we have to return hot objects first.
+	 * If the request size 'n' is known at build time, the above comparison
+	 * ensures that n > cache->len here, so omit RTE_MIN().
+	 */
+	len = __extension__(__builtin_constant_p(n)) ? cache->len :
+			RTE_MIN(n, cache->len);
 	cache->len -= len;
-	remaining -= len;
+	remaining = n - len;
 	for (index = 0; index < len; index++)
 		*obj_table++ = *--cache_objs;
 
-	if (remaining == 0) {
+	/*
+	 * If the request size 'n' is known at build time, the case
+	 * where the entire request can be satisfied from the cache
+	 * has already been handled above, so omit handling it here.
+	 */
+	if (!__extension__(__builtin_constant_p(n)) && remaining == 0) {
 		/* The entire request is satisfied from the cache. */
 
 		RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
Signature · 1 line
2.17.1
#18 lore raw

18/04/2023 22:09, Morten Brørup:

Quoted 11 lines

When getting objects from the mempool, the number of objects to get is often constant at build time.

This patch adds another code path for this case, so the compiler can optimize more, e.g. unroll the copy loop when the entire request is satisfied from the cache.

On an Intel(R) Xeon(R) E5-2620 v4 CPU, and compiled with gcc 9.4.0, mempool_perf_test with constant n shows an increase in rate_persec by an average of 17 %, minimum 9.5 %, maximum 24 %.

Quoted 5 lines

The code path where the number of objects to get is unknown at build time remains essentially unchanged.

Applied with suggested added comment, thanks.