Skip to contentSkip to conversation

mempool: micro-optimize put function

PATCH Superseded Morten Brørup 4 participants 16 messages Nov 16, 2022 – Dec 27, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mempool: micro-optimize put function

Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

---
 lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +18 -17
index 9f530db24b..aba90dbb5b 100644
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 {
 	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without crossing
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
-
-	if (cache->len + n <= cache->flushthresh) {
+	if (likely(cache->len + n <= cache->flushthresh)) {
+		/*
+		 * The objects can be added to the cache without crossing the
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
 
 	/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 
 driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);
 }
 
Signature · 1 line
2.17.1
#1 lore raw
Quoted 22 lines · Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

---
  lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
  1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +1 -1
index 9f530db24b..aba90dbb5b 100644
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
  {
  	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
  	if (unlikely(cache == NULL))

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1377

Quoted 10 lines
lib/mempool/rte_mempool.h +1 -4
  		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
  	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
  	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;

I've kept the check here since it protects against overflow in len plus n below if n is really huge.

Quoted 48 lines
lib/mempool/rte_mempool.h +16 -12
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without crossing
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
-
-	if (cache->len + n <= cache->flushthresh) {
+	if (likely(cache->len + n <= cache->flushthresh)) {
+		/*
+		 * The objects can be added to the cache without crossing the
+		 * flush threshold.
+		 */
  		cache_objs = &cache->objs[cache->len];
  		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
  		cache_objs = &cache->objs[0];
  		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
  		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
  	}
 
  	/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 
  driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
  	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
  	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
  driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
  	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);
  }
 
#2 lore raw
Quoted 35 lines

From: Andrew Rybchenko [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 12.05

Quoted 2 lines · Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by

moving an

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool

Quoted 14 lines

unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

---
  lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
  1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..aba90dbb5b 100644
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool

*mp, void * const *obj_table,

Quoted 13 lines
lib/mempool/rte_mempool.h +2 -2
  {
  	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
  	if (unlikely(cache == NULL))
  		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
  	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
  	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1377

Quoted 3 lines
lib/mempool/rte_mempool.h +0 -3
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;

I've kept the check here since it protects against overflow in len plus n below if n is really huge.

We can fix that, see below.

Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 5 lines
lib/mempool/rte_mempool.h +0 -4
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1375

Quoted 6 lines
lib/mempool/rte_mempool.h +1 -5
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
-
-	if (cache->len + n <= cache->flushthresh) {
+	if (likely(cache->len + n <= cache->flushthresh)) {

It is an invariant that cache->len <= cache->flushthresh, so the above comparison can be rewritten to protect against overflow:

if (likely(n <= cache->flushthresh - cache->len)) {

Quoted 41 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool · line 1377

Quoted 2 lines
lib/mempool/rte_mempool.h +2 -0
+		/*
+		 * The objects can be added to the cache without crossing

the

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool

Quoted 21 lines
lib/mempool/rte_mempool.h +11 -1
+		 * flush threshold.
+		 */
  		cache_objs = &cache->objs[cache->len];
  		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
  		cache_objs = &cache->objs[0];
  		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
  		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
  	}
 
  	/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool

*mp, void * const *obj_table,

Quoted 15 lines
lib/mempool/rte_mempool.h +2 -2
  driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
  	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
  	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
  driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
  	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);
  }
 
#3 lore raw
Quoted 59 lines · Morten Brørup
Quoted 35 lines

From: Andrew Rybchenko [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 12.05

Quoted 2 lines · Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by

moving an

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool

Quoted 14 lines

unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

---
   lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
   1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..aba90dbb5b 100644
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool

*mp, void * const *obj_table,

Quoted 13 lines
lib/mempool/rte_mempool.h +2 -2
   {
   	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
   	if (unlikely(cache == NULL))
   		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
   	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
   	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1377

Quoted 3 lines
lib/mempool/rte_mempool.h +0 -3
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;

I've kept the check here since it protects against overflow in len plus n below if n is really huge.

We can fix that, see below.

Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 5 lines
lib/mempool/rte_mempool.h +0 -4
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1375

Quoted 6 lines
lib/mempool/rte_mempool.h +1 -5
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
-
-	if (cache->len + n <= cache->flushthresh) {
+	if (likely(cache->len + n <= cache->flushthresh)) {

It is an invariant that cache->len <= cache->flushthresh, so the above comparison can be rewritten to protect against overflow:

Quoted 2 lines

if (likely(n <= cache->flushthresh - cache->len)) {

True, but it would be useful to highlight the usage of the invariant here using either a comment or an assert.

IMHO it is wrong to use likely here since, as far as I know, it makes else branch very expensive, but crossing the flush threshold is an expected branch and it must not be that
expensive.

Quoted 42 lines
Quoted 41 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool · line 1377

Quoted 2 lines
lib/mempool/rte_mempool.h +2 -0
+		/*
+		 * The objects can be added to the cache without crossing

the

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool

Quoted 21 lines
lib/mempool/rte_mempool.h +11 -1
+		 * flush threshold.
+		 */
   		cache_objs = &cache->objs[cache->len];
   		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
   		cache_objs = &cache->objs[0];
   		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
   		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
   	}
 
   	/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool

*mp, void * const *obj_table,

Quoted 15 lines
lib/mempool/rte_mempool.h +2 -2
   driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
   	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
   	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
   driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
   	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);
   }
 
#4 lore raw

PATCH v2 1/1 mempool: micro-optimize put function

In reply to Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed likely() from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++----------------
 1 file changed, 20 insertions(+), 16 deletions(-)
lib/mempool/rte_mempool.h +20 -16
index 9f530db24b..dd1a3177d6 100644
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 {
 	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without crossing
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the comparison below. */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without crossing the
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
 
 	/* Add the objects to the cache. */
@@ -1399,13 +1403,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 
 driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);
 }
 
Signature · 1 line
2.17.1
#5 lore raw

<snip>

Quoted 54 lines

Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed likely() from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++----------------
 1 file changed, 20 insertions(+), 16 deletions(-)
lib/mempool/rte_mempool.h +4 -12
index 9f530db24b..dd1a3177d6 100644
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without crossing
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the comparison below.
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1386

Quoted 10 lines
lib/mempool/rte_mempool.h +6 -2
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without crossing the
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In the PMDs I have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.

Quoted 33 lines
lib/mempool/rte_mempool.h +8 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
lib/mempool/rte_mempool.h +2 -3
 	/* Add the objects to the cache. */
@@ -1399,13 +1403,13 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,
 
 driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#6 lore raw
Quoted 69 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 16.51

<snip>

Quoted 3 lines

Micro-optimization:
Reduced the most likely code path in the generic put function by

moving an

Quoted 6 lines

unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len

is

Quoted 4 lines

non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed

likely()

Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static branch

prediction

Quoted 22 lines

here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++----------------
 1 file changed, 20 insertions(+), 16 deletions(-)
lib/mempool/rte_mempool.h +2 -2
index 9f530db24b..dd1a3177d6 100644
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381

Quoted 10 lines
lib/mempool/rte_mempool.h +0 -7
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 4 lines
lib/mempool/rte_mempool.h +1 -3
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the comparison

below.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +4 -1
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without crossing

the

Quoted 10 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -1
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In the PMDs I have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.

There could be applications using a mempool for something else than mbufs.

In that case, the application should be allowed to get/put many objects in one transaction.

Quoted 34 lines
Quoted 34 lines
lib/mempool/rte_mempool.h +8 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
lib/mempool/rte_mempool.h +2 -3
 	/* Add the objects to the cache. */
@@ -1399,13 +1403,13 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,
 
 driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#7 lore raw

<snip>

Quoted 77 lines
Quoted 66 lines
Quoted 3 lines

Micro-optimization:
Reduced the most likely code path in the generic put function by

moving an

Quoted 7 lines

unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len

is

Quoted 4 lines

non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed

likely()

Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static branch

prediction

Quoted 22 lines

here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++----------------
 1 file changed, 20 insertions(+), 16 deletions(-)
lib/mempool/rte_mempool.h +2 -2
index 9f530db24b..dd1a3177d6 100644
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381

Quoted 10 lines
lib/mempool/rte_mempool.h +0 -7
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 4 lines
lib/mempool/rte_mempool.h +1 -3
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the comparison

below.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +4 -1
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without crossing

the

Quoted 10 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -1
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In the PMDs I have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.

Quoted 1 line

There could be applications using a mempool for something else than mbufs.

Agree

Quoted 3 lines

In that case, the application should be allowed to get/put many objects in one transaction.

Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which is more than 'x'. It should be possible to change the threshold configuration or reduce the request size.

If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of using the per-core cache.

With this check, we are introducing an additional memcpy as well. I am not sure if reusing the latest buffers is better than having an memcpy.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/mempool/rte_mempool.h +8 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}

/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

driver_enqueue:

lib/mempool/rte_mempool.h +2 -3
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#8 lore raw
Quoted 97 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27

<snip>

Quoted 17 lines
Quoted 17 lines
Quoted 3 lines

Micro-optimization:
Reduced the most likely code path in the generic put function by

moving an

Quoted 7 lines

unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len

is

Quoted 4 lines

non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed

likely()

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static branch

prediction

Quoted 5 lines

here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-------------

---

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
 1 file changed, 20 insertions(+), 16 deletions(-)
(quoted diff) +0 -0

b/lib/mempool/rte_mempool.h

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

index 9f530db24b..dd1a3177d6 100644

lib/mempool/rte_mempool.h +0 -0
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
Quoted 7 lines
Quoted 7 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +2 -2
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds.

*/

Quoted 11 lines
Quoted 11 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 10 lines
lib/mempool/rte_mempool.h +0 -7
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382

Quoted 1 line
lib/mempool/rte_mempool.h +0 -1
-	 *      the flush threshold, flush the cache to the

backend.

Quoted 3 lines
Quoted 3 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 3 lines
lib/mempool/rte_mempool.h +1 -2
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the

comparison

Quoted 8 lines
Quoted 8 lines

below.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +4 -1
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without

crossing

Quoted 8 lines
Quoted 8 lines

the

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -1
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In the

PMDs I

Quoted 5 lines
Quoted 3 lines

have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.

There could be applications using a mempool for something else than

mbufs.
Agree

Quoted 2 lines

In that case, the application should be allowed to get/put many

objects in

Quoted 1 line

one transaction.

Quoted 8 lines

Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which is more than 'x'. It should be possible to change the threshold configuration or reduce the request size.

If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of using the per-core cache.

Correct. I suppose this is the intended behavior of this API.

The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.

[1]: http://inbox.dpdk.org/dev/[email protected]/T/#u

Quoted 3 lines

With this check, we are introducing an additional memcpy as well. I am not sure if reusing the latest buffers is better than having an memcpy.

There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.

Please note that this check is not new, it has just been moved. Before this patch, it was checked on every call (if a cache is present); with this patch, it is only checked if the entire request cannot go directly into the cache.

Quoted 39 lines
Quoted 6 lines
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the

backend, so

Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390

Quoted 1 line
lib/mempool/rte_mempool.h +1 -0
+		 * adding the objects does not cross the flush

threshold.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h +4 -0
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
Quoted 8 lines
Quoted 8 lines

lib/mempool/rte_mempool.h

Quoted 8 lines

/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

driver_enqueue:

lib/mempool/rte_mempool.h +1 -1
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1 -2
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#9 lore raw

PING mempool maintainers.

If you don't provide ACK or Review to patches, how should Thomas know that it is ready for merging?

-Morten

Quoted 177 lines

From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 18.39

Quoted 8 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27

<snip>

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Micro-optimization:
Reduced the most likely code path in the generic put function

by

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

moving an

Quoted 1 line

unlikely check out of the most likely code path and further

down.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge

and

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

len

is

Quoted 2 lines

non-zero.
* Added assertion about the invariant preventing overflow in

the

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

comparison.
* Crossing the threshold is not extremely unlikely, so removed

likely()

Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static branch

prediction

Quoted 5 lines

here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-----------

--

Quoted 64 lines

---

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
 1 file changed, 20 insertions(+), 16 deletions(-)
(quoted diff) +0 -0

b/lib/mempool/rte_mempool.h

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

index 9f530db24b..dd1a3177d6 100644

lib/mempool/rte_mempool.h +0 -0
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
Quoted 7 lines
Quoted 7 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +2 -2
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds.

*/

Quoted 11 lines
Quoted 11 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 10 lines
lib/mempool/rte_mempool.h +0 -7
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382

Quoted 1 line
lib/mempool/rte_mempool.h +0 -1
-	 *      the flush threshold, flush the cache to the

backend.

Quoted 3 lines
Quoted 3 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 3 lines
lib/mempool/rte_mempool.h +1 -2
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the

comparison

Quoted 8 lines
Quoted 8 lines

below.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +4 -1
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without

crossing

Quoted 8 lines
Quoted 8 lines

the

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -1
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In the

PMDs I

Quoted 5 lines
Quoted 3 lines

have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.

There could be applications using a mempool for something else than

mbufs.
Agree

Quoted 1 line
Quoted 5 lines
Quoted 1 line

In that case, the application should be allowed to get/put many

objects in

Quoted 1 line

one transaction.

Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which

is

Quoted 5 lines

more than 'x'. It should be possible to change the threshold configuration or reduce the request size.

If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of

using

Quoted 1 line

the per-core cache.

Correct. I suppose this is the intended behavior of this API.

The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.

[1]: http://inbox.dpdk.org/dev/20221115161822.70886-1-
[email protected]/T/#u

Quoted 2 lines

With this check, we are introducing an additional memcpy as well. I

am

Quoted 1 line

not sure if reusing the latest buffers is better than having an

memcpy.

There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.

Please note that this check is not new, it has just been moved. Before this patch, it was checked on every call (if a cache is present); with this patch, it is only checked if the entire request cannot go directly into the cache.

Quoted 39 lines
Quoted 6 lines
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the

backend, so

Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390

Quoted 1 line
lib/mempool/rte_mempool.h +1 -0
+		 * adding the objects does not cross the flush

threshold.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h +4 -0
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
Quoted 8 lines
Quoted 8 lines

lib/mempool/rte_mempool.h

Quoted 8 lines

/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

driver_enqueue:

lib/mempool/rte_mempool.h +1 -1
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1 -2
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#10 lore raw

Hi Morten,

Quoted 5 lines

PING mempool maintainers.

If you don't provide ACK or Review to patches, how should Thomas know that it is ready for merging?

-Morten

The code changes itself looks ok to me.
Though I don't think it would really bring any noticeable speedup. But yes, it looks a bit nicer this way, especially with extra comments. One question though, why do you feel this assert:
RTE_ASSERT(cache->len <= cache->flushthresh);
is necessary?
I can't see any way it could happen, unless something is totally broken within the app.

Quoted 176 lines
Quoted 176 lines

From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 18.39

Quoted 7 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Micro-optimization:
Reduced the most likely code path in the generic put function

by

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

moving an

Quoted 1 line

unlikely check out of the most likely code path and further

down.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge

and

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

len

is

Quoted 2 lines

non-zero.
* Added assertion about the invariant preventing overflow in

the

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

comparison.
* Crossing the threshold is not extremely unlikely, so removed

likely()

Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static branch

prediction

Quoted 5 lines

here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-----------

--

Quoted 64 lines

---

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
 1 file changed, 20 insertions(+), 16 deletions(-)
(quoted diff) +0 -0

b/lib/mempool/rte_mempool.h

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

index 9f530db24b..dd1a3177d6 100644

lib/mempool/rte_mempool.h +0 -0
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
Quoted 7 lines
Quoted 7 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +2 -2
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds.

*/

Quoted 11 lines
Quoted 11 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 10 lines
lib/mempool/rte_mempool.h +0 -7
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without

crossing

Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382

Quoted 1 line
lib/mempool/rte_mempool.h +0 -1
-	 *      the flush threshold, flush the cache to the

backend.

Quoted 3 lines
Quoted 3 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 3 lines
lib/mempool/rte_mempool.h +1 -2
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the

comparison

Quoted 8 lines
Quoted 8 lines

below.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 7 lines
lib/mempool/rte_mempool.h +4 -1
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without

crossing

Quoted 8 lines
Quoted 8 lines

the

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -1
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In the

PMDs I

Quoted 5 lines
Quoted 3 lines

have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.

There could be applications using a mempool for something else than

mbufs.
Agree

Quoted 1 line
Quoted 5 lines
Quoted 1 line

In that case, the application should be allowed to get/put many

objects in

Quoted 1 line

one transaction.

Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which

is

Quoted 5 lines

more than 'x'. It should be possible to change the threshold configuration or reduce the request size.

If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of

using

Quoted 1 line

the per-core cache.

Correct. I suppose this is the intended behavior of this API.

The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.

[1]: http://inbox.dpdk.org/dev/20221115161822.70886-1-
[email protected]/T/#u

Quoted 2 lines

With this check, we are introducing an additional memcpy as well. I

am

Quoted 1 line

not sure if reusing the latest buffers is better than having an

memcpy.

There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.

Please note that this check is not new, it has just been moved. Before this patch, it was checked on every call (if a cache is present); with this patch, it is only checked if the entire request cannot go directly into the cache.

Quoted 39 lines
Quoted 6 lines
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the

backend, so

Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390

Quoted 1 line
lib/mempool/rte_mempool.h +1 -0
+		 * adding the objects does not cross the flush

threshold.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h +4 -0
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache-
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
Quoted 8 lines
Quoted 8 lines

lib/mempool/rte_mempool.h

Quoted 8 lines

/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,

driver_enqueue:

lib/mempool/rte_mempool.h +1 -1
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1 -2
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#11 lore raw
Quoted 7 lines

From: Konstantin Ananyev [mailto:[email protected]]
Sent: Thursday, 22 December 2022 14.52

Hi Morten,

Quoted 2 lines

PING mempool maintainers.

Quoted 8 lines
Quoted 1 line

If you don't provide ACK or Review to patches, how should Thomas know

that it is ready for merging?

Quoted 2 lines

-Morten

The code changes itself looks ok to me.
Though I don't think it would really bring any noticeable speedup. But yes, it looks a bit nicer this way, especially with extra comments.

Agree, removing the compare and branch instructions from the likely code path provides no noticeable speedup, but makes the code more clean.

Quoted 3 lines

One question though, why do you feel this assert:
RTE_ASSERT(cache->len <= cache->flushthresh);
is necessary?

I could have written it into the comment above it, but instead chose to add the assertion as an improved comment.

Quoted 2 lines

I can't see any way it could happen, unless something is totally broken within the app.

Agree. These are the circumstances where assertions can come into action. With more assertions in the code, it is likely that less code executes before hitting an assertion, making it easier to find the root cause of such errors.

Please note that RTE_ASSERTs are omitted unless built with RTE_ENABLE_ASSERT, so this assertion is omitted and thus has no performance impact for a normal build.

Quoted 204 lines
Quoted 11 lines
Quoted 11 lines

From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 18.39

Quoted 7 lines

From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Micro-optimization:
Reduced the most likely code path in the generic put

function

Quoted 9 lines
Quoted 9 lines

by

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

moving an

Quoted 1 line

unlikely check out of the most likely code path and further

down.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Also updated the comments in the function.

v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really

huge

Quoted 5 lines
Quoted 5 lines

and

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

len

is

Quoted 2 lines

non-zero.
* Added assertion about the invariant preventing overflow

in

Quoted 3 lines
Quoted 3 lines

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

comparison.
* Crossing the threshold is not extremely unlikely, so

removed

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

likely()

Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static

branch

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

prediction

Quoted 5 lines

here anyway.

---
 lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-------

----

Quoted 13 lines
Quoted 13 lines

--

Quoted 12 lines

---

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
 1 file changed, 20 insertions(+), 16 deletions(-)
(quoted diff) +0 -0

b/lib/mempool/rte_mempool.h

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

index 9f530db24b..dd1a3177d6 100644

lib/mempool/rte_mempool.h +0 -0
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct
 rte_mempool *mp, void * const *obj_table,  {
 	void **cache_objs;
 
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1371

Quoted 6 lines
lib/mempool/rte_mempool.h +1 -2
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always

success */

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1371

Quoted 1 line
lib/mempool/rte_mempool.h +1 -0
+	/* Increment stats now, adding in mempool always

succeeds.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

*/

Quoted 10 lines
Quoted 10 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381

Quoted 10 lines
lib/mempool/rte_mempool.h +0 -7
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache

without

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

crossing

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382

Quoted 1 line
lib/mempool/rte_mempool.h +0 -1
-	 *      the flush threshold, flush the cache to the

backend.

Quoted 3 lines
Quoted 3 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1375

Quoted 3 lines
lib/mempool/rte_mempool.h +1 -2
-	 *   2. Add the objects to the cache.
-	 */
+	/* Assert the invariant preventing overflow in the

comparison

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

below.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381

Quoted 7 lines
lib/mempool/rte_mempool.h +4 -1
 */
+	RTE_ASSERT(cache->len <= cache->flushthresh);
 
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache

without

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

crossing

Quoted 8 lines
Quoted 8 lines

the

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1386

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -1
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {

IMO, this is a misconfiguration on the application part. In

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

PMDs I

Quoted 1 line
Quoted 1 line

have looked at, max value of 'n' is controlled by compile

time

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

constants. Application could do a compile time check on the

cache

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

threshold or we could have another RTE_ASSERT on this.

There could be applications using a mempool for something else

than

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

mbufs.
Agree

Quoted 2 lines

In that case, the application should be allowed to get/put many

objects in

Quoted 1 line

one transaction.

Still, this is a misconfiguration on the application. On one hand

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

threshold is configured for 'x' but they are sending a request

which

Quoted 5 lines
Quoted 5 lines

is

Quoted 4 lines

more than 'x'. It should be possible to change the threshold configuration or reduce the request size.

If the application does not fix the misconfiguration, it is

possible

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

that it will always hit this case and does not get the benefit of

using

Quoted 1 line

the per-core cache.

Correct. I suppose this is the intended behavior of this API.

Quoted 8 lines
Quoted 8 lines

The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.

[1]: http://inbox.dpdk.org/dev/20221115161822.70886-1-
[email protected]/T/#u

Quoted 2 lines

With this check, we are introducing an additional memcpy as well.

I

Quoted 8 lines
Quoted 8 lines

am

Quoted 1 line

not sure if reusing the latest buffers is better than having an

memcpy.

There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.

Please note that this check is not new, it has just been moved.

Before

Quoted 1 line
Quoted 1 line

this patch, it was checked on every call (if a cache is present);

with

Quoted 1 line
Quoted 1 line

this patch, it is only checked if the entire request cannot go

directly

Quoted 9 lines
Quoted 9 lines

into the cache.

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1389

Quoted 6 lines
lib/mempool/rte_mempool.h +3 -0
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the

backend, so

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390

Quoted 1 line
lib/mempool/rte_mempool.h +1 -0
+		 * adding the objects does not cross the flush

threshold.

Quoted 3 lines
Quoted 3 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1393

Quoted 3 lines
lib/mempool/rte_mempool.h +1 -0
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs,

cache-

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1397

Quoted 4 lines
lib/mempool/rte_mempool.h +2 -0
 >len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache.

*/

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/mempool/rte_mempool.h +1 -0
+		goto driver_enqueue_stats_incremented;
 	}

/* Add the objects to the cache. */ @@ -1399,13

+1403,13 @@

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

rte_mempool_do_generic_put(struct rte_mempool *mp, void *

const

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/mempool/rte_mempool.h

Quoted 5 lines

*obj_table,

driver_enqueue:

lib/mempool/rte_mempool.h +0 -1
-	/* increment stat now, adding in mempool always

success */

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/mempool/rte_mempool.h

Quoted 1 line
lib/mempool/rte_mempool.h +1 -0
+	/* Increment stats now, adding in mempool always

succeeds.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1 -2
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);  }
 
--

2.17.1

#12 lore raw
Quoted 24 lines
Quoted 6 lines

Hi Morten,

Quoted 2 lines

PING mempool maintainers.

Quoted 8 lines
Quoted 1 line

If you don't provide ACK or Review to patches, how should Thomas know

that it is ready for merging?

Quoted 2 lines

-Morten

The code changes itself looks ok to me.
Though I don't think it would really bring any noticeable speedup. But yes, it looks a bit nicer this way, especially with extra comments.

Agree, removing the compare and branch instructions from the likely code path provides no noticeable speedup, but makes the code more clean.

Quoted 3 lines

One question though, why do you feel this assert:
RTE_ASSERT(cache->len <= cache->flushthresh);
is necessary?

I could have written it into the comment above it, but instead chose to add the assertion as an improved comment.

Quoted 8 lines
Quoted 2 lines

I can't see any way it could happen, unless something is totally broken within the app.

Agree. These are the circumstances where assertions can come into action. With more assertions in the code, it is likely that less code executes before hitting an assertion, making it easier to find the root cause of such errors.

Please note that RTE_ASSERTs are omitted unless built with RTE_ENABLE_ASSERT, so this assertion is omitted and thus has no performance impact for a normal build.

I am aware that RTE_ASSERT is not enabled by default.
My question was more about - why do you feel assert() is necessary in that case in particular. Does it guard for some specific scenario or so.
Anyway, I am ok either way: with and without assert() here.

#13 lore raw

PATCH v3 1/1 mempool: micro-optimize put function

In reply to Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v3 (feedback from Konstantin Ananyev):
* Removed assertion and comment about the invariant preventing overflow in the comparison. They were more confusing than enlightening.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed likely() from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.

---
 lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +18 -17
index 9f530db24b..61ca0c6b65 100644
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 {
 	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */
 	if (unlikely(cache == NULL))
 		goto driver_enqueue;
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
 	RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
 
-	/* The request itself is too big for the cache */
-	if (unlikely(n > cache->flushthresh))
-		goto driver_enqueue_stats_incremented;
-
-	/*
-	 * The cache follows the following algorithm:
-	 *   1. If the objects cannot be added to the cache without crossing
-	 *      the flush threshold, flush the cache to the backend.
-	 *   2. Add the objects to the cache.
-	 */
-
-	if (cache->len + n <= cache->flushthresh) {
+	if (n <= cache->flushthresh - cache->len) {
+		/*
+		 * The objects can be added to the cache without crossing the
+		 * flush threshold.
+		 */
 		cache_objs = &cache->objs[cache->len];
 		cache->len += n;
-	} else {
+	} else if (likely(n <= cache->flushthresh)) {
+		/*
+		 * The request itself fits into the cache.
+		 * But first, the cache must be flushed to the backend, so
+		 * adding the objects does not cross the flush threshold.
+		 */
 		cache_objs = &cache->objs[0];
 		rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
 		cache->len = n;
+	} else {
+		/* The request itself is too big for the cache. */
+		goto driver_enqueue_stats_incremented;
 	}
 
 	/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
 
 driver_enqueue:
 
-	/* increment stat now, adding in mempool always success */
+	/* Increment stats now, adding in mempool always succeeds. */
 	RTE_MEMPOOL_STAT_ADD(mp, put_bulk, 1);
 	RTE_MEMPOOL_STAT_ADD(mp, put_objs, n);
 
 driver_enqueue_stats_incremented:
 
-	/* push objects to the backend */
+	/* Push the objects to the backend. */
 	rte_mempool_ops_enqueue_bulk(mp, obj_table, n);
 }
 
Signature · 1 line
2.17.1
Reviewed-by on v3 1/1
#14 lore raw
Quoted 19 lines · Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v3 (feedback from Konstantin Ananyev):
* Removed assertion and comment about the invariant preventing overflow in the comparison. They were more confusing than enlightening.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed likely() from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.

Quoted 2 lines

Thanks for optimizing it further.

Quoted 9 lines
---
  lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
  1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..61ca0c6b65 100644

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1367

Quoted 5 lines
lib/mempool/rte_mempool.h +1 -1
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
  {
  	void **cache_objs;
 
-	/* No cache provided */
+	/* No cache provided? */

IMHO such changes do not add value and just add noise. There are few similar cases below.
No strong opinion in any case.

#15 lore raw
Quoted 48 lines

From: Andrew Rybchenko [mailto:[email protected]]
Sent: Tuesday, 27 December 2022 09.54

Quoted 2 lines · Morten Brørup

Micro-optimization:
Reduced the most likely code path in the generic put function by

moving an

Quoted 6 lines

unlikely check out of the most likely code path and further down.

Also updated the comments in the function.

v3 (feedback from Konstantin Ananyev):
* Removed assertion and comment about the invariant preventing

overflow

Quoted 3 lines

in the comparison. They were more confusing than enlightening.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len

is

Quoted 4 lines

non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed

likely()

Quoted 2 lines

from that comparison.
The compiler will generate code with optimal static branch

prediction

Quoted 4 lines

here anyway.

Thanks for optimizing it further.

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool

Quoted 9 lines
---
  lib/mempool/rte_mempool.h | 35 ++++++++++++++++++-----------------
  1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +0 -0
index 9f530db24b..61ca0c6b65 100644
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool

*mp, void * const *obj_table,

Quoted 3 lines
lib/mempool/rte_mempool.h +0 -0
  {
  	void **cache_objs;
 
Quoted 7 lines

lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1367

Quoted 2 lines
lib/mempool/rte_mempool.h +1 -1
-	/* No cache provided */
+	/* No cache provided? */

IMHO such changes do not add value and just add noise. There are few similar cases below.
No strong opinion in any case.

This patch is obsolete, because the zero-copy patch v5 [1] uses the zero-copy put function, which is optimized similarly, in rte_mempool_do_generic_put().

[1]: https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/