mempool: micro-optimize put function
PATCH 1/1 mempool: micro-optimize put function
Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +18 -17
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, {void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing -* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ - -if (cache->len + n <= cache->flushthresh) { +if (likely (cache->len + n <= cache->flushthresh)) { +/* +* The objects can be added to the cache without crossing the +* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) { +/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache->len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); }
Signature · 1 line
2.17.1
Quoted 22 lines · Morten Brørup
Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.Also updated the comments in the function.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)lib/mempool/rte_mempool.h +1
-1@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, {void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL ))
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1377
Quoted 10 lines
lib/mempool/rte_mempool.h +1
-4goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented;
I've kept the check here since it protects against overflow in len plus n below if n is really huge.
Quoted 48 lines
lib/mempool/rte_mempool.h +16
-12- -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing -* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ - -if (cache->len + n <= cache->flushthresh) { +if (likely (cache->len + n <= cache->flushthresh)) { +/* +* The objects can be added to the cache without crossing the +* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) { +/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache->len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */ @@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); }
Quoted 35 lines
From: Andrew Rybchenko [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 12.05Quoted 2 lines · Morten Brørup
Micro-optimization:
Reduced the most likely code path in the generic put function bymoving an
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool
Quoted 14 lines
unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool*mp, void * const
*obj_table,Quoted 13 lines
lib/mempool/rte_mempool.h +2
-2{void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n);
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1377
Quoted 3 lines
lib/mempool/rte_mempool.h +0
-3-/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented;I've kept the check here since it protects against overflow in len plus n below if n is really huge.
We can fix that, see below.
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 5 lines
lib/mempool/rte_mempool.h +0
-4- -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1375
Quoted 6 lines
lib/mempool/rte_mempool.h +1
-5-* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ - -if (cache->len + n <= cache->flushthresh) { +if (likely (cache->len + n <= cache->flushthresh)) {
It is an invariant that cache->len <= cache->flushthresh, so the above comparison can be rewritten to protect against overflow:
if (likely(n <= cache->flushthresh - cache->len)) {
Quoted 41 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool · line 1377
Quoted 2 lines
lib/mempool/rte_mempool.h +2
-0+/* +* The objects can be added to the cache without crossing the
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool
Quoted 21 lines
lib/mempool/rte_mempool.h +11
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) { +/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache->len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */ @@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool*mp, void * const
*obj_table,Quoted 15 lines
lib/mempool/rte_mempool.h +2
-2driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); }
Quoted 59 lines · Morten Brørup
Quoted 35 lines
From: Andrew Rybchenko [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 12.05Quoted 2 lines · Morten Brørup
Micro-optimization:
Reduced the most likely code path in the generic put function bymoving an
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool
Quoted 14 lines
unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool*mp, void * const
*obj_table,Quoted 13 lines
lib/mempool/rte_mempool.h +2
-2{void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n);Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1377
Quoted 3 lines
lib/mempool/rte_mempool.h +0
-3-/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented;I've kept the check here since it protects against overflow in len plus n below if n is really huge.
We can fix that, see below.
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 5 lines
lib/mempool/rte_mempool.h +0
-4- -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1375
Quoted 6 lines
lib/mempool/rte_mempool.h +1
-5-* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ - -if (cache->len + n <= cache->flushthresh) { +if (likely (cache->len + n <= cache->flushthresh)) {It is an invariant that
cache->len<=cache->flushthresh, so the above comparison can be rewritten to protect against overflow:
Quoted 2 lines
if (
likely(n <= cache->flushthresh - cache->len)) {
True, but it would be useful to highlight the usage of the invariant here using either a comment or an assert.
IMHO it is wrong to use likely here since, as far as I know, it makes else branch very expensive, but crossing the flush threshold is an expected branch and it must not be that
expensive.
Quoted 42 lines
Quoted 41 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool · line 1377
Quoted 2 lines
lib/mempool/rte_mempool.h +2
-0+/* +* The objects can be added to the cache without crossing the
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool
Quoted 21 lines
lib/mempool/rte_mempool.h +11
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) { +/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache->len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */ @@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool*mp, void * const
*obj_table,Quoted 15 lines
lib/mempool/rte_mempool.h +2
-2driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); }
PATCH v2 1/1 mempool: micro-optimize put function
In reply to Morten BrørupMicro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed likely() from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.
--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)
lib/mempool/rte_mempool.h +20 -16
@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, {void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing -* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison below. */ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing the +* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) { +/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache->len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */
@@ -1399,13 +1403,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); }
Signature · 1 line
2.17.1
<snip>
Quoted 54 lines
Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removedlikely()from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)lib/mempool/rte_mempool.h +4
-12@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing -* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison below. */ +RTE_ASSERT (cache->len <= cache->flushthresh);
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1386
Quoted 10 lines
lib/mempool/rte_mempool.h +6
-2-if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing the +* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {
IMO, this is a misconfiguration on the application part. In the PMDs I have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another RTE_ASSERT on this.
Quoted 33 lines
lib/mempool/rte_mempool.h +8
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache- >len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }lib/mempool/rte_mempool.h +2
-3/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
Quoted 69 lines
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 16.51<snip>
Quoted 3 lines
Micro-optimization:
Reduced the most likely code path in the generic put function bymoving an
Quoted 6 lines
unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and lenis
Quoted 4 lines
non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed
likely()Quoted 2 lines
from that comparison.
The compiler will generate code with optimal static branchprediction
Quoted 22 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)lib/mempool/rte_mempool.h +2
-2@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381
Quoted 10 lines
lib/mempool/rte_mempool.h +0
-7RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 4 lines
lib/mempool/rte_mempool.h +1
-3-* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison below.
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +4
-1*/ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing the
Quoted 10 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {IMO, this is a misconfiguration on the application part. In the PMDs I have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another
RTE_ASSERTon this.
There could be applications using a mempool for something else than mbufs.
In that case, the application should be allowed to get/put many objects in one transaction.
Quoted 34 lines
Quoted 34 lines
lib/mempool/rte_mempool.h +8
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache- >len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }lib/mempool/rte_mempool.h +2
-3/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
<snip>
Quoted 77 lines
Quoted 66 lines
Quoted 3 lines
Micro-optimization:
Reduced the most likely code path in the generic put function bymoving an
Quoted 7 lines
unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and lenis
Quoted 4 lines
non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed
likely()Quoted 2 lines
from that comparison.
The compiler will generate code with optimal static branchprediction
Quoted 22 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-)lib/mempool/rte_mempool.h +2
-2@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381
Quoted 10 lines
lib/mempool/rte_mempool.h +0
-7RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 4 lines
lib/mempool/rte_mempool.h +1
-3-* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison below.
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +4
-1*/ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing the
Quoted 10 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {IMO, this is a misconfiguration on the application part. In the PMDs I have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another
RTE_ASSERTon this.
Quoted 1 line
There could be applications using a mempool for something else than mbufs.
Agree
Quoted 3 lines
In that case, the application should be allowed to get/put many objects in one transaction.
Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which is more than 'x'. It should be possible to change the threshold configuration or reduce the request size.
If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of using the per-core cache.
With this check, we are introducing an additional memcpy as well. I am not sure if reusing the latest buffers is better than having an memcpy.
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/mempool/rte_mempool.h +8
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache- >len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(structrte_mempool*mp, void * const*obj_table,
driver_enqueue:lib/mempool/rte_mempool.h +2
-3-/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
Quoted 97 lines
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27<snip>
Quoted 17 lines
Quoted 17 lines
Quoted 3 lines
Micro-optimization:
Reduced the most likely code path in the generic put function bymoving an
Quoted 7 lines
unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and lenis
Quoted 4 lines
non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed
likely()Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
from that comparison.
The compiler will generate code with optimal static branchprediction
Quoted 5 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++----------------
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
1 file changed, 20 insertions(+), 16 deletions(-)(quoted diff) +0
-0
b/lib/mempool/rte_mempool.hQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
index
9f530db24b..dd1a3177d6100644lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs;Quoted 7 lines
Quoted 7 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +2
-2-/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 10 lines
lib/mempool/rte_mempool.h +0
-7RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382
Quoted 1 line
lib/mempool/rte_mempool.h +0
-1-* the flush threshold, flush the cache to the backend.
Quoted 3 lines
Quoted 3 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 3 lines
lib/mempool/rte_mempool.h +1
-2-* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison
Quoted 8 lines
Quoted 8 lines
below.
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +4
-1*/ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing
Quoted 8 lines
Quoted 8 lines
the
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {IMO, this is a misconfiguration on the application part. In the
PMDs I
Quoted 5 lines
Quoted 3 lines
have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another
RTE_ASSERTon this.There could be applications using a mempool for something else than
mbufs.
AgreeQuoted 2 lines
In that case, the application should be allowed to get/put many
objects in
Quoted 1 line
one transaction.
Quoted 8 lines
Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which is more than 'x'. It should be possible to change the threshold configuration or reduce the request size.
If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of using the per-core cache.
Correct. I suppose this is the intended behavior of this API.
The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.
[1]: http://inbox.dpdk.org/dev/[email protected]/T/#u
Quoted 3 lines
With this check, we are introducing an additional memcpy as well. I am not sure if reusing the latest buffers is better than having an memcpy.
There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.
Please note that this check is not new, it has just been moved. Before this patch, it was checked on every call (if a cache is present); with this patch, it is only checked if the entire request cannot go directly into the cache.
Quoted 39 lines
Quoted 6 lines
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390
Quoted 1 line
lib/mempool/rte_mempool.h +1
-0+* adding the objects does not cross the flush threshold.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h +4
-0+*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache- >len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }Quoted 8 lines
Quoted 8 lines
lib/mempool/rte_mempool.h
Quoted 8 lines
/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(structrte_mempool*mp, void * const*obj_table,
driver_enqueue:lib/mempool/rte_mempool.h +1
-1-/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1
-2RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
PING mempool maintainers.
If you don't provide ACK or Review to patches, how should Thomas know that it is ready for merging?
-Morten
Quoted 177 lines
From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 18.39Quoted 8 lines
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27<snip>
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Micro-optimization:
Reduced the most likely code path in the generic put functionby
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
moving an
Quoted 1 line
unlikely check out of the most likely code path and further
down.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really hugeand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
len
is
Quoted 2 lines
non-zero.
* Added assertion about the invariant preventing overflow inthe
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
comparison.
* Crossing the threshold is not extremely unlikely, so removed
likely()Quoted 2 lines
from that comparison.
The compiler will generate code with optimal static branchprediction
Quoted 5 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-------------
Quoted 64 lines
---
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
1 file changed, 20 insertions(+), 16 deletions(-)(quoted diff) +0
-0
b/lib/mempool/rte_mempool.hQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
index
9f530db24b..dd1a3177d6100644lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs;Quoted 7 lines
Quoted 7 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +2
-2-/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 10 lines
lib/mempool/rte_mempool.h +0
-7RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382
Quoted 1 line
lib/mempool/rte_mempool.h +0
-1-* the flush threshold, flush the cache to the backend.
Quoted 3 lines
Quoted 3 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 3 lines
lib/mempool/rte_mempool.h +1
-2-* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison
Quoted 8 lines
Quoted 8 lines
below.
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +4
-1*/ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing
Quoted 8 lines
Quoted 8 lines
the
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {IMO, this is a misconfiguration on the application part. In the
PMDs I
Quoted 5 lines
Quoted 3 lines
have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another
RTE_ASSERTon this.There could be applications using a mempool for something else than
mbufs.
AgreeQuoted 1 line
Quoted 5 lines
Quoted 1 line
In that case, the application should be allowed to get/put many
objects in
Quoted 1 line
one transaction.
Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which
is
Quoted 5 lines
more than 'x'. It should be possible to change the threshold configuration or reduce the request size.
If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of
using
Quoted 1 line
the per-core cache.
Correct. I suppose this is the intended behavior of this API.
The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.
[1]: http://inbox.dpdk.org/dev/20221115161822.70886-1-
[email protected]/T/#uQuoted 2 lines
With this check, we are introducing an additional memcpy as well. I
am
Quoted 1 line
not sure if reusing the latest buffers is better than having an
memcpy.
There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.
Please note that this check is not new, it has just been moved. Before this patch, it was checked on every call (if a cache is present); with this patch, it is only checked if the entire request cannot go directly into the cache.
Quoted 39 lines
Quoted 6 lines
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390
Quoted 1 line
lib/mempool/rte_mempool.h +1
-0+* adding the objects does not cross the flush threshold.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h +4
-0+*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache- >len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }Quoted 8 lines
Quoted 8 lines
lib/mempool/rte_mempool.h
Quoted 8 lines
/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(structrte_mempool*mp, void * const*obj_table,
driver_enqueue:lib/mempool/rte_mempool.h +1
-1-/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1
-2RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
Hi Morten,
Quoted 5 lines
PING mempool maintainers.
If you don't provide ACK or Review to patches, how should Thomas know that it is ready for merging?
-Morten
The code changes itself looks ok to me.
Though I don't think it would really bring any noticeable speedup. But yes, it looks a bit nicer this way, especially with extra comments. One question though, why do you feel this assert:RTE_ASSERT(cache->len <= cache->flushthresh);
is necessary?
I can't see any way it could happen, unless something is totally broken within the app.
Quoted 176 lines
Quoted 176 lines
From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 18.39Quoted 7 lines
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Micro-optimization:
Reduced the most likely code path in the generic put functionby
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
moving an
Quoted 1 line
unlikely check out of the most likely code path and further
down.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really hugeand
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
len
is
Quoted 2 lines
non-zero.
* Added assertion about the invariant preventing overflow inthe
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
comparison.
* Crossing the threshold is not extremely unlikely, so removed
likely()Quoted 2 lines
from that comparison.
The compiler will generate code with optimal static branchprediction
Quoted 5 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-------------
Quoted 64 lines
---
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
1 file changed, 20 insertions(+), 16 deletions(-)(quoted diff) +0
-0
b/lib/mempool/rte_mempool.hQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
index
9f530db24b..dd1a3177d6100644lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs;Quoted 7 lines
Quoted 7 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +2
-2-/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 10 lines
lib/mempool/rte_mempool.h +0
-7RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382
Quoted 1 line
lib/mempool/rte_mempool.h +0
-1-* the flush threshold, flush the cache to the backend.
Quoted 3 lines
Quoted 3 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 3 lines
lib/mempool/rte_mempool.h +1
-2-* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison
Quoted 8 lines
Quoted 8 lines
below.
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 7 lines
lib/mempool/rte_mempool.h +4
-1*/ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing
Quoted 8 lines
Quoted 8 lines
the
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1385
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {IMO, this is a misconfiguration on the application part. In the
PMDs I
Quoted 5 lines
Quoted 3 lines
have looked at, max value of 'n' is controlled by compile time constants. Application could do a compile time check on the cache threshold or we could have another
RTE_ASSERTon this.There could be applications using a mempool for something else than
mbufs.
AgreeQuoted 1 line
Quoted 5 lines
Quoted 1 line
In that case, the application should be allowed to get/put many
objects in
Quoted 1 line
one transaction.
Still, this is a misconfiguration on the application. On one hand the threshold is configured for 'x' but they are sending a request which
is
Quoted 5 lines
more than 'x'. It should be possible to change the threshold configuration or reduce the request size.
If the application does not fix the misconfiguration, it is possible that it will always hit this case and does not get the benefit of
using
Quoted 1 line
the per-core cache.
Correct. I suppose this is the intended behavior of this API.
The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.
[1]: http://inbox.dpdk.org/dev/20221115161822.70886-1-
[email protected]/T/#uQuoted 2 lines
With this check, we are introducing an additional memcpy as well. I
am
Quoted 1 line
not sure if reusing the latest buffers is better than having an
memcpy.
There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.
Please note that this check is not new, it has just been moved. Before this patch, it was checked on every call (if a cache is present); with this patch, it is only checked if the entire request cannot go directly into the cache.
Quoted 39 lines
Quoted 6 lines
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390
Quoted 1 line
lib/mempool/rte_mempool.h +1
-0+* adding the objects does not cross the flush threshold.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h +4
-0+*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache- >len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }Quoted 8 lines
Quoted 8 lines
lib/mempool/rte_mempool.h
Quoted 8 lines
/* Add the objects to the cache. */ @@ -1399,13 +1403,13 @@
rte_mempool_do_generic_put(structrte_mempool*mp, void * const*obj_table,
driver_enqueue:lib/mempool/rte_mempool.h +1
-1-/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1
-2RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
Quoted 7 lines
From: Konstantin Ananyev [mailto:[email protected]]
Sent: Thursday, 22 December 2022 14.52Hi Morten,
Quoted 2 lines
PING mempool maintainers.
Quoted 8 lines
Quoted 1 line
If you don't provide ACK or Review to patches, how should Thomas know
that it is ready for merging?
Quoted 2 lines
-Morten
The code changes itself looks ok to me.
Though I don't think it would really bring any noticeable speedup. But yes, it looks a bit nicer this way, especially with extra comments.
Agree, removing the compare and branch instructions from the likely code path provides no noticeable speedup, but makes the code more clean.
Quoted 3 lines
One question though, why do you feel this assert:
RTE_ASSERT(cache->len <= cache->flushthresh);
is necessary?
I could have written it into the comment above it, but instead chose to add the assertion as an improved comment.
Quoted 2 lines
I can't see any way it could happen, unless something is totally broken within the app.
Agree. These are the circumstances where assertions can come into action. With more assertions in the code, it is likely that less code executes before hitting an assertion, making it easier to find the root cause of such errors.
Please note that RTE_ASSERTs are omitted unless built with RTE_ENABLE_ASSERT, so this assertion is omitted and thus has no performance impact for a normal build.
Quoted 204 lines
Quoted 11 lines
Quoted 11 lines
From: Morten Brørup [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 18.39Quoted 7 lines
From: Honnappa Nagarahalli [mailto:[email protected]]
Sent: Wednesday, 16 November 2022 17.27Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Micro-optimization:
Reduced the most likely code path in the generic putfunction
Quoted 9 lines
Quoted 9 lines
by
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
moving an
Quoted 1 line
unlikely check out of the most likely code path and further
down.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Also updated the comments in the function.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is reallyhuge
Quoted 5 lines
Quoted 5 lines
and
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
len
is
Quoted 2 lines
non-zero.
* Added assertion about the invariant preventing overflowin
Quoted 3 lines
Quoted 3 lines
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
comparison.
* Crossing the threshold is not extremely unlikely, soremoved
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
likely()Quoted 2 lines
from that comparison.
The compiler will generate code with optimal staticbranch
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
prediction
Quoted 5 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>--- lib/mempool/rte_mempool.h | 36 ++++++++++++++++++++-----------
Quoted 13 lines
Quoted 13 lines
--
Quoted 12 lines
---
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
1 file changed, 20 insertions(+), 16 deletions(-)(quoted diff) +0
-0
b/lib/mempool/rte_mempool.hQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
index
9f530db24b..dd1a3177d6100644lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,36 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, { void **cache_objs;Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1371
Quoted 6 lines
lib/mempool/rte_mempool.h +1
-2-/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1371
Quoted 1 line
lib/mempool/rte_mempool.h +1
-0+/* Increment stats now, adding in mempool always succeeds.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
*/
Quoted 10 lines
Quoted 10 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381
Quoted 10 lines
lib/mempool/rte_mempool.h +0
-7RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
crossing
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1382
Quoted 1 line
lib/mempool/rte_mempool.h +0
-1-* the flush threshold, flush the cache to the backend.
Quoted 3 lines
Quoted 3 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1375
Quoted 3 lines
lib/mempool/rte_mempool.h +1
-2-* 2. Add the objects to the cache. -*/ +/* Assert the invariant preventing overflow in the comparison
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
below.
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1381
Quoted 7 lines
lib/mempool/rte_mempool.h +4
-1*/ +RTE_ASSERT (cache->len <= cache->flushthresh); -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
crossing
Quoted 8 lines
Quoted 8 lines
the
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1386
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-1+* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) {IMO, this is a misconfiguration on the application part. In
the
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
PMDs I
Quoted 1 line
Quoted 1 line
have looked at, max value of 'n' is controlled by compile
time
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
constants. Application could do a compile time check on the
cache
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
threshold or we could have another
RTE_ASSERTon this.There could be applications using a mempool for something else
than
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mbufs.
AgreeQuoted 2 lines
In that case, the application should be allowed to get/put many
objects in
Quoted 1 line
one transaction.
Still, this is a misconfiguration on the application. On one hand
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
threshold is configured for 'x' but they are sending a request
which
Quoted 5 lines
Quoted 5 lines
is
Quoted 4 lines
more than 'x'. It should be possible to change the threshold configuration or reduce the request size.
If the application does not fix the misconfiguration, it is
possible
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
that it will always hit this case and does not get the benefit of
using
Quoted 1 line
the per-core cache.
Correct. I suppose this is the intended behavior of this API.
Quoted 8 lines
Quoted 8 lines
The zero-copy API proposed in another patch [1] has stricter requirements to the bulk size.
[1]: http://inbox.dpdk.org/dev/20221115161822.70886-1-
[email protected]/T/#uQuoted 2 lines
With this check, we are introducing an additional memcpy as well.
I
Quoted 8 lines
Quoted 8 lines
am
Quoted 1 line
not sure if reusing the latest buffers is better than having an
memcpy.
There is no additional memcpy. The large bulk transfer is stored directly in the backend pool, bypassing the mempool cache.
Please note that this check is not new, it has just been moved.
Before
Quoted 1 line
Quoted 1 line
this patch, it was checked on every call (if a cache is present);
with
Quoted 1 line
Quoted 1 line
this patch, it is only checked if the entire request cannot go
directly
Quoted 9 lines
Quoted 9 lines
into the cache.
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1389
Quoted 6 lines
lib/mempool/rte_mempool.h +3
-0+/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1390
Quoted 1 line
lib/mempool/rte_mempool.h +1
-0+* adding the objects does not cross the flush threshold.
Quoted 3 lines
Quoted 3 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1393
Quoted 3 lines
lib/mempool/rte_mempool.h +1
-0+*/ cache_objs = &cache->objs[0 ]; rte_mempool_ops_enqueue_bulk(mp, cache_objs,cache-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct · line 1397
Quoted 4 lines
lib/mempool/rte_mempool.h +2
-0>len); cache->len = n; + }else { +/* The request itself is too big for the cache. */
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/mempool/rte_mempool.h +1
-0+ goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */ @@ -1399,13
+1403,13 @@
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
rte_mempool_do_generic_put(struct
rte_mempool*mp, void *const
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
lib/mempool/rte_mempool.h
Quoted 5 lines
*obj_table,
driver_enqueue:lib/mempool/rte_mempool.h +0
-1-/* increment stat now, adding in mempool always success */
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
lib/mempool/rte_mempool.h
Quoted 1 line
lib/mempool/rte_mempool.h +1
-0+/* Increment stats now, adding in mempool always succeeds.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
*/
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
lib/mempool/rte_mempool.h +1
-2RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); } --2.17.1
Konstantin Ananyev
Acked-byQuoted 24 lines
Quoted 6 lines
Hi Morten,
Quoted 2 lines
PING mempool maintainers.
Quoted 8 lines
Quoted 1 line
If you don't provide ACK or Review to patches, how should Thomas know
that it is ready for merging?
Quoted 2 lines
-Morten
The code changes itself looks ok to me.
Though I don't think it would really bring any noticeable speedup. But yes, it looks a bit nicer this way, especially with extra comments.Agree, removing the compare and branch instructions from the likely code path provides no noticeable speedup, but makes the code more clean.
Quoted 3 lines
One question though, why do you feel this assert:
RTE_ASSERT(cache->len <= cache->flushthresh);
is necessary?I could have written it into the comment above it, but instead chose to add the assertion as an improved comment.
Quoted 8 lines
Quoted 2 lines
I can't see any way it could happen, unless something is totally broken within the app.
Agree. These are the circumstances where assertions can come into action. With more assertions in the code, it is likely that less code executes before hitting an assertion, making it easier to find the root cause of such errors.
Please note that
RTE_ASSERTsare omitted unless built withRTE_ENABLE_ASSERT, so this assertion is omitted and thus has no performance impact for a normal build.
I am aware that RTE_ASSERT is not enabled by default.
My question was more about - why do you feel assert() is necessary in that case in particular. Does it guard for some specific scenario or so.
Anyway, I am ok either way: with and without assert() here.
PATCH v3 1/1 mempool: micro-optimize put function
In reply to Morten BrørupMicro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
v3 (feedback from Konstantin Ananyev):
* Removed assertion and comment about the invariant preventing overflow in the comparison. They were more confusing than enlightening.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed likely() from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.
--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)
lib/mempool/rte_mempool.h +18 -17
@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, {void **cache_objs; -/* No cache provided */ +/* No cache provided? */ if (unlikely (cache ==NULL )) goto driver_enqueue; -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_bulk,1 );RTE_MEMPOOL_CACHE_STAT_ADD (cache, put_objs, n); -/* The request itself is too big for the cache */ -if (unlikely (n > cache->flushthresh)) - goto driver_enqueue_stats_incremented; - -/* -* The cache follows the following algorithm: -* 1. If the objects cannot be added to the cache without crossing -* the flush threshold, flush the cache to the backend. -* 2. Add the objects to the cache. -*/ - -if (cache->len + n <= cache->flushthresh) { +if (n <= cache->flushthresh - cache->len) { +/* +* The objects can be added to the cache without crossing the +* flush threshold. +*/ cache_objs = &cache->objs[cache->len]; cache->len += n; - }else { + }else if (likely (n <= cache->flushthresh)) { +/* +* The request itself fits into the cache. +* But first, the cache must be flushed to the backend, so +* adding the objects does not cross the flush threshold. +*/ cache_objs = &cache->objs[0 ];rte_mempool_ops_enqueue_bulk (mp, cache_objs, cache->len); cache->len = n; + }else { +/* The request itself is too big for the cache. */ + goto driver_enqueue_stats_incremented; }/* Add the objects to the cache. */
@@ -1399,13 +1400,13 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, driver_enqueue: -/* increment stat now, adding in mempool always success */ +/* Increment stats now, adding in mempool always succeeds. */ RTE_MEMPOOL_STAT_ADD (mp, put_bulk,1 );RTE_MEMPOOL_STAT_ADD (mp, put_objs, n); driver_enqueue_stats_incremented: -/* push objects to the backend */ +/* Push the objects to the backend. */ rte_mempool_ops_enqueue_bulk (mp, obj_table, n); }
Signature · 1 line
2.17.1
Quoted 19 lines · Morten Brørup
Micro-optimization:
Reduced the most likely code path in the generic put function by moving an unlikely check out of the most likely code path and further down.Also updated the comments in the function.
v3 (feedback from Konstantin Ananyev):
* Removed assertion and comment about the invariant preventing overflow in the comparison. They were more confusing than enlightening.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and len is non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removedlikely()from that comparison.
The compiler will generate code with optimal static branch prediction here anyway.
Quoted 2 lines
Signed-off-by Morten Brørup <[email protected]>
Thanks for optimizing it further.
Quoted 9 lines
--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)lib/mempool/rte_mempool.h +0
-0
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1367
Quoted 5 lines
lib/mempool/rte_mempool.h +1
-1@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, {void **cache_objs; -/* No cache provided */ +/* No cache provided? */
IMHO such changes do not add value and just add noise. There are few similar cases below.
No strong opinion in any case.
Quoted 48 lines
From: Andrew Rybchenko [mailto:[email protected]]
Sent: Tuesday, 27 December 2022 09.54Quoted 2 lines · Morten Brørup
Micro-optimization:
Reduced the most likely code path in the generic put function bymoving an
Quoted 6 lines
unlikely check out of the most likely code path and further down.
Also updated the comments in the function.
v3 (feedback from Konstantin Ananyev):
* Removed assertion and comment about the invariant preventingoverflow
Quoted 3 lines
in the comparison. They were more confusing than enlightening.
v2 (feedback from Andrew Rybchenko):
* Modified comparison to prevent overflow if n is really huge and lenis
Quoted 4 lines
non-zero.
* Added assertion about the invariant preventing overflow in the comparison.
* Crossing the threshold is not extremely unlikely, so removed
likely()Quoted 2 lines
from that comparison.
The compiler will generate code with optimal static branchprediction
Quoted 4 lines
here anyway.
Signed-off-by Morten Brørup <[email protected]>Thanks for optimizing it further.
Reviewed-by Andrew Rybchenko <[email protected]>lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool
Quoted 9 lines
--- lib/mempool/rte_mempool.h | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)lib/mempool/rte_mempool.h +0
-0@@ -1364,32 +1364,33 @@ rte_mempool_do_generic_put(struct rte_mempool*mp, void * const
*obj_table,Quoted 3 lines
lib/mempool/rte_mempool.h +0
-0{void **cache_objs;
Quoted 7 lines
lib/mempool/rte_mempool.h rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table, · line 1367
Quoted 2 lines
lib/mempool/rte_mempool.h +1
-1-/* No cache provided */ +/* No cache provided? */ IMHO such changes do not add value and just add noise. There are few similar cases below.
No strong opinion in any case.
This patch is obsolete, because the zero-copy patch v5 [1] uses the zero-copy put function, which is optimized similarly, in rte_mempool_do_generic_put().
[1]: https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/