Skip to contentSkip to conversation

ring: make ring implementation non-inlined

RFC In review Konstantin Ananyev 7 participants 12 messages Mar 20, 2020 – Jul 1, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 ring: make ring implementation non-inlined

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 8.75/43.23
legacy APIs: MP/MC: single: 56.18/80.44

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 37.36/53.37 legacy APIs: SP/SC: burst (size: 32): 93.97/117.30 legacy APIs: MP/MC: burst (size: 8): 78.23/91.45 legacy APIs: MP/MC: burst (size: 32): 131.59/152.49

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 37.29/54.48 legacy APIs: SP/SC: bulk (size: 32): 92.68/113.01 legacy APIs: MP/MC: bulk (size: 8): 78.40/93.50 legacy APIs: MP/MC: bulk (size: 32): 131.49/154.25

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 4.00/16.86
legacy APIs: MP/MC: bulk (size: 8): 7.01/15.55

### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 10.64/17.56
legacy APIs: MP/MC: bulk (size: 8): 15.30/16.69
legacy APIs: SP/SC: bulk (size: 32): 5.84/7.09
legacy APIs: MP/MC: bulk (size: 32): 6.34/7.54

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 24.34/42.40
legacy APIs: MP/MC: bulk (size: 8): 70.34/71.82
legacy APIs: SP/SC: bulk (size: 32): 12.67/14.68
legacy APIs: MP/MC: bulk (size: 32): 22.41/17.93

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 10.65/41.96 elem APIs: element size 16B: MP/MC: single: 44.33/81.36

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 39.20/58.52 elem APIs: element size 16B: SP/SC: burst (size: 32): 123.19/142.79 elem APIs: element size 16B: MP/MC: burst (size: 8): 80.72/101.36 elem APIs: element size 16B: MP/MC: burst (size: 32): 169.21/185.38

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 41.64/58.46 elem APIs: element size 16B: SP/SC: bulk (size: 32): 122.74/142.52 elem APIs: element size 16B: MP/MC: bulk (size: 8): 80.60/103.14 elem APIs: element size 16B: MP/MC: bulk (size: 32): 169.39/186.67

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 5.01/17.17 elem APIs: element size 16B: MP/MC: bulk (size: 8): 6.01/14.80

### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.02/17.18 elem APIs: element size 16B: MP/MC: bulk (size: 8): 16.81/21.14 elem APIs: element size 16B: SP/SC: bulk (size: 32): 7.87/9.01 elem APIs: element size 16B: MP/MC: bulk (size: 32): 8.22/10.57

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 27.00/51.94 elem APIs: element size 16B: MP/MC: bulk (size: 8): 78.24/74.48 elem APIs: element size 16B: SP/SC: bulk (size: 32): 15.41/16.14 elem APIs: element size 16B: MP/MC: bulk (size: 32): 18.72/21.64

---
 app/proc-info/main.c                 |  20 +-
 app/test-pmd/cmdline.c               |   2 +-
 app/test/commands.c                  |   2 +-
 app/test/test_pdump.c                |   6 +-
 app/test/test_ring_stress.h          |   2 +-
 config/common_base                   |   2 +-
 drivers/crypto/ccp/ccp_pmd_ops.c     |   2 +-
 drivers/crypto/meson.build           |   1 -
 drivers/net/ring/rte_eth_ring.c      |  17 +-
 lib/librte_eventdev/rte_event_ring.c |   6 +-
 lib/librte_eventdev/rte_event_ring.h |  17 +-
 lib/librte_pdump/rte_pdump.c         |   3 +-
 lib/librte_port/rte_port_ring.c      |  29 +-
 lib/librte_ring/Makefile             |   6 +-
 lib/librte_ring/meson.build          |   6 +-
 lib/librte_ring/ring_elem.c          | 919 +++++++++++++++++++++++++++
 lib/librte_ring/ring_impl.c          | 750 ++++++++++++++++++++++
 lib/librte_ring/ring_impl.h          |  64 ++
 lib/librte_ring/rte_ring.c           |   3 +-
 lib/librte_ring/rte_ring.h           | 412 ++----------
 lib/librte_ring/rte_ring_elem.h      | 519 ++-------------
 lib/librte_ring/rte_ring_version.map |  46 ++
 22 files changed, 1938 insertions(+), 896 deletions(-)
 create mode 100644 lib/librte_ring/ring_elem.c
 create mode 100644 lib/librte_ring/ring_impl.c
 create mode 100644 lib/librte_ring/ring_impl.h
app/proc-info/main.c +1 -19
index abeca4aab..aa8f113c1 100644
@@ -1137,25 +1137,7 @@ show_ring(char *name)
 	if (name != NULL) {
 		struct rte_ring *ptr = rte_ring_lookup(name);
 		if (ptr != NULL) {
-			printf("  - Name (%s) on socket (%d)\n"
-				"  - flags:\n"
-				"\t  -- Single Producer Enqueue (%u)\n"
-				"\t  -- Single Consmer Dequeue (%u)\n",
-				ptr->name,
-				ptr->memzone->socket_id,
-				ptr->flags & RING_F_SP_ENQ,
-				ptr->flags & RING_F_SC_DEQ);
-			printf("  - size (%u) mask (0x%x) capacity (%u)\n",
-				ptr->size,
-				ptr->mask,
-				ptr->capacity);
-			printf("  - count (%u) free count (%u)\n",
-				rte_ring_count(ptr),
-				rte_ring_free_count(ptr));
-			printf("  - full (%d) empty (%d)\n",
-				rte_ring_full(ptr),
-				rte_ring_empty(ptr));
-
+			rte_ring_dump(stdout, ptr);
 			STATS_BDR_STR(50, "");
 			return;
 		}
app/test-pmd/cmdline.c +1 -1
index a037a55c6..000853067 100644
@@ -9562,7 +9562,7 @@ dump_struct_sizes(void)
 #define DUMP_SIZE(t) printf("sizeof(" #t ") = %u\n", (unsigned)sizeof(t));
 	DUMP_SIZE(struct rte_mbuf);
 	DUMP_SIZE(struct rte_mempool);
-	DUMP_SIZE(struct rte_ring);
+	//DUMP_SIZE(struct rte_ring);
 #undef DUMP_SIZE
 }
 
app/test/commands.c +1 -1
index 3bf767bf7..89f93cdb2 100644
@@ -107,7 +107,7 @@ dump_struct_sizes(void)
 #define DUMP_SIZE(t) printf("sizeof(" #t ") = %u\n", (unsigned)sizeof(t));
 	DUMP_SIZE(struct rte_mbuf);
 	DUMP_SIZE(struct rte_mempool);
-	DUMP_SIZE(struct rte_ring);
+	//DUMP_SIZE(struct rte_ring);
 #undef DUMP_SIZE
 }
 
app/test/test_pdump.c +1 -5
index ad183184c..6a1180bcb 100644
@@ -57,8 +57,7 @@ run_pdump_client_tests(void)
 	if (ret < 0)
 		return -1;
 	mp->flags = 0x0000;
-	ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(),
-				      RING_F_SP_ENQ | RING_F_SC_DEQ);
+	ring_client = rte_ring_create("SR0", RING_SIZE, rte_socket_id(), 0);
 	if (ring_client == NULL) {
 		printf("rte_ring_create SR0 failed");
 		return -1;
@@ -71,9 +70,6 @@ run_pdump_client_tests(void)
 	}
 	rte_eth_dev_probing_finish(eth_dev);
 
-	ring_client->prod.single = 0;
-	ring_client->cons.single = 0;
-
 	printf("\n***** flags = RTE_PDUMP_FLAG_TX *****\n");
 
 	for (itr = 0; itr < NUM_ITR; itr++) {
app/test/test_ring_stress.h +1 -1
index c6f0bc9f1..27d4adfb2 100644
@@ -350,7 +350,7 @@ mt1_init(struct rte_ring **rng, void **data, uint32_t num)
 	/* alloc ring */
 	nr = 2 * num;
 	sz = rte_ring_get_memsize(nr);
-	r = rte_zmalloc(NULL, sz, alignof(*r));
+	r = rte_zmalloc(NULL, sz, RTE_CACHE_LINE_SIZE);
 	if (r == NULL) {
 		printf("%s: alloca(%zu) for FIFO with %u elems failed",
 			__func__, sz, nr);
config/common_base +1 -1
index 7ca2f28b1..87ab0766d 100644
@@ -670,7 +670,7 @@ CONFIG_RTE_LIBRTE_PMD_ZUC=n
 
 # Compile PMD for Crypto Scheduler device
 #
-CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER=y
+CONFIG_RTE_LIBRTE_PMD_CRYPTO_SCHEDULER=n
 
 #
 # Compile PMD for NULL Crypto device
drivers/crypto/ccp/ccp_pmd_ops.c +1 -1
index a19e85ecb..938977dc0 100644
@@ -666,7 +666,7 @@ ccp_pmd_qp_create_batch_info_ring(struct ccp_qp *qp,
 
 	r = rte_ring_lookup(qp->name);
 	if (r) {
-		if (r->size >= ring_size) {
+		if (rte_ring_get_size(r) >= ring_size) {
 			CCP_LOG_INFO(
 				"Reusing ring %s for processed packets",
 				qp->name);
drivers/crypto/meson.build +0 -1
index 7fa1fbe26..8c8b78db3 100644
@@ -16,7 +16,6 @@ drivers = ['aesni_gcm',
 	   'octeontx2',
 	   'openssl',
 	   'qat',
-	   'scheduler',
 	   'snow3g',
 	   'virtio',
 	   'zuc']
drivers/net/ring/rte_eth_ring.c +13 -4
index 41acbc513..0eaac0dc3 100644
@@ -41,6 +41,8 @@ struct ring_queue {
 	struct rte_ring *rng;
 	rte_atomic64_t rx_pkts;
 	rte_atomic64_t tx_pkts;
+	uint32_t rx_sync_type;
+	uint32_t tx_sync_type;
 };
 
 struct pmd_internals {
@@ -74,7 +76,7 @@ eth_ring_rx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
 	struct ring_queue *r = q;
 	const uint16_t nb_rx = (uint16_t)rte_ring_dequeue_burst(r->rng,
 			ptrs, nb_bufs, NULL);
-	if (r->rng->flags & RING_F_SC_DEQ)
+	if (r->rx_sync_type == RTE_RING_SYNC_ST)
 		r->rx_pkts.cnt += nb_rx;
 	else
 		rte_atomic64_add(&(r->rx_pkts), nb_rx);
@@ -88,7 +90,7 @@ eth_ring_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
 	struct ring_queue *r = q;
 	const uint16_t nb_tx = (uint16_t)rte_ring_enqueue_burst(r->rng,
 			ptrs, nb_bufs, NULL);
-	if (r->rng->flags & RING_F_SP_ENQ)
+	if (r->tx_sync_type == RTE_RING_SYNC_ST)
 		r->tx_pkts.cnt += nb_tx;
 	else
 		rte_atomic64_add(&(r->tx_pkts), nb_tx);
@@ -306,10 +308,14 @@ do_eth_dev_ring_create(const char *name,
 	internals->max_tx_queues = nb_tx_queues;
 	for (i = 0; i < nb_rx_queues; i++) {
 		internals->rx_ring_queues[i].rng = rx_queues[i];
+		internals->rx_ring_queues[i].rx_sync_type =
+			rte_ring_get_cons_sync_type(rx_queues[i]);
 		data->rx_queues[i] = &internals->rx_ring_queues[i];
 	}
 	for (i = 0; i < nb_tx_queues; i++) {
 		internals->tx_ring_queues[i].rng = tx_queues[i];
+		internals->tx_ring_queues[i].tx_sync_type =
+			rte_ring_get_prod_sync_type(rx_queues[i]);
 		data->tx_queues[i] = &internals->tx_ring_queues[i];
 	}
 
@@ -403,8 +409,11 @@ rte_eth_from_rings(const char *name, struct rte_ring *const rx_queues[],
 int
 rte_eth_from_ring(struct rte_ring *r)
 {
-	return rte_eth_from_rings(r->name, &r, 1, &r, 1,
-			r->memzone ? r->memzone->socket_id : SOCKET_ID_ANY);
+	const struct rte_memzone *mz;
+
+	mz = rte_ring_get_memzone(r);
+	return rte_eth_from_rings(rte_ring_get_name(r), &r, 1, &r, 1,
+			mz ? mz->socket_id : SOCKET_ID_ANY);
 }
 
 static int
lib/librte_eventdev/rte_event_ring.c +1 -5
index d27e23901..99700a18c 100644
@@ -16,12 +16,8 @@ int
 rte_event_ring_init(struct rte_event_ring *r, const char *name,
 	unsigned int count, unsigned int flags)
 {
-	/* compilation-time checks */
-	RTE_BUILD_BUG_ON((sizeof(struct rte_event_ring) &
-			  RTE_CACHE_LINE_MASK) != 0);
-
 	/* init the ring structure */
-	return rte_ring_init(&r->r, name, count, flags);
+	return rte_ring_init(r, name, count, flags);
 }
 
 /* create the ring */
lib/librte_eventdev/rte_event_ring.h +8 -9
index c0861b0ec..b31b33b3e 100644
@@ -32,9 +32,8 @@
  * used inside software eventdev implementations and by applications
  * directly as needed.
  */
-struct rte_event_ring {
-	struct rte_ring r;
-};
+
+#define rte_event_ring	rte_ring
 
 /**
  * Returns the number of events in the ring
@@ -47,7 +46,7 @@ struct rte_event_ring {
 static __rte_always_inline unsigned int
 rte_event_ring_count(const struct rte_event_ring *r)
 {
-	return rte_ring_count(&r->r);
+	return rte_ring_count(r);
 }
 
 /**
@@ -62,7 +61,7 @@ rte_event_ring_count(const struct rte_event_ring *r)
 static __rte_always_inline unsigned int
 rte_event_ring_free_count(const struct rte_event_ring *r)
 {
-	return rte_ring_free_count(&r->r);
+	return rte_ring_free_count(r);
 }
 
 /**
@@ -93,7 +92,7 @@ rte_event_ring_enqueue_burst(struct rte_event_ring *r,
 	unsigned int num;
 	uint32_t space;
 
-	num = rte_ring_enqueue_burst_elem(&r->r, events,
+	num = rte_ring_enqueue_burst_elem(r, events,
 				sizeof(struct rte_event), n,
 				&space);
 
@@ -129,7 +128,7 @@ rte_event_ring_dequeue_burst(struct rte_event_ring *r,
 	unsigned int num;
 	uint32_t remaining;
 
-	num = rte_ring_dequeue_burst_elem(&r->r, events,
+	num = rte_ring_dequeue_burst_elem(r, events,
 				sizeof(struct rte_event), n,
 				&remaining);
 
@@ -250,7 +249,7 @@ rte_event_ring_free(struct rte_event_ring *r);
 static inline unsigned int
 rte_event_ring_get_size(const struct rte_event_ring *r)
 {
-	return rte_ring_get_size(&r->r);
+	return rte_ring_get_size(r);
 }
 
 /**
@@ -264,6 +263,6 @@ rte_event_ring_get_size(const struct rte_event_ring *r)
 static inline unsigned int
 rte_event_ring_get_capacity(const struct rte_event_ring *r)
 {
-	return rte_ring_get_capacity(&r->r);
+	return rte_ring_get_capacity(r);
 }
 #endif
lib/librte_pdump/rte_pdump.c +2 -1
index 8a01ac510..43aa07793 100644
@@ -380,7 +380,8 @@ pdump_validate_ring_mp(struct rte_ring *ring, struct rte_mempool *mp)
 		rte_errno = EINVAL;
 		return -1;
 	}
-	if (ring->prod.single || ring->cons.single) {
+	if (rte_ring_get_prod_sync_type(ring) == RTE_RING_SYNC_ST ||
+			rte_ring_get_cons_sync_type(ring) == RTE_RING_SYNC_ST) {
 		PDUMP_LOG(ERR, "ring with either SP or SC settings"
 		" is not valid for pdump, should have MP and MC settings\n");
 		rte_errno = EINVAL;
lib/librte_port/rte_port_ring.c +19 -10
index 47fcdd06a..16fdee0bd 100644
@@ -40,12 +40,15 @@ rte_port_ring_reader_create_internal(void *params, int socket_id,
 	struct rte_port_ring_reader_params *conf =
 			params;
 	struct rte_port_ring_reader *port;
+	uint32_t ring_sync;
+
+	ring_sync = rte_ring_get_cons_sync_type(conf->ring);
 
 	/* Check input parameters */
 	if ((conf == NULL) ||
 		(conf->ring == NULL) ||
-		(conf->ring->cons.single && is_multi) ||
-		(!(conf->ring->cons.single) && !is_multi)) {
+			(ring_sync == RTE_RING_SYNC_ST && is_multi) ||
+			(ring_sync != RTE_RING_SYNC_ST && !is_multi)) {
 		RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
 		return NULL;
 	}
@@ -167,13 +170,16 @@ rte_port_ring_writer_create_internal(void *params, int socket_id,
 	struct rte_port_ring_writer_params *conf =
 			params;
 	struct rte_port_ring_writer *port;
+	uint32_t ring_sync;
+
+	ring_sync = rte_ring_get_prod_sync_type(conf->ring);
 
 	/* Check input parameters */
 	if ((conf == NULL) ||
-		(conf->ring == NULL) ||
-		(conf->ring->prod.single && is_multi) ||
-		(!(conf->ring->prod.single) && !is_multi) ||
-		(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
+			(conf->ring == NULL) ||
+			(ring_sync == RTE_RING_SYNC_ST && is_multi) ||
+			(ring_sync != RTE_RING_SYNC_ST && !is_multi) ||
+			(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
 		RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
 		return NULL;
 	}
@@ -436,13 +442,16 @@ rte_port_ring_writer_nodrop_create_internal(void *params, int socket_id,
 	struct rte_port_ring_writer_nodrop_params *conf =
 			params;
 	struct rte_port_ring_writer_nodrop *port;
+	uint32_t ring_sync;
+
+	ring_sync = rte_ring_get_prod_sync_type(conf->ring);
 
 	/* Check input parameters */
 	if ((conf == NULL) ||
-		(conf->ring == NULL) ||
-		(conf->ring->prod.single && is_multi) ||
-		(!(conf->ring->prod.single) && !is_multi) ||
-		(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
+			(conf->ring == NULL) ||
+			(ring_sync == RTE_RING_SYNC_ST && is_multi) ||
+			(ring_sync != RTE_RING_SYNC_ST && !is_multi) ||
+			(conf->tx_burst_sz > RTE_PORT_IN_BURST_SIZE_MAX)) {
 		RTE_LOG(ERR, PORT, "%s: Invalid Parameters\n", __func__);
 		return NULL;
 	}
lib/librte_ring/Makefile +3 -3
index 917c560ad..8c3cd523d 100644
@@ -13,11 +13,11 @@ EXPORT_MAP := rte_ring_version.map
 
 # all source are stored in SRCS-y
 SRCS-$(CONFIG_RTE_LIBRTE_RING) := rte_ring.c
+SRCS-$(CONFIG_RTE_LIBRTE_RING) += ring_impl.c
+SRCS-$(CONFIG_RTE_LIBRTE_RING) += ring_elem.c
 
 # install includes
 SYMLINK-$(CONFIG_RTE_LIBRTE_RING)-include := rte_ring.h \
-					rte_ring_elem.h \
-					rte_ring_generic.h \
-					rte_ring_c11_mem.h
+					rte_ring_elem.h
 
 include $(RTE_SDK)/mk/rte.lib.mk
lib/librte_ring/meson.build +2 -4
index f2f3ccc88..8adbdfc9d 100644
@@ -1,11 +1,9 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-sources = files('rte_ring.c')
+sources = files('rte_ring.c', 'ring_impl.c', 'ring_elem.c')
 headers = files('rte_ring.h',
-		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h')
+		'rte_ring_elem.h')
 
 # rte_ring_create_elem and rte_ring_get_memsize_elem are experimental
 allow_experimental_apis = true
lib/librte_ring/ring_elem.c new file +919 -0

919 lines not on this page · raw message

lib/librte_ring/ring_impl.c new file +750 -0

750 lines not on this page · raw message

lib/librte_ring/ring_impl.h new file +64 -0

64 lines not on this page · raw message

lib/librte_ring/rte_ring.c +1 -2
index 77e5de099..ec9392585 100644
@@ -32,8 +32,7 @@
 #include <rte_spinlock.h>
 #include <rte_tailq.h>
 
-#include "rte_ring.h"
-#include "rte_ring_elem.h"
+#include "ring_impl.h"
 
 TAILQ_HEAD(rte_ring_list, rte_tailq_entry);
 
lib/librte_ring/rte_ring.h +67 -345

591 lines not on this page · raw message

lib/librte_ring/rte_ring_elem.h +36 -483

361 lines not on this page · raw message

lib/librte_ring/rte_ring_version.map +46 -0

46 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.17.1

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 79 lines · On Fri, 20 Mar 2020 16:41:38 +0000 Konstantin Ananyev…

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 8.75/43.23
legacy APIs: MP/MC: single: 56.18/80.44

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 37.36/53.37 legacy APIs: SP/SC: burst (size: 32): 93.97/117.30 legacy APIs: MP/MC: burst (size: 8): 78.23/91.45 legacy APIs: MP/MC: burst (size: 32): 131.59/152.49

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 37.29/54.48 legacy APIs: SP/SC: bulk (size: 32): 92.68/113.01 legacy APIs: MP/MC: bulk (size: 8): 78.40/93.50 legacy APIs: MP/MC: bulk (size: 32): 131.49/154.25

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 4.00/16.86
legacy APIs: MP/MC: bulk (size: 8): 7.01/15.55

### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 10.64/17.56
legacy APIs: MP/MC: bulk (size: 8): 15.30/16.69
legacy APIs: SP/SC: bulk (size: 32): 5.84/7.09
legacy APIs: MP/MC: bulk (size: 32): 6.34/7.54

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 24.34/42.40
legacy APIs: MP/MC: bulk (size: 8): 70.34/71.82
legacy APIs: SP/SC: bulk (size: 32): 12.67/14.68
legacy APIs: MP/MC: bulk (size: 32): 22.41/17.93

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 10.65/41.96 elem APIs: element size 16B: MP/MC: single: 44.33/81.36

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 39.20/58.52 elem APIs: element size 16B: SP/SC: burst (size: 32): 123.19/142.79 elem APIs: element size 16B: MP/MC: burst (size: 8): 80.72/101.36 elem APIs: element size 16B: MP/MC: burst (size: 32): 169.21/185.38

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 41.64/58.46 elem APIs: element size 16B: SP/SC: bulk (size: 32): 122.74/142.52 elem APIs: element size 16B: MP/MC: bulk (size: 8): 80.60/103.14 elem APIs: element size 16B: MP/MC: bulk (size: 32): 169.39/186.67

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 5.01/17.17 elem APIs: element size 16B: MP/MC: bulk (size: 8): 6.01/14.80

### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.02/17.18 elem APIs: element size 16B: MP/MC: bulk (size: 8): 16.81/21.14 elem APIs: element size 16B: SP/SC: bulk (size: 32): 7.87/9.01 elem APIs: element size 16B: MP/MC: bulk (size: 32): 8.22/10.57

Quoted 7 lines

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 27.00/51.94 elem APIs: element size 16B: MP/MC: bulk (size: 8): 78.24/74.48 elem APIs: element size 16B: SP/SC: bulk (size: 32): 15.41/16.14 elem APIs: element size 16B: MP/MC: bulk (size: 32): 18.72/21.64

What is impact with LTO? I suspect compiler might have a chance to get speed back with LTO.

#2 lore raw
Quoted 97 lines

-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Friday, March 20, 2020 5:55 PM
To: Ananyev, Konstantin <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC] ring: make ring implementation non-inlined

Quoted 85 lines · On Fri, 20 Mar 2020 16:41:38 +0000 Konstantin Ananyev…

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)

### Testing single element enq/deq ###
legacy APIs: SP/SC: single: 8.75/43.23
legacy APIs: MP/MC: single: 56.18/80.44

### Testing burst enq/deq ###
legacy APIs: SP/SC: burst (size: 8): 37.36/53.37 legacy APIs: SP/SC: burst (size: 32): 93.97/117.30 legacy APIs: MP/MC: burst (size: 8): 78.23/91.45 legacy APIs: MP/MC: burst (size: 32): 131.59/152.49

### Testing bulk enq/deq ###
legacy APIs: SP/SC: bulk (size: 8): 37.29/54.48 legacy APIs: SP/SC: bulk (size: 32): 92.68/113.01 legacy APIs: MP/MC: bulk (size: 8): 78.40/93.50 legacy APIs: MP/MC: bulk (size: 32): 131.49/154.25

### Testing empty bulk deq ###
legacy APIs: SP/SC: bulk (size: 8): 4.00/16.86
legacy APIs: MP/MC: bulk (size: 8): 7.01/15.55

### Testing using two hyperthreads ###
legacy APIs: SP/SC: bulk (size: 8): 10.64/17.56
legacy APIs: MP/MC: bulk (size: 8): 15.30/16.69
legacy APIs: SP/SC: bulk (size: 32): 5.84/7.09
legacy APIs: MP/MC: bulk (size: 32): 6.34/7.54

### Testing using two physical cores ###
legacy APIs: SP/SC: bulk (size: 8): 24.34/42.40
legacy APIs: MP/MC: bulk (size: 8): 70.34/71.82
legacy APIs: SP/SC: bulk (size: 32): 12.67/14.68
legacy APIs: MP/MC: bulk (size: 32): 22.41/17.93

### Testing single element enq/deq ###
elem APIs: element size 16B: SP/SC: single: 10.65/41.96 elem APIs: element size 16B: MP/MC: single: 44.33/81.36

### Testing burst enq/deq ###
elem APIs: element size 16B: SP/SC: burst (size: 8): 39.20/58.52 elem APIs: element size 16B: SP/SC: burst (size: 32): 123.19/142.79 elem APIs: element size 16B: MP/MC: burst (size: 8): 80.72/101.36 elem APIs: element size 16B: MP/MC: burst (size: 32): 169.21/185.38

### Testing bulk enq/deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 41.64/58.46 elem APIs: element size 16B: SP/SC: bulk (size: 32): 122.74/142.52 elem APIs: element size 16B: MP/MC: bulk (size: 8): 80.60/103.14 elem APIs: element size 16B: MP/MC: bulk (size: 32): 169.39/186.67

### Testing empty bulk deq ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 5.01/17.17 elem APIs: element size 16B: MP/MC: bulk (size: 8): 6.01/14.80

### Testing using two hyperthreads ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 12.02/17.18 elem APIs: element size 16B: MP/MC: bulk (size: 8): 16.81/21.14 elem APIs: element size 16B: SP/SC: bulk (size: 32): 7.87/9.01 elem APIs: element size 16B: MP/MC: bulk (size: 32): 8.22/10.57

### Testing using two physical cores ###
elem APIs: element size 16B: SP/SC: bulk (size: 8): 27.00/51.94 elem APIs: element size 16B: MP/MC: bulk (size: 8): 78.24/74.48 elem APIs: element size 16B: SP/SC: bulk (size: 32): 15.41/16.14 elem APIs: element size 16B: MP/MC: bulk (size: 32): 18.72/21.64

Quoted 4 lines
Quoted 1 line

What is impact with LTO? I suspect compiler might have a chance to get speed back with LTO.

Might be, but LTO is not enabled by default. So don't see much point in digging any further here.

#3 lore raw
In reply to Konstantin Ananyev
Quoted 10 lines · Konstantin Ananyev

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect

Quoted 8 lines

the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

+1

My reasoning is a bit different, DPDK is using in embedded boxes too where performance has
more weight than ABI stuff. I think we need to focus first on slow path APIs ABI stuff.

I spend a few cycles to apply this patch +
http://mails.dpdk.org/archives/dev/2020-February/158586.html
on top of the tree, there are a lot of conflicts. If I get a mergeable patch then I will test it on an arm64 box.

Quoted 3 lines

Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)

#4 lore raw

Hi Jerin,

Quoted 29 lines
Quoted 10 lines · Konstantin Ananyev

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect

Quoted 8 lines

the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

+1

My reasoning is a bit different, DPDK is using in embedded boxes too where performance has
more weight than ABI stuff. I think we need to focus first on slow path APIs ABI stuff.

Quoted 4 lines

I spend a few cycles to apply this patch +
http://mails.dpdk.org/archives/dev/2020-February/158586.html
on top of the tree, there are a lot of conflicts. If I get a mergeable patch then I will test it on an arm64 box.

You don’t need to apply previous patch series. They are completely unrelated.
Just apply that one (http://patches.dpdk.org/patch/66982/)
on top of dpdk.org master.
It should be applied cleanly (at least it does for me). Konstantin

Quoted 8 lines
Quoted 8 lines

Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)

#5 lore raw
In reply to Jerin Jacob
Quoted 24 lines

From: dev [mailto:[email protected]] On Behalf Of Jerin Jacob
Sent: Wednesday, March 25, 2020 10:09 PM

Quoted 10 lines · Konstantin Ananyev

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect

Quoted 8 lines

the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

Quoted 5 lines

+1

My reasoning is a bit different, DPDK is using in embedded boxes too where performance has
more weight than ABI stuff.

As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability. ABI stability is irrelevant for us; and API instability is a non-recurring engineering cost each time we choose to switch to a new DPDK version, which we only do if we cannot avoid it, e.g. due to new drivers, security fixes or new features that we want to use.

For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated compile time configurability. I do understand the reasoning behind it, and the impact is minimal, so we accept it.

However, if DPDK starts sacrificing performance of the core libraries for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.

Quoted 15 lines

I think we need to focus first on slow
path APIs ABI stuff.

I spend a few cycles to apply this patch +
http://mails.dpdk.org/archives/dev/2020-February/158586.html
on top of the tree, there are a lot of conflicts. If I get a mergeable patch then I will test it on an arm64 box.

Quoted 3 lines

Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz
ring_perf_autotest (without patch/with patch)

#6 lore raw

26/03/2020 09:04, Morten Brørup:

Quoted 22 lines

From: Jerin Jacob

Quoted 21 lines · On Fri, Mar 20, 2020 Konstantin Ananyev wrote:
Quoted 10 lines

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect

Quoted 8 lines

the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

+1

Konstantin, thank you for doing some measures

Quoted 11 lines
Quoted 2 lines

My reasoning is a bit different, DPDK is using in embedded boxes too where performance has more weight than ABI stuff.

As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability.
ABI stability is irrelevant for us;
and API instability is a non-recurring engineering cost each time we choose to switch to a new DPDK version, which we only do if we cannot avoid it, e.g. due to new drivers, security fixes or new features that we want to use.

Quoted 4 lines

For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated compile time configurability. I do understand the reasoning behind it, and the impact is minimal, so we accept it.

The code can be optimized by removing some instructions with #ifdef. But the complexity of managing #ifdef enabling/disabling, depending on the platform and the use case, would be huge. We try to have a reasonable code "always enabled" which performs well in all cases. This is a design choice which makes DPDK a library, not a pool of code to cherry-pick.

Quoted 4 lines

However, if DPDK starts sacrificing performance of the core libraries for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.

The initial choice regarding ABI compatibility was "do not care". Recently, the decision was done to care about ABI compatibility as priority number 2. The priority number 1 remains the performance. That's a reason for allowing some ABI breakages in some specific releases announced in advance.

Quoted 2 lines
Quoted 2 lines

I think we need to focus first on slow
path APIs ABI stuff.

Yes we should not degrade fast path performance for the sake of avoiding uncertain future ABI issues.

Morten, Jerin, thank you for the feedback.

#7 lore raw

<snip>

Quoted 63 lines

Subject: Re: [dpdk-dev] [RFC] ring: make ring implementation non-inlined

26/03/2020 09:04, Morten Brørup:

Quoted 22 lines

From: Jerin Jacob

Quoted 21 lines · On Fri, Mar 20, 2020 Konstantin Ananyev wrote:
Quoted 10 lines

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain ABI stability in future.
This is just a POC to measure the impact of proposed idea, proper implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra for enqueue+dequeue pair. On some more realistic code, I suspect

Quoted 8 lines

the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though for SP/SC and/or small transfers it is more then noticable (see exact numbers below).
From my perspective we'd probably keep it inlined for now to avoid any non-anticipated perfomance degradations.
Though intersted to see perf results and opinions from other interested parties.

+1

Konstantin, thank you for doing some measures

Quoted 11 lines
Quoted 2 lines

My reasoning is a bit different, DPDK is using in embedded boxes too where performance has more weight than ABI stuff.

As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability.
ABI stability is irrelevant for us;
and API instability is a non-recurring engineering cost each time we choose to switch to a new DPDK version, which we only do if we cannot avoid it, e.g. due to new drivers, security fixes or new features that we want to use.

Quoted 4 lines

For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated compile time configurability. I do understand the reasoning behind it, and the impact is minimal, so we accept it.

The code can be optimized by removing some instructions with #ifdef. But the complexity of managing #ifdef enabling/disabling, depending on the platform and the use case, would be huge.
We try to have a reasonable code "always enabled" which performs well in all cases. This is a design choice which makes DPDK a library, not a pool of code to cherry-pick.

Quoted 4 lines

However, if DPDK starts sacrificing performance of the core libraries for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.

The initial choice regarding ABI compatibility was "do not care". Recently, the decision was done to care about ABI compatibility as priority number 2. The priority number 1 remains the performance.
That's a reason for allowing some ABI breakages in some specific releases announced in advance.

Quoted 6 lines
Quoted 1 line
Quoted 1 line

I think we need to focus first on slow path APIs ABI stuff.

Yes we should not degrade fast path performance for the sake of avoiding uncertain future ABI issues.

Morten, Jerin, thank you for the feedback.

I think we have a consensus here not to make any changes to inline functions for now. Should we mark this as 'Deferred or Rejected'?

Quoted 1 line
#8 lore raw
Quoted 90 lines

From: dev [mailto:[email protected]] On Behalf Of Honnappa Nagarahalli
Sent: Wednesday, July 1, 2020 1:16 AM

<snip>

Quoted 1 line

Subject: Re: [dpdk-dev] [RFC] ring: make ring implementation non-

inlined

Quoted 9 lines

26/03/2020 09:04, Morten Brørup:

Quoted 7 lines

From: Jerin Jacob

Quoted 6 lines · On Fri, Mar 20, 2020 Konstantin Ananyev wrote:
Quoted 5 lines

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain

ABI

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

stability in future.
This is just a POC to measure the impact of proposed idea,

proper

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra

for

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

enqueue+dequeue pair. On some more realistic code, I suspect the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though

for

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP/SC and/or small transfers it is more then noticable (see

exact

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

numbers below).
From my perspective we'd probably keep it inlined for now to

avoid

Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

+1

Konstantin, thank you for doing some measures

Quoted 1 line
Quoted 1 line

My reasoning is a bit different, DPDK is using in embedded boxes

too

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

where performance has more weight than ABI stuff.

As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability.
ABI stability is irrelevant for us;
and API instability is a non-recurring engineering cost each time

we

Quoted 1 line
Quoted 1 line

choose to switch to a new DPDK version, which we only do if we

cannot

Quoted 1 line
Quoted 1 line

avoid it, e.g. due to new drivers, security fixes or new features

that

Quoted 4 lines
Quoted 4 lines

we want to use.

For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated

compile

Quoted 1 line
Quoted 1 line

time configurability. I do understand the reasoning behind it, and

the

Quoted 4 lines
Quoted 1 line

impact is minimal, so we accept it.

The code can be optimized by removing some instructions with #ifdef. But the complexity of managing #ifdef enabling/disabling, depending

on the

Quoted 2 lines

platform and the use case, would be huge.
We try to have a reasonable code "always enabled" which performs well

in all

Quoted 1 line

cases. This is a design choice which makes DPDK a library, not a pool

of code

Quoted 3 lines

to cherry-pick.

Quoted 1 line

However, if DPDK starts sacrificing performance of the core

libraries

Quoted 6 lines
Quoted 3 lines

for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.

The initial choice regarding ABI compatibility was "do not care". Recently, the decision was done to care about ABI compatibility as

priority

Quoted 2 lines

number 2. The priority number 1 remains the performance. That's a reason for allowing some ABI breakages in some specific

releases

Quoted 4 lines

announced in advance.

Quoted 1 line
Quoted 1 line

I think we need to focus first on slow path APIs ABI stuff.

Quoted 8 lines
Quoted 1 line

Yes we should not degrade fast path performance for the sake of

avoiding

Quoted 3 lines

uncertain future ABI issues.

Morten, Jerin, thank you for the feedback.

I think we have a consensus here not to make any changes to inline functions for now.
Should we mark this as 'Deferred or Rejected'?

Rejected.

There is no need for this modification now, and no actual use cases for it in the road map. In other words: This modification has no use cases; it is purely academic. Many other suggestions have been rejected for the reason that they have no current use cases.

As Thomas mentioned, DPDK has transitioned towards being a library, rather than a pool of code to cherry-pick from. I have learned to live with this.

Being a library doesn't mean that functions cannot be exposed as inline code in the library header files. DPDK is mainly a high performance library with a tradition of exposing many of its internals in its API, and we should keep it this way. We certainly don't want an opaque API hiding all of its internals, passing around void pointers.

However, it was still an interesting experiment to investigate the performance cost.

#9 lore raw
Quoted 105 lines
Quoted 91 lines

From: dev [mailto:[email protected]] On Behalf Of Honnappa Nagarahalli
Sent: Wednesday, July 1, 2020 1:16 AM

<snip>

Quoted 1 line

Subject: Re: [dpdk-dev] [RFC] ring: make ring implementation non-

inlined

Quoted 9 lines

26/03/2020 09:04, Morten Brørup:

Quoted 7 lines

From: Jerin Jacob

Quoted 6 lines · On Fri, Mar 20, 2020 Konstantin Ananyev wrote:
Quoted 5 lines

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain

ABI

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

stability in future.
This is just a POC to measure the impact of proposed idea,

proper

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra

for

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

enqueue+dequeue pair. On some more realistic code, I suspect the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though

for

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP/SC and/or small transfers it is more then noticable (see

exact

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

numbers below).
From my perspective we'd probably keep it inlined for now to

avoid

Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

+1

Konstantin, thank you for doing some measures

Quoted 1 line
Quoted 1 line

My reasoning is a bit different, DPDK is using in embedded boxes

too

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

where performance has more weight than ABI stuff.

As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability.
ABI stability is irrelevant for us;
and API instability is a non-recurring engineering cost each time

we

Quoted 1 line
Quoted 1 line

choose to switch to a new DPDK version, which we only do if we

cannot

Quoted 1 line
Quoted 1 line

avoid it, e.g. due to new drivers, security fixes or new features

that

Quoted 4 lines
Quoted 4 lines

we want to use.

For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated

compile

Quoted 1 line
Quoted 1 line

time configurability. I do understand the reasoning behind it, and

the

Quoted 4 lines
Quoted 1 line

impact is minimal, so we accept it.

The code can be optimized by removing some instructions with #ifdef. But the complexity of managing #ifdef enabling/disabling, depending

on the

Quoted 2 lines

platform and the use case, would be huge.
We try to have a reasonable code "always enabled" which performs well

in all

Quoted 1 line

cases. This is a design choice which makes DPDK a library, not a pool

of code

Quoted 3 lines

to cherry-pick.

Quoted 1 line

However, if DPDK starts sacrificing performance of the core

libraries

Quoted 6 lines
Quoted 3 lines

for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.

The initial choice regarding ABI compatibility was "do not care". Recently, the decision was done to care about ABI compatibility as

priority

Quoted 2 lines

number 2. The priority number 1 remains the performance. That's a reason for allowing some ABI breakages in some specific

releases

Quoted 4 lines

announced in advance.

Quoted 1 line
Quoted 1 line

I think we need to focus first on slow path APIs ABI stuff.

Quoted 8 lines
Quoted 1 line

Yes we should not degrade fast path performance for the sake of

avoiding

Quoted 3 lines

uncertain future ABI issues.

Morten, Jerin, thank you for the feedback.

I think we have a consensus here not to make any changes to inline functions for now.
Should we mark this as 'Deferred or Rejected'?

Rejected.

There is no need for this modification now, and no actual use cases for it in the road map. In other words: This modification has no use cases; it is purely academic. Many other suggestions have been rejected for the reason that they have no current use cases.

Quoted 8 lines

As Thomas mentioned, DPDK has transitioned towards being a library, rather than a pool of code to cherry-pick from. I have learned to live with this.

Being a library doesn't mean that functions cannot be exposed as inline code in the library header files. DPDK is mainly a high performance library with a tradition of exposing many of its internals in its API, and we should keep it this way. We certainly don't want an opaque API hiding all of its internals, passing around void pointers.

However, it was still an interesting experiment to investigate the performance cost.

Yes, please reject it.
Konstantin

#10 lore raw

<snip>

Quoted 117 lines
Quoted 99 lines
Quoted 99 lines
Quoted 2 lines

Subject: Re: [dpdk-dev] [RFC] ring: make ring implementation non-

inlined

Quoted 10 lines

26/03/2020 09:04, Morten Brørup:

Quoted 8 lines

From: Jerin Jacob

Quoted 7 lines · On Fri, Mar 20, 2020 Konstantin Ananyev wrote:
Quoted 6 lines

As was discussed here:
http://mails.dpdk.org/archives/dev/2020-February/158586.html
this RFC aimed to hide ring internals into .c and make all ring functions non-inlined. In theory that might help to maintain

ABI

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

stability in future.
This is just a POC to measure the impact of proposed idea,

proper

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

implementation would definetly need some extra effort. On IA box (SKX) ring_perf_autotest shows ~20-30 cycles extra

for

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

enqueue+dequeue pair. On some more realistic code, I suspect the impact it might be a bit higher.
For MP/MC bulk transfers degradation seems quite small, though

for

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

SP/SC and/or small transfers it is more then noticable (see

exact

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

numbers below).
From my perspective we'd probably keep it inlined for now to

avoid

Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

any non-anticipated perfomance degradations. Though intersted to see perf results and opinions from other interested parties.

+1

Konstantin, thank you for doing some measures

Quoted 2 lines
Quoted 2 lines

My reasoning is a bit different, DPDK is using in embedded boxes

too

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

where performance has more weight than ABI stuff.

As a network appliance vendor I can confirm that we certainly care more about performance than ABI stability.
ABI stability is irrelevant for us; and API instability is a non-recurring engineering cost each time

we

Quoted 1 line
Quoted 1 line

choose to switch to a new DPDK version, which we only do if we

cannot

Quoted 2 lines
Quoted 2 lines

avoid it, e.g. due to new drivers, security fixes or new features

that

Quoted 5 lines
Quoted 5 lines

we want to use.

For us, the trend pointed in the wrong direction when DPDK switched the preference towards runtime configurability and deprecated

compile

Quoted 2 lines
Quoted 2 lines

time configurability. I do understand the reasoning behind it, and

the

Quoted 5 lines
Quoted 1 line

impact is minimal, so we accept it.

The code can be optimized by removing some instructions with #ifdef. But the complexity of managing #ifdef enabling/disabling, depending

on the

Quoted 3 lines

platform and the use case, would be huge.
We try to have a reasonable code "always enabled" which performs well

in all

Quoted 2 lines

cases. This is a design choice which makes DPDK a library, not a pool

of code

Quoted 3 lines

to cherry-pick.

Quoted 1 line

However, if DPDK starts sacrificing performance of the core

libraries

Quoted 6 lines
Quoted 3 lines

for the benefits of the GNU/Linux distributors, network appliance vendors may put more effort into sticking with old DPDK versions instead of updating.

The initial choice regarding ABI compatibility was "do not care". Recently, the decision was done to care about ABI compatibility as

priority

Quoted 2 lines

number 2. The priority number 1 remains the performance. That's a reason for allowing some ABI breakages in some specific

releases

Quoted 5 lines

announced in advance.

Quoted 1 line
Quoted 1 line

I think we need to focus first on slow path APIs ABI stuff.

Yes we should not degrade fast path performance for the sake of

avoiding

Quoted 3 lines

uncertain future ABI issues.

Morten, Jerin, thank you for the feedback.

Quoted 8 lines
Quoted 3 lines

I think we have a consensus here not to make any changes to inline functions for now.
Should we mark this as 'Deferred or Rejected'?

Rejected.

There is no need for this modification now, and no actual use cases for it in the road map. In other words: This modification has no use cases; it

is purely academic. Many other suggestions have been rejected for the reason that they have no current use cases.

Quoted 3 lines

As Thomas mentioned, DPDK has transitioned towards being a library, rather than a pool of code to cherry-pick from. I have learned to live with

this.

Quoted 4 lines

Being a library doesn't mean that functions cannot be exposed as inline code in the library header files. DPDK is mainly a high performance library with a tradition of exposing many of its internals in its

Quoted 7 lines

API, and we should keep it this way. We certainly don't want an opaque API hiding all of its internals, passing around void pointers.

Quoted 2 lines

However, it was still an interesting experiment to investigate the

performance cost.

Yes, please reject it.

I just tried to mark it rejected in patchwork, I do not have the permissions (probably you are the owner of the patch). Can you please mark it?

Quoted 2 lines

Konstantin

#11 lore raw
Quoted 2 lines · Honnappa Nagarahalli
Quoted 1 line

Yes, please reject it.

I just tried to mark it rejected in patchwork, I do not have the permissions (probably you are the owner of the patch). Can you please mark it?

Indeed, Konstantin should be able to do it.
I just flagged it Rejected.

There are still some ring patches waiting in pw, could you and Konstantin agree on/flag them?
Thanks.

Signature · 1 line
David Marchand