Skip to contentSkip to conversation

refactoring ring library

PATCH Superseded Feifei Wang 3 participants 23 messages Dec 21, 2020 – Jan 29, 2021 Arm/Amlogic Meson SoC support

lore
Share & tools
lore

Feifei Wang

author
#0 lore raw

cover refactoring ring library

Do some work for ring refactoring, which includes:
1. add rte prefix before update tail API
2. Move all private APIs into new header files

Feifei Wang (3):
test/ring: reduce iteration numbers to make test duration shorter
ring: add rte prefix before update tail API
ring: rename and refactor ring library

 app/test/test_ring_perf.c                     |   2 +-
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |  13 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |  10 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
 ...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
 ...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 13 files changed, 651 insertions(+), 640 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (94%)
 create mode 100644 lib/librte_ring/ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (95%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h} (62%)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/3 test/ring: reduce iteration numbers to make test duration shorter

When testing ring performance in the case that multiple lcores are mapped to the same physical core, e.g. --lcores '(0-3)@10', it takes a very long time to wait for the "enqueue_dequeue_bulk_helper" to finish. This is because too much iteration numbers and extremely low efficiency for enqueue and dequeue with this kind of core mapping. Following are the test results to show the above phenomenon:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test --lcores '(0-1)@25'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           7s    7s    7s    8s    9s     16s    47s    170s   660s   >0.5h   >1h
legacy APIs: SP/SC: 37    11    6     40525 40525  40209  40367  40407  40541  NoData  NoData
legacy APIs: MP/MC: 56    14    11    50657 40526  40526  40526  40625  40585  NoData  NoData

aarch64-n1sdp:
$sudo ./app/test/dpdk-test --lcore '(0-1)@1'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    9s    9s     14s    34s    111s   418s   25min   >1h
legacy APIs: SP/SC: 0.4   0.2   0.1   488   488    488    488    488    489    489     NoData
legacy APIs: MP/MC: 0.4   0.3   0.2   488   488    488    488    490    489    489     NoData

As the number of iterations increases, so does the time which is required to run the program. Currently (iter_shift = 23), it will take more than 1 hour to wait for the test to finish. To fix this, the "iter_shift" should decrease and ensure enough iterations to keep the test data stable. In order to achieve this, we also test with "-l" EAL argument:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test -l 25-26
Testing using two NUMA nodes(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           6s    6s    6s    6s    6s     6s     6s     7s     8s     11s     27s
legacy APIs: SP/SC: 47    20    13    22    54     83     91     73     81     75      95
legacy APIs: MP/MC: 44    18    18    240   245    270    250    249    252    250     253

aarch64-n1sdp:
$sudo ./app/test/dpdk-test -l 1-2
Testing using two physical cores(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    8s    8s     8s     8s     9s     9s     11s     23s
legacy APIs: SP/SC: 0.7   0.4   1.2   1.8   2.0    2.0    2.0    2.0    2.0    2.0     2.0
legacy APIs: MP/MC: 0.3   0.4   1.3   1.9   2.9    2.9    2.9    2.9    2.9    2.9     2.9

According to above test data, when "iter_shift" is set as "15", the test run time is reduced to less than 1 minute and the test result can keep stable in x86 and aarch64 servers.

Fixes 1fa5d0099efc ("test/ring: add custom element size performance tests")
Cc [email protected]
Cc [email protected]
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 app/test/test_ring_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
app/test/test_ring_perf.c +1 -1
index e63e25a86..fd82e2041 100644
@@ -178,7 +178,7 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 	struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 15;
 	const unsigned int iterations = 1 << iter_shift;
 	struct rte_ring *r = p->r;
 	unsigned int bsize = p->size;
Signature · 1 line
2.17.1
Acked-by on v1 1/3
#2 lore raw

Re: [dpdk-dev] [PATCH v1 1/3] test/ring: reduce iteration numbers to make test duration shorter

Quoted 66 lines

When testing ring performance in the case that multiple lcores are mapped to the same physical core, e.g. --lcores '(0-3)@10', it takes a very long time to wait for the "enqueue_dequeue_bulk_helper" to finish. This is because too much iteration numbers and extremely low efficiency for enqueue and dequeue with this kind of core mapping. Following are the test results to show the above phenomenon:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test --lcores '(0-1)@25'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           7s    7s    7s    8s    9s     16s    47s    170s   660s   >0.5h   >1h
legacy APIs: SP/SC: 37    11    6     40525 40525  40209  40367  40407  40541  NoData  NoData
legacy APIs: MP/MC: 56    14    11    50657 40526  40526  40526  40625  40585  NoData  NoData

aarch64-n1sdp:
$sudo ./app/test/dpdk-test --lcore '(0-1)@1'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    9s    9s     14s    34s    111s   418s   25min   >1h
legacy APIs: SP/SC: 0.4   0.2   0.1   488   488    488    488    488    489    489     NoData
legacy APIs: MP/MC: 0.4   0.3   0.2   488   488    488    488    490    489    489     NoData

As the number of iterations increases, so does the time which is required to run the program. Currently (iter_shift = 23), it will take more than 1 hour to wait for the test to finish. To fix this, the "iter_shift" should decrease and ensure enough iterations to keep the test data stable. In order to achieve this, we also test with "-l" EAL argument:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test -l 25-26
Testing using two NUMA nodes(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           6s    6s    6s    6s    6s     6s     6s     7s     8s     11s     27s
legacy APIs: SP/SC: 47    20    13    22    54     83     91     73     81     75      95
legacy APIs: MP/MC: 44    18    18    240   245    270    250    249    252    250     253

aarch64-n1sdp:
$sudo ./app/test/dpdk-test -l 1-2
Testing using two physical cores(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    8s    8s     8s     8s     9s     9s     11s     23s
legacy APIs: SP/SC: 0.7   0.4   1.2   1.8   2.0    2.0    2.0    2.0    2.0    2.0     2.0
legacy APIs: MP/MC: 0.3   0.4   1.3   1.9   2.9    2.9    2.9    2.9    2.9    2.9     2.9

According to above test data, when "iter_shift" is set as "15", the test run time is reduced to less than 1 minute and the test result can keep stable in x86 and aarch64 servers.

Fixes 1fa5d0099efc ("test/ring: add custom element size performance tests")
Cc [email protected]
Cc [email protected]
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 app/test/test_ring_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
app/test/test_ring_perf.c +0 -0
index e63e25a86..fd82e2041 100644

app/test/test_ring_perf.c enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,

Quoted 9 lines
app/test/test_ring_perf.c +1 -2
@@ -178,7 +178,7 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 	struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 15;
 	const unsigned int iterations = 1 << iter_shift;
 	struct rte_ring *r = p->r;
 	unsigned int bsize = p->size;
--

I think it would be better to rework the test(s)
to terminate after some timeout (30s or so), and report number of ops per timeout. Anyway, as a short term fix, I am ok with it.

Acked-by Konstantin Ananyev <[email protected]>
Quoted 1 line

2.17.1

author
#3 lore raw

[dpdk-dev] 回复: [PATCH v1 1/3] test/ring: reduce iteration numbers to make test duration shorter

Hi, Konstantin

Quoted 97 lines

-----邮件原件-----
发件人: Ananyev, Konstantin <[email protected]> 发送时间: 2021年1月22日 21:16
收件人: Feifei Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>; Olivier Matz <[email protected]>; Gavin Hu <[email protected]>
抄送: [email protected]; nd <[email protected]>; [email protected]
主题: RE: [PATCH v1 1/3] test/ring: reduce iteration numbers to make test duration shorter

Quoted 7 lines

When testing ring performance in the case that multiple lcores are mapped to the same physical core, e.g. --lcores '(0-3)@10', it takes a very long time to wait for the "enqueue_dequeue_bulk_helper" to finish. This is because too much iteration numbers and extremely low efficiency for enqueue and dequeue with this kind of core mapping. Following are the test results to show the above phenomenon:

Quoted 6 lines

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test --lcores '(0-1)@25'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           7s    7s    7s    8s    9s     16s    47s    170s   660s   >0.5h   >1h
legacy APIs: SP/SC: 37    11    6     40525 40525  40209  40367  40407  40541

NoData NoData

Quoted 1 line

legacy APIs: MP/MC: 56 14 11 50657 40526 40526 40526 40625 40585

NoData NoData

Quoted 7 lines

aarch64-n1sdp:
$sudo ./app/test/dpdk-test --lcore '(0-1)@1'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    9s    9s     14s    34s    111s   418s   25min   >1h
legacy APIs: SP/SC: 0.4   0.2   0.1   488   488    488    488    488    489    489

NoData

Quoted 1 line

legacy APIs: MP/MC: 0.4 0.3 0.2 488 488 488 488 490 489 489

NoData

Quoted 6 lines

As the number of iterations increases, so does the time which is required to run the program. Currently (iter_shift = 23), it will take more than 1 hour to wait for the test to finish. To fix this, the "iter_shift" should decrease and ensure enough iterations to keep the test data stable. In order to achieve this, we also test with "-l" EAL

argument:

Quoted 8 lines

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test -l 25-26
Testing using two NUMA nodes(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           6s    6s    6s    6s    6s     6s     6s     7s     8s     11s     27s
legacy APIs: SP/SC: 47    20    13    22    54     83     91     73     81     75      95
legacy APIs: MP/MC: 44    18    18    240   245    270    250    249    252    250

253

app/test/test_ring_perf.c enqueue_dequeue_bulk_helper(const unsigned int

Quoted 30 lines

aarch64-n1sdp:
$sudo ./app/test/dpdk-test -l 1-2
Testing using two physical cores(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    8s    8s     8s     8s     9s     9s     11s     23s
legacy APIs: SP/SC: 0.7   0.4   1.2   1.8   2.0    2.0    2.0    2.0    2.0    2.0     2.0
legacy APIs: MP/MC: 0.3   0.4   1.3   1.9   2.9    2.9    2.9    2.9    2.9    2.9     2.9

According to above test data, when "iter_shift" is set as "15", the test run time is reduced to less than 1 minute and the test result can keep stable in x86 and aarch64 servers.

Fixes 1fa5d0099efc ("test/ring: add custom element size performance

tests")
Cc: [email protected]
Cc: [email protected]

Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 app/test/test_ring_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
app/test/test_ring_perf.c +0 -0
index e63e25a86..fd82e2041 100644
@@ -178,7 +178,7 @@ enqueue_dequeue_bulk_helper(const unsigned int

flag, const int esize,

app/test/test_ring_perf.c enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,

Quoted 9 lines
app/test/test_ring_perf.c +1 -2
 	struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 15;
 	const unsigned int iterations = 1 << iter_shift;
 	struct rte_ring *r = p->r;
 	unsigned int bsize = p->size;
--

I think it would be better to rework the test(s) to terminate after some timeout (30s or so), and report number of ops per timeout. Anyway, as a short term fix, I am ok with it.

Quoted 1 line
Acked-by Konstantin Ananyev <[email protected]>

Ok, thanks very much.

Best Regards
Feifei

Quoted 3 lines
Quoted 3 lines

2.17.1

author
#4 lore raw

PATCH 2/3 ring: add rte prefix before update tail API

In reply to Feifei Wang

Add __rte prefix before update_tail API because it is a internal function.

Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/rte_ring_c11_mem.h | 4 ++--
 lib/librte_ring/rte_ring_elem.h    | 4 ++--
 lib/librte_ring/rte_ring_generic.h | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_c11_mem.h +2 -2
index 0fb73a337..7f5eba262 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_C11_MEM_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	RTE_SET_USED(enqueue);
lib/librte_ring/rte_ring_elem.h +2 -2
index 7034d29c0..57344d47d 100644
@@ -423,7 +423,7 @@ __rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
 
 	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
 
-	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
 	if (free_space != NULL)
 		*free_space = free_entries - n;

@@ -470,7 +470,7 @@ __rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
 
 	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
 
-	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
 end:
 	if (available != NULL)
lib/librte_ring/rte_ring_generic.h +2 -2
index 953cdbbd5..37c62b8d6 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_GENERIC_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	if (enqueue)
 		rte_smp_wmb();
Signature · 1 line
2.17.1
Acked-by on v1 2/3
#5 lore raw

Re: [dpdk-dev] [PATCH v1 2/3] ring: add rte prefix before update tail API

Quoted 56 lines

Add __rte prefix before update_tail API because it is a internal function.

Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/rte_ring_c11_mem.h | 4 ++--
 lib/librte_ring/rte_ring_elem.h    | 4 ++--
 lib/librte_ring/rte_ring_generic.h | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_c11_mem.h +2 -2
index 0fb73a337..7f5eba262 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_C11_MEM_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	RTE_SET_USED(enqueue);
lib/librte_ring/rte_ring_elem.h +2 -2
index 7034d29c0..57344d47d 100644
@@ -423,7 +423,7 @@ __rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
 
 	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
 
-	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
 	if (free_space != NULL)
 		*free_space = free_entries - n;

@@ -470,7 +470,7 @@ __rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
 
 	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
 
-	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
 end:
 	if (available != NULL)
lib/librte_ring/rte_ring_generic.h +0 -0
index 953cdbbd5..37c62b8d6 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_GENERIC_H_

lib/librte_ring/rte_ring_generic.h

Quoted 10 lines
lib/librte_ring/rte_ring_generic.h +2 -3
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	if (enqueue)
 		rte_smp_wmb();
--
Acked-by Konstantin Ananyev <[email protected]>
Quoted 1 line

2.17.1

author
#6 lore raw

PATCH 3/3 ring: rename and refactor ring library

In reply to Feifei Wang

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |   9 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |   6 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
 ...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
 ...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 12 files changed, 646 insertions(+), 635 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (96%)
 create mode 100644 lib/librte_ring/ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (98%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h} (62%)
lib/librte_ring/meson.build +8 -7
index 36fdcb6a5..98eac5810 100644
@@ -2,15 +2,16 @@
 # Copyright(c) 2017 Intel Corporation
 
 sources = files('rte_ring.c')
-headers = files('rte_ring.h',
+headers = files('ring_c11_pvt.h',
+		'ring_elem_pvt.h',
+		'ring_generic_pvt.h',
+		'ring_hts_elem_pvt.h',
+		'ring_peek_elem_pvt.h',
+		'ring_rts_elem_pvt.h',
+		'rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
 		'rte_ring_peek_zc.h',
-		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts.h')
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +3 -6
similarity index 96%
rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/ring_c11_pvt.h
index 7f5eba262..9f2f5318f 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RING_C11_PVT_H_
+#define _RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 		/* Ensure the head is read before tail */
 		__atomic_thread_fence(__ATOMIC_ACQUIRE);
 
-		/* load-acquire synchronize with store-release of ht->tail
-		 * in update_tail.
-		 */
 		cons_tail = __atomic_load_n(&r->cons.tail,
 					__ATOMIC_ACQUIRE);
 

@@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RING_C11_PVT_H_ */
lib/librte_ring/ring_elem_pvt.h new file +385 -0

385 lines not on this page · raw message

lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +3 -3
similarity index 98%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/ring_generic_pvt.h
index 37c62b8d6..fc46a27b2 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RING_GENERIC_PVT_H_
+#define _RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RING_GENERIC_PVT_H_ */
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +85 -3

97 lines not on this page · raw message

lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +72 -3

84 lines not on this page · raw message

lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +85 -3

97 lines not on this page · raw message

lib/librte_ring/rte_ring_elem.h +1 -373

386 lines not on this page · raw message

lib/librte_ring/rte_ring_hts.h +1 -83

90 lines not on this page · raw message

lib/librte_ring/rte_ring_peek.h +1 -70

83 lines not on this page · raw message

lib/librte_ring/rte_ring_peek_zc.h +1 -1
index cb3bbd067..bc8252a18 100644
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
lib/librte_ring/rte_ring_rts.h +1 -83

90 lines not on this page · raw message

Signature · 1 line
2.17.1
#7 lore raw

Re: [dpdk-dev] [PATCH v1 3/3] ring: rename and refactor ring library

Quoted 24 lines

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |   9 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |   6 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
 ...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
 ...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
Quoted 9 lines
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 12 files changed, 646 insertions(+), 635 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (96%)
 create mode 100644 lib/librte_ring/ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (98%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h} (62%)

Sorry, but I don't understand the purpose of that patch. As I remember by DPDK naming convention all installable headers should have 'rte_' prefix. Same for public defines (RTE_). Why to abandon it here?

Quoted 1,468 lines
lib/librte_ring/meson.build +8 -7
index 36fdcb6a5..98eac5810 100644
@@ -2,15 +2,16 @@
 # Copyright(c) 2017 Intel Corporation
 
 sources = files('rte_ring.c')
-headers = files('rte_ring.h',
+headers = files('ring_c11_pvt.h',
+		'ring_elem_pvt.h',
+		'ring_generic_pvt.h',
+		'ring_hts_elem_pvt.h',
+		'ring_peek_elem_pvt.h',
+		'ring_rts_elem_pvt.h',
+		'rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
 		'rte_ring_peek_zc.h',
-		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts.h')
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +3 -6
similarity index 96%
rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/ring_c11_pvt.h
index 7f5eba262..9f2f5318f 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RING_C11_PVT_H_
+#define _RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct rte_ring *r, unsigned int is_sp,
 		/* Ensure the head is read before tail */
 		__atomic_thread_fence(__ATOMIC_ACQUIRE);
 
-		/* load-acquire synchronize with store-release of ht->tail
-		 * in update_tail.
-		 */
 		cons_tail = __atomic_load_n(&r->cons.tail,
 					__ATOMIC_ACQUIRE);
 

@@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RING_C11_PVT_H_ */
lib/librte_ring/ring_elem_pvt.h new file +385 -0
new file mode 100644
index 000000000..8003e5edc
@@ -0,0 +1,385 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2017,2018 HXT-semitech Corporation.
+ * Copyright (c) 2007-2009 Kip Macy [email protected]
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RING_ELEM_PVT_H_
+#define _RING_ELEM_PVT_H_
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	const uint32_t *obj = (const uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+			ring[idx + 4] = obj[i + 4];
+			ring[idx + 5] = obj[i + 5];
+			ring[idx + 6] = obj[i + 6];
+			ring[idx + 7] = obj[i + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 6:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 5:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 4:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	const unaligned_uint64_t *obj = (const unaligned_uint64_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++];
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+	}
+}
+
+/* the actual enqueue of elements on the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t esize, uint32_t num)
+{
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_enqueue_elems_64(r, prod_head, obj_table, num);
+	else if (esize == 16)
+		__rte_ring_enqueue_elems_128(r, prod_head, obj_table, num);
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = prod_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	uint32_t *obj = (uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+			obj[i + 4] = ring[idx + 4];
+			obj[i + 5] = ring[idx + 5];
+			obj[i + 6] = ring[idx + 6];
+			obj[i + 7] = ring[idx + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 6:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 5:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 4:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	rte_int128_t *obj = (rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+	}
+}
+
+/* the actual dequeue of elements from the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
+		void *obj_table, uint32_t esize, uint32_t num)
+{
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_dequeue_elems_64(r, cons_head, obj_table, num);
+	else if (esize == 16)
+		__rte_ring_dequeue_elems_128(r, cons_head, obj_table, num);
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = cons_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+/* Between load and load. there might be cpu reorder in weak model
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier
+ * It depends on performance test results.
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "ring_c11_pvt.h"
+#else
+#include "ring_generic_pvt.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head update
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
+		unsigned int *free_space)
+{
+	uint32_t prod_head, prod_next;
+	uint32_t free_entries;
+
+	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+			&prod_head, &prod_next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param is_sc
+ *   Indicates whether to use single consumer or multi-consumer head update
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
+		unsigned int *available)
+{
+	uint32_t cons_head, cons_next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+			&cons_head, &cons_next, &entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_ELEM_PVT_H_ */
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +3 -3
similarity index 98%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/ring_generic_pvt.h
index 37c62b8d6..fc46a27b2 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RING_GENERIC_PVT_H_
+#define _RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RING_GENERIC_PVT_H_ */
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +85 -3
similarity index 60%
rename from lib/librte_ring/rte_ring_hts_c11_mem.h
rename to lib/librte_ring/ring_hts_elem_pvt.h
index 16e54b6ff..9268750b0 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_HTS_C11_MEM_H_
-#define _RTE_RING_HTS_C11_MEM_H_
+#ifndef _RING_HTS_ELEM_PVT_H_
+#define _RING_HTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_hts_c11_mem.h

@@ -161,4 +161,86 @@ __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,
 	return n;
 }
 
-#endif /* _RTE_RING_HTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries);
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_HTS_ELEM_PVT_H_ */
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +72 -3
similarity index 62%
rename from lib/librte_ring/rte_ring_peek_c11_mem.h
rename to lib/librte_ring/ring_peek_elem_pvt.h
index 283c7e70b..1c57bcdd6 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_PEEK_C11_MEM_H_
-#define _RTE_RING_PEEK_C11_MEM_H_
+#ifndef _RING_PEEK_ELEM_PVT_H_
+#define _RING_PEEK_ELEM_PVT_H_
 
 /**
  * @file rte_ring_peek_c11_mem.h

@@ -107,4 +107,73 @@ __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,
 	__atomic_store_n(&ht->ht.raw, p.raw, __ATOMIC_RELEASE);
 }
 
-#endif /* _RTE_RING_PEEK_C11_MEM_H_ */
+/**
+ * @internal This function moves prod head value.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
+		enum rte_ring_queue_behavior behavior, uint32_t *free_space)
+{
+	uint32_t free, head, next;
+
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &free);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
+			&head, &free);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		free = 0;
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal This function moves cons head value and copies up to *n*
+ * objects from the ring to the user provided obj_table.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t avail, head, next;
+
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &avail);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
+			&head, &avail);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		avail = 0;
+	}
+
+	if (n != 0)
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+
+	if (available != NULL)
+		*available = avail - n;
+	return n;
+}
+
+#endif /* _RING_PEEK_ELEM_PVT_H_ */
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +85 -3
similarity index 62%
rename from lib/librte_ring/rte_ring_rts_c11_mem.h
rename to lib/librte_ring/ring_rts_elem_pvt.h
index 327f22796..cbcec73eb 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_RTS_C11_MEM_H_
-#define _RTE_RING_RTS_C11_MEM_H_
+#ifndef _RING_RTS_ELEM_PVT_H_
+#define _RING_RTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_rts_c11_mem.h

@@ -176,4 +176,86 @@ __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,
 	return n;
 }
 
-#endif /* _RTE_RING_RTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_prod);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has finished
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_cons);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_RTS_ELEM_PVT_H_ */
lib/librte_ring/rte_ring_elem.h +1 -373
index 57344d47d..119b5c0b6 100644
@@ -21,6 +21,7 @@ extern "C" {
 #endif
 
 #include <rte_ring_core.h>
+#include <ring_elem_pvt.h>
 
 /**
  * Calculate the memory size needed for a ring with given element size

@@ -105,379 +106,6 @@ ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);
 struct rte_ring *rte_ring_create_elem(const char *name, unsigned int esize,
 			unsigned int count, int socket_id, unsigned int flags);
 
-static __rte_always_inline void
-__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	const uint32_t *obj = (const uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-			ring[idx + 4] = obj[i + 4];
-			ring[idx + 5] = obj[i + 5];
-			ring[idx + 6] = obj[i + 6];
-			ring[idx + 7] = obj[i + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 6:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 5:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 4:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	const unaligned_uint64_t *obj = (const unaligned_uint64_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++];
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-	}
-}
-
-/* the actual enqueue of elements on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.
- */
-static __rte_always_inline void
-__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_enqueue_elems_64(r, prod_head, obj_table, num);
-	else if (esize == 16)
-		__rte_ring_enqueue_elems_128(r, prod_head, obj_table, num);
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = prod_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	uint32_t *obj = (uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-			obj[i + 4] = ring[idx + 4];
-			obj[i + 5] = ring[idx + 5];
-			obj[i + 6] = ring[idx + 6];
-			obj[i + 7] = ring[idx + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 6:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 5:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 4:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	rte_int128_t *obj = (rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-	}
-}
-
-/* the actual dequeue of elements from the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.
- */
-static __rte_always_inline void
-__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
-		void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_dequeue_elems_64(r, cons_head, obj_table, num);
-	else if (esize == 16)
-		__rte_ring_dequeue_elems_128(r, cons_head, obj_table, num);
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = cons_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-/* Between load and load. there might be cpu reorder in weak model
- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
- * @param is_sp
- *   Indicates whether to use single producer or multi-producer head update
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
-		unsigned int *free_space)
-{
-	uint32_t prod_head, prod_next;
-	uint32_t free_entries;
-
-	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
-			&prod_head, &prod_next, &free_entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
-end:
-	if (free_space != NULL)
-		*free_space = free_entries - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
- * @param is_sc
- *   Indicates whether to use single consumer or multi-consumer head update
- * @param available
- *   returns the number of remaining ring entries after the dequeue has finished
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
-		unsigned int *available)
-{
-	uint32_t cons_head, cons_next;
-	uint32_t entries;
-
-	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
-			&cons_head, &cons_next, &entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
-
-end:
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
-
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
lib/librte_ring/rte_ring_hts.h +1 -83
index 359b15771..bdbdafc9f 100644
@@ -29,89 +29,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_hts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
- * @param available
- *   returns the number of remaining ring entries after the dequeue has finished
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head, &entries);
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_hts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the HTS ring (multi-producers safe).
lib/librte_ring/rte_ring_peek.h +1 -70
index 45f707dc7..0dd402be4 100644
@@ -48,39 +48,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
-
-/**
- * @internal This function moves prod head value.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
-		enum rte_ring_queue_behavior behavior, uint32_t *free_space)
-{
-	uint32_t free, head, next;
-
-	switch (r->prod.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &free);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
-			&head, &free);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		free = 0;
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Start to enqueue several objects on the ring.

@@ -248,43 +216,6 @@ rte_ring_enqueue_finish(struct rte_ring *r, void * const *obj_table,
 	rte_ring_enqueue_elem_finish(r, obj_table, sizeof(uintptr_t), n);
 }
 
-/**
- * @internal This function moves cons head value and copies up to *n*
- * objects from the ring to the user provided obj_table.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t avail, head, next;
-
-	switch (r->cons.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &avail);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
-			&head, &avail);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		avail = 0;
-	}
-
-	if (n != 0)
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-
-	if (available != NULL)
-		*available = avail - n;
-	return n;
-}
-
 /**
  * Start to dequeue several objects from the ring.
  * Note that user has to call appropriate dequeue_finish()
lib/librte_ring/rte_ring_peek_zc.h +1 -1
index cb3bbd067..bc8252a18 100644
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
lib/librte_ring/rte_ring_rts.h +1 -84
index afc12abe2..83d9903e2 100644
@@ -56,89 +56,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_rts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a ring
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible from ring
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_prod);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a ring
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible from ring
- * @param available
- *   returns the number of remaining ring entries after the dequeue has finished
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head, &entries);
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_cons);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_rts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the RTS ring (multi-producers safe).
--

2.17.1

author
#8 lore raw

[dpdk-dev] 回复: [PATCH v1 3/3] ring: rename and refactor ring library

Hi, Konstantin

Quoted 43 lines

-----邮件原件-----
发件人: Ananyev, Konstantin <[email protected]> 发送时间: 2021年1月22日 21:09
收件人: Feifei Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>
抄送: [email protected]; nd <[email protected]>
主题: RE: [PATCH v1 3/3] ring: rename and refactor ring library

Quoted 25 lines

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |   9 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |   6 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 12 files changed, 646 insertions(+), 635 deletions(-)  rename
Quoted 8 lines

lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (96%) create mode 100644 lib/librte_ring/ring_elem_pvt.h rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (98%) rename lib/librte_ring/{rte_ring_hts_c11_mem.h => ring_hts_elem_pvt.h} (60%) rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%) rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h} (62%)

Quoted 4 lines

Sorry, but I don't understand the purpose of that patch.
As I remember by DPDK naming convention all installable headers should have 'rte_' prefix. Same for public defines (RTE_).
Why to abandon it here?

This change refers to you and Honnappa's discussion about ring:
https://mails.dpdk.org/archives/dev/2020-May/166803.html

And it is to separate the external APIs that can be called by users, from internal APIs that cannot be called by users.
The internal functions are included in the files with xx_pvt.h.

Best Regards
Feifei

Quoted 1,530 lines
Quoted 47 lines
lib/librte_ring/meson.build +8 -7
index 36fdcb6a5..98eac5810 100644
@@ -2,15 +2,16 @@
 # Copyright(c) 2017 Intel Corporation
 
 sources = files('rte_ring.c')
-headers = files('rte_ring.h',
+headers = files('ring_c11_pvt.h',
+		'ring_elem_pvt.h',
+		'ring_generic_pvt.h',
+		'ring_hts_elem_pvt.h',
+		'ring_peek_elem_pvt.h',
+		'ring_rts_elem_pvt.h',
+		'rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
 		'rte_ring_peek_zc.h',
-		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts.h')
(quoted diff) +0 -0

b/lib/librte_ring/ring_c11_pvt.h similarity index 96% rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/ring_c11_pvt.h index 7f5eba262..9f2f5318f 100644

lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +2 -2
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RING_C11_PVT_H_
+#define _RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct rte_ring

*r, unsigned int is_sp,

lib/librte_ring/ring_c11_pvt.h __rte_ring_move_cons_head(struct rte_ring *r, int

Quoted 10 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -3
 		/* Ensure the head is read before tail */
 		__atomic_thread_fence(__ATOMIC_ACQUIRE);
 
-		/* load-acquire synchronize with store-release of ht->tail
-		 * in update_tail.
-		 */
 		cons_tail = __atomic_load_n(&r->cons.tail,
 					__ATOMIC_ACQUIRE);

@@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int

is_sc,

Quoted 64 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RING_C11_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/ring_elem_pvt.h new file mode 100644 index 000000000..8003e5edc

lib/librte_ring/ring_elem_pvt.h new file +53 -0
@@ -0,0 +1,385 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2017,2018 HXT-semitech Corporation.
+ * Copyright (c) 2007-2009 Kip Macy [email protected]
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RING_ELEM_PVT_H_
+#define _RING_ELEM_PVT_H_
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, const void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	const uint32_t *obj = (const uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+			ring[idx + 4] = obj[i + 4];
+			ring[idx + 5] = obj[i + 5];
+			ring[idx + 6] = obj[i + 6];
+			ring[idx + 7] = obj[i + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 6:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 5:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 4:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}

lib/librte_ring/ring_elem_pvt.h line 63

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 55 lines
lib/librte_ring/ring_elem_pvt.h new file +55 -0
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++];
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+	}
+}
+
+/* the actual enqueue of elements on the ring.
+ * Placed here since identical code needed in both

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_enqueue_elems_64(r, prod_head, obj_table,

num);

lib/librte_ring/ring_elem_pvt.h line 130

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_enqueue_elems_128(r, prod_head, obj_table,

num);

Quoted 116 lines
lib/librte_ring/ring_elem_pvt.h new file +116 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = prod_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	uint32_t *obj = (uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+			obj[i + 4] = ring[idx + 4];
+			obj[i + 5] = ring[idx + 5];
+			obj[i + 6] = ring[idx + 6];
+			obj[i + 7] = ring[idx + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 6:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 5:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 4:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	rte_int128_t *obj = (rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+	}
+}
+
+/* the actual dequeue of elements from the ring.
+ * Placed here since identical code needed in both

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
+		void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_dequeue_elems_64(r, cons_head, obj_table,

num);

lib/librte_ring/ring_elem_pvt.h line 258

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_dequeue_elems_128(r, cons_head, obj_table,

num);

Quoted 32 lines
lib/librte_ring/ring_elem_pvt.h new file +32 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = cons_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+/* Between load and load. there might be cpu reorder in weak model
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier
+ * It depends on performance test results.
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "ring_c11_pvt.h"
+#else
+#include "ring_generic_pvt.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_elem_pvt.h line 301

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_elem_pvt.h line 303

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head

update

Quoted 34 lines
lib/librte_ring/ring_elem_pvt.h new file +34 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
+		unsigned int *free_space)
+{
+	uint32_t prod_head, prod_next;
+	uint32_t free_entries;
+
+	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+			&prod_head, &prod_next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_elem_pvt.h line 348

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_elem_pvt.h line 350

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sc
+ *   Indicates whether to use single consumer or multi-consumer head

update

lib/librte_ring/ring_elem_pvt.h line 352

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

Quoted 48 lines
lib/librte_ring/ring_elem_pvt.h new file +29 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
+		unsigned int *available)
+{
+	uint32_t cons_head, cons_next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+			&cons_head, &cons_next, &entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_ELEM_PVT_H_ */
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +2 -2
similarity index 98%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/ring_generic_pvt.h
index 37c62b8d6..fc46a27b2 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RING_GENERIC_PVT_H_
+#define _RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct

rte_ring *r, unsigned int is_sc,

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring

Quoted 25 lines
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RING_GENERIC_PVT_H_ */
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -2
similarity index 60%
rename from lib/librte_ring/rte_ring_hts_c11_mem.h
rename to lib/librte_ring/ring_hts_elem_pvt.h
index 16e54b6ff..9268750b0 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_HTS_C11_MEM_H_
-#define _RTE_RING_HTS_C11_MEM_H_
+#ifndef _RING_HTS_ELEM_PVT_H_
+#define _RING_HTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_hts_c11_mem.h

@@ -161,4 +161,86 @@ __rte_ring_hts_move_cons_head(struct rte_ring

*r, unsigned int num,

Quoted 9 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -1
 	return n;
 }
 
-#endif /* _RTE_RING_HTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 10 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 179

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 187

Quoted 8 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +8 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +22 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 10 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 220

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 222

Quoted 2 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 12 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +12 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct

Quoted 32 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +12 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_HTS_ELEM_PVT_H_ */
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_peek_c11_mem.h
rename to lib/librte_ring/ring_peek_elem_pvt.h
index 283c7e70b..1c57bcdd6 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_PEEK_C11_MEM_H_
-#define _RTE_RING_PEEK_C11_MEM_H_
+#ifndef _RING_PEEK_ELEM_PVT_H_
+#define _RING_PEEK_ELEM_PVT_H_
 
 /**
  * @file rte_ring_peek_c11_mem.h

@@ -107,4 +107,73 @@ __rte_ring_hts_set_head_tail(struct

rte_ring_hts_headtail *ht, uint32_t tail,

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 9 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +6 -1
 	__atomic_store_n(&ht->ht.raw, p.raw, __ATOMIC_RELEASE);  }
 
-#endif /* _RTE_RING_PEEK_C11_MEM_H_ */
+/**
+ * @internal This function moves prod head value.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
+		enum rte_ring_queue_behavior behavior, uint32_t

*free_space) {

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring

Quoted 83 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +63 -0
+	uint32_t free, head, next;
+
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &free);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
+			&head, &free);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		free = 0;
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal This function moves cons head value and copies up to *n*
+ * objects from the ring to the user provided obj_table.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t avail, head, next;
+
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &avail);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
+			&head, &avail);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		avail = 0;
+	}
+
+	if (n != 0)
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+
+	if (available != NULL)
+		*available = avail - n;
+	return n;
+}
+
+#endif /* _RING_PEEK_ELEM_PVT_H_ */
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_rts_c11_mem.h
rename to lib/librte_ring/ring_rts_elem_pvt.h
index 327f22796..cbcec73eb 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_RTS_C11_MEM_H_
-#define _RTE_RING_RTS_C11_MEM_H_
+#ifndef _RING_RTS_ELEM_PVT_H_
+#define _RING_RTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_rts_c11_mem.h

@@ -176,4 +176,86 @@ __rte_ring_rts_move_cons_head(struct rte_ring

*r, uint32_t num,

Quoted 9 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -1
 	return n;
 }
 
-#endif /* _RTE_RING_RTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 194

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 202

Quoted 8 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +8 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +22 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_prod);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 235

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 237

Quoted 2 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 12 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +12 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 24 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +12 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_cons);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_RTS_ELEM_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_elem.h index 57344d47d..119b5c0b6 100644

lib/librte_ring/rte_ring_elem.h +1 -0
@@ -21,6 +21,7 @@ extern "C" {
 #endif
 
 #include <rte_ring_core.h>
+#include <ring_elem_pvt.h>
 
 /**
  * Calculate the memory size needed for a ring with given element
Quoted 1 line

size @@ -105,379 +106,6 @@ ssize_t

rte_ring_get_memsize_elem(unsigned

Quoted 1 line

int esize, unsigned int count); struct rte_ring *rte_ring_create_elem(const

char *name, unsigned int esize,

Quoted 44 lines
lib/librte_ring/rte_ring_elem.h +0 -42
 			unsigned int count, int socket_id, unsigned int flags);
 
-static __rte_always_inline void
-__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	const uint32_t *obj = (const uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-			ring[idx + 4] = obj[i + 4];
-			ring[idx + 5] = obj[i + 5];
-			ring[idx + 6] = obj[i + 6];
-			ring[idx + 7] = obj[i + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 6:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 5:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 4:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 56 lines
lib/librte_ring/rte_ring_elem.h +0 -56
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++];
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-	}
-}
-
-/* the actual enqueue of elements on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- */
-static __rte_always_inline void
-__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_enqueue_elems_64(r, prod_head, obj_table,

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_enqueue_elems_128(r, prod_head, obj_table,

num);

Quoted 118 lines
lib/librte_ring/rte_ring_elem.h +0 -118
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = prod_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	uint32_t *obj = (uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-			obj[i + 4] = ring[idx + 4];
-			obj[i + 5] = ring[idx + 5];
-			obj[i + 6] = ring[idx + 6];
-			obj[i + 7] = ring[idx + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 6:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 5:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 4:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	rte_int128_t *obj = (rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-	}
-}
-
-/* the actual dequeue of elements from the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- */
-static __rte_always_inline void
-__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
-		void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_dequeue_elems_64(r, cons_head, obj_table,

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_dequeue_elems_128(r, cons_head, obj_table,

num);

Quoted 33 lines
lib/librte_ring/rte_ring_elem.h +0 -33
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = cons_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-/* Between load and load. there might be cpu reorder in weak model
- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

Quoted 2 lines

- * @param is_sp
- * Indicates whether to use single producer or multi-producer head

update

Quoted 34 lines

- * @param free_space
- * returns the amount of space after the enqueue operation has finished
- * @return
- * Actual number of objects enqueued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -28
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
-		unsigned int *free_space)
-{
-	uint32_t prod_head, prod_next;
-	uint32_t free_entries;
-
-	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
-			&prod_head, &prod_next, &free_entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
-end:
-	if (free_space != NULL)
-		*free_space = free_entries - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

Quoted 2 lines

- * @param is_sc
- * Indicates whether to use single consumer or multi-consumer head

update

Quoted 2 lines

- * @param available
- * returns the number of remaining ring entries after the dequeue has

finished

Quoted 46 lines

- * @return
- * - Actual number of objects dequeued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -24
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
-		unsigned int *available)
-{
-	uint32_t cons_head, cons_next;
-	uint32_t entries;
-
-	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
-			&cons_head, &cons_next, &entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
-
-end:
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
-
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_hts.h index 359b15771..bdbdafc9f 100644

lib/librte_ring/rte_ring_hts.h +0 -7
@@ -29,89 +29,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_hts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_hts.h line 48

Quoted 10 lines
lib/librte_ring/rte_ring_hts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/rte_ring_hts.h line 49

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/rte_ring_hts.h line 57

Quoted 8 lines
lib/librte_ring/rte_ring_hts.h +0 -8
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/rte_ring_hts.h +0 -22
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_hts.h

Quoted 10 lines
lib/librte_ring/rte_ring_hts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/rte_ring_hts.h line 90

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/rte_ring_hts.h line 92

Quoted 2 lines
lib/librte_ring/rte_ring_hts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/rte_ring_hts.h

Quoted 12 lines
lib/librte_ring/rte_ring_hts.h +0 -12
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 20 lines
lib/librte_ring/rte_ring_hts.h +1 -10
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_hts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the HTS ring (multi-producers safe).
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_peek.h index 45f707dc7..0dd402be4 100644

lib/librte_ring/rte_ring_peek.h +0 -0
@@ -48,39 +48,7 @@
 extern "C" {

lib/librte_ring/rte_ring_peek.h line 58

Quoted 10 lines
lib/librte_ring/rte_ring_peek.h +0 -8
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
-
-/**
- * @internal This function moves prod head value.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
-		enum rte_ring_queue_behavior behavior, uint32_t

*free_space)

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, void *

Quoted 30 lines
lib/librte_ring/rte_ring_peek.h +1 -25
-{
-	uint32_t free, head, next;
-
-	switch (r->prod.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &free);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
-			&head, &free);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		free = 0;
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Start to enqueue several objects on the ring.

@@ -248,43 +216,6 @@ rte_ring_enqueue_finish(struct rte_ring *r, void *

const *obj_table,

Quoted 71 lines
lib/librte_ring/rte_ring_peek.h +0 -37
 	rte_ring_enqueue_elem_finish(r, obj_table, sizeof(uintptr_t), n);  }
 
-/**
- * @internal This function moves cons head value and copies up to *n*
- * objects from the ring to the user provided obj_table.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t avail, head, next;
-
-	switch (r->cons.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &avail);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
-			&head, &avail);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		avail = 0;
-	}
-
-	if (n != 0)
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-
-	if (available != NULL)
-		*available = avail - n;
-	return n;
-}
-
 /**
  * Start to dequeue several objects from the ring.
  * Note that user has to call appropriate dequeue_finish() diff --git

a/lib/librte_ring/rte_ring_peek_zc.h
b/lib/librte_ring/rte_ring_peek_zc.h
index cb3bbd067..bc8252a18 100644

lib/librte_ring/rte_ring_peek_zc.h +1 -1
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_rts.h index afc12abe2..83d9903e2 100644

lib/librte_ring/rte_ring_rts.h +0 -7
@@ -56,89 +56,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_rts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_rts.h line 75

Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/rte_ring_rts.h line 76

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/rte_ring_rts.h line 84

Quoted 8 lines
lib/librte_ring/rte_ring_rts.h +0 -8
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/rte_ring_rts.h +0 -22
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_prod);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_rts.h

Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/rte_ring_rts.h line 117

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/rte_ring_rts.h line 119

Quoted 2 lines
lib/librte_ring/rte_ring_rts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/rte_ring_rts.h

Quoted 12 lines
lib/librte_ring/rte_ring_rts.h +0 -12
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 16 lines
lib/librte_ring/rte_ring_rts.h +1 -11
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_cons);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_rts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the RTS ring (multi-producers safe).
--

2.17.1

#9 lore raw

Re: [dpdk-dev] [PATCH v1 3/3] ring: rename and refactor ring library

Hi Feifei,

Quoted 39 lines
Quoted 34 lines
Quoted 25 lines

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |   9 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |   6 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 12 files changed, 646 insertions(+), 635 deletions(-)  rename
Quoted 8 lines

lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (96%) create mode 100644 lib/librte_ring/ring_elem_pvt.h rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (98%) rename lib/librte_ring/{rte_ring_hts_c11_mem.h => ring_hts_elem_pvt.h} (60%) rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%) rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h} (62%)

Quoted 4 lines

Sorry, but I don't understand the purpose of that patch.
As I remember by DPDK naming convention all installable headers should have 'rte_' prefix. Same for public defines (RTE_).
Why to abandon it here?

Quoted 6 lines

This change refers to you and Honnappa's discussion about ring:
https://mails.dpdk.org/archives/dev/2020-May/166803.html

And it is to separate the external APIs that can be called by users, from internal APIs that cannot be called by users.
The internal functions are included in the files with xx_pvt.h.

Even these xx_pvt.h files contain internal API and not supposed to be included directly, they have to be installed into public dir (<dpdk-install-dir/include).
So I think we need to keep DPDK public prefixes for them (and inside them):
rte_xx_pvt.h (_RTE_).

Konstantin

Quoted 1,533 lines

Best Regards
Feifei

Quoted 1,530 lines
Quoted 47 lines
lib/librte_ring/meson.build +8 -7
index 36fdcb6a5..98eac5810 100644
@@ -2,15 +2,16 @@
 # Copyright(c) 2017 Intel Corporation
 
 sources = files('rte_ring.c')
-headers = files('rte_ring.h',
+headers = files('ring_c11_pvt.h',
+		'ring_elem_pvt.h',
+		'ring_generic_pvt.h',
+		'ring_hts_elem_pvt.h',
+		'ring_peek_elem_pvt.h',
+		'ring_rts_elem_pvt.h',
+		'rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
 		'rte_ring_peek_zc.h',
-		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts.h')
(quoted diff) +0 -0

b/lib/librte_ring/ring_c11_pvt.h similarity index 96% rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/ring_c11_pvt.h index 7f5eba262..9f2f5318f 100644

lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +2 -2
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RING_C11_PVT_H_
+#define _RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct rte_ring

*r, unsigned int is_sp,

lib/librte_ring/ring_c11_pvt.h __rte_ring_move_cons_head(struct rte_ring *r, int

Quoted 10 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -3
 		/* Ensure the head is read before tail */
 		__atomic_thread_fence(__ATOMIC_ACQUIRE);
 
-		/* load-acquire synchronize with store-release of ht->tail
-		 * in update_tail.
-		 */
 		cons_tail = __atomic_load_n(&r->cons.tail,
 					__ATOMIC_ACQUIRE);

@@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int

is_sc,

Quoted 64 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RING_C11_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/ring_elem_pvt.h new file mode 100644 index 000000000..8003e5edc

lib/librte_ring/ring_elem_pvt.h new file +53 -0
@@ -0,0 +1,385 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2017,2018 HXT-semitech Corporation.
+ * Copyright (c) 2007-2009 Kip Macy [email protected]
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RING_ELEM_PVT_H_
+#define _RING_ELEM_PVT_H_
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, const void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	const uint32_t *obj = (const uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+			ring[idx + 4] = obj[i + 4];
+			ring[idx + 5] = obj[i + 5];
+			ring[idx + 6] = obj[i + 6];
+			ring[idx + 7] = obj[i + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 6:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 5:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 4:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}

lib/librte_ring/ring_elem_pvt.h line 63

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 55 lines
lib/librte_ring/ring_elem_pvt.h new file +55 -0
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++];
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+	}
+}
+
+/* the actual enqueue of elements on the ring.
+ * Placed here since identical code needed in both

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_enqueue_elems_64(r, prod_head, obj_table,

num);

lib/librte_ring/ring_elem_pvt.h line 130

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_enqueue_elems_128(r, prod_head, obj_table,

num);

Quoted 116 lines
lib/librte_ring/ring_elem_pvt.h new file +116 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = prod_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	uint32_t *obj = (uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+			obj[i + 4] = ring[idx + 4];
+			obj[i + 5] = ring[idx + 5];
+			obj[i + 6] = ring[idx + 6];
+			obj[i + 7] = ring[idx + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 6:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 5:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 4:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	rte_int128_t *obj = (rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+	}
+}
+
+/* the actual dequeue of elements from the ring.
+ * Placed here since identical code needed in both

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
+		void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_dequeue_elems_64(r, cons_head, obj_table,

num);

lib/librte_ring/ring_elem_pvt.h line 258

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_dequeue_elems_128(r, cons_head, obj_table,

num);

Quoted 32 lines
lib/librte_ring/ring_elem_pvt.h new file +32 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = cons_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+/* Between load and load. there might be cpu reorder in weak model
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier
+ * It depends on performance test results.
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "ring_c11_pvt.h"
+#else
+#include "ring_generic_pvt.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_elem_pvt.h line 301

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_elem_pvt.h line 303

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head

update

Quoted 34 lines
lib/librte_ring/ring_elem_pvt.h new file +34 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
+		unsigned int *free_space)
+{
+	uint32_t prod_head, prod_next;
+	uint32_t free_entries;
+
+	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+			&prod_head, &prod_next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_elem_pvt.h line 348

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_elem_pvt.h line 350

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sc
+ *   Indicates whether to use single consumer or multi-consumer head

update

lib/librte_ring/ring_elem_pvt.h line 352

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

Quoted 48 lines
lib/librte_ring/ring_elem_pvt.h new file +29 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
+		unsigned int *available)
+{
+	uint32_t cons_head, cons_next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+			&cons_head, &cons_next, &entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_ELEM_PVT_H_ */
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +2 -2
similarity index 98%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/ring_generic_pvt.h
index 37c62b8d6..fc46a27b2 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RING_GENERIC_PVT_H_
+#define _RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct

rte_ring *r, unsigned int is_sc,

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring

Quoted 25 lines
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RING_GENERIC_PVT_H_ */
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -2
similarity index 60%
rename from lib/librte_ring/rte_ring_hts_c11_mem.h
rename to lib/librte_ring/ring_hts_elem_pvt.h
index 16e54b6ff..9268750b0 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_HTS_C11_MEM_H_
-#define _RTE_RING_HTS_C11_MEM_H_
+#ifndef _RING_HTS_ELEM_PVT_H_
+#define _RING_HTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_hts_c11_mem.h

@@ -161,4 +161,86 @@ __rte_ring_hts_move_cons_head(struct rte_ring

*r, unsigned int num,

Quoted 9 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -1
 	return n;
 }
 
-#endif /* _RTE_RING_HTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 10 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 179

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 187

Quoted 8 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +8 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +22 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 10 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 220

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 222

Quoted 2 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 12 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +12 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct

Quoted 32 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +12 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_HTS_ELEM_PVT_H_ */
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_peek_c11_mem.h
rename to lib/librte_ring/ring_peek_elem_pvt.h
index 283c7e70b..1c57bcdd6 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_PEEK_C11_MEM_H_
-#define _RTE_RING_PEEK_C11_MEM_H_
+#ifndef _RING_PEEK_ELEM_PVT_H_
+#define _RING_PEEK_ELEM_PVT_H_
 
 /**
  * @file rte_ring_peek_c11_mem.h

@@ -107,4 +107,73 @@ __rte_ring_hts_set_head_tail(struct

rte_ring_hts_headtail *ht, uint32_t tail,

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 9 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +6 -1
 	__atomic_store_n(&ht->ht.raw, p.raw, __ATOMIC_RELEASE);  }
 
-#endif /* _RTE_RING_PEEK_C11_MEM_H_ */
+/**
+ * @internal This function moves prod head value.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
+		enum rte_ring_queue_behavior behavior, uint32_t

*free_space) {

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring

Quoted 83 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +63 -0
+	uint32_t free, head, next;
+
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &free);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
+			&head, &free);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		free = 0;
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal This function moves cons head value and copies up to *n*
+ * objects from the ring to the user provided obj_table.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t avail, head, next;
+
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &avail);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
+			&head, &avail);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		avail = 0;
+	}
+
+	if (n != 0)
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+
+	if (available != NULL)
+		*available = avail - n;
+	return n;
+}
+
+#endif /* _RING_PEEK_ELEM_PVT_H_ */
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_rts_c11_mem.h
rename to lib/librte_ring/ring_rts_elem_pvt.h
index 327f22796..cbcec73eb 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_RTS_C11_MEM_H_
-#define _RTE_RING_RTS_C11_MEM_H_
+#ifndef _RING_RTS_ELEM_PVT_H_
+#define _RING_RTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_rts_c11_mem.h

@@ -176,4 +176,86 @@ __rte_ring_rts_move_cons_head(struct rte_ring

*r, uint32_t num,

Quoted 9 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -1
 	return n;
 }
 
-#endif /* _RTE_RING_RTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 194

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 202

Quoted 8 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +8 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +22 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_prod);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 235

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 237

Quoted 2 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 12 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +12 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 24 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +12 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_cons);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_RTS_ELEM_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_elem.h index 57344d47d..119b5c0b6 100644

lib/librte_ring/rte_ring_elem.h +1 -0
@@ -21,6 +21,7 @@ extern "C" {
 #endif
 
 #include <rte_ring_core.h>
+#include <ring_elem_pvt.h>
 
 /**
  * Calculate the memory size needed for a ring with given element
Quoted 1 line

size @@ -105,379 +106,6 @@ ssize_t

rte_ring_get_memsize_elem(unsigned

Quoted 1 line

int esize, unsigned int count); struct rte_ring *rte_ring_create_elem(const

char *name, unsigned int esize,

Quoted 44 lines
lib/librte_ring/rte_ring_elem.h +0 -42
 			unsigned int count, int socket_id, unsigned int flags);
 
-static __rte_always_inline void
-__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	const uint32_t *obj = (const uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-			ring[idx + 4] = obj[i + 4];
-			ring[idx + 5] = obj[i + 5];
-			ring[idx + 6] = obj[i + 6];
-			ring[idx + 7] = obj[i + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 6:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 5:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 4:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 56 lines
lib/librte_ring/rte_ring_elem.h +0 -56
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++];
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-	}
-}
-
-/* the actual enqueue of elements on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- */
-static __rte_always_inline void
-__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_enqueue_elems_64(r, prod_head, obj_table,

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_enqueue_elems_128(r, prod_head, obj_table,

num);

Quoted 118 lines
lib/librte_ring/rte_ring_elem.h +0 -118
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = prod_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	uint32_t *obj = (uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-			obj[i + 4] = ring[idx + 4];
-			obj[i + 5] = ring[idx + 5];
-			obj[i + 6] = ring[idx + 6];
-			obj[i + 7] = ring[idx + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 6:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 5:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 4:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	rte_int128_t *obj = (rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-	}
-}
-
-/* the actual dequeue of elements from the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- */
-static __rte_always_inline void
-__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
-		void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_dequeue_elems_64(r, cons_head, obj_table,

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_dequeue_elems_128(r, cons_head, obj_table,

num);

Quoted 33 lines
lib/librte_ring/rte_ring_elem.h +0 -33
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = cons_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-/* Between load and load. there might be cpu reorder in weak model
- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

Quoted 2 lines

- * @param is_sp
- * Indicates whether to use single producer or multi-producer head

update

Quoted 34 lines

- * @param free_space
- * returns the amount of space after the enqueue operation has finished
- * @return
- * Actual number of objects enqueued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -28
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
-		unsigned int *free_space)
-{
-	uint32_t prod_head, prod_next;
-	uint32_t free_entries;
-
-	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
-			&prod_head, &prod_next, &free_entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
-end:
-	if (free_space != NULL)
-		*free_space = free_entries - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

Quoted 2 lines

- * @param is_sc
- * Indicates whether to use single consumer or multi-consumer head

update

Quoted 2 lines

- * @param available
- * returns the number of remaining ring entries after the dequeue has

finished

Quoted 46 lines

- * @return
- * - Actual number of objects dequeued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -24
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
-		unsigned int *available)
-{
-	uint32_t cons_head, cons_next;
-	uint32_t entries;
-
-	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
-			&cons_head, &cons_next, &entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
-
-end:
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
-
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_hts.h index 359b15771..bdbdafc9f 100644

lib/librte_ring/rte_ring_hts.h +0 -7
@@ -29,89 +29,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_hts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_hts.h line 48

Quoted 10 lines
lib/librte_ring/rte_ring_hts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/rte_ring_hts.h line 49

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/rte_ring_hts.h line 57

Quoted 8 lines
lib/librte_ring/rte_ring_hts.h +0 -8
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/rte_ring_hts.h +0 -22
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_hts.h

Quoted 10 lines
lib/librte_ring/rte_ring_hts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/rte_ring_hts.h line 90

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/rte_ring_hts.h line 92

Quoted 2 lines
lib/librte_ring/rte_ring_hts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/rte_ring_hts.h

Quoted 12 lines
lib/librte_ring/rte_ring_hts.h +0 -12
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 20 lines
lib/librte_ring/rte_ring_hts.h +1 -10
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_hts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the HTS ring (multi-producers safe).
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_peek.h index 45f707dc7..0dd402be4 100644

lib/librte_ring/rte_ring_peek.h +0 -0
@@ -48,39 +48,7 @@
 extern "C" {

lib/librte_ring/rte_ring_peek.h line 58

Quoted 10 lines
lib/librte_ring/rte_ring_peek.h +0 -8
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
-
-/**
- * @internal This function moves prod head value.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
-		enum rte_ring_queue_behavior behavior, uint32_t

*free_space)

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, void *

Quoted 30 lines
lib/librte_ring/rte_ring_peek.h +1 -25
-{
-	uint32_t free, head, next;
-
-	switch (r->prod.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &free);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
-			&head, &free);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		free = 0;
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Start to enqueue several objects on the ring.

@@ -248,43 +216,6 @@ rte_ring_enqueue_finish(struct rte_ring *r, void *

const *obj_table,

Quoted 71 lines
lib/librte_ring/rte_ring_peek.h +0 -37
 	rte_ring_enqueue_elem_finish(r, obj_table, sizeof(uintptr_t), n);  }
 
-/**
- * @internal This function moves cons head value and copies up to *n*
- * objects from the ring to the user provided obj_table.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t avail, head, next;
-
-	switch (r->cons.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &avail);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
-			&head, &avail);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		avail = 0;
-	}
-
-	if (n != 0)
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-
-	if (available != NULL)
-		*available = avail - n;
-	return n;
-}
-
 /**
  * Start to dequeue several objects from the ring.
  * Note that user has to call appropriate dequeue_finish() diff --git

a/lib/librte_ring/rte_ring_peek_zc.h
b/lib/librte_ring/rte_ring_peek_zc.h
index cb3bbd067..bc8252a18 100644

lib/librte_ring/rte_ring_peek_zc.h +1 -1
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_rts.h index afc12abe2..83d9903e2 100644

lib/librte_ring/rte_ring_rts.h +0 -7
@@ -56,89 +56,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_rts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_rts.h line 75

Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/rte_ring_rts.h line 76

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/rte_ring_rts.h line 84

Quoted 8 lines
lib/librte_ring/rte_ring_rts.h +0 -8
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/rte_ring_rts.h +0 -22
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_prod);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_rts.h

Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/rte_ring_rts.h line 117

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/rte_ring_rts.h line 119

Quoted 2 lines
lib/librte_ring/rte_ring_rts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/rte_ring_rts.h

Quoted 12 lines
lib/librte_ring/rte_ring_rts.h +0 -12
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 16 lines
lib/librte_ring/rte_ring_rts.h +1 -11
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_cons);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_rts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the RTS ring (multi-producers safe).
--

2.17.1

#10 lore raw

Re: [dpdk-dev] [PATCH v1 3/3] ring: rename and refactor ring library

Quoted 47 lines

Hi Feifei,

Quoted 42 lines
Quoted 34 lines
Quoted 25 lines

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |   9 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |   6 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 12 files changed, 646 insertions(+), 635 deletions(-)  rename
Quoted 8 lines

lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (96%) create mode 100644 lib/librte_ring/ring_elem_pvt.h rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (98%) rename lib/librte_ring/{rte_ring_hts_c11_mem.h => ring_hts_elem_pvt.h} (60%) rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%) rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h} (62%)

Quoted 4 lines

Sorry, but I don't understand the purpose of that patch.
As I remember by DPDK naming convention all installable headers should have 'rte_' prefix. Same for public defines (RTE_).
Why to abandon it here?

This change refers to you and Honnappa's discussion about ring:
https://mails.dpdk.org/archives/dev/2020-May/166803.html

Quoted 8 lines
Quoted 3 lines

And it is to separate the external APIs that can be called by users, from internal APIs that cannot be called by users.
The internal functions are included in the files with xx_pvt.h.

Even these xx_pvt.h files contain internal API and not supposed to be included directly, they have to be installed into public dir (<dpdk-install-dir/include).
So I think we need to keep DPDK public prefixes for them (and inside them):
rte_xx_pvt.h (_RTE_).

As another thought - it is probably good to introduce new generic naming convention for such sort of files (internal ones that still have to be exposed). Might be '_rte_' prefix or so...
Though I suppose it would require broader discussion first.

Quoted 1,536 lines

Konstantin

Quoted 1,533 lines

Best Regards
Feifei

Quoted 1,530 lines
Quoted 47 lines
lib/librte_ring/meson.build +8 -7
index 36fdcb6a5..98eac5810 100644
@@ -2,15 +2,16 @@
 # Copyright(c) 2017 Intel Corporation
 
 sources = files('rte_ring.c')
-headers = files('rte_ring.h',
+headers = files('ring_c11_pvt.h',
+		'ring_elem_pvt.h',
+		'ring_generic_pvt.h',
+		'ring_hts_elem_pvt.h',
+		'ring_peek_elem_pvt.h',
+		'ring_rts_elem_pvt.h',
+		'rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
 		'rte_ring_peek_zc.h',
-		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts.h')
(quoted diff) +0 -0

b/lib/librte_ring/ring_c11_pvt.h similarity index 96% rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/ring_c11_pvt.h index 7f5eba262..9f2f5318f 100644

lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +2 -2
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RING_C11_PVT_H_
+#define _RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct rte_ring

*r, unsigned int is_sp,

lib/librte_ring/ring_c11_pvt.h __rte_ring_move_cons_head(struct rte_ring *r, int

Quoted 10 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -3
 		/* Ensure the head is read before tail */
 		__atomic_thread_fence(__ATOMIC_ACQUIRE);
 
-		/* load-acquire synchronize with store-release of ht->tail
-		 * in update_tail.
-		 */
 		cons_tail = __atomic_load_n(&r->cons.tail,
 					__ATOMIC_ACQUIRE);

@@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int

is_sc,

Quoted 64 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RING_C11_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/ring_elem_pvt.h new file mode 100644 index 000000000..8003e5edc

lib/librte_ring/ring_elem_pvt.h new file +53 -0
@@ -0,0 +1,385 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2017,2018 HXT-semitech Corporation.
+ * Copyright (c) 2007-2009 Kip Macy [email protected]
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RING_ELEM_PVT_H_
+#define _RING_ELEM_PVT_H_
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, const void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	const uint32_t *obj = (const uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+			ring[idx + 4] = obj[i + 4];
+			ring[idx + 5] = obj[i + 5];
+			ring[idx + 6] = obj[i + 6];
+			ring[idx + 7] = obj[i + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 6:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 5:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 4:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}

lib/librte_ring/ring_elem_pvt.h line 63

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 55 lines
lib/librte_ring/ring_elem_pvt.h new file +55 -0
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++];
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+	}
+}
+
+/* the actual enqueue of elements on the ring.
+ * Placed here since identical code needed in both

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_enqueue_elems_64(r, prod_head, obj_table,

num);

lib/librte_ring/ring_elem_pvt.h line 130

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_enqueue_elems_128(r, prod_head, obj_table,

num);

Quoted 116 lines
lib/librte_ring/ring_elem_pvt.h new file +116 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = prod_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
+		uint32_t idx, void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	uint32_t *obj = (uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+			obj[i + 4] = ring[idx + 4];
+			obj[i + 5] = ring[idx + 5];
+			obj[i + 6] = ring[idx + 6];
+			obj[i + 7] = ring[idx + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 6:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 5:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 4:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	rte_int128_t *obj = (rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
+	}
+}
+
+/* the actual dequeue of elements from the ring.
+ * Placed here since identical code needed in both

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * single and multi producer enqueue functions.
+ */
+static __rte_always_inline void
+__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
+		void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_dequeue_elems_64(r, cons_head, obj_table,

num);

lib/librte_ring/ring_elem_pvt.h line 258

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_dequeue_elems_128(r, cons_head, obj_table,

num);

Quoted 32 lines
lib/librte_ring/ring_elem_pvt.h new file +32 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;
+		idx = cons_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+/* Between load and load. there might be cpu reorder in weak model
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier
+ * It depends on performance test results.
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "ring_c11_pvt.h"
+#else
+#include "ring_generic_pvt.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_elem_pvt.h line 301

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_elem_pvt.h line 303

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head

update

Quoted 34 lines
lib/librte_ring/ring_elem_pvt.h new file +34 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
+		unsigned int *free_space)
+{
+	uint32_t prod_head, prod_next;
+	uint32_t free_entries;
+
+	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+			&prod_head, &prod_next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_elem_pvt.h line 348

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_elem_pvt.h line 350

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sc
+ *   Indicates whether to use single consumer or multi-consumer head

update

lib/librte_ring/ring_elem_pvt.h line 352

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

Quoted 48 lines
lib/librte_ring/ring_elem_pvt.h new file +29 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
+		unsigned int *available)
+{
+	uint32_t cons_head, cons_next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+			&cons_head, &cons_next, &entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
+
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_ELEM_PVT_H_ */
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +2 -2
similarity index 98%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/ring_generic_pvt.h
index 37c62b8d6..fc46a27b2 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RING_GENERIC_PVT_H_
+#define _RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct

rte_ring *r, unsigned int is_sc,

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring

Quoted 25 lines
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RING_GENERIC_PVT_H_ */
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -2
similarity index 60%
rename from lib/librte_ring/rte_ring_hts_c11_mem.h
rename to lib/librte_ring/ring_hts_elem_pvt.h
index 16e54b6ff..9268750b0 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_HTS_C11_MEM_H_
-#define _RTE_RING_HTS_C11_MEM_H_
+#ifndef _RING_HTS_ELEM_PVT_H_
+#define _RING_HTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_hts_c11_mem.h

@@ -161,4 +161,86 @@ __rte_ring_hts_move_cons_head(struct rte_ring

*r, unsigned int num,

Quoted 9 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -1
 	return n;
 }
 
-#endif /* _RTE_RING_HTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 10 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 179

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 187

Quoted 8 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +8 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +22 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 10 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 220

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring · line 222

Quoted 2 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 12 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +12 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct

Quoted 32 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +12 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_HTS_ELEM_PVT_H_ */
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_peek_c11_mem.h
rename to lib/librte_ring/ring_peek_elem_pvt.h
index 283c7e70b..1c57bcdd6 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_PEEK_C11_MEM_H_
-#define _RTE_RING_PEEK_C11_MEM_H_
+#ifndef _RING_PEEK_ELEM_PVT_H_
+#define _RING_PEEK_ELEM_PVT_H_
 
 /**
  * @file rte_ring_peek_c11_mem.h

@@ -107,4 +107,73 @@ __rte_ring_hts_set_head_tail(struct

rte_ring_hts_headtail *ht, uint32_t tail,

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 9 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +6 -1
 	__atomic_store_n(&ht->ht.raw, p.raw, __ATOMIC_RELEASE);  }
 
-#endif /* _RTE_RING_PEEK_C11_MEM_H_ */
+/**
+ * @internal This function moves prod head value.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
+		enum rte_ring_queue_behavior behavior, uint32_t

*free_space) {

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring

Quoted 83 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +63 -0
+	uint32_t free, head, next;
+
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &free);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
+			&head, &free);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		free = 0;
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal This function moves cons head value and copies up to *n*
+ * objects from the ring to the user provided obj_table.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t avail, head, next;
+
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
+			behavior, &head, &next, &avail);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
+			&head, &avail);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		avail = 0;
+	}
+
+	if (n != 0)
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+
+	if (available != NULL)
+		*available = avail - n;
+	return n;
+}
+
+#endif /* _RING_PEEK_ELEM_PVT_H_ */
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_rts_c11_mem.h
rename to lib/librte_ring/ring_rts_elem_pvt.h
index 327f22796..cbcec73eb 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_RTS_C11_MEM_H_
-#define _RTE_RING_RTS_C11_MEM_H_
+#ifndef _RING_RTS_ELEM_PVT_H_
+#define _RING_RTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_rts_c11_mem.h

@@ -176,4 +176,86 @@ __rte_ring_rts_move_cons_head(struct rte_ring

*r, uint32_t num,

Quoted 9 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -1
 	return n;
 }
 
-#endif /* _RTE_RING_RTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 194

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 202

Quoted 8 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +8 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has finished
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +22 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_prod);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring. Otherwise
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 235

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring · line 237

Quoted 2 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 12 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +12 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 24 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +12 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
+		__rte_ring_rts_update_tail(&r->rts_cons);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_RTS_ELEM_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_elem.h index 57344d47d..119b5c0b6 100644

lib/librte_ring/rte_ring_elem.h +1 -0
@@ -21,6 +21,7 @@ extern "C" {
 #endif
 
 #include <rte_ring_core.h>
+#include <ring_elem_pvt.h>
 
 /**
  * Calculate the memory size needed for a ring with given element
Quoted 1 line

size @@ -105,379 +106,6 @@ ssize_t

rte_ring_get_memsize_elem(unsigned

Quoted 1 line

int esize, unsigned int count); struct rte_ring *rte_ring_create_elem(const

char *name, unsigned int esize,

Quoted 44 lines
lib/librte_ring/rte_ring_elem.h +0 -42
 			unsigned int count, int socket_id, unsigned int flags);
 
-static __rte_always_inline void
-__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	const uint32_t *obj = (const uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-			ring[idx + 4] = obj[i + 4];
-			ring[idx + 5] = obj[i + 5];
-			ring[idx + 6] = obj[i + 6];
-			ring[idx + 7] = obj[i + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 6:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 5:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 4:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 56 lines
lib/librte_ring/rte_ring_elem.h +0 -56
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++];
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-	}
-}
-
-/* the actual enqueue of elements on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- */
-static __rte_always_inline void
-__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_enqueue_elems_64(r, prod_head, obj_table,

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_enqueue_elems_128(r, prod_head, obj_table,

num);

Quoted 118 lines
lib/librte_ring/rte_ring_elem.h +0 -118
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = prod_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t size,
-		uint32_t idx, void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	uint32_t *obj = (uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-			obj[i + 4] = ring[idx + 4];
-			obj[i + 5] = ring[idx + 5];
-			obj[i + 6] = ring[idx + 6];
-			obj[i + 7] = ring[idx + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 6:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 5:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 4:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t prod_head,
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	rte_int128_t *obj = (rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx), 16);
-	}
-}
-
-/* the actual dequeue of elements from the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- */
-static __rte_always_inline void
-__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
-		void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_dequeue_elems_64(r, cons_head, obj_table,

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_dequeue_elems_128(r, cons_head, obj_table,

num);

Quoted 33 lines
lib/librte_ring/rte_ring_elem.h +0 -33
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;
-		idx = cons_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-/* Between load and load. there might be cpu reorder in weak model
- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

Quoted 2 lines

- * @param is_sp
- * Indicates whether to use single producer or multi-producer head

update

Quoted 34 lines

- * @param free_space
- * returns the amount of space after the enqueue operation has finished
- * @return
- * Actual number of objects enqueued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -28
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sp,
-		unsigned int *free_space)
-{
-	uint32_t prod_head, prod_next;
-	uint32_t free_entries;
-
-	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
-			&prod_head, &prod_next, &free_entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
-end:
-	if (free_space != NULL)
-		*free_space = free_entries - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

Quoted 2 lines

- * @param is_sc
- * Indicates whether to use single consumer or multi-consumer head

update

Quoted 2 lines

- * @param available
- * returns the number of remaining ring entries after the dequeue has

finished

Quoted 46 lines

- * @return
- * - Actual number of objects dequeued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -24
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned int is_sc,
-		unsigned int *available)
-{
-	uint32_t cons_head, cons_next;
-	uint32_t entries;
-
-	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
-			&cons_head, &cons_next, &entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
-
-	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
-
-end:
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
-
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_hts.h index 359b15771..bdbdafc9f 100644

lib/librte_ring/rte_ring_hts.h +0 -7
@@ -29,89 +29,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_hts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_hts.h line 48

Quoted 10 lines
lib/librte_ring/rte_ring_hts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/rte_ring_hts.h line 49

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/rte_ring_hts.h line 57

Quoted 8 lines
lib/librte_ring/rte_ring_hts.h +0 -8
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/rte_ring_hts.h +0 -22
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_prod, head, n, 1);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the HTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_hts.h

Quoted 10 lines
lib/librte_ring/rte_ring_hts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/rte_ring_hts.h line 90

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/rte_ring_hts.h line 92

Quoted 2 lines
lib/librte_ring/rte_ring_hts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/rte_ring_hts.h

Quoted 12 lines
lib/librte_ring/rte_ring_hts.h +0 -12
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 20 lines
lib/librte_ring/rte_ring_hts.h +1 -10
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_hts_update_tail(&r->hts_cons, head, n, 0);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_hts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the HTS ring (multi-producers safe).
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_peek.h index 45f707dc7..0dd402be4 100644

lib/librte_ring/rte_ring_peek.h +0 -0
@@ -48,39 +48,7 @@
 extern "C" {

lib/librte_ring/rte_ring_peek.h line 58

Quoted 10 lines
lib/librte_ring/rte_ring_peek.h +0 -8
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
-
-/**
- * @internal This function moves prod head value.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
-		enum rte_ring_queue_behavior behavior, uint32_t

*free_space)

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, void *

Quoted 30 lines
lib/librte_ring/rte_ring_peek.h +1 -25
-{
-	uint32_t free, head, next;
-
-	switch (r->prod.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_prod_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &free);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
-			&head, &free);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		free = 0;
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Start to enqueue several objects on the ring.

@@ -248,43 +216,6 @@ rte_ring_enqueue_finish(struct rte_ring *r, void *

const *obj_table,

Quoted 71 lines
lib/librte_ring/rte_ring_peek.h +0 -37
 	rte_ring_enqueue_elem_finish(r, obj_table, sizeof(uintptr_t), n);  }
 
-/**
- * @internal This function moves cons head value and copies up to *n*
- * objects from the ring to the user provided obj_table.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t avail, head, next;
-
-	switch (r->cons.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_cons_head(r, RTE_RING_SYNC_ST, n,
-			behavior, &head, &next, &avail);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
-			&head, &avail);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		avail = 0;
-	}
-
-	if (n != 0)
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-
-	if (available != NULL)
-		*available = avail - n;
-	return n;
-}
-
 /**
  * Start to dequeue several objects from the ring.
  * Note that user has to call appropriate dequeue_finish() diff --git

a/lib/librte_ring/rte_ring_peek_zc.h
b/lib/librte_ring/rte_ring_peek_zc.h
index cb3bbd067..bc8252a18 100644

lib/librte_ring/rte_ring_peek_zc.h +1 -1
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_rts.h index afc12abe2..83d9903e2 100644

lib/librte_ring/rte_ring_rts.h +0 -7
@@ -56,89 +56,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_rts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_rts.h line 75

Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items from a

ring

lib/librte_ring/rte_ring_rts.h line 76

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as possible

from ring

lib/librte_ring/rte_ring_rts.h line 84

Quoted 8 lines
lib/librte_ring/rte_ring_rts.h +0 -8
- * @param free_space
- *   returns the amount of space after the enqueue operation has finished
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 22 lines
lib/librte_ring/rte_ring_rts.h +0 -22
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head, &free);
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_prod);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the RTS ring.
- *
- * @param r
- *   A pointer to the ring structure.

lib/librte_ring/rte_ring_rts.h

Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring. Otherwise
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items from a

ring

lib/librte_ring/rte_ring_rts.h line 117

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as possible

from ring

lib/librte_ring/rte_ring_rts.h line 119

Quoted 2 lines
lib/librte_ring/rte_ring_rts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue has

finished

lib/librte_ring/rte_ring_rts.h

Quoted 12 lines
lib/librte_ring/rte_ring_rts.h +0 -12
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior behavior,
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

Quoted 16 lines
lib/librte_ring/rte_ring_rts.h +1 -11
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize, n);
-		__rte_ring_rts_update_tail(&r->rts_cons);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_rts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the RTS ring (multi-producers safe).
--

2.17.1

author
#11 lore raw

[dpdk-dev] 回复: [PATCH v1 3/3] ring: rename and refactor ring library

Hi, Konstantin

Quoted 70 lines

-----邮件原件-----
发件人: Ananyev, Konstantin <[email protected]> 发送时间: 2021年1月25日 18:29
收件人: Ananyev, Konstantin <[email protected]>; Feifei Wang <[email protected]>; Honnappa Nagarahalli <[email protected]>
抄送: [email protected]; nd <[email protected]>; nd <[email protected]>
主题: RE: [PATCH v1 3/3] ring: rename and refactor ring library

Quoted 8 lines

Hi Feifei,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all

private

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

APIs into these new header files.

Suggested-by Honnappa Nagarahalli

<[email protected]>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli

<[email protected]>

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  15 +-
 .../{rte_ring_c11_mem.h => ring_c11_pvt.h}    |   9 +-
 lib/librte_ring/ring_elem_pvt.h               | 385 ++++++++++++++++++
 ...{rte_ring_generic.h => ring_generic_pvt.h} |   6 +-
 ...ring_hts_c11_mem.h => ring_hts_elem_pvt.h} |  88 +++-
...ng_peek_c11_mem.h => ring_peek_elem_pvt.h} |  75 +++-
...ring_rts_c11_mem.h => ring_rts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_hts.h                |  84 +---
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 12 files changed, 646 insertions(+), 635 deletions(-)  rename

lib/librte_ring/{rte_ring_c11_mem.h => ring_c11_pvt.h} (96%) create mode 100644 lib/librte_ring/ring_elem_pvt.h rename lib/librte_ring/{rte_ring_generic.h => ring_generic_pvt.h} (98%) rename lib/librte_ring/{rte_ring_hts_c11_mem.h =>

ring_hts_elem_pvt.h}

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

(60%) rename lib/librte_ring/{rte_ring_peek_c11_mem.h => ring_peek_elem_pvt.h} (62%) rename lib/librte_ring/{rte_ring_rts_c11_mem.h => ring_rts_elem_pvt.h}

(62%)

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Sorry, but I don't understand the purpose of that patch. As I remember by DPDK naming convention all installable headers

should

Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

have 'rte_' prefix. Same for public defines (RTE_). Why to abandon it here?

This change refers to you and Honnappa's discussion about ring:
https://mails.dpdk.org/archives/dev/2020-May/166803.html

And it is to separate the external APIs that can be called by users, from internal APIs that cannot be called by users.
The internal functions are included in the files with xx_pvt.h.

Even these xx_pvt.h files contain internal API and not supposed to be

included directly,

Quoted 2 lines

they have to be installed into public dir (<dpdk-install-dir/include). So I think we need to keep DPDK public prefixes for them (and inside

Quoted 9 lines

them):

Quoted 1 line

rte_xx_pvt.h (_RTE_).

As another thought - it is probably good to introduce new generic naming convention
for such sort of files (internal ones that still have to be exposed). Might be '_rte_' prefix or so...
Though I suppose it would require broader discussion first.

Thanks very much for your comments. According to this, I think maybe we can only separate the external APIs from internal APIs in this patch and keep DPDK public prefixes for this files. Thus, though these private files can be exposed, the "xx__pvt" suffix can remind users not to use them.

On the other side, for new generic naming convention, we can leave it for later discussion.

Best Regards
Feifei

Quoted 1,678 lines
Quoted 46 lines

Konstantin

Quoted 43 lines

Best Regards
Feifei

Quoted 40 lines
Quoted 40 lines
lib/librte_ring/meson.build +8 -7
index 36fdcb6a5..98eac5810 100644
@@ -2,15 +2,16 @@
 # Copyright(c) 2017 Intel Corporation
 
 sources = files('rte_ring.c')
-headers = files('rte_ring.h',
+headers = files('ring_c11_pvt.h',
+		'ring_elem_pvt.h',
+		'ring_generic_pvt.h',
+		'ring_hts_elem_pvt.h',
+		'ring_peek_elem_pvt.h',
+		'ring_rts_elem_pvt.h',
+		'rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
 		'rte_ring_peek_zc.h',
-		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts.h')
(quoted diff) +0 -0

b/lib/librte_ring/ring_c11_pvt.h similarity index 96% rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/ring_c11_pvt.h index 7f5eba262..9f2f5318f 100644

lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -0
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +2 -2
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RING_C11_PVT_H_
+#define _RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t

old_val, @@ -69,9 +69,6 @@ __rte_ring_move_prod_head(struct

rte_ring

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*r, unsigned int is_sp,

Quoted 3 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -0
 		/* Ensure the head is read before tail */
 		__atomic_thread_fence(__ATOMIC_ACQUIRE);
Quoted 8 lines
Quoted 1 line
Quoted 1 line

lib/librte_ring/ring_c11_pvt.h

Quoted 1 line
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -1
-		/* load-acquire synchronize with store-release of ht-

tail

Quoted 6 lines
Quoted 6 lines

lib/librte_ring/ring_c11_pvt.h __rte_ring_move_cons_head(struct rte_ring

Quoted 6 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +0 -2
-		 * in update_tail.
-		 */
 		cons_tail = __atomic_load_n(&r->cons.tail,
 					__ATOMIC_ACQUIRE);

@@ -178,4 +175,4 @@ __rte_ring_move_cons_head(struct rte_ring

*r, int

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

is_sc,

Quoted 17 lines
lib/librte_ring/ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RING_C11_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/ring_elem_pvt.h new file mode 100644 index 000000000..8003e5edc

lib/librte_ring/ring_elem_pvt.h new file +6 -0
@@ -0,0 +1,385 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ *
+ * Copyright (c) 2017,2018 HXT-semitech Corporation.
+ * Copyright (c) 2007-2009 Kip Macy [email protected]
+ * All rights reserved.
+ * Derived from FreeBSD's bufring.h
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+ * Used as BSD-3 Licensed with permission from Kip Macy.
+ */
+
+#ifndef _RING_ELEM_PVT_H_
+#define _RING_ELEM_PVT_H_
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t

size,

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/librte_ring/ring_elem_pvt.h new file +34 -0
+		uint32_t idx, const void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	const uint32_t *obj = (const uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+			ring[idx + 4] = obj[i + 4];
+			ring[idx + 5] = obj[i + 5];
+			ring[idx + 6] = obj[i + 6];
+			ring[idx + 7] = obj[i + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 6:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 5:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 4:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t

prod_head,

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 7 lines
lib/librte_ring/ring_elem_pvt.h new file +7 -0
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 18 lines
lib/librte_ring/ring_elem_pvt.h new file +18 -0
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			ring[idx] = obj[i];
+			ring[idx + 1] = obj[i + 1];
+			ring[idx + 2] = obj[i + 2];
+			ring[idx + 3] = obj[i + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 2:
+			ring[idx++] = obj[i++]; /* fallthrough */
+		case 1:
+			ring[idx++] = obj[i++];
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			ring[idx] = obj[i];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			ring[idx] = obj[i];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t

prod_head,

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
lib/librte_ring/ring_elem_pvt.h new file +31 -0
+		const void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 32);
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(ring + idx),
+				(const void *)(obj + i), 16);
+	}
+}
+
+/* the actual enqueue of elements on the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions.
+ */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+static __rte_always_inline void
+__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
+		const void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_enqueue_elems_64(r, prod_head,

obj_table,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

num);

lib/librte_ring/ring_elem_pvt.h line 130

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_enqueue_elems_128(r, prod_head,

obj_table,

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

num);

Quoted 6 lines
lib/librte_ring/ring_elem_pvt.h new file +6 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);
+		nr_num = num * scale;

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+		idx = prod_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t

size,

Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
lib/librte_ring/ring_elem_pvt.h new file +34 -0
+		uint32_t idx, void *obj_table, uint32_t n) {
+	unsigned int i;
+	uint32_t *ring = (uint32_t *)&r[1];
+	uint32_t *obj = (uint32_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+			obj[i + 4] = ring[idx + 4];
+			obj[i + 5] = ring[idx + 5];
+			obj[i + 6] = ring[idx + 6];
+			obj[i + 7] = ring[idx + 7];
+		}
+		switch (n & 0x7) {
+		case 7:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 6:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 5:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 4:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t

prod_head,

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/librte_ring/ring_elem_pvt.h new file +25 -0
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	uint64_t *ring = (uint64_t *)&r[1];
+	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
+			obj[i] = ring[idx];
+			obj[i + 1] = ring[idx + 1];
+			obj[i + 2] = ring[idx + 2];
+			obj[i + 3] = ring[idx + 3];
+		}
+		switch (n & 0x3) {
+		case 3:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 2:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		case 1:
+			obj[i++] = ring[idx++]; /* fallthrough */
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			obj[i] = ring[idx];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			obj[i] = ring[idx];
+	}
+}
+
+static __rte_always_inline void
+__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t

prod_head,

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+		void *obj_table, uint32_t n)
+{
+	unsigned int i;
+	const uint32_t size = r->size;
+	uint32_t idx = prod_head & r->mask;
+	rte_int128_t *ring = (rte_int128_t *)&r[1];
+	rte_int128_t *obj = (rte_int128_t *)obj_table;
+	if (likely(idx + n < size)) {
+		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
+			memcpy((void *)(obj + i), (void *)(ring + idx),

32);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 3 lines
lib/librte_ring/ring_elem_pvt.h new file +3 -0
+		switch (n & 0x1) {
+		case 1:
+			memcpy((void *)(obj + i), (void *)(ring + idx),

16);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 4 lines
lib/librte_ring/ring_elem_pvt.h new file +4 -0
+		}
+	} else {
+		for (i = 0; idx < size; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx),

16);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 3 lines
lib/librte_ring/ring_elem_pvt.h new file +3 -0
+		/* Start at the beginning */
+		for (idx = 0; i < n; i++, idx++)
+			memcpy((void *)(obj + i), (void *)(ring + idx),

16);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
lib/librte_ring/ring_elem_pvt.h new file +7 -0
+	}
+}
+
+/* the actual dequeue of elements from the ring.
+ * Placed here since identical code needed in both
+ * single and multi producer enqueue functions.
+ */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+static __rte_always_inline void
+__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
+		void *obj_table, uint32_t esize, uint32_t num) {
+	/* 8B and 16B copies implemented individually to retain
+	 * the current performance.
+	 */
+	if (esize == 8)
+		__rte_ring_dequeue_elems_64(r, cons_head,

obj_table,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

num);

lib/librte_ring/ring_elem_pvt.h line 258

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+	else if (esize == 16)
+		__rte_ring_dequeue_elems_128(r, cons_head,

obj_table,

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

num);

Quoted 5 lines
lib/librte_ring/ring_elem_pvt.h new file +5 -0
+	else {
+		uint32_t idx, scale, nr_idx, nr_num, nr_size;
+
+		/* Normalize to uint32_t */
+		scale = esize / sizeof(uint32_t);

lib/librte_ring/ring_elem_pvt.h

Quoted 10 lines
lib/librte_ring/ring_elem_pvt.h new file +10 -0
+		nr_num = num * scale;
+		idx = cons_head & r->mask;
+		nr_idx = idx * scale;
+		nr_size = r->size * scale;
+		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
+				obj_table, nr_num);
+	}
+}
+
+/* Between load and load. there might be cpu reorder in weak

model

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
lib/librte_ring/ring_elem_pvt.h new file +14 -0
+ * (powerpc/arm).
+ * There are 2 choices for the users
+ * 1.use rmb() memory barrier
+ * 2.use one-direction load_acquire/store_release barrier
+ * It depends on performance test results.
+ */
+#ifdef RTE_USE_C11_MEM_MODEL
+#include "ring_c11_pvt.h"
+#else
+#include "ring_generic_pvt.h"
+#endif
+
+/**
+ * @internal Enqueue several objects on the ring
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 5 lines
lib/librte_ring/ring_elem_pvt.h new file +5 -0
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as

possible

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

from ring

lib/librte_ring/ring_elem_pvt.h line 303

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sp
+ *   Indicates whether to use single producer or multi-producer head

update

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has

finished

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 6 lines
lib/librte_ring/ring_elem_pvt.h new file +6 -0
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_ring/ring_elem_pvt.h line 313

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned

int is_sp,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 11 lines
lib/librte_ring/ring_elem_pvt.h new file +11 -0
+		unsigned int *free_space)
+{
+	uint32_t prod_head, prod_next;
+	uint32_t free_entries;
+
+	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
+			&prod_head, &prod_next, &free_entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize,

n);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_ring/ring_elem_pvt.h line 326

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next,

is_sp, 1);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+end:
+	if (free_space != NULL)
+		*free_space = free_entries - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the ring
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 5 lines
lib/librte_ring/ring_elem_pvt.h new file +5 -0
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/ring_elem_pvt.h new file +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param is_sc
+ *   Indicates whether to use single consumer or multi-consumer

head

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

update

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue

has

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

finished

lib/librte_ring/ring_elem_pvt.h

Quoted 8 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
+		unsigned int esize, unsigned int n,
+		enum rte_ring_queue_behavior behavior, unsigned

int is_sc,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ring/ring_elem_pvt.h

Quoted 11 lines
lib/librte_ring/ring_elem_pvt.h new file +11 -0
+		unsigned int *available)
+{
+	uint32_t cons_head, cons_next;
+	uint32_t entries;
+
+	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
+			&cons_head, &cons_next, &entries);
+	if (n == 0)
+		goto end;
+
+	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize,

n);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_ring/ring_elem_pvt.h line 373

Quoted 2 lines
lib/librte_ring/ring_elem_pvt.h new file +2 -0
+
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next,

is_sc, 0);

Quoted 46 lines
Quoted 46 lines
Quoted 46 lines
Quoted 27 lines
lib/librte_ring/ring_elem_pvt.h new file +8 -0
+
+end:
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_ELEM_PVT_H_ */
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +2 -2
similarity index 98%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/ring_generic_pvt.h
index 37c62b8d6..fc46a27b2 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RING_GENERIC_PVT_H_
+#define _RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t
Quoted 1 line

old_val, @@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct

rte_ring *r, unsigned int is_sc,

Quoted 17 lines
lib/librte_ring/ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +1 -1
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RING_GENERIC_PVT_H_ */
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +0 -0
similarity index 60%
rename from lib/librte_ring/rte_ring_hts_c11_mem.h
rename to lib/librte_ring/ring_hts_elem_pvt.h
index 16e54b6ff..9268750b0 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct

Quoted 8 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -2
-#ifndef _RTE_RING_HTS_C11_MEM_H_
-#define _RTE_RING_HTS_C11_MEM_H_
+#ifndef _RING_HTS_ELEM_PVT_H_
+#define _RING_HTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_hts_c11_mem.h

@@ -161,4 +161,86 @@ __rte_ring_hts_move_cons_head(struct

rte_ring

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*r, unsigned int num,

Quoted 3 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +0 -0
 	return n;
 }
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 11 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +10 -1
-#endif /* _RTE_RING_HTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the HTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 178

Quoted 5 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -0
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has

finished

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 187

Quoted 6 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +6 -0
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 188

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 5 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -0
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head,

&free);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 3 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +3 -0
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize,

n);

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 197

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+		__rte_ring_hts_update_tail(&r->hts_prod, head, n,

1);

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +9 -0
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the HTS ring.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 214

Quoted 8 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +8 -0
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 219

Quoted 5 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -0
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue

has

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

finished

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 228

Quoted 6 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +6 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void

*obj_table,

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 229

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 5 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +5 -0
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct rte_ring *r, unsigned int num,

Quoted 3 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +3 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize,

n);

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/ring_hts_elem_pvt.h __rte_ring_hts_move_cons_head(struct · line 238

Quoted 1 line
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +1 -0
+		__rte_ring_hts_update_tail(&r->hts_cons, head, n,

0);

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/librte_ring/ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +8 -0
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_HTS_ELEM_PVT_H_ */
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +2 -2
similarity index 62%
rename from lib/librte_ring/rte_ring_peek_c11_mem.h
rename to lib/librte_ring/ring_peek_elem_pvt.h
index 283c7e70b..1c57bcdd6 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_PEEK_C11_MEM_H_
-#define _RTE_RING_PEEK_C11_MEM_H_
+#ifndef _RING_PEEK_ELEM_PVT_H_
+#define _RING_PEEK_ELEM_PVT_H_
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct

Quoted 3 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +0 -0
 /**
  * @file rte_ring_peek_c11_mem.h

@@ -107,4 +107,73 @@ __rte_ring_hts_set_head_tail(struct

rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 1 line
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +0 -0
 	__atomic_store_n(&ht->ht.raw, p.raw,

__ATOMIC_RELEASE); }

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 8 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +6 -1
-#endif /* _RTE_RING_PEEK_C11_MEM_H_ */
+/**
+ * @internal This function moves prod head value.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
+		enum rte_ring_queue_behavior behavior, uint32_t

*free_space) {

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 5 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +5 -0
+	uint32_t free, head, next;
+
+	switch (r->prod.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_prod_head(r,

RTE_RING_SYNC_ST, n,

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +14 -0
+			behavior, &head, &next, &free);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
+			&head, &free);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		free = 0;
+	}
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct · line 140

Quoted 8 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +8 -0
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal This function moves cons head value and copies up to

*n*

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 5 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +5 -0
+ * objects from the ring to the user provided obj_table.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 7 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +7 -0
+	uint32_t *available)
+{
+	uint32_t avail, head, next;
+
+	switch (r->cons.sync_type) {
+	case RTE_RING_SYNC_ST:
+		n = __rte_ring_move_cons_head(r,

RTE_RING_SYNC_ST, n,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +9 -0
+			behavior, &head, &next, &avail);
+		break;
+	case RTE_RING_SYNC_MT_HTS:
+		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
+			&head, &avail);
+		break;
+	case RTE_RING_SYNC_MT:
+	case RTE_RING_SYNC_MT_RTS:
+	default:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_peek_elem_pvt.h __rte_ring_hts_set_head_tail(struct rte_ring_hts_headtail *ht, uint32_t tail,

Quoted 8 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +8 -0
+		/* unsupported mode, shouldn't be here */
+		RTE_ASSERT(0);
+		n = 0;
+		avail = 0;
+	}
+
+	if (n != 0)
+		__rte_ring_dequeue_elems(r, head, obj_table, esize,

n);

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
lib/librte_ring/ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +7 -0
+
+	if (available != NULL)
+		*available = avail - n;
+	return n;
+}
+
+#endif /* _RING_PEEK_ELEM_PVT_H_ */
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +0 -0
similarity index 62%
rename from lib/librte_ring/rte_ring_rts_c11_mem.h
rename to lib/librte_ring/ring_rts_elem_pvt.h
index 327f22796..cbcec73eb 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct

Quoted 8 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -2
-#ifndef _RTE_RING_RTS_C11_MEM_H_
-#define _RTE_RING_RTS_C11_MEM_H_
+#ifndef _RING_RTS_ELEM_PVT_H_
+#define _RING_RTS_ELEM_PVT_H_
 
 /**
  * @file rte_ring_rts_c11_mem.h

@@ -176,4 +176,86 @@ __rte_ring_rts_move_cons_head(struct

rte_ring

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

*r, uint32_t num,

Quoted 3 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +0 -0
 	return n;
 }
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct rte_ring *r, uint32_t num,

Quoted 11 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -1
-#endif /* _RTE_RING_RTS_C11_MEM_H_ */
+/**
+ * @internal Enqueue several objects on the RTS ring.
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 193

Quoted 5 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -0
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to add in the ring from the obj_table.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -0
+ * @param free_space
+ *   returns the amount of space after the enqueue operation has

finished

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 202

Quoted 6 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +6 -0
+ * @return
+ *   Actual number of objects enqueued.
+ *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 203

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 208

Quoted 5 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -0
+	uint32_t *free_space)
+{
+	uint32_t free, head;
+
+	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head,

&free);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 211

Quoted 3 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +3 -0
+
+	if (n != 0) {
+		__rte_ring_enqueue_elems(r, head, obj_table, esize,

n);

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +10 -0
+		__rte_ring_rts_update_tail(&r->rts_prod);
+	}
+
+	if (free_space != NULL)
+		*free_space = free - n;
+	return n;
+}
+
+/**
+ * @internal Dequeue several objects from the RTS ring.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 229

Quoted 8 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +8 -0
+ *
+ * @param r
+ *   A pointer to the ring structure.
+ * @param obj_table
+ *   A pointer to a table of objects.
+ * @param esize
+ *   The size of ring element, in bytes. It must be a multiple of 4.
+ *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 234

Quoted 5 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -0
+ *   the results are undefined.
+ * @param n
+ *   The number of objects to pull from the ring.
+ * @param behavior
+ *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+ *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +2 -0
+ * @param available
+ *   returns the number of remaining ring entries after the dequeue

has

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

finished

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 243

Quoted 6 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +6 -0
+ * @return
+ *   - Actual number of objects dequeued.
+ *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
+ */
+static __rte_always_inline unsigned int
+__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void

*obj_table,

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 244

Quoted 1 line
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +1 -0
+	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 249

Quoted 5 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +5 -0
+	uint32_t *available)
+{
+	uint32_t entries, head;
+
+	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/ring_rts_elem_pvt.h __rte_ring_rts_move_cons_head(struct · line 252

Quoted 3 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +3 -0
+
+	if (n != 0) {
+		__rte_ring_dequeue_elems(r, head, obj_table, esize,

n);

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
lib/librte_ring/ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +9 -0
+		__rte_ring_rts_update_tail(&r->rts_cons);
+	}
+
+	if (available != NULL)
+		*available = entries - n;
+	return n;
+}
+
+#endif /* _RING_RTS_ELEM_PVT_H_ */
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_elem.h index 57344d47d..119b5c0b6 100644

lib/librte_ring/rte_ring_elem.h +0 -0
@@ -21,6 +21,7 @@ extern "C" {
 #endif
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
lib/librte_ring/rte_ring_elem.h +1 -0
 #include <rte_ring_core.h>
+#include <ring_elem_pvt.h>
 
 /**
  * Calculate the memory size needed for a ring with given element

size @@ -105,379 +106,6 @@ ssize_t

rte_ring_get_memsize_elem(unsigned

Quoted 1 line

int esize, unsigned int count); struct rte_ring

*rte_ring_create_elem(const

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

char *name, unsigned int esize,

Quoted 1 line

unsigned int count, int socket_id, unsigned

int flags);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_elem.h

Quoted 3 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-static __rte_always_inline void
-__rte_ring_enqueue_elems_32(struct rte_ring *r, const uint32_t

size,

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/librte_ring/rte_ring_elem.h +0 -35
-		uint32_t idx, const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	const uint32_t *obj = (const uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-			ring[idx + 4] = obj[i + 4];
-			ring[idx + 5] = obj[i + 5];
-			ring[idx + 6] = obj[i + 6];
-			ring[idx + 7] = obj[i + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 6:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 5:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 4:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_64(struct rte_ring *r, uint32_t

prod_head,

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 7 lines
lib/librte_ring/rte_ring_elem.h +0 -7
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	const unaligned_uint64_t *obj = (const unaligned_uint64_t

*)obj_table;

Quoted 18 lines
lib/librte_ring/rte_ring_elem.h +0 -18
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			ring[idx] = obj[i];
-			ring[idx + 1] = obj[i + 1];
-			ring[idx + 2] = obj[i + 2];
-			ring[idx + 3] = obj[i + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 2:
-			ring[idx++] = obj[i++]; /* fallthrough */
-		case 1:
-			ring[idx++] = obj[i++];
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			ring[idx] = obj[i];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			ring[idx] = obj[i];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_enqueue_elems_128(struct rte_ring *r, uint32_t

prod_head,

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
lib/librte_ring/rte_ring_elem.h +0 -32
-		const void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	const rte_int128_t *obj = (const rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 32);
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(ring + idx),
-				(const void *)(obj + i), 16);
-	}
-}
-
-/* the actual enqueue of elements on the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.
- */
-static __rte_always_inline void
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-__rte_ring_enqueue_elems(struct rte_ring *r, uint32_t prod_head,
-		const void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_enqueue_elems_64(r, prod_head,

obj_table,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_enqueue_elems_128(r, prod_head,

obj_table,

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

num);

Quoted 6 lines
lib/librte_ring/rte_ring_elem.h +0 -6
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);
-		nr_num = num * scale;

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
-		idx = prod_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_enqueue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_32(struct rte_ring *r, const uint32_t

size,

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
lib/librte_ring/rte_ring_elem.h +0 -35
-		uint32_t idx, void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	uint32_t *ring = (uint32_t *)&r[1];
-	uint32_t *obj = (uint32_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x7); i += 8, idx += 8) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-			obj[i + 4] = ring[idx + 4];
-			obj[i + 5] = ring[idx + 5];
-			obj[i + 6] = ring[idx + 6];
-			obj[i + 7] = ring[idx + 7];
-		}
-		switch (n & 0x7) {
-		case 7:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 6:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 5:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 4:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_64(struct rte_ring *r, uint32_t

prod_head,

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
lib/librte_ring/rte_ring_elem.h +0 -25
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	uint64_t *ring = (uint64_t *)&r[1];
-	unaligned_uint64_t *obj = (unaligned_uint64_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x3); i += 4, idx += 4) {
-			obj[i] = ring[idx];
-			obj[i + 1] = ring[idx + 1];
-			obj[i + 2] = ring[idx + 2];
-			obj[i + 3] = ring[idx + 3];
-		}
-		switch (n & 0x3) {
-		case 3:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 2:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		case 1:
-			obj[i++] = ring[idx++]; /* fallthrough */
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			obj[i] = ring[idx];
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			obj[i] = ring[idx];
-	}
-}
-
-static __rte_always_inline void
-__rte_ring_dequeue_elems_128(struct rte_ring *r, uint32_t

prod_head,

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
-		void *obj_table, uint32_t n)
-{
-	unsigned int i;
-	const uint32_t size = r->size;
-	uint32_t idx = prod_head & r->mask;
-	rte_int128_t *ring = (rte_int128_t *)&r[1];
-	rte_int128_t *obj = (rte_int128_t *)obj_table;
-	if (likely(idx + n < size)) {
-		for (i = 0; i < (n & ~0x1); i += 2, idx += 2)
-			memcpy((void *)(obj + i), (void *)(ring + idx),

32);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 3 lines
lib/librte_ring/rte_ring_elem.h +0 -3
-		switch (n & 0x1) {
-		case 1:
-			memcpy((void *)(obj + i), (void *)(ring + idx),

16);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 4 lines
lib/librte_ring/rte_ring_elem.h +0 -4
-		}
-	} else {
-		for (i = 0; idx < size; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx),

16);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 3 lines
lib/librte_ring/rte_ring_elem.h +0 -3
-		/* Start at the beginning */
-		for (idx = 0; i < n; i++, idx++)
-			memcpy((void *)(obj + i), (void *)(ring + idx),

16);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-	}
-}
-
-/* the actual dequeue of elements from the ring.
- * Placed here since identical code needed in both
- * single and multi producer enqueue functions.
- */
-static __rte_always_inline void
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-__rte_ring_dequeue_elems(struct rte_ring *r, uint32_t cons_head,
-		void *obj_table, uint32_t esize, uint32_t num)
-{
-	/* 8B and 16B copies implemented individually to retain
-	 * the current performance.
-	 */
-	if (esize == 8)
-		__rte_ring_dequeue_elems_64(r, cons_head,

obj_table,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

num);

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-	else if (esize == 16)
-		__rte_ring_dequeue_elems_128(r, cons_head,

obj_table,

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

num);

Quoted 5 lines
lib/librte_ring/rte_ring_elem.h +0 -5
-	else {
-		uint32_t idx, scale, nr_idx, nr_num, nr_size;
-
-		/* Normalize to uint32_t */
-		scale = esize / sizeof(uint32_t);

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 10 lines
lib/librte_ring/rte_ring_elem.h +0 -10
-		nr_num = num * scale;
-		idx = cons_head & r->mask;
-		nr_idx = idx * scale;
-		nr_size = r->size * scale;
-		__rte_ring_dequeue_elems_32(r, nr_size, nr_idx,
-				obj_table, nr_num);
-	}
-}
-
-/* Between load and load. there might be cpu reorder in weak

model

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

- * (powerpc/arm).
- * There are 2 choices for the users
- * 1.use rmb() memory barrier
- * 2.use one-direction load_acquire/store_release barrier
- * It depends on performance test results.
- * By default, move common functions to rte_ring_generic.h
- */

lib/librte_ring/rte_ring_elem.h +0 -8
-#ifdef RTE_USE_C11_MEM_MODEL
-#include "rte_ring_c11_mem.h"
-#else
-#include "rte_ring_generic.h"
-#endif
-
-/**
- * @internal Enqueue several objects on the ring
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

- *
- * @param r
- * A pointer to the ring structure.
- * @param obj_table
- * A pointer to a table of objects.
- * @param esize
- * The size of ring element, in bytes. It must be a multiple of 4.
- * This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

- * the results are undefined.
- * @param n
- * The number of objects to add in the ring from the obj_table.
- * @param behavior
- * RTE_RING_QUEUE_FIXED: Enqueue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Enqueue as many items as

possible

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

from ring

Quoted 2 lines

- * @param is_sp
- * Indicates whether to use single producer or multi-producer head

update

Quoted 2 lines

- * @param free_space
- * returns the amount of space after the enqueue operation has

finished

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_ring/rte_ring_elem.h

Quoted 6 lines

- * @return
- * Actual number of objects enqueued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -2
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_elem(struct rte_ring *r, const void

*obj_table,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned

int is_sp,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 11 lines
lib/librte_ring/rte_ring_elem.h +0 -11
-		unsigned int *free_space)
-{
-	uint32_t prod_head, prod_next;
-	uint32_t free_entries;
-
-	n = __rte_ring_move_prod_head(r, is_sp, n, behavior,
-			&prod_head, &prod_next, &free_entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize,

n);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-
-	__rte_ring_update_tail(&r->prod, prod_head, prod_next,

is_sp, 1);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
lib/librte_ring/rte_ring_elem.h +0 -8
-end:
-	if (free_space != NULL)
-		*free_space = free_entries - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the ring
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

- *
- * @param r
- * A pointer to the ring structure.
- * @param obj_table
- * A pointer to a table of objects.
- * @param esize
- * The size of ring element, in bytes. It must be a multiple of 4.
- * This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

- * the results are undefined.
- * @param n
- * The number of objects to pull from the ring.
- * @param behavior
- * RTE_RING_QUEUE_FIXED: Dequeue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line

- * RTE_RING_QUEUE_VARIABLE: Dequeue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines

- * @param is_sc
- * Indicates whether to use single consumer or multi-consumer

head

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

update

Quoted 2 lines

- * @param available
- * returns the number of remaining ring entries after the dequeue

has

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

finished

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 8 lines

- * @return
- * - Actual number of objects dequeued.
- * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */

lib/librte_ring/rte_ring_elem.h +0 -4
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
-		unsigned int esize, unsigned int n,
-		enum rte_ring_queue_behavior behavior, unsigned

int is_sc,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

lib/librte_ring/rte_ring_elem.h ssize_t rte_ring_get_memsize_elem(unsigned int esize, unsigned int count);

Quoted 11 lines
lib/librte_ring/rte_ring_elem.h +0 -11
-		unsigned int *available)
-{
-	uint32_t cons_head, cons_next;
-	uint32_t entries;
-
-	n = __rte_ring_move_cons_head(r, (int)is_sc, n, behavior,
-			&cons_head, &cons_next, &entries);
-	if (n == 0)
-		goto end;
-
-	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize,

n);

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_ring/rte_ring_elem.h

Quoted 2 lines
lib/librte_ring/rte_ring_elem.h +0 -2
-
-	__rte_ring_update_tail(&r->cons, cons_head, cons_next,

is_sc, 0);

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
lib/librte_ring/rte_ring_elem.h +0 -7
-
-end:
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
-
 /**
  * Enqueue several objects on the ring (multi-producers safe).
  *
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_hts.h index 359b15771..bdbdafc9f 100644

lib/librte_ring/rte_ring_hts.h +0 -0
@@ -29,89 +29,7 @@
 extern "C" {
 #endif
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/librte_ring/rte_ring_hts.h +0 -4
-#include <rte_ring_hts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the HTS ring.
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_hts.h line 43

Quoted 8 lines
lib/librte_ring/rte_ring_hts.h +0 -8
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_hts.h line 48

Quoted 5 lines
lib/librte_ring/rte_ring_hts.h +0 -5
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/rte_ring_hts.h +0 -2
- * @param free_space
- *   returns the amount of space after the enqueue operation has

finished

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_hts.h line 57

Quoted 6 lines
lib/librte_ring/rte_ring_hts.h +0 -6
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

lib/librte_ring/rte_ring_hts.h line 58

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_hts.h

Quoted 5 lines
lib/librte_ring/rte_ring_hts.h +0 -5
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_hts_move_prod_head(r, n, behavior, &head,

&free);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_hts.h

Quoted 3 lines
lib/librte_ring/rte_ring_hts.h +0 -3
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize,

n);

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/rte_ring_hts.h line 67

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
-		__rte_ring_hts_update_tail(&r->hts_prod, head, n,

1);

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/librte_ring/rte_ring_hts.h +0 -9
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the HTS ring.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_hts.h line 84

Quoted 8 lines
lib/librte_ring/rte_ring_hts.h +0 -8
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_hts.h line 89

Quoted 5 lines
lib/librte_ring/rte_ring_hts.h +0 -5
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/rte_ring_hts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue

has

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

finished

lib/librte_ring/rte_ring_hts.h line 98

Quoted 6 lines
lib/librte_ring/rte_ring_hts.h +0 -6
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_hts_dequeue_elem(struct rte_ring *r, void

*obj_table,

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/rte_ring_hts.h line 99

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_ring/rte_ring_hts.h

Quoted 5 lines
lib/librte_ring/rte_ring_hts.h +0 -5
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_hts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/rte_ring_hts.h

Quoted 3 lines
lib/librte_ring/rte_ring_hts.h +0 -3
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize,

n);

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/rte_ring_hts.h line 108

Quoted 1 line
lib/librte_ring/rte_ring_hts.h +0 -1
-		__rte_ring_hts_update_tail(&r->hts_cons, head, n,

0);

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 16 lines
lib/librte_ring/rte_ring_hts.h +1 -6
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_hts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the HTS ring (multi-producers safe).
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_peek.h index 45f707dc7..0dd402be4 100644

lib/librte_ring/rte_ring_peek.h +0 -0
@@ -48,39 +48,7 @@
 extern "C" {

lib/librte_ring/rte_ring_peek.h line 58

Quoted 10 lines
lib/librte_ring/rte_ring_peek.h +0 -8
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
-
-/**
- * @internal This function moves prod head value.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_enqueue_start(struct rte_ring *r, uint32_t n,
-		enum rte_ring_queue_behavior behavior, uint32_t

*free_space)

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

lib/librte_ring/rte_ring_peek.h

Quoted 6 lines
lib/librte_ring/rte_ring_peek.h +0 -6
-{
-	uint32_t free, head, next;
-
-	switch (r->prod.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_prod_head(r,

RTE_RING_SYNC_ST, n,

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
lib/librte_ring/rte_ring_peek.h +1 -19
-			behavior, &head, &next, &free);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_prod_head(r, n, behavior,
-			&head, &free);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		free = 0;
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
+#include <ring_peek_elem_pvt.h>
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r,

Quoted 3 lines
lib/librte_ring/rte_ring_peek.h +0 -0
 /**
  * Start to enqueue several objects on the ring.

@@ -248,43 +216,6 @@ rte_ring_enqueue_finish(struct rte_ring *r,

void *

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

const *obj_table,

Quoted 1 line
lib/librte_ring/rte_ring_peek.h +0 -0
 	rte_ring_enqueue_elem_finish(r, obj_table,

sizeof(uintptr_t), n); }

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, · line 250

Quoted 3 lines
lib/librte_ring/rte_ring_peek.h +0 -2
-/**
- * @internal This function moves cons head value and copies up to

*n*

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, void * const *obj_table,

Quoted 5 lines
lib/librte_ring/rte_ring_peek.h +0 -5
- * objects from the ring to the user provided obj_table.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_dequeue_start(struct rte_ring *r, void *obj_table,
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, void * const *obj_table,

Quoted 7 lines
lib/librte_ring/rte_ring_peek.h +0 -7
-	uint32_t *available)
-{
-	uint32_t avail, head, next;
-
-	switch (r->cons.sync_type) {
-	case RTE_RING_SYNC_ST:
-		n = __rte_ring_move_cons_head(r,

RTE_RING_SYNC_ST, n,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
lib/librte_ring/rte_ring_peek.h +0 -9
-			behavior, &head, &next, &avail);
-		break;
-	case RTE_RING_SYNC_MT_HTS:
-		n =  __rte_ring_hts_move_cons_head(r, n, behavior,
-			&head, &avail);
-		break;
-	case RTE_RING_SYNC_MT:
-	case RTE_RING_SYNC_MT_RTS:
-	default:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_peek.h rte_ring_enqueue_finish(struct rte_ring *r, void * const *obj_table,

Quoted 8 lines
lib/librte_ring/rte_ring_peek.h +0 -8
-		/* unsupported mode, shouldn't be here */
-		RTE_ASSERT(0);
-		n = 0;
-		avail = 0;
-	}
-
-	if (n != 0)
-		__rte_ring_dequeue_elems(r, head, obj_table, esize,

n);

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
lib/librte_ring/rte_ring_peek.h +0 -6
-
-	if (available != NULL)
-		*available = avail - n;
-	return n;
-}
-
 /**
  * Start to dequeue several objects from the ring.
  * Note that user has to call appropriate dequeue_finish() diff --git

a/lib/librte_ring/rte_ring_peek_zc.h
b/lib/librte_ring/rte_ring_peek_zc.h
index cb3bbd067..bc8252a18 100644

lib/librte_ring/rte_ring_peek_zc.h +1 -1
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
(quoted diff) +0 -0

b/lib/librte_ring/rte_ring_rts.h index afc12abe2..83d9903e2 100644

lib/librte_ring/rte_ring_rts.h +0 -0
@@ -56,89 +56,7 @@
 extern "C" {
 #endif
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
lib/librte_ring/rte_ring_rts.h +0 -4
-#include <rte_ring_rts_c11_mem.h>
-
-/**
- * @internal Enqueue several objects on the RTS ring.
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_rts.h line 70

Quoted 8 lines
lib/librte_ring/rte_ring_rts.h +0 -8
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_rts.h line 75

Quoted 5 lines
lib/librte_ring/rte_ring_rts.h +0 -5
- *   the results are undefined.
- * @param n
- *   The number of objects to add in the ring from the obj_table.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Enqueue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Enqueue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/rte_ring_rts.h +0 -2
- * @param free_space
- *   returns the amount of space after the enqueue operation has

finished

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_rts.h line 84

Quoted 6 lines
lib/librte_ring/rte_ring_rts.h +0 -6
- * @return
- *   Actual number of objects enqueued.
- *   If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_enqueue_elem(struct rte_ring *r, const void

*obj_table,

lib/librte_ring/rte_ring_rts.h line 85

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_rts.h line 90

Quoted 5 lines
lib/librte_ring/rte_ring_rts.h +0 -5
-	uint32_t *free_space)
-{
-	uint32_t free, head;
-
-	n =  __rte_ring_rts_move_prod_head(r, n, behavior, &head,

&free);

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

lib/librte_ring/rte_ring_rts.h line 93

Quoted 3 lines
lib/librte_ring/rte_ring_rts.h +0 -3
-
-	if (n != 0) {
-		__rte_ring_enqueue_elems(r, head, obj_table, esize,

n);

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
lib/librte_ring/rte_ring_rts.h +0 -10
-		__rte_ring_rts_update_tail(&r->rts_prod);
-	}
-
-	if (free_space != NULL)
-		*free_space = free - n;
-	return n;
-}
-
-/**
- * @internal Dequeue several objects from the RTS ring.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

lib/librte_ring/rte_ring_rts.h line 111

Quoted 8 lines
lib/librte_ring/rte_ring_rts.h +0 -8
- *
- * @param r
- *   A pointer to the ring structure.
- * @param obj_table
- *   A pointer to a table of objects.
- * @param esize
- *   The size of ring element, in bytes. It must be a multiple of 4.
- *   This must be the same value used while creating the ring.

Otherwise

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/librte_ring/rte_ring_rts.h line 116

Quoted 5 lines
lib/librte_ring/rte_ring_rts.h +0 -5
- *   the results are undefined.
- * @param n
- *   The number of objects to pull from the ring.
- * @param behavior
- *   RTE_RING_QUEUE_FIXED:    Dequeue a fixed number of items

from a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ring

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
- *   RTE_RING_QUEUE_VARIABLE: Dequeue as many items as

possible

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

from ring

Quoted 2 lines
lib/librte_ring/rte_ring_rts.h +0 -2
- * @param available
- *   returns the number of remaining ring entries after the dequeue

has

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

finished

lib/librte_ring/rte_ring_rts.h line 125

Quoted 6 lines
lib/librte_ring/rte_ring_rts.h +0 -6
- * @return
- *   - Actual number of objects dequeued.
- *     If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only.
- */
-static __rte_always_inline unsigned int
-__rte_ring_do_rts_dequeue_elem(struct rte_ring *r, void

*obj_table,

Quoted 1 line
Quoted 1 line
Quoted 1 line

lib/librte_ring/rte_ring_rts.h line 126

Quoted 1 line
lib/librte_ring/rte_ring_rts.h +0 -1
-	uint32_t esize, uint32_t n, enum rte_ring_queue_behavior

behavior,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

lib/librte_ring/rte_ring_rts.h line 131

Quoted 5 lines
lib/librte_ring/rte_ring_rts.h +0 -5
-	uint32_t *available)
-{
-	uint32_t entries, head;
-
-	n = __rte_ring_rts_move_cons_head(r, n, behavior, &head,

&entries);

lib/librte_ring/rte_ring_rts.h line 134

Quoted 3 lines
lib/librte_ring/rte_ring_rts.h +0 -3
-
-	if (n != 0) {
-		__rte_ring_dequeue_elems(r, head, obj_table, esize,

n);

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
lib/librte_ring/rte_ring_rts.h +1 -8
-		__rte_ring_rts_update_tail(&r->rts_cons);
-	}
-
-	if (available != NULL)
-		*available = entries - n;
-	return n;
-}
+#include <ring_rts_elem_pvt.h>
 
 /**
  * Enqueue several objects on the RTS ring (multi-producers safe).
--

2.17.1

author
#12 lore raw

v2 cover refactoring ring library

In reply to Feifei Wang

Do some work for ring refactoring, which includes:
1. add rte prefix before update tail API
2. Move all private APIs into new header files

v2:
1. add rte_ prefixes for private files (Konstantin)

Feifei Wang (3):
test/ring: reduce iteration numbers to make test duration shorter
ring: add rte prefix before update tail API
ring: refactor ring library

 app/test/test_ring_perf.c                     |   2 +-
 lib/librte_ring/meson.build                   |  11 +-
 ...{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} |  10 +-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_elem_pvt.h           | 385 ++++++++++++++++++
 ..._ring_generic.h => rte_ring_generic_pvt.h} |  10 +-
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 ..._hts_c11_mem.h => rte_ring_hts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 ...eek_c11_mem.h => rte_ring_peek_elem_pvt.h} |  75 +++-
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 ..._rts_c11_mem.h => rte_ring_rts_elem_pvt.h} |  88 +++-
 13 files changed, 649 insertions(+), 635 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} (96%)
 create mode 100644 lib/librte_ring/rte_ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => rte_ring_generic_pvt.h} (95%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => rte_ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => rte_ring_peek_elem_pvt.h} (61%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => rte_ring_rts_elem_pvt.h} (62%)
Signature · 1 line
2.25.1
author
#13 lore raw

PATCH v2 1/3 test/ring: reduce iteration numbers to make test duration shorter

When testing ring performance in the case that multiple lcores are mapped to the same physical core, e.g. --lcores '(0-3)@10', it takes a very long time to wait for the "enqueue_dequeue_bulk_helper" to finish. This is because too much iteration numbers and extremely low efficiency for enqueue and dequeue with this kind of core mapping. Following are the test results to show the above phenomenon:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test --lcores '(0-1)@25'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           7s    7s    7s    8s    9s     16s    47s    170s   660s   >0.5h   >1h
legacy APIs: SP/SC: 37    11    6     40525 40525  40209  40367  40407  40541  NoData  NoData
legacy APIs: MP/MC: 56    14    11    50657 40526  40526  40526  40625  40585  NoData  NoData

aarch64-n1sdp:
$sudo ./app/test/dpdk-test --lcore '(0-1)@1'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    9s    9s     14s    34s    111s   418s   25min   >1h
legacy APIs: SP/SC: 0.4   0.2   0.1   488   488    488    488    488    489    489     NoData
legacy APIs: MP/MC: 0.4   0.3   0.2   488   488    488    488    490    489    489     NoData

As the number of iterations increases, so does the time which is required to run the program. Currently (iter_shift = 23), it will take more than 1 hour to wait for the test to finish. To fix this, the "iter_shift" should decrease and ensure enough iterations to keep the test data stable. In order to achieve this, we also test with "-l" EAL argument:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test -l 25-26
Testing using two NUMA nodes(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           6s    6s    6s    6s    6s     6s     6s     7s     8s     11s     27s
legacy APIs: SP/SC: 47    20    13    22    54     83     91     73     81     75      95
legacy APIs: MP/MC: 44    18    18    240   245    270    250    249    252    250     253

aarch64-n1sdp:
$sudo ./app/test/dpdk-test -l 1-2
Testing using two physical cores(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    8s    8s     8s     8s     9s     9s     11s     23s
legacy APIs: SP/SC: 0.7   0.4   1.2   1.8   2.0    2.0    2.0    2.0    2.0    2.0     2.0
legacy APIs: MP/MC: 0.3   0.4   1.3   1.9   2.9    2.9    2.9    2.9    2.9    2.9     2.9

According to above test data, when "iter_shift" is set as "15", the test run time is reduced to less than 1 minute and the test result can keep stable in x86 and aarch64 servers.

Fixes 1fa5d0099efc ("test/ring: add custom element size performance tests")
Cc [email protected]
Cc [email protected]
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
Acked-by Konstantin Ananyev <[email protected]>
---
 app/test/test_ring_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
app/test/test_ring_perf.c +1 -1
index e63e25a86..fd82e2041 100644
@@ -178,7 +178,7 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 	struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 15;
 	const unsigned int iterations = 1 << iter_shift;
 	struct rte_ring *r = p->r;
 	unsigned int bsize = p->size;
Signature · 1 line
2.25.1
author
#14 lore raw

PATCH v2 2/3 ring: add rte prefix before update tail API

In reply to Feifei Wang

Add __rte prefix before update_tail API because it is a internal function.

Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
Acked-by Konstantin Ananyev <[email protected]>
---
 lib/librte_ring/rte_ring_c11_mem.h | 4 ++--
 lib/librte_ring/rte_ring_elem.h    | 4 ++--
 lib/librte_ring/rte_ring_generic.h | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_c11_mem.h +2 -2
index 0fb73a337..7f5eba262 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_C11_MEM_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	RTE_SET_USED(enqueue);
lib/librte_ring/rte_ring_elem.h +2 -2
index 7034d29c0..57344d47d 100644
@@ -423,7 +423,7 @@ __rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
 
 	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
 
-	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
 	if (free_space != NULL)
 		*free_space = free_entries - n;

@@ -470,7 +470,7 @@ __rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
 
 	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
 
-	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
 end:
 	if (available != NULL)
lib/librte_ring/rte_ring_generic.h +2 -2
index 953cdbbd5..37c62b8d6 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_GENERIC_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	if (enqueue)
 		rte_smp_wmb();
Signature · 1 line
2.25.1
author
#15 lore raw

PATCH v2 3/3 ring: refactor ring library

In reply to Feifei Wang

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Finally, the external APIs and internal APIs will be separated from each other. This can remind users not to use internal APIs and make ring library easier to maintain.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
---
 lib/librte_ring/meson.build                   |  11 +-
 ...{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} |   6 +-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_elem_pvt.h           | 385 ++++++++++++++++++
 ..._ring_generic.h => rte_ring_generic_pvt.h} |   6 +-
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 ..._hts_c11_mem.h => rte_ring_hts_elem_pvt.h} |  88 +++-
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 ...eek_c11_mem.h => rte_ring_peek_elem_pvt.h} |  75 +++-
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 ..._rts_c11_mem.h => rte_ring_rts_elem_pvt.h} |  88 +++-
 12 files changed, 644 insertions(+), 630 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} (98%)
 create mode 100644 lib/librte_ring/rte_ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => rte_ring_generic_pvt.h} (97%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => rte_ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => rte_ring_peek_elem_pvt.h} (61%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => rte_ring_rts_elem_pvt.h} (62%)
lib/librte_ring/meson.build +6 -5
index 36fdcb6a5..f007893b5 100644
@@ -5,12 +5,13 @@ sources = files('rte_ring.c')
 headers = files('rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
+		'rte_ring_elem_pvt.h',
+		'rte_ring_c11_pvt.h',
+		'rte_ring_generic_pvt.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
+		'rte_ring_hts_elem_pvt.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
+		'rte_ring_peek_elem_pvt.h',
 		'rte_ring_peek_zc.h',
 		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts_elem_pvt.h')
lib/librte_ring/rte_ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +3 -3
similarity index 98%
rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/rte_ring_c11_pvt.h
index 7f5eba262..759192f4c 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RTE_RING_C11_PVT_H_
+#define _RTE_RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -178,4 +178,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RTE_RING_C11_PVT_H_ */
lib/librte_ring/rte_ring_elem.h +1 -373

386 lines not on this page · raw message

lib/librte_ring/rte_ring_elem_pvt.h new file +385 -0

385 lines not on this page · raw message

lib/librte_ring/rte_ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +3 -3
similarity index 97%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/rte_ring_generic_pvt.h
index 37c62b8d6..532deb5e7 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RTE_RING_GENERIC_PVT_H_
+#define _RTE_RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RTE_RING_GENERIC_PVT_H_ */
lib/librte_ring/rte_ring_hts.h +1 -83

90 lines not on this page · raw message

lib/librte_ring/rte_ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +85 -3

97 lines not on this page · raw message

lib/librte_ring/rte_ring_peek.h +1 -70

83 lines not on this page · raw message

lib/librte_ring/rte_ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +72 -3

84 lines not on this page · raw message

lib/librte_ring/rte_ring_peek_zc.h +1 -1
index cb3bbd067..be677a3e1 100644
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <rte_ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
lib/librte_ring/rte_ring_rts.h +1 -83

90 lines not on this page · raw message

lib/librte_ring/rte_ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +85 -3

97 lines not on this page · raw message

Signature · 1 line
2.25.1
Acked-by on v2 3/3
#16 lore raw

Re: [dpdk-dev] [PATCH v2 3/3] ring: refactor ring library

Quoted 12 lines

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Finally, the external APIs and internal APIs will be separated from each other. This can remind users not to use internal APIs and make ring library easier to maintain.

Quoted 5 lines
Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>

---

Acked-by Konstantin Ananyev <[email protected]>
Quoted 2 lines

--
2.25.1

#17 lore raw
In reply to Feifei Wang
Quoted 7 lines · Feifei Wang

Do some work for ring refactoring, which includes:
1. add rte prefix before update tail API
2. Move all private APIs into new header files

v2:
1. add rte_ prefixes for private files (Konstantin)

This series seems simple enough and safe for taking in rc2. Honnappa and Konstantin seem ok with it, so I'll proceed unless I hear any objection by tomorrow.

Thanks.

Signature · 1 line
David Marchand
author
#18 lore raw

v3 cover refactoring ring library

In reply to Feifei Wang

Do some work for ring refactoring, which includes:
1. add rte prefix before update tail API
2. Move all private APIs into new header files

v2:
1. add rte_ prefixes for private files (Konstantin)

v3:
1. fix compile issue of file name bug

Feifei Wang (3):
test/ring: reduce iteration numbers to make test duration shorter
ring: add rte prefix before update tail API
ring: refactor ring library

 app/test/test_ring_perf.c                     |   2 +-
 lib/librte_ring/meson.build                   |  11 +-
 ...{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} |  10 +-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_elem_pvt.h           | 385 ++++++++++++++++++
 ..._ring_generic.h => rte_ring_generic_pvt.h} |  10 +-
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 ..._hts_c11_mem.h => rte_ring_hts_elem_pvt.h} |  90 +++-
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 ...eek_c11_mem.h => rte_ring_peek_elem_pvt.h} |  77 +++-
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 ..._rts_c11_mem.h => rte_ring_rts_elem_pvt.h} |  90 +++-
 13 files changed, 652 insertions(+), 638 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} (96%)
 create mode 100644 lib/librte_ring/rte_ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => rte_ring_generic_pvt.h} (95%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => rte_ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => rte_ring_peek_elem_pvt.h} (61%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => rte_ring_rts_elem_pvt.h} (62%)
Signature · 1 line
2.25.1
author
#19 lore raw

PATCH v3 1/3 test/ring: reduce iteration numbers to make test duration shorter

When testing ring performance in the case that multiple lcores are mapped to the same physical core, e.g. --lcores '(0-3)@10', it takes a very long time to wait for the "enqueue_dequeue_bulk_helper" to finish. This is because too much iteration numbers and extremely low efficiency for enqueue and dequeue with this kind of core mapping. Following are the test results to show the above phenomenon:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test --lcores '(0-1)@25'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           7s    7s    7s    8s    9s     16s    47s    170s   660s   >0.5h   >1h
legacy APIs: SP/SC: 37    11    6     40525 40525  40209  40367  40407  40541  NoData  NoData
legacy APIs: MP/MC: 56    14    11    50657 40526  40526  40526  40625  40585  NoData  NoData

aarch64-n1sdp:
$sudo ./app/test/dpdk-test --lcore '(0-1)@1'
Testing using two hyperthreads(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    9s    9s     14s    34s    111s   418s   25min   >1h
legacy APIs: SP/SC: 0.4   0.2   0.1   488   488    488    488    488    489    489     NoData
legacy APIs: MP/MC: 0.4   0.3   0.2   488   488    488    488    490    489    489     NoData

As the number of iterations increases, so does the time which is required to run the program. Currently (iter_shift = 23), it will take more than 1 hour to wait for the test to finish. To fix this, the "iter_shift" should decrease and ensure enough iterations to keep the test data stable. In order to achieve this, we also test with "-l" EAL argument:

x86-Intel(R) Xeon(R) Gold 6240:
$sudo ./app/test/dpdk-test -l 25-26
Testing using two NUMA nodes(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           6s    6s    6s    6s    6s     6s     6s     7s     8s     11s     27s
legacy APIs: SP/SC: 47    20    13    22    54     83     91     73     81     75      95
legacy APIs: MP/MC: 44    18    18    240   245    270    250    249    252    250     253

aarch64-n1sdp:
$sudo ./app/test/dpdk-test -l 1-2
Testing using two physical cores(bulk (size: 8):)

iter_shift:         3     5     7     9     11     13    *15     17     19     21      23
run time:           8s    8s    8s    8s    8s     8s     8s     9s     9s     11s     23s
legacy APIs: SP/SC: 0.7   0.4   1.2   1.8   2.0    2.0    2.0    2.0    2.0    2.0     2.0
legacy APIs: MP/MC: 0.3   0.4   1.3   1.9   2.9    2.9    2.9    2.9    2.9    2.9     2.9

According to above test data, when "iter_shift" is set as "15", the test run time is reduced to less than 1 minute and the test result can keep stable in x86 and aarch64 servers.

Fixes 1fa5d0099efc ("test/ring: add custom element size performance tests")
Cc [email protected]
Cc [email protected]
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
Acked-by Konstantin Ananyev <[email protected]>
---
 app/test/test_ring_perf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
app/test/test_ring_perf.c +1 -1
index e63e25a86..fd82e2041 100644
@@ -178,7 +178,7 @@ enqueue_dequeue_bulk_helper(const unsigned int flag, const int esize,
 	struct thread_params *p)
 {
 	int ret;
-	const unsigned int iter_shift = 23;
+	const unsigned int iter_shift = 15;
 	const unsigned int iterations = 1 << iter_shift;
 	struct rte_ring *r = p->r;
 	unsigned int bsize = p->size;
Signature · 1 line
2.25.1
author
#20 lore raw

PATCH v3 2/3 ring: add rte prefix before update tail API

In reply to Feifei Wang

Add __rte prefix before update_tail API because it is a internal function.

Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
Acked-by Konstantin Ananyev <[email protected]>
---
 lib/librte_ring/rte_ring_c11_mem.h | 4 ++--
 lib/librte_ring/rte_ring_elem.h    | 4 ++--
 lib/librte_ring/rte_ring_generic.h | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
lib/librte_ring/rte_ring_c11_mem.h +2 -2
index 0fb73a337..7f5eba262 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_C11_MEM_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	RTE_SET_USED(enqueue);
lib/librte_ring/rte_ring_elem.h +2 -2
index 7034d29c0..57344d47d 100644
@@ -423,7 +423,7 @@ __rte_ring_do_enqueue_elem(struct rte_ring *r, const void *obj_table,
 
 	__rte_ring_enqueue_elems(r, prod_head, obj_table, esize, n);
 
-	update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
+	__rte_ring_update_tail(&r->prod, prod_head, prod_next, is_sp, 1);
 end:
 	if (free_space != NULL)
 		*free_space = free_entries - n;

@@ -470,7 +470,7 @@ __rte_ring_do_dequeue_elem(struct rte_ring *r, void *obj_table,
 
 	__rte_ring_dequeue_elems(r, cons_head, obj_table, esize, n);
 
-	update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
+	__rte_ring_update_tail(&r->cons, cons_head, cons_next, is_sc, 0);
 
 end:
 	if (available != NULL)
lib/librte_ring/rte_ring_generic.h +2 -2
index 953cdbbd5..37c62b8d6 100644
@@ -11,8 +11,8 @@
 #define _RTE_RING_GENERIC_H_
 
 static __rte_always_inline void
-update_tail(struct rte_ring_headtail *ht, uint32_t old_val, uint32_t new_val,
-		uint32_t single, uint32_t enqueue)
+__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
+		uint32_t new_val, uint32_t single, uint32_t enqueue)
 {
 	if (enqueue)
 		rte_smp_wmb();
Signature · 1 line
2.25.1
author
#21 lore raw

PATCH v3 3/3 ring: refactor ring library

In reply to Feifei Wang

For legacy modes, rename ring_generic/c11 to ring_generic/c11_pvt. Furthermore, add new file ring_elem_pvt.h which includes ring_do_eq/deq and ring element copy/delete APIs.

For other modes, rename xx_c11_mem to xx_elem_pvt. Move all private APIs into these new header files.

Finally, the external APIs and internal APIs will be separated from each other. This can remind users not to use internal APIs and make ring library easier to maintain.

Suggested-by Honnappa Nagarahalli <[email protected]>
Signed-off-by Feifei Wang <[email protected]>
Reviewed-by Honnappa Nagarahalli <[email protected]>
Reviewed-by Ruifeng Wang <[email protected]>
Acked-by Konstantin Ananyev <[email protected]>
---
 lib/librte_ring/meson.build                   |  11 +-
 ...{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} |   6 +-
 lib/librte_ring/rte_ring_elem.h               | 374 +----------------
 lib/librte_ring/rte_ring_elem_pvt.h           | 385 ++++++++++++++++++
 ..._ring_generic.h => rte_ring_generic_pvt.h} |   6 +-
 lib/librte_ring/rte_ring_hts.h                |  84 +---
 ..._hts_c11_mem.h => rte_ring_hts_elem_pvt.h} |  90 +++-
 lib/librte_ring/rte_ring_peek.h               |  71 +---
 ...eek_c11_mem.h => rte_ring_peek_elem_pvt.h} |  77 +++-
 lib/librte_ring/rte_ring_peek_zc.h            |   2 +-
 lib/librte_ring/rte_ring_rts.h                |  84 +---
 ..._rts_c11_mem.h => rte_ring_rts_elem_pvt.h} |  90 +++-
 12 files changed, 647 insertions(+), 633 deletions(-)
 rename lib/librte_ring/{rte_ring_c11_mem.h => rte_ring_c11_pvt.h} (98%)
 create mode 100644 lib/librte_ring/rte_ring_elem_pvt.h
 rename lib/librte_ring/{rte_ring_generic.h => rte_ring_generic_pvt.h} (97%)
 rename lib/librte_ring/{rte_ring_hts_c11_mem.h => rte_ring_hts_elem_pvt.h} (60%)
 rename lib/librte_ring/{rte_ring_peek_c11_mem.h => rte_ring_peek_elem_pvt.h} (61%)
 rename lib/librte_ring/{rte_ring_rts_c11_mem.h => rte_ring_rts_elem_pvt.h} (62%)
lib/librte_ring/meson.build +6 -5
index 36fdcb6a5..f007893b5 100644
@@ -5,12 +5,13 @@ sources = files('rte_ring.c')
 headers = files('rte_ring.h',
 		'rte_ring_core.h',
 		'rte_ring_elem.h',
-		'rte_ring_c11_mem.h',
-		'rte_ring_generic.h',
+		'rte_ring_elem_pvt.h',
+		'rte_ring_c11_pvt.h',
+		'rte_ring_generic_pvt.h',
 		'rte_ring_hts.h',
-		'rte_ring_hts_c11_mem.h',
+		'rte_ring_hts_elem_pvt.h',
 		'rte_ring_peek.h',
-		'rte_ring_peek_c11_mem.h',
+		'rte_ring_peek_elem_pvt.h',
 		'rte_ring_peek_zc.h',
 		'rte_ring_rts.h',
-		'rte_ring_rts_c11_mem.h')
+		'rte_ring_rts_elem_pvt.h')
lib/librte_ring/rte_ring_c11_pvt.h renamed from lib/librte_ring/rte_ring_c11_mem.h +3 -3
similarity index 98%
rename from lib/librte_ring/rte_ring_c11_mem.h
rename to lib/librte_ring/rte_ring_c11_pvt.h
index 7f5eba262..759192f4c 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_C11_MEM_H_
-#define _RTE_RING_C11_MEM_H_
+#ifndef _RTE_RING_C11_PVT_H_
+#define _RTE_RING_C11_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -178,4 +178,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_C11_MEM_H_ */
+#endif /* _RTE_RING_C11_PVT_H_ */
lib/librte_ring/rte_ring_elem.h +1 -373

386 lines not on this page · raw message

lib/librte_ring/rte_ring_elem_pvt.h new file +385 -0

385 lines not on this page · raw message

lib/librte_ring/rte_ring_generic_pvt.h renamed from lib/librte_ring/rte_ring_generic.h +3 -3
similarity index 97%
rename from lib/librte_ring/rte_ring_generic.h
rename to lib/librte_ring/rte_ring_generic_pvt.h
index 37c62b8d6..532deb5e7 100644
@@ -7,8 +7,8 @@
  * Used as BSD-3 Licensed with permission from Kip Macy.
  */
 
-#ifndef _RTE_RING_GENERIC_H_
-#define _RTE_RING_GENERIC_H_
+#ifndef _RTE_RING_GENERIC_PVT_H_
+#define _RTE_RING_GENERIC_PVT_H_
 
 static __rte_always_inline void
 __rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,

@@ -170,4 +170,4 @@ __rte_ring_move_cons_head(struct rte_ring *r, unsigned int is_sc,
 	return n;
 }
 
-#endif /* _RTE_RING_GENERIC_H_ */
+#endif /* _RTE_RING_GENERIC_PVT_H_ */
lib/librte_ring/rte_ring_hts.h +1 -83

90 lines not on this page · raw message

lib/librte_ring/rte_ring_hts_elem_pvt.h renamed from lib/librte_ring/rte_ring_hts_c11_mem.h +86 -4

101 lines not on this page · raw message

lib/librte_ring/rte_ring_peek.h +1 -70

83 lines not on this page · raw message

lib/librte_ring/rte_ring_peek_elem_pvt.h renamed from lib/librte_ring/rte_ring_peek_c11_mem.h +73 -4

88 lines not on this page · raw message

lib/librte_ring/rte_ring_peek_zc.h +1 -1
index cb3bbd067..be677a3e1 100644
@@ -72,7 +72,7 @@
 extern "C" {
 #endif
 
-#include <rte_ring_peek_c11_mem.h>
+#include <rte_ring_peek_elem_pvt.h>
 
 /**
  * Ring zero-copy information structure.
lib/librte_ring/rte_ring_rts.h +1 -83

90 lines not on this page · raw message

lib/librte_ring/rte_ring_rts_elem_pvt.h renamed from lib/librte_ring/rte_ring_rts_c11_mem.h +86 -4

101 lines not on this page · raw message

Signature · 1 line
2.25.1
#22 lore raw
In reply to Feifei Wang
Quoted 10 lines · Feifei Wang

Do some work for ring refactoring, which includes:
1. add rte prefix before update tail API
2. Move all private APIs into new header files

v2:
1. add rte_ prefixes for private files (Konstantin)

v3:
1. fix compile issue of file name bug

Series applied, thanks.

Signature · 1 line
David Marchand