Skip to contentSkip to conversation

doc: announce ring ABI and API changes

PATCH In review Gage Eads 5 participants 14 messages Jan 16, 2019 – May 13, 2019

lore
Share & tools
lore

Gage Eads

author
#0 lore raw

PATCH 1/1 doc: announce ring ABI and API changes

In order to support the non-blocking ring[1], one ABI change and one API change are required in librte_ring. This commit updates the deprecation notice to pave the way for their inclusion in 19.05.

[1] http://mails.dpdk.org/archives/dev/2019-January/123475.html

---
 doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)
doc/guides/rel_notes/deprecation.rst +11 -0
index d4aea4b46..d74cff467 100644
@@ -83,3 +83,14 @@ Deprecation Notices
   - The size and layout of ``rte_cryptodev_qp_conf`` and syntax of
     ``rte_cryptodev_queue_pair_setup`` will change to to allow to use
     two different mempools for crypto and device private sessions.
+
+* ring: two changes are planned for rte_ring in v19.05:
+
+  - The ring head and tail values are planned to be changed from ``uint32_t``
+    to ``size_t``. This reduces the likelihood of wrap-around to effectively
+    zero for 64-bit builds, which is important in avoiding the ABA problem in
+    the upcoming non-blocking ring implementation. (32-bit builds are
+    unaffected by this change.)
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per entry
+    (such as the upcoming non-blocking ring).
Signature · 1 line
2.13.6
#1 lore raw
Quoted 20 lines · On Tue, 15 Jan 2019 17:59:34 -0600 Gage Eads <[email protected]> wrote:

In order to support the non-blocking ring[1], one ABI change and one API change are required in librte_ring. This commit updates the deprecation notice to pave the way for their inclusion in 19.05.

[1] http://mails.dpdk.org/archives/dev/2019-January/123475.html

---
 doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)
doc/guides/rel_notes/deprecation.rst +1 -0
index d4aea4b46..d74cff467 100644
@@ -83,3 +83,14 @@ Deprecation Notices
   - The size and layout of ``rte_cryptodev_qp_conf`` and syntax of
     ``rte_cryptodev_queue_pair_setup`` will change to to allow to use
     two different mempools for crypto and device private sessions.
+

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 96

Quoted 10 lines
doc/guides/rel_notes/deprecation.rst +10 -0
+* ring: two changes are planned for rte_ring in v19.05:
+
+  - The ring head and tail values are planned to be changed from ``uint32_t``
+    to ``size_t``. This reduces the likelihood of wrap-around to effectively
+    zero for 64-bit builds, which is important in avoiding the ABA problem in
+    the upcoming non-blocking ring implementation. (32-bit builds are
+    unaffected by this change.)
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per entry
+    (such as the upcoming non-blocking ring).

Would it be possible to support new and old ring types, either through naming tricks and/or new ring flag? Changing things like ring buffer and mbuf are basically a flag day for all users.

I admit to having a personal interest in this since the API/ABI churn is this project causes vendors to stay on older code. And older code does not correctly support newer networks.

author
#2 lore raw
Quoted 48 lines

-----Original Message-----
From: Stephen Hemminger [mailto:[email protected]]
Sent: Tuesday, January 15, 2019 6:34 PM
To: Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>
Subject: Re: [PATCH] doc: announce ring ABI and API changes

Quoted 20 lines · On Tue, 15 Jan 2019 17:59:34 -0600 Gage Eads <[email protected]> wrote:

In order to support the non-blocking ring[1], one ABI change and one API change are required in librte_ring. This commit updates the deprecation notice to pave the way for their inclusion in 19.05.

[1] http://mails.dpdk.org/archives/dev/2019-January/123475.html

---
 doc/guides/rel_notes/deprecation.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)
doc/guides/rel_notes/deprecation.rst +0 -0
index d4aea4b46..d74cff467 100644
@@ -83,3 +83,14 @@ Deprecation Notices
   - The size and layout of ``rte_cryptodev_qp_conf`` and syntax of
     ``rte_cryptodev_queue_pair_setup`` will change to to allow to use
     two different mempools for crypto and device private sessions.

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 95

Quoted 10 lines
doc/guides/rel_notes/deprecation.rst +10 -0
+
+* ring: two changes are planned for rte_ring in v19.05:
+
+  - The ring head and tail values are planned to be changed from ``uint32_t``
+    to ``size_t``. This reduces the likelihood of wrap-around to effectively
+    zero for 64-bit builds, which is important in avoiding the ABA problem in
+    the upcoming non-blocking ring implementation. (32-bit builds are
+    unaffected by this change.)
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per

entry

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 96

Quoted 1 line
doc/guides/rel_notes/deprecation.rst +1 -0
+    (such as the upcoming non-blocking ring).
Quoted 7 lines

Would it be possible to support new and old ring types, either through naming tricks and/or new ring flag? Changing things like ring buffer and mbuf are basically a flag day for all users.

I admit to having a personal interest in this since the API/ABI churn is this project causes vendors to stay on older code. And older code does not correctly support newer networks.

Fair enough -- I appreciate the additional context wrt avoiding churn.

This might be doable with the following change:

"
@@ -70,6 +70,15 @@ struct rte_ring_headtail {
uint32_t single; /**< True if single prod/cons */
};

+/* 64-bit version of rte_ring_headtail, for use by rings that need to avoid
+ * head/tail wrap-around.
+ */
+struct rte_ring_headtail_64 {

+       volatile uint64_t head;  /**< Prod/consumer head. */
+       volatile uint64_t tail;  /**< Prod/consumer tail. */
+       uint32_t single;       /**< True if single prod/cons */

+};
+
/**
* An RTE ring structure.
*
@@ -97,11 +106,19 @@ struct rte_ring {
char pad0 __rte_cache_aligned; /**< empty cache line */

/** Ring producer status. */

-       struct rte_ring_headtail prod __rte_cache_aligned;
+       RTE_STD_C11
+       union {

+ struct rte_ring_headtail prod __rte_cache_aligned;
+ struct rte_ring_headtail_64 prod_64 __rte_cache_aligned;
+ };
char pad1 __rte_cache_aligned; /**< empty cache line */

/** Ring consumer status. */

-       struct rte_ring_headtail cons __rte_cache_aligned;
+       RTE_STD_C11
+       union {

+ struct rte_ring_headtail cons __rte_cache_aligned;
+ struct rte_ring_headtail_64 cons_64 __rte_cache_aligned;
+ };
char pad2 __rte_cache_aligned; /**< empty cache line */
};
"

The ABI compatibility hinges on the fact that today's prod and cons are both padded out to a full cache line, and the 64-bit version fits within a single cache line. (Confirmed with pahole.)

abi-compliance-checker reports two issues, but both appear to be false positives:
1. "Field cons has been removed from this type"
2. "Field prod has been removed from this type"

I need to do more work to see whether/how the ring functions are affected by such a change, but I first want to check if the community agrees with this approach. Note that I don't see any way to avoid the API change to rte_ring_get_memsize, but I doubt that would have near the impact of a ring data structure change.

Thanks,
Gage

author
#3 lore raw

PATCH v2 1/1 doc: announce ring API change

In reply to Gage Eads

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.05.

[1] http://mails.dpdk.org/archives/dev/2019-January/123774.html

---
v2:
- Drop the ABI change notice

 doc/guides/rel_notes/deprecation.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/rel_notes/deprecation.rst +6 -0
index d4aea4b46..5b74a2aa7 100644
@@ -83,3 +83,9 @@ Deprecation Notices
   - The size and layout of ``rte_cryptodev_qp_conf`` and syntax of
     ``rte_cryptodev_queue_pair_setup`` will change to to allow to use
     two different mempools for crypto and device private sessions.
+
+* ring: two changes are planned for rte_ring in v19.05:
+
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per entry
+    (such as the upcoming non-blocking ring).
Signature · 1 line
2.13.6
author
#4 lore raw

PATCH v3 1/1 doc: announce ring API change

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.05.

[1] http://mails.dpdk.org/archives/dev/2019-January/123774.html

---
v3:
- "two changes are planned" -> "one change is planned"
v2:
- Drop the ABI change notice

 doc/guides/rel_notes/deprecation.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/rel_notes/deprecation.rst +6 -0
index d4aea4b46..91e048a6a 100644
@@ -83,3 +83,9 @@ Deprecation Notices
   - The size and layout of ``rte_cryptodev_qp_conf`` and syntax of
     ``rte_cryptodev_queue_pair_setup`` will change to to allow to use
     two different mempools for crypto and device private sessions.
+
+* ring: one change is planned for rte_ring in v19.05:
+
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per entry
+    (such as the upcoming non-blocking ring).
Signature · 1 line
2.13.6
#5 lore raw

Re: [PATCH v3] doc: announce ring API change

18/01/2019 16:31, Gage Eads:

Quoted 8 lines

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.05.

[1] http://mails.dpdk.org/archives/dev/2019-January/123774.html

Nobody agreed on this change.

Gage, do you still want to push non-blocking ring in 19.05?

---

Quoted 5 lines
doc/guides/rel_notes/deprecation.rst +5 -0
+* ring: one change is planned for rte_ring in v19.05:
+
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per entry
+    (such as the upcoming non-blocking ring).
author
#6 lore raw

Re: [PATCH v3] doc: announce ring API change

Quoted 18 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Friday, February 1, 2019 5:17 AM
To: [email protected]; Eads, Gage <[email protected]>
Cc: [email protected]; [email protected]; Richardson, Bruce <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; Mcnamara, John <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3] doc: announce ring API change

18/01/2019 16:31, Gage Eads:

Quoted 5 lines

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.05.

Quoted 8 lines
Quoted 3 lines

[1] http://mails.dpdk.org/archives/dev/2019-January/123774.html

Nobody agreed on this change.

Gage, do you still want to push non-blocking ring in 19.05?

It was optimistic to try for agreement before the 19.02 release :) -- no, let's delay it until 19.08. I'll edit and resubmit this patch.

Quoted 13 lines

---

doc/guides/rel_notes/deprecation.rst Deprecation Notices

Quoted 4 lines
doc/guides/rel_notes/deprecation.rst +4 -0
+* ring: one change is planned for rte_ring in v19.05:
+
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per

entry

Quoted 1 line

+ (such as the upcoming non-blocking ring).

author
#7 lore raw

PATCH v4 1/1 doc: announce ring API change

In reply to Gage Eads

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

---
v4:
- 19.05 -> 19.08
v3:
- "two changes are planned" -> "one change is planned"
v2:
- Drop the ABI change notice

 doc/guides/rel_notes/deprecation.rst | 6 ++++++
 1 file changed, 6 insertions(+)
doc/guides/rel_notes/deprecation.rst +6 -0
index d4aea4b46..93509e23a 100644
@@ -83,3 +83,9 @@ Deprecation Notices
   - The size and layout of ``rte_cryptodev_qp_conf`` and syntax of
     ``rte_cryptodev_queue_pair_setup`` will change to to allow to use
     two different mempools for crypto and device private sessions.
+
+* ring: one change is planned for rte_ring in v19.08:
+
+  - rte_ring_get_memsize() will get a new ``flags`` parameter, so it can
+    calculate the memory required for rings that require more than 8B per entry
+    (such as the upcoming non-blocking ring).
Signature · 1 line
2.13.6
#8 lore raw

Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

01/02/2019 15:36, Gage Eads:

Quoted 8 lines

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

There is still no agreement on this change?

author
#9 lore raw

Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 6 lines

01/02/2019 15:36, Gage Eads:

Quoted 5 lines

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

Quoted 6 lines
Quoted 3 lines

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

There is still no agreement on this change?

Still none. I was hoping this discussion (http://mails.dpdk.org/archives/dev/2019-April/129229.html) would lead to some clear direction, but at this point the effort is stalled.

#10 lore raw

Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 6 lines · Gage
Quoted 6 lines

01/02/2019 15:36, Gage Eads:

Quoted 5 lines

In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

There is still no agreement on this change?

Still none. I was hoping this discussion (http://mails.dpdk.org/archives/dev/2019-April/129229.html) would lead to some clear direction, but at this point the effort is stalled.

The fundamental tradeoff is between non-blocking rings and ABI breakage.

Why not have a new ring type for non-blocking rings since non-blocking rings are not necessary for all use cases.

#11 lore raw

Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 13 lines · Stephen Hemminger

On Fri, 10 May 2019 14:53:56 +0000
"Eads, Gage" <[email protected]> wrote:

01/02/2019 15:36, Gage Eads:
In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

Quoted 8 lines

There is still no agreement on this change?

Still none. I was hoping this discussion (
http://mails.dpdk.org/archives/dev/2019-April/129229.html) would lead to some clear direction, but at this point the effort is stalled.

The fundamental tradeoff is between non-blocking rings and ABI breakage.

It is also possible to do "non-blocking" (but not lock-free) rings with the original element size (a pointer per ring slot) as implemented here:
https://github.com/ARM-software/progress64/blob/master/src/p64_ringbuf.c
Some extra (head&tail) metadata is required but I think there is space for that in the rte_ring structure.

Quoted 2 lines

Why not have a new ring type for non-blocking rings since non-blocking rings are not necessary for all use cases.

I proposed a new library ("rte_lfring") with lock-free rings here:
https://mails.dpdk.org/archives/dev/2019-January/124242.html
The lock-free design should be the same as in Gage's patch.

rte_lfring could of course be part of the rte_ring library.

Signature · 2 lines
Ola Liljedahl, Networking System Architect, Arm
Phone +46706866373, Skype ola.liljedahl
author
#12 lore raw

Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 42 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, May 10, 2019 10:19 AM
To: [email protected]; Eads, Gage <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 13 lines · Stephen Hemminger

On Fri, 10 May 2019 14:53:56 +0000
"Eads, Gage" <[email protected]> wrote:

01/02/2019 15:36, Gage Eads:
In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

Quoted 8 lines

There is still no agreement on this change?

Still none. I was hoping this discussion (
http://mails.dpdk.org/archives/dev/2019-April/129229.html) would lead to some clear direction, but at this point the effort is stalled.

The fundamental tradeoff is between non-blocking rings and ABI breakage.

It is also possible to do "non-blocking" (but not lock-free) rings with the original element size (a pointer per ring slot) as implemented here:
https://github.com/ARM-
software/progress64/blob/master/src/p64_ringbuf.c
Some extra (head&tail) metadata is required but I think there is space for that in the rte_ring structure.

Quoted 9 lines
Quoted 2 lines

Why not have a new ring type for non-blocking rings since non-blocking rings are not necessary for all use cases.

I proposed a new library ("rte_lfring") with lock-free rings here:
https://mails.dpdk.org/archives/dev/2019-January/124242.html
The lock-free design should be the same as in Gage's patch.

rte_lfring could of course be part of the rte_ring library.

Just read through the API/ABI stability discussion (https://mails.dpdk.org/archives/dev/2019-April/128969.html). I'll drop my patchset and work on supporting this lfring API instead.

Thanks,
Gage

Quoted 4 lines

--
Ola Liljedahl, Networking System Architect, Arm Phone +46706866373, Skype ola.liljedahl

#13 lore raw

Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 49 lines · Gage
Quoted 49 lines

-----Original Message-----
From: Ola Liljedahl [mailto:[email protected]]
Sent: Friday, May 10, 2019 10:19 AM
To: [email protected]; Eads, Gage <[email protected]>
Cc: [email protected]; nd <[email protected]>; Ananyev, Konstantin <[email protected]>; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Honnappa Nagarahalli <[email protected]>; Gavin Hu (Arm Technology China) <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v4] doc: announce ring API change

Quoted 13 lines · Stephen Hemminger

On Fri, 10 May 2019 14:53:56 +0000
"Eads, Gage" <[email protected]> wrote:

01/02/2019 15:36, Gage Eads:
In order to support the non-blocking ring[1], an API change (additional argument to rte_ring_get_memsize()) is required in librte_ring. This commit updates the deprecation notice to pave the way for its inclusion in 19.08.

[1] http://mails.dpdk.org/archives/dev/2019-January/124162.html

Quoted 8 lines

There is still no agreement on this change?

Still none. I was hoping this discussion (
http://mails.dpdk.org/archives/dev/2019-April/129229.html) would lead to some clear direction, but at this point the effort is stalled.

The fundamental tradeoff is between non-blocking rings and ABI breakage.

It is also possible to do "non-blocking" (but not lock-free) rings with the original element size (a pointer per ring slot) as implemented here:
https://github.com/ARM-
software/progress64/blob/master/src/p64_ringbuf.c
Some extra (head&tail) metadata is required but I think there is space for that in the rte_ring structure.

Quoted 2 lines

Why not have a new ring type for non-blocking rings since non-blocking rings are not necessary for all use cases.

I proposed a new library ("rte_lfring") with lock-free rings here:
https://mails.dpdk.org/archives/dev/2019-January/124242.html
The lock-free design should be the same as in Gage's patch.

Quoted 6 lines
Quoted 2 lines

rte_lfring could of course be part of the rte_ring library.

Just read through the API/ABI stability discussion
(https://mails.dpdk.org/archives/dev/2019-April/128969.html). I'll drop my patchset and work on supporting this lfring API instead.

+1

Given the discussions related to ABI stability, it looks better to implement this in another library.

Olivier