atomic operations

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does not allow for writing optimized code for all the CPU architectures supported in DPDK. DPDK will adopt C11 atomic operations semantics and provide wrappers using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.

* rte_smp_*mb: These APIs provide full barrier functionality. However, many use cases do not require full barriers. To support such use cases, DPDK will adopt C11 barrier semantics and provide wrappers using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.
"

Should we keep these notifications forever?

It is very difficult to find which wrapper to use.

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#locks-and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-model/
https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc. Please who could work on improving the documentation?

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does not allow for writing optimized code for all the CPU architectures supported in DPDK. DPDK will adopt C11 atomic operations semantics and provide wrappers using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.

* rte_smp_*mb: These APIs provide full barrier functionality. However, many use cases do not require full barriers. To support such use cases, DPDK will adopt C11 barrier semantics and provide wrappers using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.
"

Should we keep these notifications forever?

It is very difficult to find which wrapper to use.

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#locks-and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-model/
https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc.

Please who could work on improving the documentation?

Quoted 8 lines · Thomas Monjalon

It is very difficult to find which wrapper to use.

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#locks-and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-model/
https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Quoted 3 lines

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc. Please who could work on improving the documentation?

Maybe there needs to be a DPDK version of the infamous kernel documentation of memory barriers.

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Sunday, July 4, 2021 8:38 AM
To: Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>; Phil Yang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; David Christensen <[email protected]>
Subject: Re: [dpdk-dev] atomic operations

03/07/2021 13:29, Thomas Monjalon:

Quoted 3 lines

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter.

This does

Quoted 1 line

not allow for writing optimized code for all the CPU architectures

supported

Quoted 1 line

in DPDK. DPDK will adopt C11 atomic operations semantics and provide

wrappers

Quoted 5 lines

using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.

* rte_smp_*mb: These APIs provide full barrier functionality. However,

many

Quoted 2 lines

use cases do not require full barriers. To support such use cases, DPDK will adopt C11 barrier semantics and provide wrappers using C11 atomic built-

ins.

Quoted 1 line

These wrappers must be used for patches that need to be merged in

20.08

Quoted 2 lines

onwards. This change will not introduce any performance degradation.
"

The only new wrapper is rte_atomic_thread_fence(). What else?

We are missing clear recommendations.

Quoted 1 line

Should we keep these notifications forever?

Quoted 2 lines

It is very difficult to find which wrapper to use.

We should make function names explicit instead of "These".

Quoted 5 lines

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#loc
ks-and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-

model/

Quoted 4 lines

https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc.

Precision: I meant "there is *no* example".

Quoted 1 line

Please who could work on improving the documentation?

One simple example: increment a counter atomically. __atomic_fetch_add(&counter, 1, __ATOMIC_RELAXED); or __atomic_add_fetch(&counter, 1, __ATOMIC_RELAXED);

From: Thomas Monjalon <[email protected]>

Quoted 19 lines

03/07/2021 13:29, Thomas Monjalon:

Quoted 3 lines

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter.

This does

Quoted 1 line

not allow for writing optimized code for all the CPU architectures

supported

Quoted 1 line

in DPDK. DPDK will adopt C11 atomic operations semantics and provide

wrappers

Quoted 5 lines

using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.

* rte_smp_*mb: These APIs provide full barrier functionality. However,

many

Quoted 2 lines

use cases do not require full barriers. To support such use cases, DPDK will adopt C11 barrier semantics and provide wrappers using C11 atomic built-

ins.

Quoted 1 line

These wrappers must be used for patches that need to be merged in

Quoted 5 lines

20.08

Quoted 2 lines

onwards. This change will not introduce any performance degradation.
"

The only new wrapper is rte_atomic_thread_fence(). What else?

Yes. The decision was to use GCC atomic built-ins directly.
And rte_atomic_thread_fence() is an exception. It is a wrapper of __atomic_thread_fence(), because mem order __ATOMIC_SEQ_CST has an optimized implementation for x86.

Quoted 3 lines

We are missing clear recommendations.

Quoted 1 line

Should we keep these notifications forever?

Targeting to obsolete APIs rte_atomicNN_xxx and rte_smp_*mb. Arm is working on replace occurrences with equivalent atomic built-ins. There is still a lot work to do in drivers.

Quoted 15 lines

Quoted 1 line

It is very difficult to find which wrapper to use.

We should make function names explicit instead of "These".

Quoted 5 lines

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#loc
ks-and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-

model/

Quoted 4 lines

https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc.

Quoted 3 lines

Precision: I meant "there is *no* example".

Quoted 1 line

Please who could work on improving the documentation?

Agree that the documentation needs improve.
Add link to list of atomic built-ins and the above mentioned blog posts can be part of the improvement.

Quoted 3 lines

One simple example: increment a counter atomically. __atomic_fetch_add(&counter, 1, __ATOMIC_RELAXED); or __atomic_add_fetch(&counter, 1, __ATOMIC_RELAXED);

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Monday, July 5, 2021 3:30 PM
To: Honnappa Nagarahalli <[email protected]>; Ruifeng Wang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; David Christensen <[email protected]>; nd <[email protected]>
Subject: Re: [dpdk-dev] atomic operations

05/07/2021 09:00, Ruifeng Wang:

Quoted 13 lines

From: Thomas Monjalon <[email protected]>

Quoted 12 lines

03/07/2021 13:29, Thomas Monjalon:

Quoted 3 lines

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter.

This does

Quoted 2 lines

not allow for writing optimized code for all the CPU
architectures

supported

Quoted 2 lines

in DPDK. DPDK will adopt C11 atomic operations semantics and
provide

wrappers

Quoted 1 line

using C11 atomic built-ins. These wrappers must be used for patches

that

Quoted 1 line

Quoted 1 line

Quoted 1 line

need to be merged in 20.08 onwards. This change will not introduce

any

Quoted 6 lines

Quoted 6 lines

Quoted 4 lines

performance degradation.

* rte_smp_*mb: These APIs provide full barrier functionality. However,

many

Quoted 1 line

use cases do not require full barriers. To support such use cases, DPDK

will

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

adopt C11 barrier semantics and provide wrappers using C11
atomic built-

ins.

Quoted 2 lines

These wrappers must be used for patches that need to be merged
in

Quoted 8 lines

Quoted 5 lines

20.08

Quoted 2 lines

onwards. This change will not introduce any performance degradation.
"

The only new wrapper is rte_atomic_thread_fence(). What else?

Yes. The decision was to use GCC atomic built-ins directly. And rte_atomic_thread_fence() is an exception. It is a wrapper of

__atomic_thread_fence(), because mem order __ATOMIC_SEQ_CST has an optimized implementation for x86.

Then above deprecation is wrong.

Quoted 7 lines

Quoted 3 lines

We are missing clear recommendations.

Quoted 1 line

Should we keep these notifications forever?

Targeting to obsolete APIs rte_atomicNN_xxx and rte_smp_*mb. Arm is working on replace occurrences with equivalent atomic built-ins. There is still a lot work to do in drivers.

This is an ongoing work.
In the meantime we need clear recommendation what to use.

Quoted 10 lines

Quoted 10 lines

Quoted 1 line

It is very difficult to find which wrapper to use.

We should make function names explicit instead of "These".

Quoted 6 lines

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html
#loc
ks-and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-

memory-

Quoted 10 lines

Quoted 9 lines

model/

Quoted 4 lines

https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc.

Precision: I meant "there is *no* example".

Quoted 1 line

Please who could work on improving the documentation?

Quoted 2 lines

Agree that the documentation needs improve.
Add link to list of atomic built-ins and the above mentioned blog posts can

be part of the improvement.

It should be more than a link.
We need to know when to use what.

First thing, please fix the deprecation notice.

Quoted 4 lines

Quoted 4 lines

One simple example: increment a counter atomically. __atomic_fetch_add(&counter, 1, __ATOMIC_RELAXED); or __atomic_add_fetch(&counter, 1, __ATOMIC_RELAXED);

I really hate how atomics are "documented" in GCC doc.
For instance, it doesn't say what is returned (old or new value) in above functions.

Quoted 9 lines

Quoted 4 lines

Quoted 4 lines

One simple example: increment a counter atomically. __atomic_fetch_add(&counter, 1, __ATOMIC_RELAXED); or __atomic_add_fetch(&counter, 1, __ATOMIC_RELAXED);

I really hate how atomics are "documented" in GCC doc.
For instance, it doesn't say what is returned (old or new value) in above functions.

Returns are described as "the result of the operation" and "the value that had previously been in *ptr". It can be more concise.

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported in DPDK. DPDK will adopt C11 atomic operations semantics and provide
wrappers
using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.

* rte_smp_*mb: These APIs provide full barrier functionality. However, many use cases do not require full barriers. To support such use cases, DPDK will adopt C11 barrier semantics and provide wrappers using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.
"

Should we keep these notifications forever?

It is very difficult to find which wrapper to use.

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#locks-
and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-
model/
https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc.
Please who could work on improving the documentation?

<snip>

Quoted 12 lines

In the deprecation notices of DPDK 21.05, we can still read this:
"
* rte_atomicNN_xxx: These APIs do not take memory order parameter. This does
not allow for writing optimized code for all the CPU architectures supported in DPDK. DPDK will adopt C11 atomic operations semantics and provide
wrappers
using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.

Quoted 8 lines

* rte_smp_*mb: These APIs provide full barrier functionality. However, many use cases do not require full barriers. To support such use cases, DPDK will adopt C11 barrier semantics and provide wrappers using C11 atomic built-ins. These wrappers must be used for patches that need to be merged in 20.08 onwards. This change will not introduce any performance degradation.
"

Should we keep these notifications forever?

I do not think we need to keep them forever (unless the precedence is to keep all the older deprecations).

Quoted 2 lines

It is very difficult to find which wrapper to use.

Actually, the deprecations are incorrect on the 'wrappers'.
When the deprecations were added, the understanding was we will develop wrappers, the discussion was not concluded. When we made the decision, we decided to use the C++11 atomic built-ins. Only wrapper developed was rte_atomic_thread_fence. This is documented in the following blog.

Quoted 9 lines

This is the guide we have:
https://doc.dpdk.org/guides/prog_guide/writing_efficient_code.html#locks-
and-atomic-operations
There are 2 blog posts:
https://www.dpdk.org/blog/2021/03/26/dpdk-adopts-the-c11-memory-
model/
https://www.dpdk.org/blog/2021/06/09/reader-writer-concurrency/

Quoted 3 lines

Basically it says we should use "__atomic builtins" but there is example for simple situations like counters, memory barriers, etc.
Please who could work on improving the documentation?

There is good amount of information on how to use the __atomic builtins for counters, memory barriers etc. It would make sense to document something in DPDK if we implement our own wrappers.