Skip to contentSkip to conversation

doc/eal: add caveat about spinlocks from non-pinned threads

PATCH In review Stephen Hemminger 4 participants 7 messages Jun 10, 2022 – Jun 11, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 doc/eal: add caveat about spinlocks from non-pinned threads

Need to warn users of DPDK spinlocks from non-pinned threads. This is similar wording to Linux documentation in pthread_spin_init.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 5f0748fba1c0..45d3de8d84f6 100644
@@ -797,6 +797,16 @@ Known Issues
 
   The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+  then there is a possibility of large application delays.
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until
+  the lock holder is once more rescheduled and releases the lock.
+
+
 cgroup control
 ~~~~~~~~~~~~~~
 
Signature · 1 line
2.35.1
Acked-by on v1 1/1
#1 lore raw
Quoted 4 lines · Stephen Hemminger

Need to warn users of DPDK spinlocks from non-pinned threads. This is similar wording to Linux documentation in pthread_spin_init.

Quoted 27 lines
---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +10 -0
index 5f0748fba1c0..45d3de8d84f6 100644
@@ -797,6 +797,16 @@ Known Issues
 
   The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+  then there is a possibility of large application delays.
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until
+  the lock holder is once more rescheduled and releases the lock.
+
+
 cgroup control
 ~~~~~~~~~~~~~~
 
Signature · 1 line
2.35.1
#2 lore raw
In reply to Stephen Hemminger
Quoted 13 lines · Stephen Hemminger

Need to warn users of DPDK spinlocks from non-pinned threads. This is similar wording to Linux documentation in pthread_spin_init.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +0 -0
index 5f0748fba1c0..45d3de8d84f6 100644

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 802

Quoted 6 lines
doc/guides/prog_guide/env_abstraction_layer.rst +3 -0
@@ -797,6 +797,16 @@ Known Issues
 
   The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a

nit: suggest not using term pthread but instead just say thread as not to imply a specific platform/implementation.

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 806

Quoted 4 lines
doc/guides/prog_guide/env_abstraction_layer.rst +4 -0
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+  then there is a possibility of large application delays.
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until

'until the lock holder' -> 'until the thread holding the lock'

but i'm not really fussed, just a suggestion.

Quoted 8 lines
doc/guides/prog_guide/env_abstraction_layer.rst +3 -0
+  the lock holder is once more rescheduled and releases the lock.
+
+
 cgroup control
 ~~~~~~~~~~~~~~
 
Signature · 1 line
2.35.1
#3 lore raw
Quoted 8 lines · On Fri, 10 Jun 2022 16:48:15 -0700 Tyler Retzlaff…

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 802

Quoted 4 lines

++ locking

doc/guides/prog_guide/env_abstraction_layer.rst +2 -0
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a  

nit: suggest not using term pthread but instead just say thread as not to imply a specific platform/implementation.

Quoted 8 lines

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 806

Quoted 4 lines
doc/guides/prog_guide/env_abstraction_layer.rst +4 -0
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+  then there is a possibility of large application delays.
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until  

'until the lock holder' -> 'until the thread holding the lock'

but i'm not really fussed, just a suggestion.

Sure, that wording was from existing pthread_spin_init() man page

#4 lore raw
In reply to Stephen Hemminger
Quoted 13 lines · Stephen Hemminger

Need to warn users of DPDK spinlocks from non-pinned threads. This is similar wording to Linux documentation in pthread_spin_init.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +0 -0
index 5f0748fba1c0..45d3de8d84f6 100644

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 803

Quoted 7 lines
doc/guides/prog_guide/env_abstraction_layer.rst +4 -0
@@ -797,6 +797,16 @@ Known Issues
 
   The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)

Some APIs inherently use rte_spinlock, just like rte_malloc/rte_eal_alarm_set, Because DPDK API mainly use rte_spinlock to support thread-safty.

Suggest declare DPDK API mainly use rte_spinlock to support thread-safty, so if the caller thread is not pinned to an lcore may encount a possibility of large application delays.

Quoted 10 lines
doc/guides/prog_guide/env_abstraction_layer.rst +6 -0
+  then there is a possibility of large application delays.
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until
+  the lock holder is once more rescheduled and releases the lock.
+
+
 cgroup control
 ~~~~~~~~~~~~~~
 
#5 lore raw
Quoted 25 lines · On Sat, 11 Jun 2022 09:55:00 +0800 fengchengwen <[email protected]> wrote:
Quoted 13 lines · Stephen Hemminger

Need to warn users of DPDK spinlocks from non-pinned threads. This is similar wording to Linux documentation in pthread_spin_init.

---
 doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +0 -0
index 5f0748fba1c0..45d3de8d84f6 100644

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 803

Quoted 7 lines
doc/guides/prog_guide/env_abstraction_layer.rst +4 -0
@@ -797,6 +797,16 @@ Known Issues
 
   The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+
+  If a pthread, that is not pinned to an lcore acquires a lock such as a
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)  

Some APIs inherently use rte_spinlock, just like rte_malloc/rte_eal_alarm_set, Because DPDK API mainly use rte_spinlock to support thread-safty.

Quoted 3 lines

Suggest declare DPDK API mainly use rte_spinlock to support thread-safty, so if the caller thread is not pinned to an lcore may encount a possibility of large application delays.

I copied text from pthread_spinlock man page. The same caveats apply to pthread_spinlocks as DPDK; therefore using same wording seemed appropriate.

But it is worth mentioning that other API's may depend on spinlocks internally.

#6 lore raw
In reply to Stephen Hemminger
Quoted 13 lines · Stephen Hemminger

Need to warn users of DPDK spinlocks from non-pinned threads. This is similar wording to Linux documentation in pthread_spin_init.

---
  doc/guides/prog_guide/env_abstraction_layer.rst | 10 ++++++++++
  1 file changed, 10 insertions(+)
doc/guides/prog_guide/env_abstraction_layer.rst +0 -0
index 5f0748fba1c0..45d3de8d84f6 100644

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 801

Quoted 5 lines
doc/guides/prog_guide/env_abstraction_layer.rst +2 -0
@@ -797,6 +797,16 @@ Known Issues
 
    The debug statistics of rte_ring, rte_mempool and rte_timer are not supported in an unregistered non-EAL pthread.
 
++ locking
+

Isn't this problem more general than locks? The use of any non-preemption safe data structures potentially causes such delays. Regular DPDK rings for sure. The lock-less stack? The hash library?

Both actual and open-coded spinlocks internal to the APIs are also very common.

doc/guides/prog_guide/env_abstraction_layer.rst Known Issues · line 804

Quoted 3 lines
doc/guides/prog_guide/env_abstraction_layer.rst +3 -0
+  If a pthread, that is not pinned to an lcore acquires a lock such as a
+  DPDK based lock (rte_spinlock, rte_rwlock, rte_ticketlock, rte_mcslock)
+  then there is a possibility of large application delays.

Pinning or not doesn't matter. What matters is if the thread is preempted and thus is prevented from making progress for a long time.

Quoted 8 lines
doc/guides/prog_guide/env_abstraction_layer.rst +5 -0
+  The problem is that if a thread is scheduled off the CPU while it holds
+  a lock, then other threads will waste time spinning on the lock until
+  the lock holder is once more rescheduled and releases the lock.
+
+
  cgroup control
  ~~~~~~~~~~~~~~