Skip to contentSkip to conversation

Add support for TSAN to liburcu

PATCH Superseded Olivier Dion via lttng-dev 1 participant 69 messages May 15, 2023 – Jul 5, 2023

lore
Share & tools
lore
#0 lore raw

cover Add support for TSAN to liburcu

This patch set adds support for TSAN in liburcu.

* Here are the major changes

- Usage of compiler atomic builtins is added to the uatomic API. This is required for TSAN to understand atomic memory accesses. If the compiler supports such builtins, they are used by default. User can opt-out and use the legacy implementation of the uatomic API by using the `--disable-atomic-builtins' configuration option.

- The CMM memory model is introduced but yet formalized. It tries to be as close as possible to the C11 memory model while offering primitives such as cmm_smp_wmb(), cmm_smp_rmb() and cmm_mb() that can't be expressed in it. For example, cmm_mb() can be used for ordering memory accesses to MMIO devices, which is out of the scope of the C11 memory model.

- The CMM annotation layer is a new public API that is highly experimental and not guaranteed to be stable at this stage. It serves the dual purpose of verifying local (intra-thread) relaxed atomic accesses ordering with a memory barrier and global (inter-thread) relaxed atomic accesses with a shared state. The second purpose is necessary for TSAN to understand memory accesses ordering since it does not fully support thread fence yet.

* CMM annotation example

Consider the following pseudo-code of writer side in synchronize_rcu(). An acquire group is defined on the stack of the writer. Annotations are made onto the group to ensure ordering of relaxed memory accesses in reader_state() before the memory barrier at the end of synchronize_rcu(). It also helps TSAN to understand that the relaxed accesses in reader_state() act like acquire accesses because of the memory barrier in synchronize_rcu().

In other words, the purpose of this annotation is to convert a group of load-acquire memory operations into load-relaxed memory operations followed by a single memory barrier. This highly benefits weakly ordered architectures by having a constant number of memory barriers instead of being linearly proportional to the number of loads. This does not benefit TSO architectures.

```
enum urcu_state reader_state(unsigned long *ctr, cmm_annotate_t *acquire_group)
{
	unsigned long v;

	v = uatomic_load(ctr, CMM_RELAXED);
	cmm_annotate_group_mem_acquire(acquire_group, ctr);
	// ...
}

void wait_for_readers(..., cmm_annotate_group *acquire_group)
{
	// ...
	switch (reader_state(..., acquire_group)) {
		// ...
	}
	// ...
}

void synchronize_rcu()
{
	cmm_annotate_define(acquire_group);
	// ...
	wait_for_readers(..., &acquire_group);
	// ...
	cmm_annotate_group_mb_acquire(&acquire_group);
	cmm_smp_mb();
}
```

* Known limitation

The only known limitation is with the urcu-signal flavor. Indeed, TSAN hijacks calls to sigaction(2) and installs its own signal handler that will deliver the signals to the urcu handler at synchronization points. This is known to deadlock the urcu-signal flavor in at least one case. See commit log of `urcu/annotate: Add CMM annotation' for a minimal reproducer outside of liburcu.

Therefore, we have the intention of deprecating the urcu-signal flavor in the future, starting by disabling it by default.

Olivier Dion (11):
configure: Add --disable-atomic-builtins option urcu/uatomic: Use atomic builtins if configured urcu/compiler: Use atomic builtins if configured urcu/arch/generic: Use atomic builtins if configured urcu/system: Use atomic builtins if configured urcu/uatomic: Add CMM memory model
urcu-wait: Fix wait state load/store
tests: Use uatomic for accessing global states
benchmark: Use uatomic for accessing global states tests/unit/test_build: Quiet unused return value urcu/annotate: Add CMM annotation

 README.md                               |  11 ++
 configure.ac                            |  26 ++++
 include/Makefile.am                     |   4 +
 include/urcu/annotate.h                 | 174 ++++++++++++++++++++++++
 include/urcu/arch/generic.h             |  37 +++++
 include/urcu/compiler.h                 |  20 ++-
 include/urcu/static/pointer.h           |  40 ++----
 include/urcu/static/urcu-bp.h           |  12 +-
 include/urcu/static/urcu-common.h       |   8 +-
 include/urcu/static/urcu-mb.h           |  11 +-
 include/urcu/static/urcu-memb.h         |  26 +++-
 include/urcu/static/urcu-qsbr.h         |  29 ++--
 include/urcu/system.h                   |  21 +++
 include/urcu/uatomic.h                  |  25 +++-
 include/urcu/uatomic/builtins-generic.h | 124 +++++++++++++++++
 include/urcu/uatomic/builtins-x86.h     | 124 +++++++++++++++++
 include/urcu/uatomic/builtins.h         |  83 +++++++++++
 include/urcu/uatomic/generic.h          | 128 +++++++++++++++++
 src/rculfhash.c                         |  92 ++++++++-----
 src/urcu-bp.c                           |  17 ++-
 src/urcu-pointer.c                      |   9 +-
 src/urcu-qsbr.c                         |  31 +++--
 src/urcu-wait.h                         |  15 +-
 src/urcu.c                              |  24 ++--
 tests/benchmark/Makefile.am             |  91 +++++++------
 tests/benchmark/common-states.c         |   1 +
 tests/benchmark/common-states.h         |  51 +++++++
 tests/benchmark/test_mutex.c            |  32 +----
 tests/benchmark/test_perthreadlock.c    |  32 +----
 tests/benchmark/test_rwlock.c           |  32 +----
 tests/benchmark/test_urcu.c             |  33 +----
 tests/benchmark/test_urcu_assign.c      |  33 +----
 tests/benchmark/test_urcu_bp.c          |  33 +----
 tests/benchmark/test_urcu_defer.c       |  33 +----
 tests/benchmark/test_urcu_gc.c          |  34 +----
 tests/benchmark/test_urcu_hash.c        |   6 +-
 tests/benchmark/test_urcu_hash.h        |  15 --
 tests/benchmark/test_urcu_hash_rw.c     |  10 +-
 tests/benchmark/test_urcu_hash_unique.c |  10 +-
 tests/benchmark/test_urcu_lfq.c         |  20 +--
 tests/benchmark/test_urcu_lfs.c         |  20 +--
 tests/benchmark/test_urcu_lfs_rcu.c     |  20 +--
 tests/benchmark/test_urcu_qsbr.c        |  33 +----
 tests/benchmark/test_urcu_qsbr_gc.c     |  34 +----
 tests/benchmark/test_urcu_wfcq.c        |  22 ++-
 tests/benchmark/test_urcu_wfq.c         |  20 +--
 tests/benchmark/test_urcu_wfs.c         |  22 ++-
 tests/common/api.h                      |  12 +-
 tests/regression/rcutorture.h           | 102 ++++++++++----
 tests/unit/test_build.c                 |   8 +-
 50 files changed, 1227 insertions(+), 623 deletions(-)
 create mode 100644 include/urcu/annotate.h
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins-x86.h
 create mode 100644 include/urcu/uatomic/builtins.h
 create mode 100644 tests/benchmark/common-states.c
 create mode 100644 tests/benchmark/common-states.h
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#1 lore raw

PATCH 1/11 configure: Add --disable-atomic-builtins option

By default, if the toolchain supports atomic builtins, use them for the uatomic API. This requires that the toolchains used to compile the library and the user application supports such builtins.

The advantage of using these builtins is that they are well known synchronization primitives by several tools such as TSAN.

Change-Id Ia8e97112681f744f17816dbc4cbbec805a483331
---
 README.md    | 11 +++++++++++
 configure.ac | 26 ++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
README.md +11 -0
index ba5bb08..6ce96c9 100644
@@ -429,6 +429,17 @@ still being used to iterate on a hash table.
 This option alters the rculfhash ABI. Make sure to compile both library
 and application with matching configuration.
 
+### Usage of `--disable-atomic-builtins`
+
+By default, the configure script will check if the toolchain supports atomic
+builtins. If so, then the RCU memory model is implemented using the atomic
+builtins of the toolchain.
+
+Building liburcu with `--disable-atomic-builtins` force to use the legacy
+internal implementations for atomic accesses.
+
+This option is useful if for example the atomic builtins for a given toolchain
+version is known to be broken or to be inefficient.
 
 Make targets
 ------------
configure.ac +26 -0
index 909cf1d..4450a31 100644
@@ -230,6 +230,11 @@ AE_FEATURE([rcu-debug], [Enable internal debugging self-checks. Introduces a per
 AE_FEATURE_DEFAULT_DISABLE
 AE_FEATURE([cds-lfht-iter-debug], [Enable extra debugging checks for lock-free hash table iterator traversal. Alters the rculfhash ABI. Make sure to compile both library and application with matching configuration.])
 
+# toolchain atomic builtins
+# Enabled by default
+AE_FEATURE_DEFAULT_ENABLE
+AE_FEATURE([atomic-builtins], [Disable the usage of toolchain atomic builtins.])
+
 # When given, add -Werror to WARN_CFLAGS and WARN_CXXFLAGS.
 # Disabled by default
 AE_FEATURE_DEFAULT_DISABLE
@@ -259,6 +264,23 @@ AE_IF_FEATURE_ENABLED([cds-lfht-iter-debug], [
   AC_DEFINE([CONFIG_CDS_LFHT_ITER_DEBUG], [1], [Enable extra debugging checks for lock-free hash table iterator traversal. Alters the rculfhash ABI. Make sure to compile both library and application with matching configuration.])
 ])
 
+AE_IF_FEATURE_ENABLED([atomic-builtins], [
+  AC_COMPILE_IFELSE(
+	[AC_LANG_PROGRAM(
+		[[int x, y;]],
+		[[__atomic_store_n(&x, 0, __ATOMIC_RELAXED);
+		  __atomic_load_n(&x, __ATOMIC_RELAXED);
+		  y = __atomic_exchange_n(&x, 1, __ATOMIC_RELAXED);
+		  __atomic_compare_exchange_n(&x, &y, 0, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
+		  __atomic_add_fetch(&x, 1, __ATOMIC_RELAXED);
+		  __atomic_sub_fetch(&x, 1, __ATOMIC_RELAXED);
+		  __atomic_and_fetch(&x, 0x01, __ATOMIC_RELAXED);
+		  __atomic_or_fetch(&x, 0x01, __ATOMIC_RELAXED);
+		  __atomic_thread_fence(__ATOMIC_RELAXED);
+		  __atomic_signal_fence(__ATOMIC_RELAXED);]])],
+	[AC_DEFINE([CONFIG_RCU_USE_ATOMIC_BUILTINS], [1], [Use compiler atomic builtins.])],
+	[AE_FEATURE_DISABLE(atomic-builtins)])
+])
 
 ##                                                                          ##
 ## Set automake variables for optional feature conditionnals in Makefile.am ##
@@ -361,6 +383,10 @@ PPRINT_PROP_BOOL([Internal debugging], $value)
 AE_IS_FEATURE_ENABLED([cds-lfht-iter-debug]) && value=1 || value=0
 PPRINT_PROP_BOOL([Lock-free HT iterator debugging], $value)
 
+# atomic builtins enabled/disabled
+AE_IS_FEATURE_ENABLED([atomic-builtins]) && value=1 || value=0
+PPRINT_PROP_BOOL([Use toolchain atomic builtins], $value)
+
 PPRINT_PROP_BOOL([Multi-flavor support], 1)
 
 report_bindir="`eval eval echo $bindir`"
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#2 lore raw

PATCH 2/11 urcu/uatomic: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

Implement uatomic in term of atomic builtins if configured to do so.

Change-Id I5814494c62ee507fd5d381c3ba4ccd0a80c4f4e3
---
 include/Makefile.am                     |  3 +
 include/urcu/uatomic.h                  |  5 +-
 include/urcu/uatomic/builtins-generic.h | 85 +++++++++++++++++++++++++
 include/urcu/uatomic/builtins-x86.h     | 85 +++++++++++++++++++++++++
 include/urcu/uatomic/builtins.h         | 83 ++++++++++++++++++++++++
 5 files changed, 260 insertions(+), 1 deletion(-)
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins-x86.h
 create mode 100644 include/urcu/uatomic/builtins.h
include/Makefile.am +3 -0
index ba1fe60..fac941f 100644
@@ -63,6 +63,9 @@ nobase_include_HEADERS = \
 	urcu/uatomic/alpha.h \
 	urcu/uatomic_arch.h \
 	urcu/uatomic/arm.h \
+	urcu/uatomic/builtins.h \
+	urcu/uatomic/builtins-generic.h \
+	urcu/uatomic/builtins-x86.h \
 	urcu/uatomic/gcc.h \
 	urcu/uatomic/generic.h \
 	urcu/uatomic.h \
include/urcu/uatomic.h +4 -1
index 2fb5fd4..6b57c5f 100644
@@ -22,8 +22,11 @@
 #define _URCU_UATOMIC_H
 
 #include <urcu/arch.h>
+#include <urcu/config.h>
 
-#if defined(URCU_ARCH_X86)
+#if defined(CONFIG_RCU_USE_ATOMIC_BUILTINS)
+#include <urcu/uatomic/builtins.h>
+#elif defined(URCU_ARCH_X86)
 #include <urcu/uatomic/x86.h>
 #elif defined(URCU_ARCH_PPC)
 #include <urcu/uatomic/ppc.h>
include/urcu/uatomic/builtins-generic.h new file +85 -0

85 lines not on this page · raw message

include/urcu/uatomic/builtins-x86.h new file +85 -0

85 lines not on this page · raw message

include/urcu/uatomic/builtins.h new file +83 -0

83 lines not on this page · raw message

Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#3 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 71 lines · Olivier Dion

Implement uatomic in term of atomic builtins if configured to do so.

Change-Id I5814494c62ee507fd5d381c3ba4ccd0a80c4f4e3
---
 include/Makefile.am                     |  3 +
 include/urcu/uatomic.h                  |  5 +-
 include/urcu/uatomic/builtins-generic.h | 85 +++++++++++++++++++++++++
 include/urcu/uatomic/builtins-x86.h     | 85 +++++++++++++++++++++++++
 include/urcu/uatomic/builtins.h         | 83 ++++++++++++++++++++++++
 5 files changed, 260 insertions(+), 1 deletion(-)
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins-x86.h
 create mode 100644 include/urcu/uatomic/builtins.h
include/Makefile.am +3 -0
index ba1fe60..fac941f 100644
@@ -63,6 +63,9 @@ nobase_include_HEADERS = \
 	urcu/uatomic/alpha.h \
 	urcu/uatomic_arch.h \
 	urcu/uatomic/arm.h \
+	urcu/uatomic/builtins.h \
+	urcu/uatomic/builtins-generic.h \
+	urcu/uatomic/builtins-x86.h \
 	urcu/uatomic/gcc.h \
 	urcu/uatomic/generic.h \
 	urcu/uatomic.h \
include/urcu/uatomic.h +4 -1
index 2fb5fd4..6b57c5f 100644
@@ -22,8 +22,11 @@
 #define _URCU_UATOMIC_H
 
 #include <urcu/arch.h>
+#include <urcu/config.h>
 
-#if defined(URCU_ARCH_X86)
+#if defined(CONFIG_RCU_USE_ATOMIC_BUILTINS)
+#include <urcu/uatomic/builtins.h>
+#elif defined(URCU_ARCH_X86)
 #include <urcu/uatomic/x86.h>
 #elif defined(URCU_ARCH_PPC)
 #include <urcu/uatomic/ppc.h>
include/urcu/uatomic/builtins-generic.h new file +18 -0
new file mode 100644
index 0000000..8e6a9b5
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-generic.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

include/urcu/uatomic/builtins-generic.h line 28

Quoted 10 lines
include/urcu/uatomic/builtins-generic.h new file +10 -0
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_GENERIC_H
+#define _URCU_UATOMIC_BUILTINS_GENERIC_H
+
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

Does this lose the volatile semantics that the old-style definitions had?

Quoted 81 lines
include/urcu/uatomic/builtins-generic.h new file +57 -0
+
+#define uatomic_cmpxchg(addr, old, new)					\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = (__typeof__(*(addr)))old;	\
+		__atomic_compare_exchange_n(addr, &_old, new, 0,	\
+					    __ATOMIC_SEQ_CST,		\
+					    __ATOMIC_SEQ_CST);		\
+		_old;							\
+	})
+
+#define uatomic_xchg(addr, v)				\
+	__atomic_exchange_n(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_add_return(addr, v)			\
+	__atomic_add_fetch(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_sub_return(addr, v)			\
+	__atomic_sub_fetch(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_and(addr, mask)					\
+	(void)__atomic_and_fetch(addr, mask, __ATOMIC_RELAXED)
+
+#define uatomic_or(addr, mask)					\
+	(void)__atomic_or_fetch(addr, mask, __ATOMIC_RELAXED)
+
+#define uatomic_add(addr, v)					\
+	(void)__atomic_add_fetch(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_sub(addr, v)					\
+	(void)__atomic_sub_fetch(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_inc(addr)					\
+	(void)__atomic_add_fetch(addr, 1, __ATOMIC_RELAXED)
+
+#define uatomic_dec(addr)					\
+	(void)__atomic_sub_fetch(addr, 1, __ATOMIC_RELAXED)
+
+#define cmm_smp_mb__before_uatomic_and() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_and()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_or() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_or()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_add() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_add()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_sub() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_sub()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_inc() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_inc() cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_dec() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_dec() cmm_smp_mb()
+
+#endif /* _URCU_UATOMIC_BUILTINS_GENERIC_H */
include/urcu/uatomic/builtins-x86.h new file +18 -0
new file mode 100644
index 0000000..a70f922
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-x86.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

include/urcu/uatomic/builtins-x86.h line 28

Quoted 10 lines
include/urcu/uatomic/builtins-x86.h new file +10 -0
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_X86_H
+#define _URCU_UATOMIC_BUILTINS_X86_H
+
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

And same question here.

Thanx, Paul

Quoted 149 lines
include/urcu/uatomic/builtins-x86.h new file +57 -0
+
+#define uatomic_cmpxchg(addr, old, new)					\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = (__typeof__(*(addr)))old;	\
+		__atomic_compare_exchange_n(addr, &_old, new, 0,	\
+					    __ATOMIC_SEQ_CST,		\
+					    __ATOMIC_SEQ_CST);		\
+		_old;							\
+	})
+
+#define uatomic_xchg(addr, v)				\
+	__atomic_exchange_n(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_add_return(addr, v)			\
+	__atomic_add_fetch(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_sub_return(addr, v)			\
+	__atomic_sub_fetch(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_and(addr, mask)					\
+	(void)__atomic_and_fetch(addr, mask, __ATOMIC_SEQ_CST)
+
+#define uatomic_or(addr, mask)					\
+	(void)__atomic_or_fetch(addr, mask, __ATOMIC_SEQ_CST)
+
+#define uatomic_add(addr, v)					\
+	(void)__atomic_add_fetch(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_sub(addr, v)					\
+	(void)__atomic_sub_fetch(addr, v, __ATOMIC_SEQ_CST)
+
+#define uatomic_inc(addr)					\
+	(void)__atomic_add_fetch(addr, 1, __ATOMIC_SEQ_CST)
+
+#define uatomic_dec(addr)					\
+	(void)__atomic_sub_fetch(addr, 1, __ATOMIC_SEQ_CST)
+
+#define cmm_smp_mb__before_uatomic_and() do { } while (0)
+#define cmm_smp_mb__after_uatomic_and()  do { } while (0)
+
+#define cmm_smp_mb__before_uatomic_or() do { } while (0)
+#define cmm_smp_mb__after_uatomic_or()  do { } while (0)
+
+#define cmm_smp_mb__before_uatomic_add() do { } while (0)
+#define cmm_smp_mb__after_uatomic_add()  do { } while (0)
+
+#define cmm_smp_mb__before_uatomic_sub() do { } while (0)
+#define cmm_smp_mb__after_uatomic_sub()  do { } while (0)
+
+#define cmm_smp_mb__before_uatomic_inc() do { } while (0)
+#define cmm_smp_mb__after_uatomic_inc()  do { } while (0)
+
+#define cmm_smp_mb__before_uatomic_dec() do { } while (0)
+#define cmm_smp_mb__after_uatomic_dec()  do { } while (0)
+
+#endif /* _URCU_UATOMIC_BUILTINS_X86_H */
include/urcu/uatomic/builtins.h new file +83 -0
new file mode 100644
index 0000000..164201b
@@ -0,0 +1,83 @@
+/*
+ * urcu/uatomic/builtins.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_H
+#define _URCU_UATOMIC_BUILTINS_H
+
+#include <urcu/arch.h>
+
+#if defined(__has_builtin)
+#  if !__has_builtin(__atomic_store_n)
+#    error "Toolchain does not support __atomic_store_n."
+#  endif
+#  if !__has_builtin(__atomic_load_n)
+#    error "Toolchain does not support __atomic_load_n."
+#  endif
+#  if !__has_builtin(__atomic_exchange_n)
+#    error "Toolchain does not support __atomic_exchange_n."
+#  endif
+#  if !__has_builtin(__atomic_compare_exchange_n)
+#    error "Toolchain does not support __atomic_compare_exchange_n."
+#  endif
+#  if !__has_builtin(__atomic_add_fetch)
+#    error "Toolchain does not support __atomic_add_fetch."
+#  endif
+#  if !__has_builtin(__atomic_sub_fetch)
+#    error "Toolchain does not support __atomic_sub_fetch."
+#  endif
+#  if !__has_builtin(__atomic_or_fetch)
+#    error "Toolchain does not support __atomic_or_fetch."
+#  endif
+#  if !__has_builtin(__atomic_thread_fence)
+#    error "Toolchain does not support __atomic_thread_fence."
+#  endif
+#  if !__has_builtin(__atomic_signal_fence)
+#    error "Toolchain does not support __atomic_signal_fence."
+#  endif
+#elif defined(__GNUC__)
+#  define GCC_VERSION (__GNUC__       * 10000 + \
+		       __GNUC_MINOR__ * 100   + \
+		       __GNUC_PATCHLEVEL__)
+#  if  GCC_VERSION < 40700
+#    error "GCC version is too old. Version must be 4.7 or greater"
+#  endif
+#  undef  GCC_VERSION
+#else
+#  error "Toolchain is not supported."
+#endif
+
+#if defined(__GNUC__)
+#  define UATOMIC_HAS_ATOMIC_BYTE  __GCC_ATOMIC_CHAR_LOCK_FREE
+#  define UATOMIC_HAS_ATOMIC_SHORT __GCC_ATOMIC_SHORT_LOCK_FREE
+#elif defined(__clang__)
+#  define UATOMIC_HAS_ATOMIC_BYTE  __CLANG_ATOMIC_CHAR_LOCK_FREE
+#  define UATOMIC_HAS_ATOMIC_SHORT __CLANG_ATOMIC_SHORT_LOCK_FREE
+#else
+/* #  define UATOMIC_HAS_ATOMIC_BYTE  */
+/* #  define UATOMIC_HAS_ATOMIC_SHORT */
+#endif
+
+#if defined(URCU_ARCH_X86)
+#  include <urcu/uatomic/builtins-x86.h>
+#else
+#  include <urcu/uatomic/builtins-generic.h>
+#endif
+
+#endif	/* _URCU_UATOMIC_BUILTINS_H */
Signature · 1 line
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#4 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

On 6/21/23 19:19, Paul E. McKenney wrote:
[...]

Quoted 29 lines
Quoted 29 lines
include/urcu/uatomic/builtins-generic.h new file +23 -0
new file mode 100644
index 0000000..8e6a9b5
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-generic.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_GENERIC_H
+#define _URCU_UATOMIC_BUILTINS_GENERIC_H
+
Quoted 9 lines

include/urcu/uatomic/builtins-generic.h line 28

Quoted 5 lines
include/urcu/uatomic/builtins-generic.h new file +5 -0
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

Does this lose the volatile semantics that the old-style definitions had?

Yes.

[...]

Quoted 24 lines
Quoted 24 lines
include/urcu/uatomic/builtins-generic.h new file +23 -0
+++ b/include/urcu/uatomic/builtins-x86.h
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-x86.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_X86_H
+#define _URCU_UATOMIC_BUILTINS_X86_H
Quoted 8 lines

include/urcu/uatomic/builtins-generic.h line 28

Quoted 6 lines
include/urcu/uatomic/builtins-generic.h new file +6 -0
+
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

And same question here.

Yes, this opens interesting questions:

* what semantic do we want for uatomic_read/set ?

* what semantic do we want for CMM_LOAD_SHARED/CMM_STORE_SHARED ?

* do we want to allow load/store-shared to work on variables larger than a word ? (e.g. on a uint64_t on a 32-bit architecture, or on a structure)

* what are the guarantees of a volatile type ?

* what are the guarantees of a load/store relaxed in C11 ?

Does the delta between volatile and C11 relaxed guarantees matter ?

Is there an advantage to use C11 load/store relaxed over volatile ? Should we combine both C11 load/store _and_ volatile ? Should we use atomic_signal_fence instead ?

Thanks,

Mathieu

Quoted 4 lines

Thanx, Paul

Quoted 1 line
include/urcu/uatomic/builtins-generic.h new file +1 -0
+
Signature · 8 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#5 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 89 lines · Mathieu Desnoyers

On 6/21/23 19:19, Paul E. McKenney wrote:
[...]

Quoted 29 lines
Quoted 29 lines
include/urcu/uatomic/builtins-generic.h new file +23 -0
new file mode 100644
index 0000000..8e6a9b5
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-generic.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_GENERIC_H
+#define _URCU_UATOMIC_BUILTINS_GENERIC_H
+
Quoted 9 lines

include/urcu/uatomic/builtins-generic.h line 28

Quoted 5 lines
include/urcu/uatomic/builtins-generic.h new file +5 -0
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

Does this lose the volatile semantics that the old-style definitions had?

Yes.

[...]

Quoted 24 lines
Quoted 24 lines
include/urcu/uatomic/builtins-generic.h new file +23 -0
+++ b/include/urcu/uatomic/builtins-x86.h
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-x86.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_X86_H
+#define _URCU_UATOMIC_BUILTINS_X86_H
Quoted 8 lines

include/urcu/uatomic/builtins-generic.h line 28

Quoted 6 lines
include/urcu/uatomic/builtins-generic.h new file +6 -0
+
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

And same question here.

Yes, this opens interesting questions:

* what semantic do we want for uatomic_read/set ?

* what semantic do we want for CMM_LOAD_SHARED/CMM_STORE_SHARED ?

* do we want to allow load/store-shared to work on variables larger than a word ? (e.g. on a uint64_t on a 32-bit architecture, or on a structure)

* what are the guarantees of a volatile type ?

Quoted 7 lines

* what are the guarantees of a load/store relaxed in C11 ?

Does the delta between volatile and C11 relaxed guarantees matter ?

Is there an advantage to use C11 load/store relaxed over volatile ? Should we combine both C11 load/store _and_ volatile ? Should we use atomic_signal_fence instead ?

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#6 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 3 lines · Paul E. McKenney

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

Good catch. Seems like not a problem on GCC (yet), but Clang is extremely aggressive and seems to do store fusing on some corner cases [0].

However, I do not find any simple reproducer of load/store fusing. Do you have example of such fusing, or is this a precaution? In the meantime, back to reading the standard to be certain :-)

[0] https://godbolt.org/z/odKG9a75a

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#7 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 8 lines · Olivier Dion
Quoted 3 lines · Paul E. McKenney

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

Good catch. Seems like not a problem on GCC (yet), but Clang is extremely aggressive and seems to do store fusing on some corner cases [0].

I don't think this is an example of store fusing, but rather just that the compiler can eliminate stores to static variables which are otherwise unused, making the entire variable useless.

Thanks,

Mathieu

Quoted 7 lines

However, I do not find any simple reproducer of load/store fusing. Do you have example of such fusing, or is this a precaution? In the meantime, back to reading the standard to be certain :-)

[0] https://godbolt.org/z/odKG9a75a

Signature · 8 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#8 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 7 lines · Mathieu Desnoyers
Quoted 6 lines · Olivier Dion
Quoted 3 lines · Paul E. McKenney

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

Quoted 6 lines
Quoted 2 lines

Good catch. Seems like not a problem on GCC (yet), but Clang is extremely aggressive and seems to do store fusing on some corner cases [0].

I don't think this is an example of store fusing, but rather just that the compiler can eliminate stores to static variables which are otherwise unused, making the entire variable useless.

Indeed, that is not store fusing. It is however interesting to see that the dead store elimination is avoid when casting one of the &x with a volatile qualifier. TIL dead store can apply to atomics.

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#9 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev
Quoted 6 lines · Olivier Dion
Quoted 3 lines · Paul E. McKenney

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

Quoted 8 lines

Good catch. Seems like not a problem on GCC (yet), but Clang is extremely aggressive and seems to do store fusing on some corner cases [0].

However, I do not find any simple reproducer of load/store fusing. Do you have example of such fusing, or is this a precaution? In the meantime, back to reading the standard to be certain :-)

[0] https://godbolt.org/z/odKG9a75a

I certainly have heard a number of compiler writers thinking in terms of doing load/store fusing, some of whom were trying to get rid of the volatile variants in order to remove an impediment to their mission of optimizing all programs out of existence. ;-)

I therefore suggest taking this possibility quite seriously.

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#10 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

In reply to Paul E. McKenney via lttng-dev
Quoted 94 lines · Paul E. McKenney
Quoted 93 lines · Mathieu Desnoyers

On 6/21/23 19:19, Paul E. McKenney wrote:
[...]

Quoted 29 lines
Quoted 29 lines
include/urcu/uatomic/builtins-generic.h new file +23 -0
new file mode 100644
index 0000000..8e6a9b5
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-generic.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_GENERIC_H
+#define _URCU_UATOMIC_BUILTINS_GENERIC_H
+
Quoted 9 lines

include/urcu/uatomic/builtins-generic.h line 28

Quoted 5 lines
include/urcu/uatomic/builtins-generic.h new file +5 -0
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

Does this lose the volatile semantics that the old-style definitions had?

Yes.

[...]

Quoted 24 lines
Quoted 24 lines
include/urcu/uatomic/builtins-generic.h new file +23 -0
+++ b/include/urcu/uatomic/builtins-x86.h
@@ -0,0 +1,85 @@
+/*
+ * urcu/uatomic/builtins-x86.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_X86_H
+#define _URCU_UATOMIC_BUILTINS_X86_H
Quoted 8 lines

include/urcu/uatomic/builtins-generic.h line 28

Quoted 6 lines
include/urcu/uatomic/builtins-generic.h new file +6 -0
+
+#include <urcu/system.h>
+
+#define uatomic_set(addr, v) __atomic_store_n(addr, v, __ATOMIC_RELAXED)
+
+#define uatomic_read(addr) __atomic_load_n(addr, __ATOMIC_RELAXED)

And same question here.

Yes, this opens interesting questions:

* what semantic do we want for uatomic_read/set ?

* what semantic do we want for CMM_LOAD_SHARED/CMM_STORE_SHARED ?

* do we want to allow load/store-shared to work on variables larger than a word ? (e.g. on a uint64_t on a 32-bit architecture, or on a structure)

* what are the guarantees of a volatile type ?

* what are the guarantees of a load/store relaxed in C11 ?

Does the delta between volatile and C11 relaxed guarantees matter ?

Quoted 7 lines
Quoted 3 lines

Is there an advantage to use C11 load/store relaxed over volatile ? Should we combine both C11 load/store _and_ volatile ? Should we use atomic_signal_fence instead ?

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

I'm OK with erring towards a safer approach, but just out of curiosity, do you have examples of compilers doing load or store fusion on C11 or C++11 relaxed atomics, or is it out of caution due to lack of explicit guarantees in the standards ?

Does this lack of guarantee about fusion also apply to other MO such as acquire, release and seq.cst. ?

Thanks,

Mathieu

Signature · 8 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#11 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

In reply to Paul E. McKenney via lttng-dev
Quoted 5 lines · Paul E. McKenney
Quoted 1 line · Mathieu Desnoyers

On 6/21/23 19:19, Paul E. McKenney wrote:

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

After some testing, I got a wall of warnings:

-Wignored-qualifiers:

Warn if the return type of a function has a type qualifier such as "const". For ISO C such a type qualifier has no effect, since the value returned by a function is not an lvalue. For C++, the warning is only emitted for scalar types or "void". ISO C prohibits qualified "void" return types on function definitions, so such return types always receive a warning even without this option.

Since we are using atomic builtins, for example load:

type __atomic_load_n (type *ptr, int memorder)

If we put the qualifier volatile to TYPE, we end up with the same qualifier on the return value, triggering a warning for each atomic operation.

This seems to be only a problem when compiling in C++ [0] while in C it seems the compiler is more relaxed on this [1].

Ideas to make the toolchains happy? :-)

[0] https://godbolt.org/z/3nW14M3v1
[1] https://godbolt.org/z/TcTeMeKbW

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#12 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 2 lines · Olivier Dion

[0] https://godbolt.org/z/3nW14M3v1
[1] https://godbolt.org/z/TcTeMeKbW

Sorry. That was:

    [0] https://godbolt.org/z/ETcxnz4TW
    [1] https://godbolt.org/z/jMjh8YoM4
Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#13 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

Quoted 8 lines · Olivier Dion
Quoted 2 lines · Olivier Dion
   [0] https://godbolt.org/z/3nW14M3v1
   [1] https://godbolt.org/z/TcTeMeKbW

Sorry. That was:

     [0] https://godbolt.org/z/ETcxnz4TW

Change

(volatile __typeof__(ptr))(ptr);

for:

(volatile __typeof__(*(ptr)) *)(ptr);

and:

void love_iso(int *x)

{
     __atomic_store_n(cast_volatile(&x), 1,
                      __ATOMIC_RELAXED);
}

for

void love_iso(int *x)

{
     __atomic_store_n(cast_volatile(x), 1,
                      __ATOMIC_RELAXED);
}

Thanks,

Mathieu

Quoted 2 lines
     [1] https://godbolt.org/z/jMjh8YoM4

Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#14 lore raw

Re: [lttng-dev] [PATCH 02/11] urcu/uatomic: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev
Quoted 22 lines · Olivier Dion
Quoted 5 lines · Paul E. McKenney
Quoted 1 line · Mathieu Desnoyers

On 6/21/23 19:19, Paul E. McKenney wrote:

I suggest C11 volatile atomic load/store. Load/store fusing is permitted for non-volatile atomic loads and stores, and such fusing can ruin your code's entire day. ;-)

After some testing, I got a wall of warnings:

-Wignored-qualifiers:

Warn if the return type of a function has a type qualifier such as "const". For ISO C such a type qualifier has no effect, since the value returned by a function is not an lvalue. For C++, the warning is only emitted for scalar types or "void". ISO C prohibits qualified "void" return types on function definitions, so such return types always receive a warning even without this option.

Since we are using atomic builtins, for example load:

type __atomic_load_n (type *ptr, int memorder)

Quoted 8 lines

If we put the qualifier volatile to TYPE, we end up with the same qualifier on the return value, triggering a warning for each atomic operation.

This seems to be only a problem when compiling in C++ [0] while in C it seems the compiler is more relaxed on this [1].

Ideas to make the toolchains happy? :-)

Change:

(__typeof__(*ptr) *volatile)(ptr);

(which applies the volatile to the pointer, rather than what is pointed to)

to either:

(volatile __typeof__(*ptr) *)(ptr);

or:

(__typeof__(*ptr) volatile *)(ptr);

Thanks,

Mathieu

Quoted 4 lines
   [0] https://godbolt.org/z/3nW14M3v1
   [1] https://godbolt.org/z/TcTeMeKbW
Signature · 8 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#15 lore raw

PATCH 3/11 urcu/compiler: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

Use __atomic_signal_fence(__ATOMIC_SEQ_CST) for cmm_barrier() if configured to use atomic builtins.

Change-Id Ib168b50f1e97a8da861b92d6882c56db230ebb2c
---
 include/urcu/compiler.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
include/urcu/compiler.h +7 -1
index 2f32b38..3604488 100644
@@ -25,10 +25,16 @@
 # include <type_traits>	/* for std::remove_cv */
 #endif
 
+#include <urcu/config.h>
+
 #define caa_likely(x)	__builtin_expect(!!(x), 1)
 #define caa_unlikely(x)	__builtin_expect(!!(x), 0)
 
-#define	cmm_barrier()	__asm__ __volatile__ ("" : : : "memory")
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+#  define cmm_barrier() __atomic_signal_fence(__ATOMIC_SEQ_CST)
+#else
+#  define cmm_barrier() asm volatile ("" : : : "memory")
+#endif
 
 /*
  * Instruct the compiler to perform only a single access to a variable
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#16 lore raw

PATCH 4/11 urcu/arch/generic: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

If configured to use atomic builtins, implement SMP memory barriers in term of atomic builtins if the architecture does not implement its own version.

Change-Id Iddc4283606e0fce572e104d2d3f03b5c0d9926fb
---
 include/urcu/arch/generic.h | 8 ++++++++
 1 file changed, 8 insertions(+)
include/urcu/arch/generic.h +8 -0
index be6e41e..e292c70 100644
@@ -43,6 +43,14 @@ extern "C" {
  * GCC builtins) as well as cmm_rmb and cmm_wmb (defaulting to cmm_mb).
  */
 
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+# ifndef cmm_smp_mb
+#  define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
+# endif
+
+#endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
+
 #ifndef cmm_mb
 #define cmm_mb()    __sync_synchronize()
 #endif
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#17 lore raw

Re: [lttng-dev] [PATCH 04/11] urcu/arch/generic: Use atomic builtins if configured

Quoted 19 lines · Olivier Dion

If configured to use atomic builtins, implement SMP memory barriers in term of atomic builtins if the architecture does not implement its own version.

Change-Id Iddc4283606e0fce572e104d2d3f03b5c0d9926fb
---
 include/urcu/arch/generic.h | 8 ++++++++
 1 file changed, 8 insertions(+)
include/urcu/arch/generic.h +0 -0
index be6e41e..e292c70 100644
@@ -43,6 +43,14 @@ extern "C" {
  * GCC builtins) as well as cmm_rmb and cmm_wmb (defaulting to cmm_mb).
  */
 

include/urcu/arch/generic.h extern "C" { · line 55

Quoted 10 lines
include/urcu/arch/generic.h +8 -0
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+# ifndef cmm_smp_mb
+#  define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
+# endif
+
+#endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
+
 #ifndef cmm_mb
 #define cmm_mb()    __sync_synchronize()

Just out of curiosity, why not also implement cmm_mb() in terms of __atomic_thread_fence(__ATOMIC_SEQ_CST)? (Or is that a later patch?)

Thanx, Paul

Quoted 4 lines
include/urcu/arch/generic.h +0 -0
 #endif
Signature · 1 line
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#18 lore raw

Re: [lttng-dev] [PATCH 04/11] urcu/arch/generic: Use atomic builtins if configured

Quoted 6 lines · Paul E. McKenney
Quoted 2 lines · Olivier Dion

#ifndef cmm_mb
#define cmm_mb() __sync_synchronize()

Just out of curiosity, why not also implement cmm_mb() in terms of __atomic_thread_fence(__ATOMIC_SEQ_CST)? (Or is that a later patch?)

IIRC, Mathieu and I agree that the definition of a thread fence -- acts as a synchronization fence between threads -- is too weak for what we want here. For example, with I/O devices.

Although __sync_synchronize() is probably an alias for a SEQ_CST thread fence, its definition -- issues a full memory barrier -- is stronger.

We do not want to rely on this assumption (alias) and prefer to rely on the documented definition instead.

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#19 lore raw

Re: [lttng-dev] [PATCH 04/11] urcu/arch/generic: Use atomic builtins if configured

Quoted 12 lines · Olivier Dion
Quoted 6 lines · Paul E. McKenney
Quoted 2 lines · Olivier Dion
  #ifndef cmm_mb
  #define cmm_mb()    __sync_synchronize()

Just out of curiosity, why not also implement cmm_mb() in terms of __atomic_thread_fence(__ATOMIC_SEQ_CST)? (Or is that a later patch?)

IIRC, Mathieu and I agree that the definition of a thread fence -- acts as a synchronization fence between threads -- is too weak for what we want here. For example, with I/O devices.

Quoted 6 lines

Although __sync_synchronize() is probably an alias for a SEQ_CST thread fence, its definition -- issues a full memory barrier -- is stronger.

We do not want to rely on this assumption (alias) and prefer to rely on the documented definition instead.

We should document this rationale with a new comment near the #define, in case anyone mistakenly decides to use a thread fence there to make it similar to the rest of the code in the future.

Thanks,

Mathieu

Signature · 8 lines
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#20 lore raw

Re: [lttng-dev] [PATCH 04/11] urcu/arch/generic: Use atomic builtins if configured

Quoted 16 lines · Mathieu Desnoyers
Quoted 15 lines · Olivier Dion
Quoted 6 lines · Paul E. McKenney
Quoted 2 lines · Olivier Dion
  #ifndef cmm_mb
  #define cmm_mb()    __sync_synchronize()

Just out of curiosity, why not also implement cmm_mb() in terms of __atomic_thread_fence(__ATOMIC_SEQ_CST)? (Or is that a later patch?)

IIRC, Mathieu and I agree that the definition of a thread fence -- acts as a synchronization fence between threads -- is too weak for what we want here. For example, with I/O devices.

Although __sync_synchronize() is probably an alias for a SEQ_CST thread fence, its definition -- issues a full memory barrier -- is stronger.

Quoted 6 lines
Quoted 2 lines

We do not want to rely on this assumption (alias) and prefer to rely on the documented definition instead.

We should document this rationale with a new comment near the #define, in case anyone mistakenly decides to use a thread fence there to make it similar to the rest of the code in the future.

That would be good, thank you!

Ah, and I did not find any issues with the rest of the patchset.

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#21 lore raw

PATCH 5/11 urcu/system: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

If configured to use atomic builtins, use them for implementing the CMM_LOAD_SHARED and CMM_STORE_SHARED macros.

Change-Id I3eaaaaf0d26c47aced6e94b40fd59c7b8baa6272
---
 include/urcu/system.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
include/urcu/system.h +21 -0
index faae390..f184aad 100644
@@ -19,9 +19,28 @@
  * all copies or substantial portions of the Software.
  */
 
+#include <urcu/config.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
 
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+#define CMM_LOAD_SHARED(x)			\
+	__atomic_load_n(&(x), __ATOMIC_RELAXED)
+
+#define _CMM_LOAD_SHARED(x) CMM_LOAD_SHARED(x)
+
+#define CMM_STORE_SHARED(x, v)					\
+	__extension__						\
+	({							\
+		__typeof__(v) _v = (v);				\
+		__atomic_store_n(&(x), _v, __ATOMIC_RELAXED);	\
+		_v;						\
+	})
+
+#define _CMM_STORE_SHARED(x, v) CMM_STORE_SHARED(x, v)
+
+#else
 /*
  * Identify a shared load. A cmm_smp_rmc() or cmm_smp_mc() should come
  * before the load.
@@ -56,4 +75,6 @@
 		_v = _v;	/* Work around clang "unused result" */	\
 	})
 
+#endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
+
 #endif /* _URCU_SYSTEM_H */
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#22 lore raw

PATCH 6/11 urcu/uatomic: Add CMM memory model

In reply to Olivier Dion via lttng-dev

Introducing the URCU memory model with the following new primitives:

- uatomic_load(addr, memory_order)

- uatomic_store(addr, value, memory_order)
- uatomic_and_mo(addr, mask, memory_order)
- uatomic_or_mo(addr, mask, memory_order)
- uatomic_add_mo(addr, value, memory_order)
- uatomic_sub_mo(addr, value, memory_order)
- uatomic_inc_mo(addr, memory_order)
- uatomic_dec_mo(addr, memory_order)

- uatomic_add_return_mo(addr, value, memory_order)
- uatomic_sub_return_mo(addr, value, memory_order)

- uatomic_cmpxchg_mo(addr, old, new,
memory_order_success,
memory_order_failure)

- uatomic_cmpxchg(addr, new, memory_order)

The URCU memory model reflects the C11 memory model. The memory order can be selected through the enum rcu_memorder.

If configured with atomic builtins, the correspondence between the URCU memory model and the C11 memory model is a one to one. However, if not configured with atomic builtins, the following stipulate the memory model.

For load operations with uatomic_load(), the memory orders CMM_RELAXED, CMM_CONSUME, CMM_ACQUIRE and CMM_SEQ_CST are allowed. A barrier is maybe inserted before and after the load from memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_CONSUME: Memory barrier after read
- CMM_ACQUIRE: Memory barrier after read
- CMM_SEQ_CST: Memory barriers before and after read

For store operations with uatomic_store(), the memory orders CMM_RELAXED, CMM_RELEASE, CMM_SEQ_CST are allowed. A barrier is maybe inserted before and after the load from memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_RELEASE: Memory barrier before operation
- CMM_SEQ_CST: Memory barriers before and after operation

For store operations with uatomic_and_mo(), uatomic_or_mo(), uatomic_add_mo(), uatomic_sub_mo(), uatomic_inc_mo(), uatomic_dec_mo(), uatomic_add_return_mo() and uatomic_sub_return_mo(), all memory orders are allowed. A barrier is maybe inserted before and after the store to memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation

For the compare exchange operation uatomic_cmpxchg_mo(), the success memory order can be anything while the failure memory order cannot be CMM_RELEASE nor CMM_ACQ_REL and cannot be stronger than the success memory order.

For the exchange operation uatomic_xchg_mo(), any memory order is valid.

Change-Id I213ba19c84e82a63083f00143a3142ffbdab1d52
---
 include/urcu/static/pointer.h           |  40 +++-----
 include/urcu/uatomic.h                  |  20 ++++
 include/urcu/uatomic/builtins-generic.h |  81 +++++++++++----
 include/urcu/uatomic/builtins-x86.h     |  79 +++++++++++----
 include/urcu/uatomic/generic.h          | 128 ++++++++++++++++++++++++
 src/urcu-pointer.c                      |   9 +-
 6 files changed, 283 insertions(+), 74 deletions(-)
include/urcu/static/pointer.h +13 -27
index 9e46a57..9da8657 100644
@@ -96,23 +96,8 @@ extern "C" {
  * -Wincompatible-pointer-types errors.  Using the statement expression
  * makes it an rvalue and gets rid of the const-ness.
  */
-#ifdef __URCU_DEREFERENCE_USE_ATOMIC_CONSUME
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(__extension__ ({				\
-					__typeof__(p) __attribute__((unused)) _________p0 = { 0 }; \
-					_________p0;					\
-				})) _________p1;					\
-				__atomic_load(&(p), &_________p1, __ATOMIC_CONSUME);	\
-				(_________p1);						\
-			})
-#else
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(p) _________p1 = CMM_LOAD_SHARED(p);		\
-				cmm_smp_read_barrier_depends();				\
-				(_________p1);						\
-			})
-#endif
-
+# define _rcu_dereference(p)			\
+	uatomic_load(&(p), CMM_CONSUME)
 /**
  * _rcu_cmpxchg_pointer - same as rcu_assign_pointer, but tests if the pointer
  * is as expected by "old". If succeeds, returns the previous pointer to the
@@ -131,8 +116,9 @@ extern "C" {
 	({								\
 		__typeof__(*p) _________pold = (old);			\
 		__typeof__(*p) _________pnew = (_new);			\
-		uatomic_cmpxchg(p, _________pold, _________pnew);	\
-	})
+		uatomic_cmpxchg_mo(p, _________pold, _________pnew,	\
+				   CMM_SEQ_CST, CMM_SEQ_CST);		\
+	});
 
 /**
  * _rcu_xchg_pointer - same as rcu_assign_pointer, but returns the previous
@@ -149,17 +135,17 @@ extern "C" {
 	__extension__					\
 	({						\
 		__typeof__(*p) _________pv = (v);	\
-		uatomic_xchg(p, _________pv);		\
+		uatomic_xchg_mo(p, _________pv,		\
+				CMM_SEQ_CST);		\
 	})
 
 
-#define _rcu_set_pointer(p, v)				\
-	do {						\
-		__typeof__(*p) _________pv = (v);	\
-		if (!__builtin_constant_p(v) || 	\
-		    ((v) != NULL))			\
-			cmm_wmb();				\
-		uatomic_set(p, _________pv);		\
+#define _rcu_set_pointer(p, v)						\
+	do {								\
+		__typeof__(*p) _________pv = (v);			\
+		uatomic_store(p, _________pv,				\
+			__builtin_constant_p(v) && (v) == NULL ?	\
+			CMM_RELAXED : CMM_RELEASE);			\
 	} while (0)
 
 /**
include/urcu/uatomic.h +20 -0
index 6b57c5f..6c0d38f 100644
@@ -24,6 +24,26 @@
 #include <urcu/arch.h>
 #include <urcu/config.h>
 
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+enum cmm_memorder {
+	CMM_RELAXED = __ATOMIC_RELAXED,
+	CMM_CONSUME = __ATOMIC_CONSUME,
+	CMM_ACQUIRE = __ATOMIC_ACQUIRE,
+	CMM_RELEASE = __ATOMIC_RELEASE,
+	CMM_ACQ_REL = __ATOMIC_ACQ_REL,
+	CMM_SEQ_CST = __ATOMIC_SEQ_CST,
+};
+#else
+enum cmm_memorder {
+	CMM_RELAXED,
+	CMM_CONSUME,
+	CMM_ACQUIRE,
+	CMM_RELEASE,
+	CMM_ACQ_REL,
+	CMM_SEQ_CST,
+};
+#endif
+
 #if defined(CONFIG_RCU_USE_ATOMIC_BUILTINS)
 #include <urcu/uatomic/builtins.h>
 #elif defined(URCU_ARCH_X86)
include/urcu/uatomic/builtins-generic.h +60 -21

106 lines not on this page · raw message

include/urcu/uatomic/builtins-x86.h +59 -20

105 lines not on this page · raw message

include/urcu/uatomic/generic.h +128 -0

138 lines not on this page · raw message

src/urcu-pointer.c +3 -6
index d0854ac..cea8aeb 100644
@@ -39,19 +39,16 @@ void *rcu_dereference_sym(void *p)
 
 void *rcu_set_pointer_sym(void **p, void *v)
 {
-	cmm_wmb();
-	uatomic_set(p, v);
+	uatomic_store(p, v, CMM_RELEASE);
 	return v;
 }
 
 void *rcu_xchg_pointer_sym(void **p, void *v)
 {
-	cmm_wmb();
-	return uatomic_xchg(p, v);
+	return uatomic_xchg_mo(p, v, CMM_SEQ_CST);
 }
 
 void *rcu_cmpxchg_pointer_sym(void **p, void *old, void *_new)
 {
-	cmm_wmb();
-	return uatomic_cmpxchg(p, old, _new);
+	return uatomic_cmpxchg_mo(p, old, _new, CMM_SEQ_CST, CMM_SEQ_CST);
 }
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#23 lore raw

PATCH 7/11 urcu-wait: Fix wait state load/store

In reply to Olivier Dion via lttng-dev

The state of a wait node must be accessed atomically. Also, the action of busy loading until the teardown state is seen must follow a CMM_ACQUIRE semantic while storing the teardown must follow a CMM_RELEASE semantic.

Change-Id I9cd9cf4cd9ab2081551d7f33c0b1c23c3cf3942f
---
 src/urcu-wait.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
src/urcu-wait.h +3 -3
index ef5f7ed..4667a13 100644
@@ -135,7 +135,7 @@ void urcu_adaptative_wake_up(struct urcu_wait_node *wait)
 			urcu_die(errno);
 	}
 	/* Allow teardown of struct urcu_wait memory. */
-	uatomic_or(&wait->state, URCU_WAIT_TEARDOWN);
+	uatomic_or_mo(&wait->state, URCU_WAIT_TEARDOWN, CMM_RELEASE);
 }
 
 /*
@@ -193,7 +193,7 @@ skip_futex_wait:
 			break;
 		caa_cpu_relax();
 	}
-	while (!(uatomic_read(&wait->state) & URCU_WAIT_TEARDOWN))
+	while (!(uatomic_load(&wait->state, CMM_ACQUIRE) & URCU_WAIT_TEARDOWN))
 		poll(NULL, 0, 10);
 	urcu_posix_assert(uatomic_read(&wait->state) & URCU_WAIT_TEARDOWN);
 }
@@ -209,7 +209,7 @@ void urcu_wake_all_waiters(struct urcu_waiters *waiters)
 			caa_container_of(iter, struct urcu_wait_node, node);
 
 		/* Don't wake already running threads */
-		if (wait_node->state & URCU_WAIT_RUNNING)
+		if (uatomic_load(&wait_node->state, CMM_RELAXED) & URCU_WAIT_RUNNING)
 			continue;
 		urcu_adaptative_wake_up(wait_node);
 	}
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#24 lore raw

PATCH 8/11 tests: Use uatomic for accessing global states

In reply to Olivier Dion via lttng-dev

Global states accesses were protected via memory barriers. Use the uatomic API with the RCU memory model so that TSAN does not warns about none atomic concurrent accesses.

Also, the thread id map mutex must be unlocked after setting the new created thread id in the map. Otherwise, the new thread could observe an unset id.

Change-Id I1ecdc387b3f510621cbc116ad3b95c676f5d659a
---
 tests/common/api.h            |  12 ++--
 tests/regression/rcutorture.h | 102 ++++++++++++++++++++++++----------
 2 files changed, 80 insertions(+), 34 deletions(-)
tests/common/api.h +7 -5
index a260463..9d22b0f 100644
@@ -26,6 +26,7 @@
 
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
+#include <urcu/uatomic.h>
 
 /*
  * Machine parameters.
@@ -135,7 +136,7 @@ static int __smp_thread_id(void)
 	thread_id_t tid = pthread_self();
 
 	for (i = 0; i < NR_THREADS; i++) {
-		if (__thread_id_map[i] == tid) {
+		if (uatomic_read(&__thread_id_map[i]) == tid) {
 			long v = i + 1;  /* must be non-NULL. */
 
 			if (pthread_setspecific(thread_id_key, (void *)v) != 0) {
@@ -184,12 +185,13 @@ static thread_id_t create_thread(void *(*func)(void *), void *arg)
 		exit(-1);
 	}
 	__thread_id_map[i] = __THREAD_ID_MAP_WAITING;
-	spin_unlock(&__thread_id_map_mutex);
+
 	if (pthread_create(&tid, NULL, func, arg) != 0) {
 		perror("create_thread:pthread_create");
 		exit(-1);
 	}
-	__thread_id_map[i] = tid;
+	uatomic_set(&__thread_id_map[i], tid);
+	spin_unlock(&__thread_id_map_mutex);
 	return tid;
 }
 
@@ -199,7 +201,7 @@ static void *wait_thread(thread_id_t tid)
 	void *vp;
 
 	for (i = 0; i < NR_THREADS; i++) {
-		if (__thread_id_map[i] == tid)
+		if (uatomic_read(&__thread_id_map[i]) == tid)
 			break;
 	}
 	if (i >= NR_THREADS){
@@ -211,7 +213,7 @@ static void *wait_thread(thread_id_t tid)
 		perror("wait_thread:pthread_join");
 		exit(-1);
 	}
-	__thread_id_map[i] = __THREAD_ID_MAP_EMPTY;
+	uatomic_set(&__thread_id_map[i], __THREAD_ID_MAP_EMPTY);
 	return vp;
 }
 
tests/regression/rcutorture.h +73 -29

202 lines not on this page · raw message

Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#25 lore raw

PATCH 9/11 benchmark: Use uatomic for accessing global states

In reply to Olivier Dion via lttng-dev

Global states accesses were protected via memory barriers. Use the uatomic API with the URCU memory model so that TSAN can understand the ordering imposed by the synchronization flags.

Change-Id I1bf5702c5ac470f308c478effe39e424a3158060
---
 tests/benchmark/Makefile.am             | 91 +++++++++++++------------
 tests/benchmark/common-states.c         |  1 +
 tests/benchmark/common-states.h         | 51 ++++++++++++++
 tests/benchmark/test_mutex.c            | 32 +--------
 tests/benchmark/test_perthreadlock.c    | 32 +--------
 tests/benchmark/test_rwlock.c           | 32 +--------
 tests/benchmark/test_urcu.c             | 33 +--------
 tests/benchmark/test_urcu_assign.c      | 33 +--------
 tests/benchmark/test_urcu_bp.c          | 33 +--------
 tests/benchmark/test_urcu_defer.c       | 33 +--------
 tests/benchmark/test_urcu_gc.c          | 34 ++-------
 tests/benchmark/test_urcu_hash.c        |  6 +-
 tests/benchmark/test_urcu_hash.h        | 15 ----
 tests/benchmark/test_urcu_hash_rw.c     | 10 +--
 tests/benchmark/test_urcu_hash_unique.c | 10 +--
 tests/benchmark/test_urcu_lfq.c         | 20 ++----
 tests/benchmark/test_urcu_lfs.c         | 20 ++----
 tests/benchmark/test_urcu_lfs_rcu.c     | 20 ++----
 tests/benchmark/test_urcu_qsbr.c        | 33 +--------
 tests/benchmark/test_urcu_qsbr_gc.c     | 34 ++-------
 tests/benchmark/test_urcu_wfcq.c        | 22 +++---
 tests/benchmark/test_urcu_wfq.c         | 20 ++----
 tests/benchmark/test_urcu_wfs.c         | 22 +++---
 23 files changed, 177 insertions(+), 460 deletions(-)
 create mode 100644 tests/benchmark/common-states.c
 create mode 100644 tests/benchmark/common-states.h
tests/benchmark/Makefile.am +47 -44
index c53e025..a7f91c2 100644
@@ -1,4 +1,5 @@
 AM_CPPFLAGS += -I$(top_srcdir)/src -I$(top_srcdir)/tests/common
+AM_CPPFLAGS += -include $(top_srcdir)/tests/benchmark/common-states.h
 
 TEST_EXTENSIONS = .tap
 TAP_LOG_DRIVER_FLAGS = --merge --comments
@@ -7,6 +8,8 @@ TAP_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' \
 	URCU_TESTS_BUILDDIR='$(abs_top_builddir)/tests' \
 	$(SHELL) $(top_srcdir)/tests/utils/tap-driver.sh
 
+noinst_HEADERS = common-states.h
+
 SCRIPT_LIST = \
 	runpaul-phase1.sh \
 	runpaul-phase2.sh \
@@ -61,163 +64,163 @@ URCU_CDS_LIB=$(top_builddir)/src/liburcu-cds.la
 
 DEBUG_YIELD_LIB=$(builddir)/../common/libdebug-yield.la
 
-test_urcu_SOURCES = test_urcu.c
+test_urcu_SOURCES = test_urcu.c common-states.c
 test_urcu_LDADD = $(URCU_LIB)
 
-test_urcu_dynamic_link_SOURCES = test_urcu.c
+test_urcu_dynamic_link_SOURCES = test_urcu.c common-states.c
 test_urcu_dynamic_link_LDADD = $(URCU_LIB)
 test_urcu_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_timing_SOURCES = test_urcu_timing.c
+test_urcu_timing_SOURCES = test_urcu_timing.c common-states.c
 test_urcu_timing_LDADD = $(URCU_LIB)
 
-test_urcu_yield_SOURCES = test_urcu.c
+test_urcu_yield_SOURCES = test_urcu.c common-states.c
 test_urcu_yield_LDADD = $(URCU_LIB) $(DEBUG_YIELD_LIB)
 test_urcu_yield_CFLAGS = -DDEBUG_YIELD $(AM_CFLAGS)
 
 
-test_urcu_qsbr_SOURCES = test_urcu_qsbr.c
+test_urcu_qsbr_SOURCES = test_urcu_qsbr.c common-states.c
 test_urcu_qsbr_LDADD = $(URCU_QSBR_LIB)
 
-test_urcu_qsbr_timing_SOURCES = test_urcu_qsbr_timing.c
+test_urcu_qsbr_timing_SOURCES = test_urcu_qsbr_timing.c common-states.c
 test_urcu_qsbr_timing_LDADD = $(URCU_QSBR_LIB)
 
 
-test_urcu_mb_SOURCES = test_urcu.c
+test_urcu_mb_SOURCES = test_urcu.c common-states.c
 test_urcu_mb_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_CFLAGS = -DRCU_MB $(AM_CFLAGS)
 
 
-test_urcu_signal_SOURCES = test_urcu.c
+test_urcu_signal_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_CFLAGS = -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_signal_dynamic_link_SOURCES = test_urcu.c
+test_urcu_signal_dynamic_link_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_dynamic_link_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_dynamic_link_CFLAGS = -DRCU_SIGNAL -DDYNAMIC_LINK_TEST \
 					$(AM_CFLAGS)
 
-test_urcu_signal_timing_SOURCES = test_urcu_timing.c
+test_urcu_signal_timing_SOURCES = test_urcu_timing.c common-states.c
 test_urcu_signal_timing_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_timing_CFLAGS= -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_signal_yield_SOURCES = test_urcu.c
+test_urcu_signal_yield_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_yield_LDADD = $(URCU_SIGNAL_LIB) $(DEBUG_YIELD_LIB)
 test_urcu_signal_yield_CFLAGS = -DRCU_SIGNAL -DDEBUG_YIELD $(AM_CFLAGS)
 
-test_rwlock_timing_SOURCES = test_rwlock_timing.c
+test_rwlock_timing_SOURCES = test_rwlock_timing.c common-states.c
 test_rwlock_timing_LDADD = $(URCU_SIGNAL_LIB)
 
-test_rwlock_SOURCES = test_rwlock.c
+test_rwlock_SOURCES = test_rwlock.c common-states.c
 test_rwlock_LDADD = $(URCU_SIGNAL_LIB)
 
-test_perthreadlock_timing_SOURCES = test_perthreadlock_timing.c
+test_perthreadlock_timing_SOURCES = test_perthreadlock_timing.c common-states.c
 test_perthreadlock_timing_LDADD = $(URCU_SIGNAL_LIB)
 
-test_perthreadlock_SOURCES = test_perthreadlock.c
+test_perthreadlock_SOURCES = test_perthreadlock.c common-states.c
 test_perthreadlock_LDADD = $(URCU_SIGNAL_LIB)
 
-test_mutex_SOURCES = test_mutex.c
+test_mutex_SOURCES = test_mutex.c common-states.c
 
-test_looplen_SOURCES = test_looplen.c
+test_looplen_SOURCES = test_looplen.c common-states.c
 
-test_urcu_gc_SOURCES = test_urcu_gc.c
+test_urcu_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_gc_LDADD = $(URCU_LIB)
 
-test_urcu_signal_gc_SOURCES = test_urcu_gc.c
+test_urcu_signal_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_signal_gc_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_gc_CFLAGS = -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_mb_gc_SOURCES = test_urcu_gc.c
+test_urcu_mb_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_mb_gc_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_gc_CFLAGS = -DRCU_MB $(AM_CFLAGS)
 
-test_urcu_qsbr_gc_SOURCES = test_urcu_qsbr_gc.c
+test_urcu_qsbr_gc_SOURCES = test_urcu_qsbr_gc.c common-states.c
 test_urcu_qsbr_gc_LDADD = $(URCU_QSBR_LIB)
 
-test_urcu_qsbr_lgc_SOURCES = test_urcu_qsbr_gc.c
+test_urcu_qsbr_lgc_SOURCES = test_urcu_qsbr_gc.c common-states.c
 test_urcu_qsbr_lgc_LDADD = $(URCU_QSBR_LIB)
 test_urcu_qsbr_lgc_CFLAGS = -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_lgc_SOURCES = test_urcu_gc.c
+test_urcu_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_lgc_LDADD = $(URCU_LIB)
 test_urcu_lgc_CFLAGS = -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_signal_lgc_SOURCES = test_urcu_gc.c
+test_urcu_signal_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_signal_lgc_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_lgc_CFLAGS = -DRCU_SIGNAL -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_mb_lgc_SOURCES = test_urcu_gc.c
+test_urcu_mb_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_mb_lgc_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_lgc_CFLAGS = -DTEST_LOCAL_GC -DRCU_MB $(AM_CFLAGS)
 
-test_urcu_qsbr_dynamic_link_SOURCES = test_urcu_qsbr.c
+test_urcu_qsbr_dynamic_link_SOURCES = test_urcu_qsbr.c common-states.c
 test_urcu_qsbr_dynamic_link_LDADD = $(URCU_QSBR_LIB)
 test_urcu_qsbr_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_defer_SOURCES = test_urcu_defer.c
+test_urcu_defer_SOURCES = test_urcu_defer.c common-states.c
 test_urcu_defer_LDADD = $(URCU_LIB)
 
 test_cycles_per_loop_SOURCES = test_cycles_per_loop.c
 
-test_urcu_assign_SOURCES = test_urcu_assign.c
+test_urcu_assign_SOURCES = test_urcu_assign.c common-states.c
 test_urcu_assign_LDADD = $(URCU_LIB)
 
-test_urcu_assign_dynamic_link_SOURCES = test_urcu_assign.c
+test_urcu_assign_dynamic_link_SOURCES = test_urcu_assign.c common-states.c
 test_urcu_assign_dynamic_link_LDADD = $(URCU_LIB)
 test_urcu_assign_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_bp_SOURCES = test_urcu_bp.c
+test_urcu_bp_SOURCES = test_urcu_bp.c common-states.c
 test_urcu_bp_LDADD = $(URCU_BP_LIB)
 
-test_urcu_bp_dynamic_link_SOURCES = test_urcu_bp.c
+test_urcu_bp_dynamic_link_SOURCES = test_urcu_bp.c common-states.c
 test_urcu_bp_dynamic_link_LDADD = $(URCU_BP_LIB)
 test_urcu_bp_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_lfq_SOURCES = test_urcu_lfq.c
+test_urcu_lfq_SOURCES = test_urcu_lfq.c common-states.c
 test_urcu_lfq_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfq_dynlink_SOURCES = test_urcu_lfq.c
+test_urcu_lfq_dynlink_SOURCES = test_urcu_lfq.c common-states.c
 test_urcu_lfq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfq_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_wfq_SOURCES = test_urcu_wfq.c
+test_urcu_wfq_SOURCES = test_urcu_wfq.c common-states.c
 test_urcu_wfq_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfq_dynlink_SOURCES = test_urcu_wfq.c
+test_urcu_wfq_dynlink_SOURCES = test_urcu_wfq.c common-states.c
 test_urcu_wfq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfq_dynlink_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfcq_SOURCES = test_urcu_wfcq.c
+test_urcu_wfcq_SOURCES = test_urcu_wfcq.c common-states.c
 test_urcu_wfcq_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfcq_dynlink_SOURCES = test_urcu_wfcq.c
+test_urcu_wfcq_dynlink_SOURCES = test_urcu_wfcq.c common-states.c
 test_urcu_wfcq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfcq_dynlink_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_lfs_SOURCES = test_urcu_lfs.c
+test_urcu_lfs_SOURCES = test_urcu_lfs.c common-states.c
 test_urcu_lfs_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_rcu_SOURCES = test_urcu_lfs_rcu.c
+test_urcu_lfs_rcu_SOURCES = test_urcu_lfs_rcu.c common-states.c
 test_urcu_lfs_rcu_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_dynlink_SOURCES = test_urcu_lfs.c
+test_urcu_lfs_dynlink_SOURCES = test_urcu_lfs.c common-states.c
 test_urcu_lfs_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfs_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_rcu_dynlink_SOURCES = test_urcu_lfs_rcu.c
+test_urcu_lfs_rcu_dynlink_SOURCES = test_urcu_lfs_rcu.c common-states.c
 test_urcu_lfs_rcu_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfs_rcu_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_wfs_SOURCES = test_urcu_wfs.c
+test_urcu_wfs_SOURCES = test_urcu_wfs.c common-states.c
 test_urcu_wfs_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfs_dynlink_SOURCES = test_urcu_wfs.c
+test_urcu_wfs_dynlink_SOURCES = test_urcu_wfs.c common-states.c
 test_urcu_wfs_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfs_dynlink_LDADD = $(URCU_COMMON_LIB)
 
 test_urcu_hash_SOURCES = test_urcu_hash.c test_urcu_hash.h \
-		test_urcu_hash_rw.c test_urcu_hash_unique.c
+		test_urcu_hash_rw.c test_urcu_hash_unique.c common-states.c
 test_urcu_hash_CFLAGS = -DRCU_QSBR $(AM_CFLAGS)
 test_urcu_hash_LDADD = $(URCU_QSBR_LIB) $(URCU_COMMON_LIB) $(URCU_CDS_LIB)
 
tests/benchmark/common-states.c new file +1 -0
new file mode 100644
index 0000000..6e70351
@@ -0,0 +1 @@
+volatile int _test_go = 0, _test_stop = 0;
tests/benchmark/common-states.h new file +51 -0

51 lines not on this page · raw message

tests/benchmark/test_mutex.c +3 -29
index 55f7c38..145139c 100644
@@ -49,8 +49,6 @@ struct test_array {
 
 static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -111,19 +109,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -147,9 +132,7 @@ void *thr_reader(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int v;
@@ -182,10 +165,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		pthread_mutex_lock(&lock);
@@ -325,13 +305,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_perthreadlock.c +3 -29
index 47a512c..bf468eb 100644
@@ -53,8 +53,6 @@ struct per_thread_lock {
 
 static struct per_thread_lock *per_thread_lock;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -117,19 +115,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -175,9 +160,7 @@ void *thr_reader(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int v;
@@ -211,10 +194,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		for (tidx = 0; tidx < (long)nr_readers; tidx++) {
@@ -359,13 +339,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_rwlock.c +3 -29
index 6908ea4..f5099e8 100644
@@ -53,8 +53,6 @@ struct test_array {
  */
 pthread_rwlock_t lock;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -116,19 +114,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -147,9 +132,7 @@ void *thr_reader(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int a, ret;
@@ -194,10 +177,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		int ret;
@@ -355,13 +335,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu.c +3 -30
index ea849fa..b89513b 100644
@@ -44,8 +44,6 @@
 #endif
 #include <urcu.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -107,19 +105,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -142,10 +127,7 @@ void *thr_reader(void *_count)
 	rcu_register_thread();
 	urcu_posix_assert(!rcu_read_ongoing());
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -186,10 +168,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -337,13 +316,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_assign.c +3 -30
index 88889a8..e83b05e 100644
@@ -48,8 +48,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -111,19 +109,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -201,10 +186,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -240,10 +222,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_copy_mutex_lock();
@@ -394,13 +373,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_bp.c +3 -30
index 6f8c59d..c3b00f1 100644
@@ -44,8 +44,6 @@
 #endif
 #include <urcu-bp.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -107,19 +105,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -142,10 +127,7 @@ void *thr_reader(void *_count)
 	rcu_register_thread();
 	urcu_posix_assert(!rcu_read_ongoing());
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -182,10 +164,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -332,13 +311,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_defer.c +3 -30
index e948ebf..c501f60 100644
@@ -49,8 +49,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -112,19 +110,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -149,10 +134,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -203,10 +185,7 @@ void *thr_writer(void *data)
 		exit(-1);
 	}
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(*new));
@@ -359,13 +338,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_gc.c +4 -30
index f14f728..1cbee44 100644
@@ -33,6 +33,7 @@
 #include <urcu/arch.h>
 #include <urcu/assert.h>
 #include <urcu/tls-compat.h>
+#include <urcu/uatomic.h>
 #include "thread-id.h"
 #include "../common/debug-yield.h"
 
@@ -48,8 +49,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -120,19 +119,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -157,10 +143,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -231,10 +214,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 #ifndef TEST_LOCAL_GC
@@ -399,13 +379,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_hash.c +2 -4
index 3574b4c..1a3087e 100644
@@ -96,8 +96,6 @@ DEFINE_URCU_TLS(unsigned long, lookup_ok);
 
 struct cds_lfht *test_ht;
 
-volatile int test_go, test_stop;
-
 unsigned long wdelay;
 
 unsigned long duration;
@@ -649,14 +647,14 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	remain = duration;
 	do {
 		remain = sleep(remain);
 	} while (remain > 0);
 
-	test_stop = 1;
+	end_test();
 
 end_pthread_join:
 	for (i_thr = 0; i_thr < nr_readers_created; i_thr++) {
tests/benchmark/test_urcu_hash.h +0 -15
index 47b2ae3..73a0a6d 100644
@@ -125,8 +125,6 @@ cds_lfht_iter_get_test_node(struct cds_lfht_iter *iter)
 	return to_test_node(cds_lfht_iter_get_node(iter));
 }
 
-extern volatile int test_go, test_stop;
-
 extern unsigned long wdelay;
 
 extern unsigned long duration;
@@ -174,19 +172,6 @@ extern pthread_mutex_t affinity_mutex;
 
 void set_affinity(void);
 
-/*
- * returns 0 if test should end.
- */
-static inline int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static inline int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 extern DECLARE_URCU_TLS(unsigned long long, nr_writes);
 extern DECLARE_URCU_TLS(unsigned long long, nr_reads);
 
tests/benchmark/test_urcu_hash_rw.c +2 -8
index 862a6f0..087e869 100644
@@ -73,10 +73,7 @@ void *test_hash_rw_thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -133,10 +130,7 @@ void *test_hash_rw_thr_writer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfht_node *ret_node = NULL;
tests/benchmark/test_urcu_hash_unique.c +2 -8
index de7c427..90c0e19 100644
@@ -71,10 +71,7 @@ void *test_hash_unique_thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct lfht_test_node *node;
@@ -136,10 +133,7 @@ void *test_hash_unique_thr_writer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		/*
tests/benchmark/test_urcu_lfq.c +6 -14
index 490e8b0..50c4211 100644
@@ -47,8 +47,6 @@
 #include <urcu.h>
 #include <urcu/cds.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -110,12 +108,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -146,10 +144,7 @@ void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -202,10 +197,7 @@ void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfq_node_rcu *qnode;
@@ -375,7 +367,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -385,7 +377,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_lfs.c +6 -14
index 52239e0..48b2b23 100644
@@ -59,8 +59,6 @@ enum test_sync {
 
 static enum test_sync test_sync;
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -124,12 +122,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -159,10 +157,7 @@ static void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -261,10 +256,7 @@ static void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	urcu_posix_assert(test_pop || test_pop_all);
 
@@ -459,7 +451,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -469,7 +461,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_lfs_rcu.c +6 -14
index 7975faf..ae3dff4 100644
@@ -51,8 +51,6 @@
 
 #include <urcu/cds.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -114,12 +112,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -150,10 +148,7 @@ void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -205,10 +200,7 @@ void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfs_node_rcu *snode;
@@ -377,7 +369,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -387,7 +379,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_qsbr.c +3 -30
index 1ea369c..295e9db 100644
@@ -44,8 +44,6 @@
 #endif
 #include "urcu-qsbr.h"
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -106,19 +104,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -145,10 +130,7 @@ void *thr_reader(void *_count)
 	urcu_posix_assert(!rcu_read_ongoing());
 	rcu_thread_online();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -192,10 +174,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -343,13 +322,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_qsbr_gc.c +4 -30
index 8877a82..163405d 100644
@@ -33,6 +33,7 @@
 #include <urcu/arch.h>
 #include <urcu/assert.h>
 #include <urcu/tls-compat.h>
+#include <urcu/uatomic.h>
 #include "thread-id.h"
 #include "../common/debug-yield.h"
 
@@ -46,8 +47,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -118,19 +117,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -154,10 +140,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		_rcu_read_lock();
@@ -231,10 +214,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 #ifndef TEST_LOCAL_GC
@@ -399,13 +379,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_wfcq.c +8 -14
index 2c6e0fd..542a13a 100644
@@ -56,7 +56,7 @@ static enum test_sync test_sync;
 
 static int test_force_sync;
 
-static volatile int test_go, test_stop_enqueue, test_stop_dequeue;
+static volatile int test_stop_enqueue, test_stop_dequeue;
 
 static unsigned long rduration;
 
@@ -122,12 +122,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop_dequeue;
+	return !uatomic_load(&test_stop_dequeue, CMM_RELAXED);
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop_enqueue;
+	return !uatomic_load(&test_stop_enqueue, CMM_RELAXED);
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -155,10 +155,7 @@ static void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfcq_node *node = malloc(sizeof(*node));
@@ -266,10 +263,7 @@ static void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		if (test_dequeue && test_splice) {
@@ -482,7 +476,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -492,7 +486,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_enqueue = 1;
+	uatomic_store(&test_stop_enqueue, 1, CMM_RELEASE);
 
 	if (test_wait_empty) {
 		while (nr_enqueuers != uatomic_read(&test_enqueue_stopped)) {
@@ -503,7 +497,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_dequeue = 1;
+	uatomic_store(&test_stop_dequeue, 1, CMM_RELAXED);
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_wfq.c +6 -14
index 8381160..2d8de87 100644
@@ -51,8 +51,6 @@
 #include <urcu.h>
 #include <urcu/wfqueue.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -114,12 +112,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -143,10 +141,7 @@ void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfq_node *node = malloc(sizeof(*node));
@@ -185,10 +180,7 @@ void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfq_node *node = cds_wfq_dequeue_blocking(&q);
@@ -343,7 +335,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -353,7 +345,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_wfs.c +8 -14
index c285feb..d1a4afc 100644
@@ -59,7 +59,7 @@ static enum test_sync test_sync;
 
 static int test_force_sync;
 
-static volatile int test_go, test_stop_enqueue, test_stop_dequeue;
+static volatile int test_stop_enqueue, test_stop_dequeue;
 
 static unsigned long rduration;
 
@@ -125,12 +125,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop_dequeue;
+	return !uatomic_load(&test_stop_dequeue, CMM_RELAXED);
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop_enqueue;
+	return !uatomic_load(&test_stop_enqueue, CMM_RELAXED);
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -157,10 +157,7 @@ static void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfs_node *node = malloc(sizeof(*node));
@@ -250,10 +247,7 @@ static void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	urcu_posix_assert(test_pop || test_pop_all);
 
@@ -469,7 +463,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -479,7 +473,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_enqueue = 1;
+	uatomic_store(&test_stop_enqueue, 1, CMM_RELEASE);
 
 	if (test_wait_empty) {
 		while (nr_enqueuers != uatomic_read(&test_enqueue_stopped)) {
@@ -490,7 +484,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_dequeue = 1;
+	uatomic_store(&test_stop_dequeue, 1, CMM_RELAXED);
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#26 lore raw

PATCH 10/11 tests/unit/test_build: Quiet unused return value

In reply to Olivier Dion via lttng-dev
Change-Id Ie5a18e0ccc4b1b5ee85c5bd140561cc2ff9e2fbc
---
 tests/unit/test_build.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/unit/test_build.c +4 -4
index f6b667c..702c1ef 100644
@@ -129,10 +129,10 @@ void test_build_rcu_dereference(void)
 	static struct a_clear_struct *clear = NULL;
 	static struct a_clear_struct *const clear_const = NULL;
 
-	rcu_dereference(opaque);
-	rcu_dereference(opaque_const);
-	rcu_dereference(clear);
-	rcu_dereference(clear_const);
+	(void) rcu_dereference(opaque);
+	(void) rcu_dereference(opaque_const);
+	(void) rcu_dereference(clear);
+	(void) rcu_dereference(clear_const);
 }
 
 int main(void)
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#27 lore raw

PATCH 11/11 urcu/annotate: Add CMM annotation

In reply to Olivier Dion via lttng-dev

The CMM annotation is highly experimental and not meant to be used by user for now, even though it is exposed in the public API since some parts of the liburcu public API require those annotations.

The main primitive is the cmm_annotate_t which denotes a group of memory operations associated with a memory barrier. A group follows a state machine, starting from the `CMM_ANNOTATE_VOID' state. The following are the only valid transitions:

CMM_ANNOTATE_VOID -> CMM_ANNOTATE_MB (acquire & release MB) CMM_ANNOTATE_VOID -> CMM_ANNOTATE_LOAD (acquire memory) CMM_ANNOTATE_LOAD -> CMM_ANNOTATE_MB (acquire MB)

The macro cmm_annotate_define(name)' can be used to create an annotation object on the stack. The rest of the cmm_annotate_*' macros can be used to change the state of the group after validating that the transition is allowed. Some of these macros also inject TSAN annotations to help it understand the flow of events in the program since it does not currently support thread fence.

Sometime, a single memory access does not need to be associated with a group. In the case, the acquire/release macros variant without the `group' infix can be used to annotate memory accesses.

Note that TSAN can not be used on the liburcu-signal flavor. This is because TSAN hijacks calls to sigaction(3) and places its own handler that will deliver the signal to the application at a synchronization point.

Thus, the usage of TSAN on the signal flavor is undefined behavior. However, there's at least one known behavior which is a deadlock between readers that want to unregister them-self by locking the `rcu_registry_lock' while a synchronize RCU is made on the writer side which has already locked that mutex until all the registered readers execute a memory barrier in a signal handler defined by liburcu-signal. However, TSAN will not call the registered handler while waiting on the mutex. Therefore, the writer spin infinitely on pthread_kill(3p) because the reader simply never complete the handshake.

See the deadlock minimal reproducer below.

Deadlock reproducer:

```
#include <poll.h>
#include <signal.h>

#include <pthread.h>

#define SIGURCU SIGUSR1

static pthread_mutex_t rcu_registry_lock = PTHREAD_MUTEX_INITIALIZER;
static int need_mb = 0;

static void *reader_side(void *nil)
{
	(void) nil;

	pthread_mutex_lock(&rcu_registry_lock);
	pthread_mutex_unlock(&rcu_registry_lock);

	return NULL;
}

static void writer_side(pthread_t reader)
{
	__atomic_store_n(&need_mb, 1, __ATOMIC_RELEASE);
	while (__atomic_load_n(&need_mb, __ATOMIC_ACQUIRE)) {
		pthread_kill(reader, SIGURCU);
		(void) poll(NULL, 0, 1);
	}
	pthread_mutex_unlock(&rcu_registry_lock);

	pthread_join(reader, NULL);
}

static void sigrcu_handler(int signo, siginfo_t *siginfo, void *context)
{
	(void) signo;
	(void) siginfo;
	(void) context;

	__atomic_store_n(&need_mb, 0, __ATOMIC_SEQ_CST);
}

static void install_signal(void)
{
	struct sigaction act;

	act.sa_sigaction = sigrcu_handler;
	act.sa_flags     = SA_SIGINFO | SA_RESTART;

	sigemptyset(&act.sa_mask);

	(void) sigaction(SIGURCU, &act, NULL);
}

int main(void)
{
	pthread_t th;

	install_signal();

	pthread_mutex_lock(&rcu_registry_lock);
	pthread_create(&th, NULL, reader_side, NULL);

	writer_side(th);

	return 0;
}
```
Change-Id I9c234bb311cc0f82ea9dbefdf4fee07047ab93f9
---
 include/Makefile.am               |   1 +
 include/urcu/annotate.h           | 174 ++++++++++++++++++++++++++++++
 include/urcu/arch/generic.h       |  33 +++++-
 include/urcu/compiler.h           |  12 +++
 include/urcu/static/urcu-bp.h     |  12 ++-
 include/urcu/static/urcu-common.h |   8 +-
 include/urcu/static/urcu-mb.h     |  11 +-
 include/urcu/static/urcu-memb.h   |  26 +++--
 include/urcu/static/urcu-qsbr.h   |  29 +++--
 src/rculfhash.c                   |  92 ++++++++++------
 src/urcu-bp.c                     |  17 ++-
 src/urcu-qsbr.c                   |  31 ++++--
 src/urcu-wait.h                   |   9 +-
 src/urcu.c                        |  24 +++--
 14 files changed, 390 insertions(+), 89 deletions(-)
 create mode 100644 include/urcu/annotate.h
include/Makefile.am +1 -0
index fac941f..b1520a1 100644
@@ -1,4 +1,5 @@
 nobase_include_HEADERS = \
+	urcu/annotate.h \
 	urcu/arch/aarch64.h \
 	urcu/arch/alpha.h \
 	urcu/arch/arm.h \
include/urcu/annotate.h new file +174 -0

174 lines not on this page · raw message

include/urcu/arch/generic.h +31 -2
index e292c70..65dedf2 100644
@@ -45,9 +45,38 @@ extern "C" {
 
 #ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
 
+# ifdef CMM_SANITIZE_THREAD
+/*
+ * This makes TSAN quiet about unsupported thread fence.
+ */
+static inline void _cmm_thread_fence_wrapper(void)
+{
+#   if defined(__clang__)
+#    pragma clang diagnostic push
+#    pragma clang diagnostic ignored "-Wpragmas"
+#    pragma clang diagnostic ignored "-Wunknown-warning-option"
+#    pragma clang diagnostic ignored "-Wtsan"
+#   elif defined(__GNUC__)
+#    pragma GCC diagnostic push
+#    pragma GCC diagnostic ignored "-Wpragmas"
+#    pragma GCC diagnostic ignored "-Wtsan"
+#   endif
+	__atomic_thread_fence(__ATOMIC_SEQ_CST);
+#   if defined(__clang__)
+#    pragma clang diagnostic pop
+#   elif defined(__GNUC__)
+#    pragma GCC diagnostic pop
+#   endif
+}
+# endif	 /* CMM_SANITIZE_THREAD */
+
 # ifndef cmm_smp_mb
-#  define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
-# endif
+#  ifdef CMM_SANITIZE_THREAD
+#   define cmm_smp_mb() _cmm_thread_fence_wrapper()
+#  else
+#   define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
+#  endif /* CMM_SANITIZE_THREAD */
+# endif /* !cmm_smp_mb */
 
 #endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
 
include/urcu/compiler.h +12 -0
index 3604488..7930820 100644
@@ -129,4 +129,16 @@
 				+ __GNUC_PATCHLEVEL__)
 #endif
 
+/*
+ * Allow user to manually define CMM_SANITIZE_THREAD if their toolchain is not
+ * supported by this check.
+ */
+#ifndef CMM_SANITIZE_THREAD
+# if defined(__GNUC__) && defined(__SANITIZE_THREAD__)
+#   define CMM_SANITIZE_THREAD
+# elif defined(__clang__) && defined(__has_feature) && __has_feature(thread_sanitizer)
+#   define CMM_SANITIZE_THREAD
+# endif
+#endif	/* !CMM_SANITIZE_THREAD */
+
 #endif /* _URCU_COMPILER_H */
include/urcu/static/urcu-bp.h +9 -3
index 8ba3830..3e14ef7 100644
@@ -33,6 +33,7 @@
 #include <pthread.h>
 #include <unistd.h>
 
+#include <urcu/annotate.h>
 #include <urcu/debug.h>
 #include <urcu/config.h>
 #include <urcu/compiler.h>
@@ -117,7 +118,8 @@ static inline void urcu_bp_smp_mb_slave(void)
 		cmm_smp_mb();
 }
 
-static inline enum urcu_bp_state urcu_bp_reader_state(unsigned long *ctr)
+static inline enum urcu_bp_state urcu_bp_reader_state(unsigned long *ctr,
+						cmm_annotate_t *group)
 {
 	unsigned long v;
 
@@ -127,7 +129,9 @@ static inline enum urcu_bp_state urcu_bp_reader_state(unsigned long *ctr)
 	 * Make sure both tests below are done on the same version of *value
 	 * to insure consistency.
 	 */
-	v = CMM_LOAD_SHARED(*ctr);
+	v = uatomic_load(ctr, CMM_RELAXED);
+	cmm_annotate_group_mem_acquire(group, ctr);
+
 	if (!(v & URCU_BP_GP_CTR_NEST_MASK))
 		return URCU_BP_READER_INACTIVE;
 	if (!((v ^ urcu_bp_gp.ctr) & URCU_BP_GP_CTR_PHASE))
@@ -181,12 +185,14 @@ static inline void _urcu_bp_read_lock(void)
 static inline void _urcu_bp_read_unlock(void)
 {
 	unsigned long tmp;
+	unsigned long *ctr = &URCU_TLS(urcu_bp_reader)->ctr;
 
 	tmp = URCU_TLS(urcu_bp_reader)->ctr;
 	urcu_assert_debug(tmp & URCU_BP_GP_CTR_NEST_MASK);
 	/* Finish using rcu before decrementing the pointer. */
 	urcu_bp_smp_mb_slave();
-	_CMM_STORE_SHARED(URCU_TLS(urcu_bp_reader)->ctr, tmp - URCU_BP_GP_COUNT);
+	cmm_annotate_mem_release(ctr);
+	uatomic_store(ctr, tmp - URCU_BP_GP_COUNT, CMM_RELAXED);
 	cmm_barrier();	/* Ensure the compiler does not reorder us with mutex */
 }
 
include/urcu/static/urcu-common.h +6 -2
index 60ea8b8..32cb834 100644
@@ -34,6 +34,7 @@
 #include <unistd.h>
 #include <stdint.h>
 
+#include <urcu/annotate.h>
 #include <urcu/config.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
@@ -105,7 +106,8 @@ static inline void urcu_common_wake_up_gp(struct urcu_gp *gp)
 }
 
 static inline enum urcu_state urcu_common_reader_state(struct urcu_gp *gp,
-		unsigned long *ctr)
+						unsigned long *ctr,
+						cmm_annotate_t *group)
 {
 	unsigned long v;
 
@@ -113,7 +115,9 @@ static inline enum urcu_state urcu_common_reader_state(struct urcu_gp *gp,
 	 * Make sure both tests below are done on the same version of *value
 	 * to insure consistency.
 	 */
-	v = CMM_LOAD_SHARED(*ctr);
+	v = uatomic_load(ctr, CMM_RELAXED);
+	cmm_annotate_group_mem_acquire(group, ctr);
+
 	if (!(v & URCU_GP_CTR_NEST_MASK))
 		return URCU_READER_INACTIVE;
 	if (!((v ^ gp->ctr) & URCU_GP_CTR_PHASE))
include/urcu/static/urcu-mb.h +6 -5
index b97e42a..5bf7933 100644
@@ -108,13 +108,14 @@ static inline void _urcu_mb_read_lock(void)
  */
 static inline void _urcu_mb_read_unlock_update_and_wakeup(unsigned long tmp)
 {
+	unsigned long *ctr = &URCU_TLS(urcu_mb_reader).ctr;
+
 	if (caa_likely((tmp & URCU_GP_CTR_NEST_MASK) == URCU_GP_COUNT)) {
-		cmm_smp_mb();
-		_CMM_STORE_SHARED(URCU_TLS(urcu_mb_reader).ctr, tmp - URCU_GP_COUNT);
-		cmm_smp_mb();
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_SEQ_CST);
 		urcu_common_wake_up_gp(&urcu_mb_gp);
-	} else
-		_CMM_STORE_SHARED(URCU_TLS(urcu_mb_reader).ctr, tmp - URCU_GP_COUNT);
+	} else {
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_RELAXED);
+	}
 }
 
 /*
include/urcu/static/urcu-memb.h +20 -6
index c8d102f..8191ccc 100644
@@ -34,6 +34,7 @@
 #include <unistd.h>
 #include <stdint.h>
 
+#include <urcu/annotate.h>
 #include <urcu/debug.h>
 #include <urcu/config.h>
 #include <urcu/compiler.h>
@@ -93,11 +94,20 @@ extern DECLARE_URCU_TLS(struct urcu_reader, urcu_memb_reader);
  */
 static inline void _urcu_memb_read_lock_update(unsigned long tmp)
 {
+	unsigned long *ctr = &URCU_TLS(urcu_memb_reader).ctr;
+
 	if (caa_likely(!(tmp & URCU_GP_CTR_NEST_MASK))) {
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, _CMM_LOAD_SHARED(urcu_memb_gp.ctr));
+		unsigned long *pgctr = &urcu_memb_gp.ctr;
+		unsigned long gctr = uatomic_load(pgctr, CMM_RELAXED);
+
+		/* Paired with following mb slave. */
+		cmm_annotate_mem_acquire(pgctr);
+		uatomic_store(ctr, gctr, CMM_RELAXED);
+
 		urcu_memb_smp_mb_slave();
-	} else
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, tmp + URCU_GP_COUNT);
+	} else {
+		uatomic_store(ctr, tmp + URCU_GP_COUNT, CMM_RELAXED);
+	}
 }
 
 /*
@@ -131,13 +141,17 @@ static inline void _urcu_memb_read_lock(void)
  */
 static inline void _urcu_memb_read_unlock_update_and_wakeup(unsigned long tmp)
 {
+	unsigned long *ctr = &URCU_TLS(urcu_memb_reader).ctr;
+
 	if (caa_likely((tmp & URCU_GP_CTR_NEST_MASK) == URCU_GP_COUNT)) {
 		urcu_memb_smp_mb_slave();
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, tmp - URCU_GP_COUNT);
+		cmm_annotate_mem_release(ctr);
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_RELAXED);
 		urcu_memb_smp_mb_slave();
 		urcu_common_wake_up_gp(&urcu_memb_gp);
-	} else
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, tmp - URCU_GP_COUNT);
+	} else {
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_RELAXED);
+	}
 }
 
 /*
include/urcu/static/urcu-qsbr.h +19 -10
index b878877..864cbcf 100644
@@ -35,6 +35,7 @@
 #include <unistd.h>
 #include <stdint.h>
 
+#include <urcu/annotate.h>
 #include <urcu/debug.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
@@ -96,11 +97,14 @@ static inline void urcu_qsbr_wake_up_gp(void)
 	}
 }
 
-static inline enum urcu_state urcu_qsbr_reader_state(unsigned long *ctr)
+static inline enum urcu_state urcu_qsbr_reader_state(unsigned long *ctr,
+						cmm_annotate_t *group)
 {
 	unsigned long v;
 
-	v = CMM_LOAD_SHARED(*ctr);
+	v = uatomic_load(ctr, CMM_RELAXED);
+	cmm_annotate_group_mem_acquire(group, ctr);
+
 	if (!v)
 		return URCU_READER_INACTIVE;
 	if (v == urcu_qsbr_gp.ctr)
@@ -155,9 +159,9 @@ static inline int _urcu_qsbr_read_ongoing(void)
  */
 static inline void _urcu_qsbr_quiescent_state_update_and_wakeup(unsigned long gp_ctr)
 {
-	cmm_smp_mb();
-	_CMM_STORE_SHARED(URCU_TLS(urcu_qsbr_reader).ctr, gp_ctr);
-	cmm_smp_mb();	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
+	uatomic_store(&URCU_TLS(urcu_qsbr_reader).ctr, gp_ctr, CMM_SEQ_CST);
+
+	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
 	urcu_qsbr_wake_up_gp();
 	cmm_smp_mb();
 }
@@ -179,7 +183,8 @@ static inline void _urcu_qsbr_quiescent_state(void)
 	unsigned long gp_ctr;
 
 	urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
-	if ((gp_ctr = CMM_LOAD_SHARED(urcu_qsbr_gp.ctr)) == URCU_TLS(urcu_qsbr_reader).ctr)
+	gp_ctr = uatomic_load(&urcu_qsbr_gp.ctr, CMM_RELAXED);
+	if (gp_ctr == URCU_TLS(urcu_qsbr_reader).ctr)
 		return;
 	_urcu_qsbr_quiescent_state_update_and_wakeup(gp_ctr);
 }
@@ -195,9 +200,8 @@ static inline void _urcu_qsbr_quiescent_state(void)
 static inline void _urcu_qsbr_thread_offline(void)
 {
 	urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
-	cmm_smp_mb();
-	CMM_STORE_SHARED(URCU_TLS(urcu_qsbr_reader).ctr, 0);
-	cmm_smp_mb();	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
+	uatomic_store(&URCU_TLS(urcu_qsbr_reader).ctr, 0, CMM_SEQ_CST);
+	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
 	urcu_qsbr_wake_up_gp();
 	cmm_barrier();	/* Ensure the compiler does not reorder us with mutex */
 }
@@ -212,9 +216,14 @@ static inline void _urcu_qsbr_thread_offline(void)
  */
 static inline void _urcu_qsbr_thread_online(void)
 {
+	unsigned long *pctr = &URCU_TLS(urcu_qsbr_reader).ctr;
+	unsigned long ctr;
+
 	urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
 	cmm_barrier();	/* Ensure the compiler does not reorder us with mutex */
-	_CMM_STORE_SHARED(URCU_TLS(urcu_qsbr_reader).ctr, CMM_LOAD_SHARED(urcu_qsbr_gp.ctr));
+	ctr = uatomic_load(&urcu_qsbr_gp.ctr, CMM_RELAXED);
+	cmm_annotate_mem_acquire(&urcu_qsbr_gp.ctr);
+	uatomic_store(pctr, ctr, CMM_RELAXED);
 	cmm_smp_mb();
 }
 
src/rculfhash.c +58 -34

227 lines not on this page · raw message

src/urcu-bp.c +12 -5
index 47fad8e..08aaa88 100644
@@ -36,6 +36,7 @@
 #include <stdbool.h>
 #include <sys/mman.h>
 
+#include <urcu/annotate.h>
 #include <urcu/assert.h>
 #include <urcu/config.h>
 #include <urcu/arch.h>
@@ -220,7 +221,8 @@ static void smp_mb_master(void)
  */
 static void wait_for_readers(struct cds_list_head *input_readers,
 			struct cds_list_head *cur_snap_readers,
-			struct cds_list_head *qsreaders)
+			struct cds_list_head *qsreaders,
+			cmm_annotate_t *group)
 {
 	unsigned int wait_loops = 0;
 	struct urcu_bp_reader *index, *tmp;
@@ -235,7 +237,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 			wait_loops++;
 
 		cds_list_for_each_entry_safe(index, tmp, input_readers, node) {
-			switch (urcu_bp_reader_state(&index->ctr)) {
+			switch (urcu_bp_reader_state(&index->ctr, group)) {
 			case URCU_BP_READER_ACTIVE_CURRENT:
 				if (cur_snap_readers) {
 					cds_list_move(&index->node,
@@ -274,6 +276,8 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 
 void urcu_bp_synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(cur_snap_readers);
 	CDS_LIST_HEAD(qsreaders);
 	sigset_t newmask, oldmask;
@@ -295,13 +299,14 @@ void urcu_bp_synchronize_rcu(void)
 	 * where new ptr points to. */
 	/* Write new ptr before changing the qparity */
 	smp_mb_master();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Wait for readers to observe original parity or be quiescent.
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, &cur_snap_readers, &qsreaders);
+	wait_for_readers(&registry, &cur_snap_readers, &qsreaders, &acquire_group);
 
 	/*
 	 * Adding a cmm_smp_mb() which is _not_ formally required, but makes the
@@ -311,7 +316,8 @@ void urcu_bp_synchronize_rcu(void)
 	cmm_smp_mb();
 
 	/* Switch parity: 0 -> 1, 1 -> 0 */
-	CMM_STORE_SHARED(rcu_gp.ctr, rcu_gp.ctr ^ URCU_BP_GP_CTR_PHASE);
+	cmm_annotate_group_mem_release(&release_group, &rcu_gp.ctr);
+	uatomic_store(&rcu_gp.ctr, rcu_gp.ctr ^ URCU_BP_GP_CTR_PHASE, CMM_RELAXED);
 
 	/*
 	 * Must commit qparity update to memory before waiting for other parity
@@ -332,7 +338,7 @@ void urcu_bp_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&cur_snap_readers, NULL, &qsreaders);
+	wait_for_readers(&cur_snap_readers, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -344,6 +350,7 @@ void urcu_bp_synchronize_rcu(void)
 	 * freed.
 	 */
 	smp_mb_master();
+	cmm_annotate_group_mb_acquire(&acquire_group);
 out:
 	mutex_unlock(&rcu_registry_lock);
 	mutex_unlock(&rcu_gp_lock);
src/urcu-qsbr.c +22 -9
index 318ab29..fd50e80 100644
@@ -34,6 +34,7 @@
 #include <errno.h>
 #include <poll.h>
 
+#include <urcu/annotate.h>
 #include <urcu/assert.h>
 #include <urcu/wfcqueue.h>
 #include <urcu/map/urcu-qsbr.h>
@@ -156,7 +157,8 @@ static void wait_gp(void)
  */
 static void wait_for_readers(struct cds_list_head *input_readers,
 			struct cds_list_head *cur_snap_readers,
-			struct cds_list_head *qsreaders)
+			struct cds_list_head *qsreaders,
+			cmm_annotate_t *group)
 {
 	unsigned int wait_loops = 0;
 	struct urcu_qsbr_reader *index, *tmp;
@@ -183,7 +185,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 			cmm_smp_mb();
 		}
 		cds_list_for_each_entry_safe(index, tmp, input_readers, node) {
-			switch (urcu_qsbr_reader_state(&index->ctr)) {
+			switch (urcu_qsbr_reader_state(&index->ctr, group)) {
 			case URCU_READER_ACTIVE_CURRENT:
 				if (cur_snap_readers) {
 					cds_list_move(&index->node,
@@ -208,8 +210,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 		if (cds_list_empty(input_readers)) {
 			if (wait_loops >= RCU_QS_ACTIVE_ATTEMPTS) {
 				/* Read reader_gp before write futex */
-				cmm_smp_mb();
-				uatomic_set(&urcu_qsbr_gp.futex, 0);
+				uatomic_store(&urcu_qsbr_gp.futex, 0, CMM_RELEASE);
 			}
 			break;
 		} else {
@@ -238,6 +239,8 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 #if (CAA_BITS_PER_LONG < 64)
 void urcu_qsbr_synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(cur_snap_readers);
 	CDS_LIST_HEAD(qsreaders);
 	unsigned long was_online;
@@ -258,6 +261,7 @@ void urcu_qsbr_synchronize_rcu(void)
 		urcu_qsbr_thread_offline();
 	else
 		cmm_smp_mb();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Add ourself to gp_waiters queue of threads awaiting to wait
@@ -289,7 +293,7 @@ void urcu_qsbr_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, &cur_snap_readers, &qsreaders);
+	wait_for_readers(&registry, &cur_snap_readers, &qsreaders, &acquire_group);
 
 	/*
 	 * Must finish waiting for quiescent state for original parity
@@ -309,7 +313,8 @@ void urcu_qsbr_synchronize_rcu(void)
 	cmm_smp_mb();
 
 	/* Switch parity: 0 -> 1, 1 -> 0 */
-	CMM_STORE_SHARED(urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr ^ URCU_QSBR_GP_CTR);
+	cmm_annotate_group_mem_release(&release_group, &urcu_qsbr_gp.ctr);
+	uatomic_store(&urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr ^ URCU_QSBR_GP_CTR, CMM_RELAXED);
 
 	/*
 	 * Must commit urcu_qsbr_gp.ctr update to memory before waiting for
@@ -332,7 +337,7 @@ void urcu_qsbr_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&cur_snap_readers, NULL, &qsreaders);
+	wait_for_readers(&cur_snap_readers, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -347,6 +352,8 @@ gp_end:
 	 * Finish waiting for reader threads before letting the old ptr being
 	 * freed.
 	 */
+	cmm_annotate_group_mb_acquire(&acquire_group);
+
 	if (was_online)
 		urcu_qsbr_thread_online();
 	else
@@ -355,6 +362,8 @@ gp_end:
 #else /* !(CAA_BITS_PER_LONG < 64) */
 void urcu_qsbr_synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(qsreaders);
 	unsigned long was_online;
 	DEFINE_URCU_WAIT_NODE(wait, URCU_WAIT_WAITING);
@@ -371,6 +380,7 @@ void urcu_qsbr_synchronize_rcu(void)
 		urcu_qsbr_thread_offline();
 	else
 		cmm_smp_mb();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Add ourself to gp_waiters queue of threads awaiting to wait
@@ -398,7 +408,8 @@ void urcu_qsbr_synchronize_rcu(void)
 		goto out;
 
 	/* Increment current G.P. */
-	CMM_STORE_SHARED(urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr + URCU_QSBR_GP_CTR);
+	cmm_annotate_group_mem_release(&release_group, &urcu_qsbr_gp.ctr);
+	uatomic_store(&urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr + URCU_QSBR_GP_CTR, CMM_RELAXED);
 
 	/*
 	 * Must commit urcu_qsbr_gp.ctr update to memory before waiting for
@@ -421,7 +432,7 @@ void urcu_qsbr_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, NULL, &qsreaders);
+	wait_for_readers(&registry, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -436,6 +447,8 @@ gp_end:
 		urcu_qsbr_thread_online();
 	else
 		cmm_smp_mb();
+
+	cmm_annotate_group_mb_acquire(&acquire_group);
 }
 #endif  /* !(CAA_BITS_PER_LONG < 64) */
 
src/urcu-wait.h +4 -5
index 4667a13..1ffced4 100644
@@ -126,9 +126,8 @@ void urcu_wait_node_init(struct urcu_wait_node *node,
 static inline
 void urcu_adaptative_wake_up(struct urcu_wait_node *wait)
 {
-	cmm_smp_mb();
 	urcu_posix_assert(uatomic_read(&wait->state) == URCU_WAIT_WAITING);
-	uatomic_set(&wait->state, URCU_WAIT_WAKEUP);
+	uatomic_store(&wait->state, URCU_WAIT_WAKEUP, CMM_RELEASE);
 	if (!(uatomic_read(&wait->state) & URCU_WAIT_RUNNING)) {
 		if (futex_noasync(&wait->state, FUTEX_WAKE, 1,
 				NULL, NULL, 0) < 0)
@@ -150,11 +149,11 @@ void urcu_adaptative_busy_wait(struct urcu_wait_node *wait)
 	/* Load and test condition before read state */
 	cmm_smp_rmb();
 	for (i = 0; i < URCU_WAIT_ATTEMPTS; i++) {
-		if (uatomic_read(&wait->state) != URCU_WAIT_WAITING)
+		if (uatomic_load(&wait->state, CMM_ACQUIRE) != URCU_WAIT_WAITING)
 			goto skip_futex_wait;
 		caa_cpu_relax();
 	}
-	while (uatomic_read(&wait->state) == URCU_WAIT_WAITING) {
+	while (uatomic_load(&wait->state, CMM_ACQUIRE) == URCU_WAIT_WAITING) {
 		if (!futex_noasync(&wait->state, FUTEX_WAIT, URCU_WAIT_WAITING, NULL, NULL, 0)) {
 			/*
 			 * Prior queued wakeups queued by unrelated code
@@ -189,7 +188,7 @@ skip_futex_wait:
 	 * memory allocated for struct urcu_wait.
 	 */
 	for (i = 0; i < URCU_WAIT_ATTEMPTS; i++) {
-		if (uatomic_read(&wait->state) & URCU_WAIT_TEARDOWN)
+		if (uatomic_load(&wait->state, CMM_RELAXED) & URCU_WAIT_TEARDOWN)
 			break;
 		caa_cpu_relax();
 	}
src/urcu.c +16 -8
index c60307e..353e9bb 100644
@@ -38,6 +38,7 @@
 #include <poll.h>
 
 #include <urcu/config.h>
+#include <urcu/annotate.h>
 #include <urcu/assert.h>
 #include <urcu/arch.h>
 #include <urcu/wfcqueue.h>
@@ -300,7 +301,8 @@ end:
  */
 static void wait_for_readers(struct cds_list_head *input_readers,
 			struct cds_list_head *cur_snap_readers,
-			struct cds_list_head *qsreaders)
+			struct cds_list_head *qsreaders,
+			cmm_annotate_t *group)
 {
 	unsigned int wait_loops = 0;
 	struct urcu_reader *index, *tmp;
@@ -323,7 +325,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 		}
 
 		cds_list_for_each_entry_safe(index, tmp, input_readers, node) {
-			switch (urcu_common_reader_state(&rcu_gp, &index->ctr)) {
+			switch (urcu_common_reader_state(&rcu_gp, &index->ctr, group)) {
 			case URCU_READER_ACTIVE_CURRENT:
 				if (cur_snap_readers) {
 					cds_list_move(&index->node,
@@ -407,6 +409,8 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 
 void synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(cur_snap_readers);
 	CDS_LIST_HEAD(qsreaders);
 	DEFINE_URCU_WAIT_NODE(wait, URCU_WAIT_WAITING);
@@ -421,10 +425,11 @@ void synchronize_rcu(void)
 	 * queue before their insertion into the wait queue.
 	 */
 	if (urcu_wait_add(&gp_waiters, &wait) != 0) {
-		/* Not first in queue: will be awakened by another thread. */
+		/*
+		 * Not first in queue: will be awakened by another thread.
+		 * Implies a memory barrier after grace period.
+		 */
 		urcu_adaptative_busy_wait(&wait);
-		/* Order following memory accesses after grace period. */
-		cmm_smp_mb();
 		return;
 	}
 	/* We won't need to wake ourself up */
@@ -449,13 +454,14 @@ void synchronize_rcu(void)
 	 */
 	/* Write new ptr before changing the qparity */
 	smp_mb_master();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Wait for readers to observe original parity or be quiescent.
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, &cur_snap_readers, &qsreaders);
+	wait_for_readers(&registry, &cur_snap_readers, &qsreaders, &acquire_group);
 
 	/*
 	 * Must finish waiting for quiescent state for original parity before
@@ -474,7 +480,8 @@ void synchronize_rcu(void)
 	cmm_smp_mb();
 
 	/* Switch parity: 0 -> 1, 1 -> 0 */
-	CMM_STORE_SHARED(rcu_gp.ctr, rcu_gp.ctr ^ URCU_GP_CTR_PHASE);
+	cmm_annotate_group_mem_release(&release_group, &rcu_gp.ctr);
+	uatomic_store(&rcu_gp.ctr, rcu_gp.ctr ^ URCU_GP_CTR_PHASE, CMM_RELAXED);
 
 	/*
 	 * Must commit rcu_gp.ctr update to memory before waiting for quiescent
@@ -497,7 +504,7 @@ void synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&cur_snap_readers, NULL, &qsreaders);
+	wait_for_readers(&cur_snap_readers, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -510,6 +517,7 @@ void synchronize_rcu(void)
 	 * iterates on reader threads.
 	 */
 	smp_mb_master();
+	cmm_annotate_group_mb_acquire(&acquire_group);
 out:
 	mutex_unlock(&rcu_registry_lock);
 	mutex_unlock(&rcu_gp_lock);
Signature · 6 lines
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#28 lore raw

Re: [lttng-dev] [PATCH 11/11] urcu/annotate: Add CMM annotation

Quoted 10 lines · Tony Finch

The __has_feature pseudo-macro interacts in fun ways with preprocessor expression evaluation, because when it is undefined and you write

	#if defined(__has_feature) && __has_feature(thread_sanitizer)

macro expansion gives you a syntax error

	#if 0 && 0(thread_sanitizer)

so it must be split into two #if expressions, thus:

TIL. Thank you for that!

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#29 lore raw
In reply to Olivier Dion via lttng-dev
Quoted 2 lines · Olivier Dion

This patch set adds support for TSAN in liburcu.

Hi Olivier,

Where can I see the actual changes? Preferably side-by-side diffs with full context. Are they (or can they be) uploaded somewhere on github/gerrit?

Are there any remaining open questions?

Is there CI coverage with -fsanitize=thread?

Quoted 150 lines

* Here are the major changes

- Usage of compiler atomic builtins is added to the uatomic API. This is required for TSAN to understand atomic memory accesses. If the compiler supports such builtins, they are used by default. User can opt-out and use the legacy implementation of the uatomic API by using the `--disable-atomic-builtins' configuration option.

- The CMM memory model is introduced but yet formalized. It tries to be as close as possible to the C11 memory model while offering primitives such as cmm_smp_wmb(), cmm_smp_rmb() and cmm_mb() that can't be expressed in it. For example, cmm_mb() can be used for ordering memory accesses to MMIO devices, which is out of the scope of the C11 memory model.

- The CMM annotation layer is a new public API that is highly experimental and not guaranteed to be stable at this stage. It serves the dual purpose of verifying local (intra-thread) relaxed atomic accesses ordering with a memory barrier and global (inter-thread) relaxed atomic accesses with a shared state. The second purpose is necessary for TSAN to understand memory accesses ordering since it does not fully support thread fence yet.

* CMM annotation example

Consider the following pseudo-code of writer side in synchronize_rcu(). An acquire group is defined on the stack of the writer. Annotations are made onto the group to ensure ordering of relaxed memory accesses in reader_state() before the memory barrier at the end of synchronize_rcu(). It also helps TSAN to understand that the relaxed accesses in reader_state() act like acquire accesses because of the memory barrier in synchronize_rcu().

In other words, the purpose of this annotation is to convert a group of load-acquire memory operations into load-relaxed memory operations followed by a single memory barrier. This highly benefits weakly ordered architectures by having a constant number of memory barriers instead of being linearly proportional to the number of loads. This does not benefit TSO architectures.

```
enum urcu_state reader_state(unsigned long *ctr, cmm_annotate_t *acquire_group)
{
        unsigned long v;

        v = uatomic_load(ctr, CMM_RELAXED);
        cmm_annotate_group_mem_acquire(acquire_group, ctr);
        // ...
}

void wait_for_readers(..., cmm_annotate_group *acquire_group)
{
        // ...
        switch (reader_state(..., acquire_group)) {
                // ...
        }
        // ...
}

void synchronize_rcu()
{
        cmm_annotate_define(acquire_group);
        // ...
        wait_for_readers(..., &acquire_group);
        // ...
        cmm_annotate_group_mb_acquire(&acquire_group);
        cmm_smp_mb();
}
```

* Known limitation

The only known limitation is with the urcu-signal flavor. Indeed, TSAN hijacks calls to sigaction(2) and installs its own signal handler that will deliver the signals to the urcu handler at synchronization points. This is known to deadlock the urcu-signal flavor in at least one case. See commit log of `urcu/annotate: Add CMM annotation' for a minimal reproducer outside of liburcu.

Therefore, we have the intention of deprecating the urcu-signal flavor in the future, starting by disabling it by default.

Olivier Dion (11):
configure: Add --disable-atomic-builtins option urcu/uatomic: Use atomic builtins if configured urcu/compiler: Use atomic builtins if configured urcu/arch/generic: Use atomic builtins if configured urcu/system: Use atomic builtins if configured urcu/uatomic: Add CMM memory model
urcu-wait: Fix wait state load/store
tests: Use uatomic for accessing global states
benchmark: Use uatomic for accessing global states tests/unit/test_build: Quiet unused return value urcu/annotate: Add CMM annotation

 README.md                               |  11 ++
 configure.ac                            |  26 ++++
 include/Makefile.am                     |   4 +
 include/urcu/annotate.h                 | 174 ++++++++++++++++++++++++
 include/urcu/arch/generic.h             |  37 +++++
 include/urcu/compiler.h                 |  20 ++-
 include/urcu/static/pointer.h           |  40 ++----
 include/urcu/static/urcu-bp.h           |  12 +-
 include/urcu/static/urcu-common.h       |   8 +-
 include/urcu/static/urcu-mb.h           |  11 +-
 include/urcu/static/urcu-memb.h         |  26 +++-
 include/urcu/static/urcu-qsbr.h         |  29 ++--
 include/urcu/system.h                   |  21 +++
 include/urcu/uatomic.h                  |  25 +++-
 include/urcu/uatomic/builtins-generic.h | 124 +++++++++++++++++
 include/urcu/uatomic/builtins-x86.h     | 124 +++++++++++++++++
 include/urcu/uatomic/builtins.h         |  83 +++++++++++
 include/urcu/uatomic/generic.h          | 128 +++++++++++++++++
 src/rculfhash.c                         |  92 ++++++++-----
 src/urcu-bp.c                           |  17 ++-
 src/urcu-pointer.c                      |   9 +-
 src/urcu-qsbr.c                         |  31 +++--
 src/urcu-wait.h                         |  15 +-
 src/urcu.c                              |  24 ++--
 tests/benchmark/Makefile.am             |  91 +++++++------
 tests/benchmark/common-states.c         |   1 +
 tests/benchmark/common-states.h         |  51 +++++++
 tests/benchmark/test_mutex.c            |  32 +----
 tests/benchmark/test_perthreadlock.c    |  32 +----
 tests/benchmark/test_rwlock.c           |  32 +----
 tests/benchmark/test_urcu.c             |  33 +----
 tests/benchmark/test_urcu_assign.c      |  33 +----
 tests/benchmark/test_urcu_bp.c          |  33 +----
 tests/benchmark/test_urcu_defer.c       |  33 +----
 tests/benchmark/test_urcu_gc.c          |  34 +----
 tests/benchmark/test_urcu_hash.c        |   6 +-
 tests/benchmark/test_urcu_hash.h        |  15 --
 tests/benchmark/test_urcu_hash_rw.c     |  10 +-
 tests/benchmark/test_urcu_hash_unique.c |  10 +-
 tests/benchmark/test_urcu_lfq.c         |  20 +--
 tests/benchmark/test_urcu_lfs.c         |  20 +--
 tests/benchmark/test_urcu_lfs_rcu.c     |  20 +--
 tests/benchmark/test_urcu_qsbr.c        |  33 +----
 tests/benchmark/test_urcu_qsbr_gc.c     |  34 +----
 tests/benchmark/test_urcu_wfcq.c        |  22 ++-
 tests/benchmark/test_urcu_wfq.c         |  20 +--
 tests/benchmark/test_urcu_wfs.c         |  22 ++-
 tests/common/api.h                      |  12 +-
 tests/regression/rcutorture.h           | 102 ++++++++++----
 tests/unit/test_build.c                 |   8 +-
 50 files changed, 1227 insertions(+), 623 deletions(-)
 create mode 100644 include/urcu/annotate.h
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins-x86.h
 create mode 100644 include/urcu/uatomic/builtins.h
 create mode 100644 tests/benchmark/common-states.c
 create mode 100644 tests/benchmark/common-states.h
Quoted 3 lines

--
2.39.2

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#30 lore raw
Quoted 1 line · Dmitry Vyukov via lttng-dev

On Mon, 15 May 2023 at 22:18, Olivier Dion <[email protected]> wrote:

Hi Dmitry,

Quoted 3 lines

Where can I see the actual changes? Preferably side-by-side diffs with full context. Are they (or can they be) uploaded somewhere on github/gerrit?

Here's the link to the first patch of the set on our Gerrit
<https://review.lttng.org/c/userspace-rcu/+/9737>.

Quoted 1 line

Are there any remaining open questions?

On the Gerrit no. But I would add this for the known issues:

We have a regression test for forking. We get the following when running it:

==23432==ThreadSanitizer: starting new threads after multi-threaded fork is not supported. Dying (set die_after_fork=0 to override)

With TSAN_OPTIONS=die_after_fork=0, here are the results for GCC and Clang.

* gcc 11.3.0
==25266==ThreadSanitizer: dup thread with used id 0x7fd40dafe600

Looks like this was fixed with recent merged of TSAN in gcc 13.

* clang 14.0.6

Tests pass but are very slow. This seems to be because we're calling exit(3) in the childs. Changing it to _exit(2) solves the issue. The only thing I can think of is that exit(3) is not async-signal-safe like _exit(2), yet we do other none-async safe calls like malloc(3) and free(3) in the childs.

Here are some perf records of that:

With exit(3)

```
  12.35%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ebd22b
   8.69%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ebd8fa
   7.50%  test_urcu_fork.  [unknown]                [k] 0xffffffff8a001360
   6.11%  test_urcu_fork.  test_urcu_fork.tap       [.] __sanitizer::internal_memset
   5.85%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan::ForkChildAfter
   1.96%  test_urcu_fork.  [unknown]                [k] 0xffffffff892f8efd
   1.88%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ec8e2c
   1.29%  test_urcu_fork.  [unknown]                [k] 0xffffffff890a99d1
   0.79%  test_urcu_fork.  [unknown]                [k] 0xffffffff8918c566
   0.75%  test_urcu_fork.  test_urcu_fork.tap       [.] __sanitizer::ThreadRegistry::OnFork
   0.71%  test_urcu_fork.  [unknown]                [k] 0xffffffff89349574
   0.64%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ee424f
   0.57%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan::MetaMap::AllocBlock
   0.55%  test_urcu_fork.  [unknown]                [k] 0xffffffff89367249
   0.53%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan_read8
   0.51%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ec8e05
```

With _exit(2)

```
  12.26%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ebd8fa
   9.51%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ebd22b
   6.78%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::internal_memset
   6.49%  test_urcu_fork.  [unknown]             [k] 0xffffffff8a001360
   4.92%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::ForkChildAfter
   2.92%  test_urcu_fork.  [unknown]             [k] 0xffffffff892f8efd
   2.19%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ec8e2c
   1.88%  test_urcu_fork.  [unknown]             [k] 0xffffffff890a99d1
   0.83%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::MetaMap::AllocBlock
   0.72%  test_urcu_fork.  [unknown]             [k] 0xffffffff892f8f1c
   0.67%  test_urcu_fork.  [unknown]             [k] 0xffffffff8918c566
   0.65%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ec16dd
   0.62%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator32<__sanitizer::AP32>, __sanitizer::LargeMmapAllocatorPtrArrayStatic>::Allocate
   0.61%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_write4
   0.60%  test_urcu_fork.  [unknown]             [k] 0xffffffff89367249
   0.59%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ee424f
   0.50%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::ThreadRegistry::OnFork
   0.50%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_write8
   0.47%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::ForkBefore
   0.45%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_func_entry
```
Quoted 1 line

Is there CI coverage with -fsanitize=thread?

Since the urcu-signal flavor deadlocks with TSAN (see reproducer in commit log), the CI simply timeouts. I do however have a job with TSAN as an matrix axis here <https://ci.lttng.org/job/dev_odion_liburcu/>.

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#31 lore raw
Quoted 25 lines · Olivier Dion
Quoted 1 line · Dmitry Vyukov via lttng-dev

On Mon, 15 May 2023 at 22:18, Olivier Dion <[email protected]> wrote:

Hi Dmitry,

Quoted 3 lines

Where can I see the actual changes? Preferably side-by-side diffs with full context. Are they (or can they be) uploaded somewhere on github/gerrit?

Here's the link to the first patch of the set on our Gerrit
<https://review.lttng.org/c/userspace-rcu/+/9737>.

Quoted 1 line

Are there any remaining open questions?

On the Gerrit no. But I would add this for the known issues:

We have a regression test for forking. We get the following when running it:

==23432==ThreadSanitizer: starting new threads after multi-threaded fork is not supported. Dying (set die_after_fork=0 to override)

With TSAN_OPTIONS=die_after_fork=0, here are the results for GCC and Clang.

Quoted 6 lines

* gcc 11.3.0
==25266==ThreadSanitizer: dup thread with used id 0x7fd40dafe600

Looks like this was fixed with recent merged of TSAN in gcc 13.

* clang 14.0.6

Hi Olivier,

Forking under tsan is a bit tricky. But I see we now take a number of internal mutexes around fork (but I think still not all of them), so maybe it's not that bad. If TSAN_OPTIONS=die_after_fork=0 works reasonably reliably for you, then export it for testing.

Older compilers are missing a number of bug fixes.
So if you can restrict testing to newer compilers only, that's the way to go.

Quoted 5 lines

Tests pass but are very slow. This seems to be because we're calling exit(3) in the childs. Changing it to _exit(2) solves the issue. The only thing I can think of is that exit(3) is not async-signal-safe like _exit(2), yet we do other none-async safe calls like malloc(3) and free(3) in the childs.

By default tsan sleeps for 1 second at exit, since exit sequence is a common source of races. Perhaps it's just these sleeps. Try TSAN_OPTIONS=atexit_sleep_ms=0 (or maybe =50).

Quoted 46 lines

Here are some perf records of that:

With exit(3)

```
  12.35%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ebd22b
   8.69%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ebd8fa
   7.50%  test_urcu_fork.  [unknown]                [k] 0xffffffff8a001360
   6.11%  test_urcu_fork.  test_urcu_fork.tap       [.] __sanitizer::internal_memset
   5.85%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan::ForkChildAfter
   1.96%  test_urcu_fork.  [unknown]                [k] 0xffffffff892f8efd
   1.88%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ec8e2c
   1.29%  test_urcu_fork.  [unknown]                [k] 0xffffffff890a99d1
   0.79%  test_urcu_fork.  [unknown]                [k] 0xffffffff8918c566
   0.75%  test_urcu_fork.  test_urcu_fork.tap       [.] __sanitizer::ThreadRegistry::OnFork
   0.71%  test_urcu_fork.  [unknown]                [k] 0xffffffff89349574
   0.64%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ee424f
   0.57%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan::MetaMap::AllocBlock
   0.55%  test_urcu_fork.  [unknown]                [k] 0xffffffff89367249
   0.53%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan_read8
   0.51%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ec8e05
```

With _exit(2)

```
  12.26%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ebd8fa
   9.51%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ebd22b
   6.78%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::internal_memset
   6.49%  test_urcu_fork.  [unknown]             [k] 0xffffffff8a001360
   4.92%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::ForkChildAfter
   2.92%  test_urcu_fork.  [unknown]             [k] 0xffffffff892f8efd
   2.19%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ec8e2c
   1.88%  test_urcu_fork.  [unknown]             [k] 0xffffffff890a99d1
   0.83%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::MetaMap::AllocBlock
   0.72%  test_urcu_fork.  [unknown]             [k] 0xffffffff892f8f1c
   0.67%  test_urcu_fork.  [unknown]             [k] 0xffffffff8918c566
   0.65%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ec16dd
   0.62%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator32<__sanitizer::AP32>, __sanitizer::LargeMmapAllocatorPtrArrayStatic>::Allocate
   0.61%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_write4
   0.60%  test_urcu_fork.  [unknown]             [k] 0xffffffff89367249
   0.59%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ee424f
   0.50%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::ThreadRegistry::OnFork
   0.50%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_write8
   0.47%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::ForkBefore
   0.45%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_func_entry
```
Quoted 5 lines
Quoted 1 line

Is there CI coverage with -fsanitize=thread?

Since the urcu-signal flavor deadlocks with TSAN (see reproducer in commit log), the CI simply timeouts. I do however have a job with TSAN as an matrix axis here <https://ci.lttng.org/job/dev_odion_liburcu/>.

Sounds good.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#32 lore raw

I've skimmed through the changes. They look sane. You know what you are doing. So unless you have any other open issues, I don't have much to add.

Quoted 96 lines · Dmitry Vyukov
Quoted 21 lines

Hi Dmitry,

Quoted 3 lines

Where can I see the actual changes? Preferably side-by-side diffs with full context. Are they (or can they be) uploaded somewhere on github/gerrit?

Here's the link to the first patch of the set on our Gerrit
<https://review.lttng.org/c/userspace-rcu/+/9737>.

Quoted 1 line

Are there any remaining open questions?

On the Gerrit no. But I would add this for the known issues:

We have a regression test for forking. We get the following when running it:

==23432==ThreadSanitizer: starting new threads after multi-threaded fork is not supported. Dying (set die_after_fork=0 to override)

With TSAN_OPTIONS=die_after_fork=0, here are the results for GCC and Clang.

Quoted 6 lines

* gcc 11.3.0
==25266==ThreadSanitizer: dup thread with used id 0x7fd40dafe600

Looks like this was fixed with recent merged of TSAN in gcc 13.

* clang 14.0.6

Hi Olivier,

Forking under tsan is a bit tricky. But I see we now take a number of internal mutexes around fork (but I think still not all of them), so maybe it's not that bad. If TSAN_OPTIONS=die_after_fork=0 works reasonably reliably for you, then export it for testing.

Older compilers are missing a number of bug fixes.
So if you can restrict testing to newer compilers only, that's the way to go.

Quoted 5 lines

Tests pass but are very slow. This seems to be because we're calling exit(3) in the childs. Changing it to _exit(2) solves the issue. The only thing I can think of is that exit(3) is not async-signal-safe like _exit(2), yet we do other none-async safe calls like malloc(3) and free(3) in the childs.

By default tsan sleeps for 1 second at exit, since exit sequence is a common source of races. Perhaps it's just these sleeps. Try TSAN_OPTIONS=atexit_sleep_ms=0 (or maybe =50).

Quoted 46 lines

Here are some perf records of that:

With exit(3)

```
  12.35%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ebd22b
   8.69%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ebd8fa
   7.50%  test_urcu_fork.  [unknown]                [k] 0xffffffff8a001360
   6.11%  test_urcu_fork.  test_urcu_fork.tap       [.] __sanitizer::internal_memset
   5.85%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan::ForkChildAfter
   1.96%  test_urcu_fork.  [unknown]                [k] 0xffffffff892f8efd
   1.88%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ec8e2c
   1.29%  test_urcu_fork.  [unknown]                [k] 0xffffffff890a99d1
   0.79%  test_urcu_fork.  [unknown]                [k] 0xffffffff8918c566
   0.75%  test_urcu_fork.  test_urcu_fork.tap       [.] __sanitizer::ThreadRegistry::OnFork
   0.71%  test_urcu_fork.  [unknown]                [k] 0xffffffff89349574
   0.64%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ee424f
   0.57%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan::MetaMap::AllocBlock
   0.55%  test_urcu_fork.  [unknown]                [k] 0xffffffff89367249
   0.53%  test_urcu_fork.  test_urcu_fork.tap       [.] __tsan_read8
   0.51%  test_urcu_fork.  [unknown]                [k] 0xffffffff89ec8e05
```

With _exit(2)

```
  12.26%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ebd8fa
   9.51%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ebd22b
   6.78%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::internal_memset
   6.49%  test_urcu_fork.  [unknown]             [k] 0xffffffff8a001360
   4.92%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::ForkChildAfter
   2.92%  test_urcu_fork.  [unknown]             [k] 0xffffffff892f8efd
   2.19%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ec8e2c
   1.88%  test_urcu_fork.  [unknown]             [k] 0xffffffff890a99d1
   0.83%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::MetaMap::AllocBlock
   0.72%  test_urcu_fork.  [unknown]             [k] 0xffffffff892f8f1c
   0.67%  test_urcu_fork.  [unknown]             [k] 0xffffffff8918c566
   0.65%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ec16dd
   0.62%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator32<__sanitizer::AP32>, __sanitizer::LargeMmapAllocatorPtrArrayStatic>::Allocate
   0.61%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_write4
   0.60%  test_urcu_fork.  [unknown]             [k] 0xffffffff89367249
   0.59%  test_urcu_fork.  [unknown]             [k] 0xffffffff89ee424f
   0.50%  test_urcu_fork.  test_urcu_fork.tap    [.] __sanitizer::ThreadRegistry::OnFork
   0.50%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_write8
   0.47%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan::ForkBefore
   0.45%  test_urcu_fork.  test_urcu_fork.tap    [.] __tsan_func_entry
```
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Is there CI coverage with -fsanitize=thread?

Since the urcu-signal flavor deadlocks with TSAN (see reproducer in commit log), the CI simply timeouts. I do however have a job with TSAN as an matrix axis here <https://ci.lttng.org/job/dev_odion_liburcu/>.

Sounds good.

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#33 lore raw
Quoted 3 lines · Dmitry Vyukov via lttng-dev

I've skimmed through the changes. They look sane. You know what you are doing. So unless you have any other open issues, I don't have much to add.

Thanks for taking the time!

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#34 lore raw

Hi Dmitry,

We do have a new issue and we think it might be a limitation from TSAN.

Find attached a test program that we believe is correct. You can compile it with `gcc -fsanitize=thread test.c -pthread'.

TSAN flags a race condition between the atomic store relaxed at line 36 and the free at line 81.

We can solve the issue by replacing the atomic store relaxed with a atomic store release. However, the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store, making the release semantic redundant.

We've found an alternative to fix this by ignoring the thread during the relaxed store (line 39). However, what we would like is to annotate the memory stored (line 45).

My theory (I don't know the internal of TSAN much) is that TSAN thinks for some reason that the atomic store relaxed happen at the same epoch as the free, resulting in a false positive. If so, m

Thought?

Signature · 3 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#35 lore raw
Quoted 18 lines · Olivier Dion

Hi Dmitry,

We do have a new issue and we think it might be a limitation from TSAN.

Find attached a test program that we believe is correct. You can compile it with `gcc -fsanitize=thread test.c -pthread'.

TSAN flags a race condition between the atomic store relaxed at line 36 and the free at line 81.

We can solve the issue by replacing the atomic store relaxed with a atomic store release. However, the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store, making the release semantic redundant.

Quoted 7 lines

We've found an alternative to fix this by ignoring the thread during the relaxed store (line 39). However, what we would like is to annotate the memory stored (line 45).

My theory (I don't know the internal of TSAN much) is that TSAN thinks for some reason that the atomic store relaxed happen at the same epoch as the free, resulting in a false positive. If so, m

I don't this this is true in the C/C++ memory model:
"the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store".

std::atomic_thread_fence does affect all preceding/subsequent operations, but an atomic memory operation only affects ordering on that variable, it doesn't also serve as a standalone memory fence.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#36 lore raw

A little bit related question/problem - some of our system test jobs got significantly slower (3x-5x) with **clang-16** TSAN after we integrated liburcu to BIND 9. The GCC (13.1.1) TSAN doesn’t exhibit this behavior.

The liburcu integration in BIND 9 is very light at the moment - we use liburcu for qptrie and there’s some usage of wfcqueue for queuing jobs(callbacks) between threads.

We were not able to put a finger on it yet, but saying this aloud might help diagnose this.

Ondrej
--
Ondřej Surý <[email protected]> (He/Him)

Quoted 35 lines · Dmitry Vyukov via lttng-dev
Quoted 18 lines · Olivier Dion

Hi Dmitry,

We do have a new issue and we think it might be a limitation from TSAN.

Find attached a test program that we believe is correct. You can compile it with `gcc -fsanitize=thread test.c -pthread'.

TSAN flags a race condition between the atomic store relaxed at line 36 and the free at line 81.

We can solve the issue by replacing the atomic store relaxed with a atomic store release. However, the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store, making the release semantic redundant.

Quoted 7 lines

We've found an alternative to fix this by ignoring the thread during the relaxed store (line 39). However, what we would like is to annotate the memory stored (line 45).

My theory (I don't know the internal of TSAN much) is that TSAN thinks for some reason that the atomic store relaxed happen at the same epoch as the free, resulting in a false positive. If so, m

I don't this this is true in the C/C++ memory model:
"the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store".

Quoted 7 lines

std::atomic_thread_fence does affect all preceding/subsequent operations, but an atomic memory operation only affects ordering on that variable, it doesn't also serve as a standalone memory fence.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#37 lore raw
Quoted 6 lines · Ondřej Surý

A little bit related question/problem - some of our system test jobs got significantly slower (3x-5x) with **clang-16** TSAN after we integrated liburcu to BIND 9. The GCC (13.1.1) TSAN doesn’t exhibit this behavior.

The liburcu integration in BIND 9 is very light at the moment - we use liburcu for qptrie and there’s some usage of wfcqueue for queuing jobs(callbacks) between threads.

We were not able to put a finger on it yet, but saying this aloud might help diagnose this.

Interesting. Clang 16 got a completely new tsan runtime implementation, which is generally ~2x faster and consumes less memory.
However, of course, it's not possible to optimize for all gazillion of programs in the world. I suspect librcu tests are quite extreme in terms of synchronization.

If you collect perf profiles for both, I can look for some low hanging fruit. Or is there a reasonable way for me to reproduce?

Quoted 47 lines

Ondrej
--
Ondřej Surý <[email protected]> (He/Him)

Quoted 42 lines · Dmitry Vyukov via lttng-dev
Quoted 18 lines · Olivier Dion

Hi Dmitry,

We do have a new issue and we think it might be a limitation from TSAN.

Find attached a test program that we believe is correct. You can compile it with `gcc -fsanitize=thread test.c -pthread'.

TSAN flags a race condition between the atomic store relaxed at line 36 and the free at line 81.

We can solve the issue by replacing the atomic store relaxed with a atomic store release. However, the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store, making the release semantic redundant.

Quoted 7 lines

We've found an alternative to fix this by ignoring the thread during the relaxed store (line 39). However, what we would like is to annotate the memory stored (line 45).

My theory (I don't know the internal of TSAN much) is that TSAN thinks for some reason that the atomic store relaxed happen at the same epoch as the free, resulting in a false positive. If so, m

I don't this this is true in the C/C++ memory model:
"the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store".

std::atomic_thread_fence does affect all preceding/subsequent operations, but an atomic memory operation only affects ordering on that variable, it doesn't also serve as a standalone memory fence.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#38 lore raw
Quoted 13 lines · Dmitry Vyukov
Quoted 5 lines

A little bit related question/problem - some of our system test jobs got significantly slower (3x-5x) with **clang-16** TSAN after we integrated liburcu to BIND 9. The GCC (13.1.1) TSAN doesn’t exhibit this behavior.

The liburcu integration in BIND 9 is very light at the moment - we use liburcu for qptrie and there’s some usage of wfcqueue for queuing jobs(callbacks) between threads.

We were not able to put a finger on it yet, but saying this aloud might help diagnose this.

Interesting. Clang 16 got a completely new tsan runtime implementation, which is generally ~2x faster and consumes less memory.
However, of course, it's not possible to optimize for all gazillion of programs in the world. I suspect librcu tests are quite extreme in terms of synchronization.

Quoted 2 lines

If you collect perf profiles for both, I can look for some low hanging fruit. Or is there a reasonable way for me to reproduce?

How many CPUs do these machines have?

Quoted 39 lines
Quoted 39 lines

Ondrej
--
Ondřej Surý <[email protected]> (He/Him)

Quoted 35 lines · Dmitry Vyukov via lttng-dev
Quoted 18 lines · Olivier Dion

Hi Dmitry,

We do have a new issue and we think it might be a limitation from TSAN.

Find attached a test program that we believe is correct. You can compile it with `gcc -fsanitize=thread test.c -pthread'.

TSAN flags a race condition between the atomic store relaxed at line 36 and the free at line 81.

We can solve the issue by replacing the atomic store relaxed with a atomic store release. However, the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store, making the release semantic redundant.

Quoted 7 lines

We've found an alternative to fix this by ignoring the thread during the relaxed store (line 39). However, what we would like is to annotate the memory stored (line 45).

My theory (I don't know the internal of TSAN much) is that TSAN thinks for some reason that the atomic store relaxed happen at the same epoch as the free, resulting in a false positive. If so, m

I don't this this is true in the C/C++ memory model:
"the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store".

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

std::atomic_thread_fence does affect all preceding/subsequent operations, but an atomic memory operation only affects ordering on that variable, it doesn't also serve as a standalone memory fence.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#39 lore raw
In reply to Dmitry Vyukov via lttng-dev
Quoted 18 lines · Dmitry Vyukov
Quoted 6 lines · Ondřej Surý

A little bit related question/problem - some of our system test jobs got significantly slower (3x-5x) with **clang-16** TSAN after we integrated liburcu to BIND 9. The GCC (13.1.1) TSAN doesn’t exhibit this behavior.

The liburcu integration in BIND 9 is very light at the moment - we use liburcu for qptrie and there’s some usage of wfcqueue for queuing jobs(callbacks) between threads.

We were not able to put a finger on it yet, but saying this aloud might help diagnose this.

Interesting. Clang 16 got a completely new tsan runtime implementation, which is generally ~2x faster and consumes less memory.
However, of course, it's not possible to optimize for all gazillion of programs in the world. I suspect librcu tests are quite extreme in terms of synchronization.

Quoted 2 lines

If you collect perf profiles for both, I can look for some low hanging fruit. Or is there a reasonable way for me to reproduce?

Looks like it got all resolved by using the latest liburcu patch set with full TSAN support. I haven't realized that our CI still runs with whatever liburcu Debian bullseye have. Sorry for the confusion.

And I can no longer reproduce this locally.

FTR here are some numbers from my local system (13th Gen Intel(R) Core(TM) i9-13900K) running TSAN-enabled patched userspace-rcu[1], libuv 1.45.0 and OpenSSL 3.0.8:

gcc-13 baseline

rpz:            passed in 198.71s (0:03:18)
rpzrecurse:     passed in 310.62s (0:05:10)
rrsetorder:     passed in 200.40s (0:03:20)

gcc-13 TSAN

rpz:            passed in 305.64s (0:05:05)
rpzrecurse:     passed in 427.73s (0:07:07)
rrsetorder:     passed in 260.65s (0:04:20)

clang-16 TSAN

rpz:            passed in 290.05s (0:04:50)
rpzrecurse:     passed in 468.53s (0:07:48)
rrsetorder:     passed in 258.97s (0:04:18)

This looks quite reasonable to me.

1. git fetch https://review.lttng.org/userspace-rcu refs/changes/58/10058/1 && git checkout FETCH_HEAD

Ondrej
--
Ondřej Surý (He/Him)
[email protected]

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#40 lore raw
Quoted 45 lines · Ondřej Surý
Quoted 19 lines · Dmitry Vyukov
Quoted 6 lines · Ondřej Surý

A little bit related question/problem - some of our system test jobs got significantly slower (3x-5x) with **clang-16** TSAN after we integrated liburcu to BIND 9. The GCC (13.1.1) TSAN doesn’t exhibit this behavior.

The liburcu integration in BIND 9 is very light at the moment - we use liburcu for qptrie and there’s some usage of wfcqueue for queuing jobs(callbacks) between threads.

We were not able to put a finger on it yet, but saying this aloud might help diagnose this.

Interesting. Clang 16 got a completely new tsan runtime implementation, which is generally ~2x faster and consumes less memory.
However, of course, it's not possible to optimize for all gazillion of programs in the world. I suspect librcu tests are quite extreme in terms of synchronization.

Quoted 2 lines

If you collect perf profiles for both, I can look for some low hanging fruit. Or is there a reasonable way for me to reproduce?

Looks like it got all resolved by using the latest liburcu patch set with full TSAN support. I haven't realized that our CI still runs with whatever liburcu Debian bullseye have. Sorry for the confusion.

And I can no longer reproduce this locally.

FTR here are some numbers from my local system (13th Gen Intel(R) Core(TM) i9-13900K) running TSAN-enabled patched userspace-rcu[1], libuv 1.45.0 and OpenSSL 3.0.8:

gcc-13 baseline

rpz:            passed in 198.71s (0:03:18)
rpzrecurse:     passed in 310.62s (0:05:10)
rrsetorder:     passed in 200.40s (0:03:20)

gcc-13 TSAN

rpz:            passed in 305.64s (0:05:05)
rpzrecurse:     passed in 427.73s (0:07:07)
rrsetorder:     passed in 260.65s (0:04:20)

clang-16 TSAN

Quoted 7 lines
rpz:            passed in 290.05s (0:04:50)
rpzrecurse:     passed in 468.53s (0:07:48)
rrsetorder:     passed in 258.97s (0:04:18)

This looks quite reasonable to me.

1. git fetch https://review.lttng.org/userspace-rcu refs/changes/58/10058/1 && git checkout FETCH_HEAD

Yes, looks reasonable.
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#41 lore raw
In reply to Ondřej Surý via lttng-dev
Quoted 3 lines · Ondřej Surý

Looks like it got all resolved by using the latest liburcu patch set with full TSAN support. I haven't realized that our CI still runs with whatever liburcu

Just pointing out that we've add support for wfcqueue which seems to be quite used in BIND. New patches are on the way for supporting other data structures as well in the same vein :-)

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#42 lore raw
In reply to Dmitry Vyukov via lttng-dev
Quoted 10 lines · Dmitry Vyukov

On Tue, 23 May 2023 at 18:05, Olivier Dion <[email protected]> wrote:

I don't this this is true in the C/C++ memory model:
"the preceding atomic exchange with sequential consistency already acts as an implicit release (in term of memory barrier) for the following store".

std::atomic_thread_fence does affect all preceding/subsequent operations, but an atomic memory operation only affects ordering on that variable, it doesn't also serve as a standalone memory fence.

After reading the standard, we concur with you. We had to revisit the memory model used by URCU to understand the conflict. While doing so, we had to adapt some of the algorithms that were assuming implicit full memory barriers with operation such as CAS.

Thank for the insight!

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#43 lore raw
In reply to Dmitry Vyukov via lttng-dev
Quoted 4 lines · Dmitry Vyukov

Forking under tsan is a bit tricky. But I see we now take a number of internal mutexes around fork (but I think still not all of them), so maybe it's not that bad. If TSAN_OPTIONS=die_after_fork=0 works reasonably reliably for you, then export it for testing.

Works like a charm! Thanks for that!

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#44 lore raw

v2 cover Add support for TSAN to liburcu

In reply to Olivier Dion via lttng-dev

This patch set adds support for TSAN in liburcu.

* Change since v1

** Adding CMM_SEQ_CST_FENCE memory order to the CMM memory model

The C11 memory model is incompatible with the memory model used by liburcu, since the semantic of the C11 memory model is based on a happen-before (acquire/release) relationship of memory accesses, while liburcu is based on memory barriers and relaxed memory accesses.

To circumvent this, a new memory order called CMM_SEQ_CST_FENCE is introduced. It implies a CMM_SEQ_CST while emitting a thread fence after the operation. Operations that were documented as emitting memory barriers before and after the operation are implemented now in term of this new memory order to keep compatibility.

However, this model is redundant in some cases and the memory orders were changed internally in liburcu to simply use the CMM_SEQ_CST.

** Adding cmm_emit_legacy_smp_mb() to queue/stack APIs

The queue/stack APIs document implicit memory barriers before or after operation. These are now optionally emitted only if CONFIG_RCU_EMIT_LEGACY_MB is defined in urcu/config.h or manually defined by the user before including liburcu. That way, users can opt-in even if the system headers were configured without this feature.

However, users can not opt-out of that feature if configured by the system.

* v1

** Here are the major changes

- Usage of compiler atomic builtins is added to the uatomic API. This is required for TSAN to understand atomic memory accesses. If the compiler supports such builtins, they are used by default. User can opt-out and use the legacy implementation of the uatomic API by using the `--disable-atomic-builtins' configuration option.

- The CMM memory model is introduced but yet formalized. It tries to be as close as possible to the C11 memory model while offering primitives such as cmm_smp_wmb(), cmm_smp_rmb() and cmm_mb() that can't be expressed in it. For example, cmm_mb() can be used for ordering memory accesses to MMIO devices, which is out of the scope of the C11 memory model.

- The CMM annotation layer is a new public API that is highly experimental and not guaranteed to be stable at this stage. It serves the dual purpose of verifying local (intra-thread) relaxed atomic accesses ordering with a memory barrier and global (inter-thread) relaxed atomic accesses with a shared state. The second purpose is necessary for TSAN to understand memory accesses ordering since it does not fully support thread fence yet.

** CMM annotation example

Consider the following pseudo-code of writer side in synchronize_rcu(). An acquire group is defined on the stack of the writer. Annotations are made onto the group to ensure ordering of relaxed memory accesses in reader_state() before the memory barrier at the end of synchronize_rcu(). It also helps TSAN to understand that the relaxed accesses in reader_state() act like acquire accesses because of the memory barrier in synchronize_rcu().

In other words, the purpose of this annotation is to convert a group of load-acquire memory operations into load-relaxed memory operations followed by a single memory barrier. This highly benefits weakly ordered architectures by having a constant number of memory barriers instead of being linearly proportional to the number of loads. This does not benefit TSO architectures.

Olivier Dion (12):
configure: Add --disable-atomic-builtins option urcu/compiler: Use atomic builtins if configured urcu/arch/generic: Use atomic builtins if configured urcu/system: Use atomic builtins if configured urcu/uatomic: Add CMM memory model
urcu-wait: Fix wait state load/store
tests: Use uatomic for accessing global states
benchmark: Use uatomic for accessing global states tests/unit/test_build: Quiet unused return value urcu/annotate: Add CMM annotation
Add cmm_emit_legacy_smp_mb()
tests: Add tests for checking race conditions

 README.md                               |  11 ++
 configure.ac                            |  39 ++++
 doc/uatomic-api.md                      |   3 +-
 include/Makefile.am                     |   3 +
 include/urcu/annotate.h                 | 174 ++++++++++++++++++
 include/urcu/arch.h                     |   6 +
 include/urcu/arch/generic.h             |  37 ++++
 include/urcu/compiler.h                 |  22 ++-
 include/urcu/config.h.in                |   6 +
 include/urcu/static/lfstack.h           |  25 ++-
 include/urcu/static/pointer.h           |  40 ++--
 include/urcu/static/rculfqueue.h        |  14 +-
 include/urcu/static/rculfstack.h        |   8 +-
 include/urcu/static/urcu-bp.h           |  12 +-
 include/urcu/static/urcu-common.h       |   8 +-
 include/urcu/static/urcu-mb.h           |  11 +-
 include/urcu/static/urcu-memb.h         |  26 ++-
 include/urcu/static/urcu-qsbr.h         |  29 ++-
 include/urcu/static/wfcqueue.h          |  68 +++----
 include/urcu/static/wfqueue.h           |   9 +-
 include/urcu/static/wfstack.h           |  24 ++-
 include/urcu/system.h                   |  21 +++
 include/urcu/uatomic.h                  |  63 ++++++-
 include/urcu/uatomic/builtins-generic.h | 170 +++++++++++++++++
 include/urcu/uatomic/builtins.h         |  79 ++++++++
 include/urcu/uatomic/generic.h          | 234 ++++++++++++++++++++++++
 src/rculfhash.c                         |  92 ++++++----
 src/urcu-bp.c                           |  17 +-
 src/urcu-pointer.c                      |   9 +-
 src/urcu-qsbr.c                         |  31 +++-
 src/urcu-wait.h                         |  15 +-
 src/urcu.c                              |  24 ++-
 tests/benchmark/Makefile.am             |  91 ++++-----
 tests/benchmark/common-states.c         |   1 +
 tests/benchmark/common-states.h         |  51 ++++++
 tests/benchmark/test_mutex.c            |  32 +---
 tests/benchmark/test_perthreadlock.c    |  32 +---
 tests/benchmark/test_rwlock.c           |  32 +---
 tests/benchmark/test_urcu.c             |  33 +---
 tests/benchmark/test_urcu_assign.c      |  33 +---
 tests/benchmark/test_urcu_bp.c          |  33 +---
 tests/benchmark/test_urcu_defer.c       |  33 +---
 tests/benchmark/test_urcu_gc.c          |  34 +---
 tests/benchmark/test_urcu_hash.c        |   6 +-
 tests/benchmark/test_urcu_hash.h        |  15 --
 tests/benchmark/test_urcu_hash_rw.c     |  10 +-
 tests/benchmark/test_urcu_hash_unique.c |  10 +-
 tests/benchmark/test_urcu_lfq.c         |  20 +-
 tests/benchmark/test_urcu_lfs.c         |  20 +-
 tests/benchmark/test_urcu_lfs_rcu.c     |  20 +-
 tests/benchmark/test_urcu_qsbr.c        |  33 +---
 tests/benchmark/test_urcu_qsbr_gc.c     |  34 +---
 tests/benchmark/test_urcu_wfcq.c        |  22 +--
 tests/benchmark/test_urcu_wfq.c         |  20 +-
 tests/benchmark/test_urcu_wfs.c         |  22 +--
 tests/common/api.h                      |  12 +-
 tests/regression/rcutorture.h           | 106 +++++++----
 tests/unit/test_build.c                 |   8 +-
 tests/unit/test_lfstack.c               |  90 +++++++++
 tests/unit/test_wfcqueue.c              | 119 ++++++++++++
 tests/unit/test_wfqueue.c               |  91 +++++++++
 tests/unit/test_wfstack.c               |  90 +++++++++
 62 files changed, 1799 insertions(+), 684 deletions(-)
 create mode 100644 include/urcu/annotate.h
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins.h
 create mode 100644 tests/benchmark/common-states.c
 create mode 100644 tests/benchmark/common-states.h
 create mode 100644 tests/unit/test_lfstack.c
 create mode 100644 tests/unit/test_wfcqueue.c
 create mode 100644 tests/unit/test_wfqueue.c
 create mode 100644 tests/unit/test_wfstack.c
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#45 lore raw

Olivier,

is this somewhere in Gerrit again, please? It’s much easier for us to pull the changes from git than cherry-pick patches from the mailing list.

Ondřej
--
Ondřej Surý <[email protected]> (He/Him)

Quoted 149 lines · Olivier Dion

This patch set adds support for TSAN in liburcu.

* Change since v1

** Adding CMM_SEQ_CST_FENCE memory order to the CMM memory model

The C11 memory model is incompatible with the memory model used by liburcu, since the semantic of the C11 memory model is based on a happen-before (acquire/release) relationship of memory accesses, while liburcu is based on memory barriers and relaxed memory accesses.

To circumvent this, a new memory order called CMM_SEQ_CST_FENCE is introduced. It implies a CMM_SEQ_CST while emitting a thread fence after the operation. Operations that were documented as emitting memory barriers before and after the operation are implemented now in term of this new memory order to keep compatibility.

However, this model is redundant in some cases and the memory orders were changed internally in liburcu to simply use the CMM_SEQ_CST.

** Adding cmm_emit_legacy_smp_mb() to queue/stack APIs

The queue/stack APIs document implicit memory barriers before or after operation. These are now optionally emitted only if CONFIG_RCU_EMIT_LEGACY_MB is defined in urcu/config.h or manually defined by the user before including liburcu. That way, users can opt-in even if the system headers were configured without this feature.

However, users can not opt-out of that feature if configured by the system.

* v1

** Here are the major changes

- Usage of compiler atomic builtins is added to the uatomic API. This is required for TSAN to understand atomic memory accesses. If the compiler supports such builtins, they are used by default. User can opt-out and use the legacy implementation of the uatomic API by using the `--disable-atomic-builtins' configuration option.

- The CMM memory model is introduced but yet formalized. It tries to be as close as possible to the C11 memory model while offering primitives such as cmm_smp_wmb(), cmm_smp_rmb() and cmm_mb() that can't be expressed in it. For example, cmm_mb() can be used for ordering memory accesses to MMIO devices, which is out of the scope of the C11 memory model.

- The CMM annotation layer is a new public API that is highly experimental and not guaranteed to be stable at this stage. It serves the dual purpose of verifying local (intra-thread) relaxed atomic accesses ordering with a memory barrier and global (inter-thread) relaxed atomic accesses with a shared state. The second purpose is necessary for TSAN to understand memory accesses ordering since it does not fully support thread fence yet.

** CMM annotation example

Consider the following pseudo-code of writer side in synchronize_rcu(). An acquire group is defined on the stack of the writer. Annotations are made onto the group to ensure ordering of relaxed memory accesses in reader_state() before the memory barrier at the end of synchronize_rcu(). It also helps TSAN to understand that the relaxed accesses in reader_state() act like acquire accesses because of the memory barrier in synchronize_rcu().

In other words, the purpose of this annotation is to convert a group of load-acquire memory operations into load-relaxed memory operations followed by a single memory barrier. This highly benefits weakly ordered architectures by having a constant number of memory barriers instead of being linearly proportional to the number of loads. This does not benefit TSO architectures.

Olivier Dion (12):
configure: Add --disable-atomic-builtins option urcu/compiler: Use atomic builtins if configured urcu/arch/generic: Use atomic builtins if configured urcu/system: Use atomic builtins if configured urcu/uatomic: Add CMM memory model
urcu-wait: Fix wait state load/store
tests: Use uatomic for accessing global states
benchmark: Use uatomic for accessing global states tests/unit/test_build: Quiet unused return value urcu/annotate: Add CMM annotation
Add cmm_emit_legacy_smp_mb()
tests: Add tests for checking race conditions

README.md                               |  11 ++
configure.ac                            |  39 ++++
doc/uatomic-api.md                      |   3 +-
include/Makefile.am                     |   3 +
include/urcu/annotate.h                 | 174 ++++++++++++++++++
include/urcu/arch.h                     |   6 +
include/urcu/arch/generic.h             |  37 ++++
include/urcu/compiler.h                 |  22 ++-
include/urcu/config.h.in                |   6 +
include/urcu/static/lfstack.h           |  25 ++-
include/urcu/static/pointer.h           |  40 ++--
include/urcu/static/rculfqueue.h        |  14 +-
include/urcu/static/rculfstack.h        |   8 +-
include/urcu/static/urcu-bp.h           |  12 +-
include/urcu/static/urcu-common.h       |   8 +-
include/urcu/static/urcu-mb.h           |  11 +-
include/urcu/static/urcu-memb.h         |  26 ++-
include/urcu/static/urcu-qsbr.h         |  29 ++-
include/urcu/static/wfcqueue.h          |  68 +++----
include/urcu/static/wfqueue.h           |   9 +-
include/urcu/static/wfstack.h           |  24 ++-
include/urcu/system.h                   |  21 +++
include/urcu/uatomic.h                  |  63 ++++++-
include/urcu/uatomic/builtins-generic.h | 170 +++++++++++++++++
include/urcu/uatomic/builtins.h         |  79 ++++++++
include/urcu/uatomic/generic.h          | 234 ++++++++++++++++++++++++
src/rculfhash.c                         |  92 ++++++----
src/urcu-bp.c                           |  17 +-
src/urcu-pointer.c                      |   9 +-
src/urcu-qsbr.c                         |  31 +++-
src/urcu-wait.h                         |  15 +-
src/urcu.c                              |  24 ++-
tests/benchmark/Makefile.am             |  91 ++++-----
tests/benchmark/common-states.c         |   1 +
tests/benchmark/common-states.h         |  51 ++++++
tests/benchmark/test_mutex.c            |  32 +---
tests/benchmark/test_perthreadlock.c    |  32 +---
tests/benchmark/test_rwlock.c           |  32 +---
tests/benchmark/test_urcu.c             |  33 +---
tests/benchmark/test_urcu_assign.c      |  33 +---
tests/benchmark/test_urcu_bp.c          |  33 +---
tests/benchmark/test_urcu_defer.c       |  33 +---
tests/benchmark/test_urcu_gc.c          |  34 +---
tests/benchmark/test_urcu_hash.c        |   6 +-
tests/benchmark/test_urcu_hash.h        |  15 --
tests/benchmark/test_urcu_hash_rw.c     |  10 +-
tests/benchmark/test_urcu_hash_unique.c |  10 +-
tests/benchmark/test_urcu_lfq.c         |  20 +-
tests/benchmark/test_urcu_lfs.c         |  20 +-
tests/benchmark/test_urcu_lfs_rcu.c     |  20 +-
tests/benchmark/test_urcu_qsbr.c        |  33 +---
tests/benchmark/test_urcu_qsbr_gc.c     |  34 +---
tests/benchmark/test_urcu_wfcq.c        |  22 +--
tests/benchmark/test_urcu_wfq.c         |  20 +-
tests/benchmark/test_urcu_wfs.c         |  22 +--
tests/common/api.h                      |  12 +-
tests/regression/rcutorture.h           | 106 +++++++----
tests/unit/test_build.c                 |   8 +-
tests/unit/test_lfstack.c               |  90 +++++++++
tests/unit/test_wfcqueue.c              | 119 ++++++++++++
tests/unit/test_wfqueue.c               |  91 +++++++++
tests/unit/test_wfstack.c               |  90 +++++++++
62 files changed, 1799 insertions(+), 684 deletions(-)
Quoted 13 lines

create mode 100644 include/urcu/annotate.h
create mode 100644 include/urcu/uatomic/builtins-generic.h create mode 100644 include/urcu/uatomic/builtins.h create mode 100644 tests/benchmark/common-states.c create mode 100644 tests/benchmark/common-states.h create mode 100644 tests/unit/test_lfstack.c create mode 100644 tests/unit/test_wfcqueue.c create mode 100644 tests/unit/test_wfqueue.c create mode 100644 tests/unit/test_wfstack.c

Signature · 1 line
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#46 lore raw
Quoted 5 lines · Ondřej Surý

Olivier,

is this somewhere in Gerrit again, please? It’s much easier for us to pull the changes from git than cherry-pick patches from the mailing list.

Here https://review.lttng.org/c/userspace-rcu/+/10118/6.

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#47 lore raw

PATCH v2 1/12 configure: Add --disable-atomic-builtins option

In reply to Olivier Dion via lttng-dev

By default, if the toolchain supports atomic builtins, use them for the uatomic API. This requires that the toolchains used to compile the library and the user application supports such builtins.

The advantage of using these builtins is that they are well known synchronization primitives by several tools such as TSAN.

Change-Id Ia8e97112681f744f17816dbc4cbbec805a483331
---
 README.md                | 11 +++++++++++
 configure.ac             | 26 ++++++++++++++++++++++++++
 include/urcu/config.h.in |  3 +++
 3 files changed, 40 insertions(+)
README.md +11 -0
index ba5bb08..6ce96c9 100644
@@ -429,6 +429,17 @@ still being used to iterate on a hash table.
 This option alters the rculfhash ABI. Make sure to compile both library
 and application with matching configuration.
 
+### Usage of `--disable-atomic-builtins`
+
+By default, the configure script will check if the toolchain supports atomic
+builtins. If so, then the RCU memory model is implemented using the atomic
+builtins of the toolchain.
+
+Building liburcu with `--disable-atomic-builtins` force to use the legacy
+internal implementations for atomic accesses.
+
+This option is useful if for example the atomic builtins for a given toolchain
+version is known to be broken or to be inefficient.
 
 Make targets
 ------------
configure.ac +26 -0
index 909cf1d..4450a31 100644
@@ -230,6 +230,11 @@ AE_FEATURE([rcu-debug], [Enable internal debugging self-checks. Introduces a per
 AE_FEATURE_DEFAULT_DISABLE
 AE_FEATURE([cds-lfht-iter-debug], [Enable extra debugging checks for lock-free hash table iterator traversal. Alters the rculfhash ABI. Make sure to compile both library and application with matching configuration.])
 
+# toolchain atomic builtins
+# Enabled by default
+AE_FEATURE_DEFAULT_ENABLE
+AE_FEATURE([atomic-builtins], [Disable the usage of toolchain atomic builtins.])
+
 # When given, add -Werror to WARN_CFLAGS and WARN_CXXFLAGS.
 # Disabled by default
 AE_FEATURE_DEFAULT_DISABLE
@@ -259,6 +264,23 @@ AE_IF_FEATURE_ENABLED([cds-lfht-iter-debug], [
   AC_DEFINE([CONFIG_CDS_LFHT_ITER_DEBUG], [1], [Enable extra debugging checks for lock-free hash table iterator traversal. Alters the rculfhash ABI. Make sure to compile both library and application with matching configuration.])
 ])
 
+AE_IF_FEATURE_ENABLED([atomic-builtins], [
+  AC_COMPILE_IFELSE(
+	[AC_LANG_PROGRAM(
+		[[int x, y;]],
+		[[__atomic_store_n(&x, 0, __ATOMIC_RELAXED);
+		  __atomic_load_n(&x, __ATOMIC_RELAXED);
+		  y = __atomic_exchange_n(&x, 1, __ATOMIC_RELAXED);
+		  __atomic_compare_exchange_n(&x, &y, 0, 0, __ATOMIC_RELAXED, __ATOMIC_RELAXED);
+		  __atomic_add_fetch(&x, 1, __ATOMIC_RELAXED);
+		  __atomic_sub_fetch(&x, 1, __ATOMIC_RELAXED);
+		  __atomic_and_fetch(&x, 0x01, __ATOMIC_RELAXED);
+		  __atomic_or_fetch(&x, 0x01, __ATOMIC_RELAXED);
+		  __atomic_thread_fence(__ATOMIC_RELAXED);
+		  __atomic_signal_fence(__ATOMIC_RELAXED);]])],
+	[AC_DEFINE([CONFIG_RCU_USE_ATOMIC_BUILTINS], [1], [Use compiler atomic builtins.])],
+	[AE_FEATURE_DISABLE(atomic-builtins)])
+])
 
 ##                                                                          ##
 ## Set automake variables for optional feature conditionnals in Makefile.am ##
@@ -361,6 +383,10 @@ PPRINT_PROP_BOOL([Internal debugging], $value)
 AE_IS_FEATURE_ENABLED([cds-lfht-iter-debug]) && value=1 || value=0
 PPRINT_PROP_BOOL([Lock-free HT iterator debugging], $value)
 
+# atomic builtins enabled/disabled
+AE_IS_FEATURE_ENABLED([atomic-builtins]) && value=1 || value=0
+PPRINT_PROP_BOOL([Use toolchain atomic builtins], $value)
+
 PPRINT_PROP_BOOL([Multi-flavor support], 1)
 
 report_bindir="`eval eval echo $bindir`"
include/urcu/config.h.in +3 -0
index 99d763a..1daaa7e 100644
@@ -19,6 +19,9 @@
    Introduces a performance penalty. */
 #undef CONFIG_RCU_DEBUG
 
+/* Uatomic API uses atomic builtins? */
+#undef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
 /* Expose multi-flavor support */
 #define CONFIG_RCU_HAVE_MULTIFLAVOR 1
 
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#48 lore raw

PATCH v2 2/12 urcu/compiler: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

Use __atomic_signal_fence(__ATOMIC_SEQ_CST) for cmm_barrier() if configured to use atomic builtins.

Change-Id Ib168b50f1e97a8da861b92d6882c56db230ebb2c
---
 include/urcu/compiler.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
include/urcu/compiler.h +7 -1
index 2f32b38..3604488 100644
@@ -25,10 +25,16 @@
 # include <type_traits>	/* for std::remove_cv */
 #endif
 
+#include <urcu/config.h>
+
 #define caa_likely(x)	__builtin_expect(!!(x), 1)
 #define caa_unlikely(x)	__builtin_expect(!!(x), 0)
 
-#define	cmm_barrier()	__asm__ __volatile__ ("" : : : "memory")
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+#  define cmm_barrier() __atomic_signal_fence(__ATOMIC_SEQ_CST)
+#else
+#  define cmm_barrier() asm volatile ("" : : : "memory")
+#endif
 
 /*
  * Instruct the compiler to perform only a single access to a variable
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#49 lore raw

PATCH v2 3/12 urcu/arch/generic: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

If configured to use atomic builtins, implement SMP memory barriers in term of atomic builtins if the architecture does not implement its own version.

Change-Id Iddc4283606e0fce572e104d2d3f03b5c0d9926fb
---
 include/urcu/arch/generic.h | 8 ++++++++
 1 file changed, 8 insertions(+)
include/urcu/arch/generic.h +8 -0
index be6e41e..e292c70 100644
@@ -43,6 +43,14 @@ extern "C" {
  * GCC builtins) as well as cmm_rmb and cmm_wmb (defaulting to cmm_mb).
  */
 
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+# ifndef cmm_smp_mb
+#  define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
+# endif
+
+#endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
+
 #ifndef cmm_mb
 #define cmm_mb()    __sync_synchronize()
 #endif
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#50 lore raw

PATCH v2 4/12 urcu/system: Use atomic builtins if configured

In reply to Olivier Dion via lttng-dev

If configured to use atomic builtins, use them for implementing the CMM_LOAD_SHARED and CMM_STORE_SHARED macros.

Change-Id I3eaaaaf0d26c47aced6e94b40fd59c7b8baa6272
---
 include/urcu/system.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
include/urcu/system.h +21 -0
index faae390..f184aad 100644
@@ -19,9 +19,28 @@
  * all copies or substantial portions of the Software.
  */
 
+#include <urcu/config.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
 
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+#define CMM_LOAD_SHARED(x)			\
+	__atomic_load_n(&(x), __ATOMIC_RELAXED)
+
+#define _CMM_LOAD_SHARED(x) CMM_LOAD_SHARED(x)
+
+#define CMM_STORE_SHARED(x, v)					\
+	__extension__						\
+	({							\
+		__typeof__(v) _v = (v);				\
+		__atomic_store_n(&(x), _v, __ATOMIC_RELAXED);	\
+		_v;						\
+	})
+
+#define _CMM_STORE_SHARED(x, v) CMM_STORE_SHARED(x, v)
+
+#else
 /*
  * Identify a shared load. A cmm_smp_rmc() or cmm_smp_mc() should come
  * before the load.
@@ -56,4 +75,6 @@
 		_v = _v;	/* Work around clang "unused result" */	\
 	})
 
+#endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
+
 #endif /* _URCU_SYSTEM_H */
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#51 lore raw

Re: [lttng-dev] [PATCH v2 04/12] urcu/system: Use atomic builtins if configured

Quoted 28 lines · Olivier Dion

If configured to use atomic builtins, use them for implementing the CMM_LOAD_SHARED and CMM_STORE_SHARED macros.

Change-Id I3eaaaaf0d26c47aced6e94b40fd59c7b8baa6272
---
 include/urcu/system.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
include/urcu/system.h +7 -0
index faae390..f184aad 100644
@@ -19,9 +19,28 @@
  * all copies or substantial portions of the Software.
  */
 
+#include <urcu/config.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
 
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+#define CMM_LOAD_SHARED(x)			\
+	__atomic_load_n(&(x), __ATOMIC_RELAXED)
+
+#define _CMM_LOAD_SHARED(x) CMM_LOAD_SHARED(x)

include/urcu/system.h line 41

Quoted 10 lines
include/urcu/system.h +10 -0
+
+#define CMM_STORE_SHARED(x, v)					\
+	__extension__						\
+	({							\
+		__typeof__(v) _v = (v);				\
+		__atomic_store_n(&(x), _v, __ATOMIC_RELAXED);	\
+		_v;						\
+	})
+
+#define _CMM_STORE_SHARED(x, v) CMM_STORE_SHARED(x, v)

Same question here on loss of volatile semantics.

Thanx, Paul

Quoted 15 lines
include/urcu/system.h +4 -0
+
+#else
 /*
  * Identify a shared load. A cmm_smp_rmc() or cmm_smp_mc() should come
  * before the load.
@@ -56,4 +75,6 @@
 		_v = _v;	/* Work around clang "unused result" */	\
 	})
 
+#endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
+
 #endif /* _URCU_SYSTEM_H */
Signature · 1 line
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#52 lore raw

Re: [lttng-dev] [PATCH v2 04/12] urcu/system: Use atomic builtins if configured

Quoted 4 lines · Paul E. McKenney

On Wed, Jun 07, 2023 at 02:53:51PM -0400, Olivier Dion wrote:

Same question here on loss of volatile semantics.

This apply to all review on volatile semantics. I added a cmm_cast_volatile() macro/template for C/C++ for adding the volatile qualifier to pointers passed to every atomic builtin calls.

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#53 lore raw

Re: [lttng-dev] [PATCH v2 04/12] urcu/system: Use atomic builtins if configured

Quoted 8 lines · Olivier Dion
Quoted 3 lines · Paul E. McKenney

On Wed, Jun 07, 2023 at 02:53:51PM -0400, Olivier Dion wrote:

Same question here on loss of volatile semantics.

This apply to all review on volatile semantics. I added a cmm_cast_volatile() macro/template for C/C++ for adding the volatile qualifier to pointers passed to every atomic builtin calls.

Sounds very good, thank you!

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#54 lore raw

Re: [lttng-dev] [PATCH v2 04/12] urcu/system: Use atomic builtins if configured

Quoted 11 lines · Paul E. McKenney
Quoted 8 lines · Olivier Dion
Quoted 3 lines · Paul E. McKenney

On Wed, Jun 07, 2023 at 02:53:51PM -0400, Olivier Dion wrote:

Same question here on loss of volatile semantics.

This apply to all review on volatile semantics. I added a cmm_cast_volatile() macro/template for C/C++ for adding the volatile qualifier to pointers passed to every atomic builtin calls.

Sounds very good, thank you!

Maybe a case of synchronicity here, but I just stumble upon this
<https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0062r1.html>
where you seem to express the same concerns :-)

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#55 lore raw

Re: [lttng-dev] [PATCH v2 04/12] urcu/system: Use atomic builtins if configured

Quoted 11 lines · Olivier Dion
Quoted 10 lines · Paul E. McKenney
Quoted 8 lines · Olivier Dion
Quoted 3 lines · Paul E. McKenney

On Wed, Jun 07, 2023 at 02:53:51PM -0400, Olivier Dion wrote:

Same question here on loss of volatile semantics.

This apply to all review on volatile semantics. I added a cmm_cast_volatile() macro/template for C/C++ for adding the volatile qualifier to pointers passed to every atomic builtin calls.

Quoted 5 lines
Quoted 1 line

Sounds very good, thank you!

Maybe a case of synchronicity here, but I just stumble upon this
<https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0062r1.html>
where you seem to express the same concerns :-)

Just for completeness, my response to Hans's concern about volatile is addressed by an empty memory-clobber asm, similar to barrier() in the Linux kernel.

But yes, this has seen significant discussion over the years. ;-)

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#56 lore raw

PATCH v2 5/12 urcu/uatomic: Add CMM memory model

In reply to Olivier Dion via lttng-dev

Introducing the CMM memory model with the following new primitives:

- uatomic_load(addr, memory_order)

- uatomic_store(addr, value, memory_order)
- uatomic_and_mo(addr, mask, memory_order)
- uatomic_or_mo(addr, mask, memory_order)
- uatomic_add_mo(addr, value, memory_order)
- uatomic_sub_mo(addr, value, memory_order)
- uatomic_inc_mo(addr, memory_order)
- uatomic_dec_mo(addr, memory_order)

- uatomic_add_return_mo(addr, value, memory_order)
- uatomic_sub_return_mo(addr, value, memory_order)

- uatomic_xchg_mo(addr, value, memory_order)

- uatomic_cmpxchg_mo(addr, old, new,
memory_order_success,
memory_order_failure)

The CMM memory model reflects the C11 memory model with an additional CMM_SEQ_CST_FENCE memory order. The memory order can be selected through the enum cmm_memorder.

* With Atomic Builtins

If configured with atomic builtins, the correspondence between the CMM memory model and the C11 memory model is a one to one at the exception of the CMM_SEQ_CST_FENCE memory order which implies the memory order CMM_SEQ_CST and a thread fence after the operation.

* Without Atomic Builtins

However, if not configured with atomic builtins, the following stipulate the memory model.

For load operations with uatomic_load(), the memory orders CMM_RELAXED, CMM_CONSUME, CMM_ACQUIRE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are allowed. A barrier may be inserted before and after the load from memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_CONSUME: Memory barrier after read
- CMM_ACQUIRE: Memory barrier after read
- CMM_SEQ_CST: Memory barriers before and after read
- CMM_SEQ_CST_FENCE: Memory barriers before and after read

For store operations with uatomic_store(), the memory orders CMM_RELAXED, CMM_RELEASE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are allowed. A barrier may be inserted before and after the store to memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_RELEASE: Memory barrier before operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

For load/store operations with uatomic_and_mo(), uatomic_or_mo(), uatomic_add_mo(), uatomic_sub_mo(), uatomic_inc_mo(), uatomic_dec_mo(), uatomic_add_return_mo() and uatomic_sub_return_mo(), all memory orders are allowed. A barrier may be inserted before and after the operation depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

For the exchange operation uatomic_xchg_mo(), any memory order is valid. A barrier may be inserted before and after the exchange to memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

For the compare exchange operation uatomic_cmpxchg_mo(), the success memory order can be anything while the failure memory order cannot be CMM_RELEASE nor CMM_ACQ_REL and cannot be stronger than the success memory order. A barrier may be inserted before and after the store to memory depending on the memory orders:

Success memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

Barriers after the operations are only emitted if the compare exchange succeed.

Failure memory order:
- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

Barriers after the operations are only emitted if the compare exchange failed. Barriers before the operation are never emitted by this memory order.

Change-Id I213ba19c84e82a63083f00143a3142ffbdab1d52
---
 doc/uatomic-api.md                      |   3 +-
 include/Makefile.am                     |   2 +
 include/urcu/static/pointer.h           |  40 ++--
 include/urcu/uatomic.h                  |  63 ++++++-
 include/urcu/uatomic/builtins-generic.h | 170 +++++++++++++++++
 include/urcu/uatomic/builtins.h         |  79 ++++++++
 include/urcu/uatomic/generic.h          | 234 ++++++++++++++++++++++++
 src/urcu-pointer.c                      |   9 +-
 8 files changed, 565 insertions(+), 35 deletions(-)
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins.h
doc/uatomic-api.md +2 -1
index 0962399..7341ee8 100644
@@ -52,7 +52,8 @@ An atomic read-modify-write operation that performs this
 sequence of operations atomically: check if `addr` contains `old`.
 If true, then replace the content of `addr` by `new`. Return the
 value previously contained by `addr`. This function implies a full
-memory barrier before and after the atomic operation.
+memory barrier before and after the atomic operation. The second memory
+barrier is only emitted if the operation succeeded.
 
 
 ```c
include/Makefile.am +2 -0
index ba1fe60..b20e56d 100644
@@ -63,6 +63,8 @@ nobase_include_HEADERS = \
 	urcu/uatomic/alpha.h \
 	urcu/uatomic_arch.h \
 	urcu/uatomic/arm.h \
+	urcu/uatomic/builtins.h \
+	urcu/uatomic/builtins-generic.h \
 	urcu/uatomic/gcc.h \
 	urcu/uatomic/generic.h \
 	urcu/uatomic.h \
include/urcu/static/pointer.h +13 -27
index 9e46a57..9da8657 100644
@@ -96,23 +96,8 @@ extern "C" {
  * -Wincompatible-pointer-types errors.  Using the statement expression
  * makes it an rvalue and gets rid of the const-ness.
  */
-#ifdef __URCU_DEREFERENCE_USE_ATOMIC_CONSUME
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(__extension__ ({				\
-					__typeof__(p) __attribute__((unused)) _________p0 = { 0 }; \
-					_________p0;					\
-				})) _________p1;					\
-				__atomic_load(&(p), &_________p1, __ATOMIC_CONSUME);	\
-				(_________p1);						\
-			})
-#else
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(p) _________p1 = CMM_LOAD_SHARED(p);		\
-				cmm_smp_read_barrier_depends();				\
-				(_________p1);						\
-			})
-#endif
-
+# define _rcu_dereference(p)			\
+	uatomic_load(&(p), CMM_CONSUME)
 /**
  * _rcu_cmpxchg_pointer - same as rcu_assign_pointer, but tests if the pointer
  * is as expected by "old". If succeeds, returns the previous pointer to the
@@ -131,8 +116,9 @@ extern "C" {
 	({								\
 		__typeof__(*p) _________pold = (old);			\
 		__typeof__(*p) _________pnew = (_new);			\
-		uatomic_cmpxchg(p, _________pold, _________pnew);	\
-	})
+		uatomic_cmpxchg_mo(p, _________pold, _________pnew,	\
+				   CMM_SEQ_CST, CMM_SEQ_CST);		\
+	});
 
 /**
  * _rcu_xchg_pointer - same as rcu_assign_pointer, but returns the previous
@@ -149,17 +135,17 @@ extern "C" {
 	__extension__					\
 	({						\
 		__typeof__(*p) _________pv = (v);	\
-		uatomic_xchg(p, _________pv);		\
+		uatomic_xchg_mo(p, _________pv,		\
+				CMM_SEQ_CST);		\
 	})
 
 
-#define _rcu_set_pointer(p, v)				\
-	do {						\
-		__typeof__(*p) _________pv = (v);	\
-		if (!__builtin_constant_p(v) || 	\
-		    ((v) != NULL))			\
-			cmm_wmb();				\
-		uatomic_set(p, _________pv);		\
+#define _rcu_set_pointer(p, v)						\
+	do {								\
+		__typeof__(*p) _________pv = (v);			\
+		uatomic_store(p, _________pv,				\
+			__builtin_constant_p(v) && (v) == NULL ?	\
+			CMM_RELAXED : CMM_RELEASE);			\
 	} while (0)
 
 /**
include/urcu/uatomic.h +62 -1

71 lines not on this page · raw message

include/urcu/uatomic/builtins-generic.h new file +170 -0

170 lines not on this page · raw message

include/urcu/uatomic/builtins.h new file +79 -0

79 lines not on this page · raw message

include/urcu/uatomic/generic.h +234 -0

244 lines not on this page · raw message

src/urcu-pointer.c +3 -6
index d0854ac..cea8aeb 100644
@@ -39,19 +39,16 @@ void *rcu_dereference_sym(void *p)
 
 void *rcu_set_pointer_sym(void **p, void *v)
 {
-	cmm_wmb();
-	uatomic_set(p, v);
+	uatomic_store(p, v, CMM_RELEASE);
 	return v;
 }
 
 void *rcu_xchg_pointer_sym(void **p, void *v)
 {
-	cmm_wmb();
-	return uatomic_xchg(p, v);
+	return uatomic_xchg_mo(p, v, CMM_SEQ_CST);
 }
 
 void *rcu_cmpxchg_pointer_sym(void **p, void *old, void *_new)
 {
-	cmm_wmb();
-	return uatomic_cmpxchg(p, old, _new);
+	return uatomic_cmpxchg_mo(p, old, _new, CMM_SEQ_CST, CMM_SEQ_CST);
 }
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#57 lore raw

Re: [lttng-dev] [PATCH v2 05/12] urcu/uatomic: Add CMM memory model

Quoted 162 lines · Olivier Dion

Introducing the CMM memory model with the following new primitives:

- uatomic_load(addr, memory_order)

- uatomic_store(addr, value, memory_order)
- uatomic_and_mo(addr, mask, memory_order)
- uatomic_or_mo(addr, mask, memory_order)
- uatomic_add_mo(addr, value, memory_order)
- uatomic_sub_mo(addr, value, memory_order)
- uatomic_inc_mo(addr, memory_order)
- uatomic_dec_mo(addr, memory_order)

- uatomic_add_return_mo(addr, value, memory_order)
- uatomic_sub_return_mo(addr, value, memory_order)

- uatomic_xchg_mo(addr, value, memory_order)

- uatomic_cmpxchg_mo(addr, old, new,
memory_order_success,
memory_order_failure)

The CMM memory model reflects the C11 memory model with an additional CMM_SEQ_CST_FENCE memory order. The memory order can be selected through the enum cmm_memorder.

* With Atomic Builtins

If configured with atomic builtins, the correspondence between the CMM memory model and the C11 memory model is a one to one at the exception of the CMM_SEQ_CST_FENCE memory order which implies the memory order CMM_SEQ_CST and a thread fence after the operation.

* Without Atomic Builtins

However, if not configured with atomic builtins, the following stipulate the memory model.

For load operations with uatomic_load(), the memory orders CMM_RELAXED, CMM_CONSUME, CMM_ACQUIRE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are allowed. A barrier may be inserted before and after the load from memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_CONSUME: Memory barrier after read
- CMM_ACQUIRE: Memory barrier after read
- CMM_SEQ_CST: Memory barriers before and after read
- CMM_SEQ_CST_FENCE: Memory barriers before and after read

For store operations with uatomic_store(), the memory orders CMM_RELAXED, CMM_RELEASE, CMM_SEQ_CST and CMM_SEQ_CST_FENCE are allowed. A barrier may be inserted before and after the store to memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_RELEASE: Memory barrier before operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

For load/store operations with uatomic_and_mo(), uatomic_or_mo(), uatomic_add_mo(), uatomic_sub_mo(), uatomic_inc_mo(), uatomic_dec_mo(), uatomic_add_return_mo() and uatomic_sub_return_mo(), all memory orders are allowed. A barrier may be inserted before and after the operation depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

For the exchange operation uatomic_xchg_mo(), any memory order is valid. A barrier may be inserted before and after the exchange to memory depending on the memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

For the compare exchange operation uatomic_cmpxchg_mo(), the success memory order can be anything while the failure memory order cannot be CMM_RELEASE nor CMM_ACQ_REL and cannot be stronger than the success memory order. A barrier may be inserted before and after the store to memory depending on the memory orders:

Success memory order:

- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_RELEASE: Memory barrier before operation
- CMM_ACQ_REL: Memory barriers before and after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

Barriers after the operations are only emitted if the compare exchange succeed.

Failure memory order:
- CMM_RELAXED: No barrier
- CMM_ACQUIRE: Memory barrier after operation
- CMM_CONSUME: Memory barrier after operation
- CMM_SEQ_CST: Memory barriers before and after operation
- CMM_SEQ_CST_FENCE: Memory barriers before and after operation

Barriers after the operations are only emitted if the compare exchange failed. Barriers before the operation are never emitted by this memory order.

Change-Id I213ba19c84e82a63083f00143a3142ffbdab1d52
---
 doc/uatomic-api.md                      |   3 +-
 include/Makefile.am                     |   2 +
 include/urcu/static/pointer.h           |  40 ++--
 include/urcu/uatomic.h                  |  63 ++++++-
 include/urcu/uatomic/builtins-generic.h | 170 +++++++++++++++++
 include/urcu/uatomic/builtins.h         |  79 ++++++++
 include/urcu/uatomic/generic.h          | 234 ++++++++++++++++++++++++
 src/urcu-pointer.c                      |   9 +-
 8 files changed, 565 insertions(+), 35 deletions(-)
 create mode 100644 include/urcu/uatomic/builtins-generic.h
 create mode 100644 include/urcu/uatomic/builtins.h
doc/uatomic-api.md +2 -1
index 0962399..7341ee8 100644
@@ -52,7 +52,8 @@ An atomic read-modify-write operation that performs this
 sequence of operations atomically: check if `addr` contains `old`.
 If true, then replace the content of `addr` by `new`. Return the
 value previously contained by `addr`. This function implies a full
-memory barrier before and after the atomic operation.
+memory barrier before and after the atomic operation. The second memory
+barrier is only emitted if the operation succeeded.
 
 
 ```c
include/Makefile.am +2 -0
index ba1fe60..b20e56d 100644
@@ -63,6 +63,8 @@ nobase_include_HEADERS = \
 	urcu/uatomic/alpha.h \
 	urcu/uatomic_arch.h \
 	urcu/uatomic/arm.h \
+	urcu/uatomic/builtins.h \
+	urcu/uatomic/builtins-generic.h \
 	urcu/uatomic/gcc.h \
 	urcu/uatomic/generic.h \
 	urcu/uatomic.h \
include/urcu/static/pointer.h +0 -0
index 9e46a57..9da8657 100644

include/urcu/static/pointer.h extern "C" { · line 105

Quoted 10 lines
include/urcu/static/pointer.h +0 -7
@@ -96,23 +96,8 @@ extern "C" {
  * -Wincompatible-pointer-types errors.  Using the statement expression
  * makes it an rvalue and gets rid of the const-ness.
  */
-#ifdef __URCU_DEREFERENCE_USE_ATOMIC_CONSUME
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(__extension__ ({				\
-					__typeof__(p) __attribute__((unused)) _________p0 = { 0 }; \
-					_________p0;					\
-				})) _________p1;					\
-				__atomic_load(&(p), &_________p1, __ATOMIC_CONSUME);	\

There is talk of getting rid of memory_order_consume. But for the moment, it is what there is. Another alternative is to use a volatile load, similar to old-style CMM_LOAD_SHARED() or in-kernel READ_ONCE().

Thanx, Paul

Quoted 669 lines
include/urcu/static/pointer.h +13 -20
-				(_________p1);						\
-			})
-#else
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(p) _________p1 = CMM_LOAD_SHARED(p);		\
-				cmm_smp_read_barrier_depends();				\
-				(_________p1);						\
-			})
-#endif
-
+# define _rcu_dereference(p)			\
+	uatomic_load(&(p), CMM_CONSUME)
 /**
  * _rcu_cmpxchg_pointer - same as rcu_assign_pointer, but tests if the pointer
  * is as expected by "old". If succeeds, returns the previous pointer to the
@@ -131,8 +116,9 @@ extern "C" {
 	({								\
 		__typeof__(*p) _________pold = (old);			\
 		__typeof__(*p) _________pnew = (_new);			\
-		uatomic_cmpxchg(p, _________pold, _________pnew);	\
-	})
+		uatomic_cmpxchg_mo(p, _________pold, _________pnew,	\
+				   CMM_SEQ_CST, CMM_SEQ_CST);		\
+	});
 
 /**
  * _rcu_xchg_pointer - same as rcu_assign_pointer, but returns the previous
@@ -149,17 +135,17 @@ extern "C" {
 	__extension__					\
 	({						\
 		__typeof__(*p) _________pv = (v);	\
-		uatomic_xchg(p, _________pv);		\
+		uatomic_xchg_mo(p, _________pv,		\
+				CMM_SEQ_CST);		\
 	})
 
 
-#define _rcu_set_pointer(p, v)				\
-	do {						\
-		__typeof__(*p) _________pv = (v);	\
-		if (!__builtin_constant_p(v) || 	\
-		    ((v) != NULL))			\
-			cmm_wmb();				\
-		uatomic_set(p, _________pv);		\
+#define _rcu_set_pointer(p, v)						\
+	do {								\
+		__typeof__(*p) _________pv = (v);			\
+		uatomic_store(p, _________pv,				\
+			__builtin_constant_p(v) && (v) == NULL ?	\
+			CMM_RELAXED : CMM_RELEASE);			\
 	} while (0)
 
 /**
include/urcu/uatomic.h +62 -1
index 2fb5fd4..be857e1 100644
@@ -21,9 +21,70 @@
 #ifndef _URCU_UATOMIC_H
 #define _URCU_UATOMIC_H
 
+#include <assert.h>
+
 #include <urcu/arch.h>
+#include <urcu/config.h>
 
-#if defined(URCU_ARCH_X86)
+enum cmm_memorder {
+	CMM_RELAXED = 0,
+	CMM_CONSUME = 1,
+	CMM_ACQUIRE = 2,
+	CMM_RELEASE = 3,
+	CMM_ACQ_REL = 4,
+	CMM_SEQ_CST = 5,
+	CMM_SEQ_CST_FENCE = 6,
+};
+
+#ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
+
+/*
+ * Make sure that CMM_SEQ_CST_FENCE is not equivalent to other memory orders.
+ */
+# ifdef static_assert
+static_assert(CMM_RELAXED == __ATOMIC_RELAXED, "");
+static_assert(CMM_CONSUME == __ATOMIC_CONSUME, "");
+static_assert(CMM_ACQUIRE == __ATOMIC_ACQUIRE, "");
+static_assert(CMM_RELEASE == __ATOMIC_RELEASE, "");
+static_assert(CMM_ACQ_REL == __ATOMIC_ACQ_REL, "");
+static_assert(CMM_SEQ_CST == __ATOMIC_SEQ_CST, "");
+# endif
+
+/*
+ * This is not part of the public API. It it used internally to implement the
+ * CMM_SEQ_CST_FENCE memory order.
+ *
+ * NOTE: Using switch here instead of if statement to avoid -Wduplicated-cond
+ * warning when memory order is conditionally determined.
+ */
+static inline void cmm_seq_cst_fence_after_atomic(enum cmm_memorder mo)
+{
+	switch (mo) {
+	case CMM_SEQ_CST_FENCE:
+		cmm_smp_mb();
+		break;
+	default:
+		break;
+	}
+}
+
+#endif
+
+/*
+ * This is not part of the public API. It is used internally to convert from the
+ * CMM memory model to the C11 memory model.
+ */
+static inline int cmm_to_c11(int mo)
+{
+	if (mo == CMM_SEQ_CST_FENCE) {
+		return CMM_SEQ_CST;
+	}
+	return mo;
+}
+
+#if defined(CONFIG_RCU_USE_ATOMIC_BUILTINS)
+#include <urcu/uatomic/builtins.h>
+#elif defined(URCU_ARCH_X86)
 #include <urcu/uatomic/x86.h>
 #elif defined(URCU_ARCH_PPC)
 #include <urcu/uatomic/ppc.h>
include/urcu/uatomic/builtins-generic.h new file +170 -0
new file mode 100644
index 0000000..673e888
@@ -0,0 +1,170 @@
+/*
+ * urcu/uatomic/builtins-generic.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_GENERIC_H
+#define _URCU_UATOMIC_BUILTINS_GENERIC_H
+
+#include <urcu/system.h>
+
+#define uatomic_store(addr, v, mo)				\
+	__extension__						\
+	({							\
+		__atomic_store_n(addr, v, cmm_to_c11(mo));	\
+		cmm_seq_cst_fence_after_atomic(mo);		\
+	})
+
+#define uatomic_set(addr, v)			\
+	uatomic_store(addr, v, CMM_RELAXED)
+
+#define uatomic_load(addr, mo)						\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _value = __atomic_load_n(addr,	\
+							cmm_to_c11(mo)); \
+		cmm_seq_cst_fence_after_atomic(mo);			\
+									\
+		_value;							\
+	})
+
+#define uatomic_read(addr)			\
+	uatomic_load(addr, CMM_RELAXED)
+
+#define uatomic_cmpxchg_mo(addr, old, new, mos, mof)			\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = (__typeof__(*(addr)))old;	\
+									\
+		if (__atomic_compare_exchange_n(addr, &_old, new, 0,	\
+							cmm_to_c11(mos), \
+							cmm_to_c11(mof))) { \
+			cmm_seq_cst_fence_after_atomic(mos);		\
+		} else {						\
+			cmm_seq_cst_fence_after_atomic(mof);		\
+		}							\
+		_old;							\
+	})
+
+#define uatomic_cmpxchg(addr, old, new)					\
+	uatomic_cmpxchg_mo(addr, old, new, CMM_SEQ_CST_FENCE, CMM_RELAXED)
+
+#define uatomic_xchg_mo(addr, v, mo)					\
+	__extension__							\
+	({								\
+		__typeof__((*addr)) _old = __atomic_exchange_n(addr, v,	\
+							cmm_to_c11(mo)); \
+		cmm_seq_cst_fence_after_atomic(mo);			\
+		_old;							\
+	})
+
+#define uatomic_xchg(addr, v)						\
+	uatomic_xchg_mo(addr, v, CMM_SEQ_CST_FENCE)
+
+#define uatomic_add_return_mo(addr, v, mo)				\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = __atomic_add_fetch(addr, v,	\
+							cmm_to_c11(mo)); \
+		cmm_seq_cst_fence_after_atomic(mo);			\
+		_old;							\
+	})
+
+#define uatomic_add_return(addr, v)					\
+	uatomic_add_return_mo(addr, v, CMM_SEQ_CST_FENCE)
+
+#define uatomic_sub_return_mo(addr, v, mo)				\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = __atomic_sub_fetch(addr, v,	\
+							cmm_to_c11(mo)); \
+		cmm_seq_cst_fence_after_atomic(mo);			\
+		_old;							\
+	})
+
+#define uatomic_sub_return(addr, v)					\
+	uatomic_sub_return_mo(addr, v, CMM_SEQ_CST_FENCE)
+
+#define uatomic_and_mo(addr, mask, mo)					\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = __atomic_and_fetch(addr, mask, \
+							cmm_to_c11(mo)); \
+		cmm_seq_cst_fence_after_atomic(mo);			\
+		_old;							\
+	})
+
+#define uatomic_and(addr, mask)				\
+	(void) uatomic_and_mo(addr, mask, CMM_SEQ_CST)
+
+#define uatomic_or_mo(addr, mask, mo)					\
+	__extension__							\
+	({								\
+		__typeof__(*(addr)) _old = __atomic_or_fetch(addr, mask, \
+							cmm_to_c11(mo)); \
+		cmm_seq_cst_fence_after_atomic(mo);			\
+		_old;							\
+	})
+
+
+#define uatomic_or(addr, mask)				\
+	(void) uatomic_or_mo(addr, mask, CMM_RELAXED)
+
+#define uatomic_add_mo(addr, v, mo)			\
+	(void) uatomic_add_return_mo(addr, v, mo)
+
+#define uatomic_add(addr, v)				\
+	(void) uatomic_add_mo(addr, v, CMM_RELAXED)
+
+#define uatomic_sub_mo(addr, v, mo)			\
+	(void) uatomic_sub_return_mo(addr, v, mo)
+
+#define uatomic_sub(addr, v)				\
+	(void) uatomic_sub_mo(addr, v, CMM_RELAXED)
+
+#define uatomic_inc_mo(addr, mo)		\
+	(void) uatomic_add_mo(addr, 1, mo)
+
+#define uatomic_inc(addr)				\
+	(void) uatomic_inc_mo(addr, CMM_RELAXED)
+
+#define uatomic_dec_mo(addr, mo)		\
+	(void) uatomic_sub_mo(addr, 1, mo)
+
+#define uatomic_dec(addr)				\
+	(void) uatomic_dec_mo(addr, CMM_RELAXED)
+
+#define cmm_smp_mb__before_uatomic_and() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_and()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_or() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_or()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_add() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_add()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_sub() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_sub()  cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_inc() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_inc() cmm_smp_mb()
+
+#define cmm_smp_mb__before_uatomic_dec() cmm_smp_mb()
+#define cmm_smp_mb__after_uatomic_dec() cmm_smp_mb()
+
+#endif /* _URCU_UATOMIC_BUILTINS_X86_H */
include/urcu/uatomic/builtins.h new file +79 -0
new file mode 100644
index 0000000..82e98f8
@@ -0,0 +1,79 @@
+/*
+ * urcu/uatomic/builtins.h
+ *
+ * Copyright (c) 2023 Olivier Dion <[email protected]>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+#ifndef _URCU_UATOMIC_BUILTINS_H
+#define _URCU_UATOMIC_BUILTINS_H
+
+#include <urcu/arch.h>
+
+#if defined(__has_builtin)
+# if !__has_builtin(__atomic_store_n)
+#  error "Toolchain does not support __atomic_store_n."
+# endif
+# if !__has_builtin(__atomic_load_n)
+#  error "Toolchain does not support __atomic_load_n."
+# endif
+# if !__has_builtin(__atomic_exchange_n)
+#  error "Toolchain does not support __atomic_exchange_n."
+# endif
+# if !__has_builtin(__atomic_compare_exchange_n)
+#  error "Toolchain does not support __atomic_compare_exchange_n."
+# endif
+# if !__has_builtin(__atomic_add_fetch)
+#  error "Toolchain does not support __atomic_add_fetch."
+# endif
+# if !__has_builtin(__atomic_sub_fetch)
+#  error "Toolchain does not support __atomic_sub_fetch."
+# endif
+# if !__has_builtin(__atomic_or_fetch)
+#  error "Toolchain does not support __atomic_or_fetch."
+# endif
+# if !__has_builtin(__atomic_thread_fence)
+#  error "Toolchain does not support __atomic_thread_fence."
+# endif
+# if !__has_builtin(__atomic_signal_fence)
+#  error "Toolchain does not support __atomic_signal_fence."
+# endif
+#elif defined(__GNUC__)
+# define GCC_VERSION (__GNUC__       * 10000 + \
+		       __GNUC_MINOR__ * 100   + \
+		       __GNUC_PATCHLEVEL__)
+# if  GCC_VERSION < 40700
+#  error "GCC version is too old. Version must be 4.7 or greater"
+# endif
+# undef  GCC_VERSION
+#else
+# error "Toolchain is not supported."
+#endif
+
+#if defined(__GNUC__)
+# define UATOMIC_HAS_ATOMIC_BYTE  __GCC_ATOMIC_CHAR_LOCK_FREE
+# define UATOMIC_HAS_ATOMIC_SHORT __GCC_ATOMIC_SHORT_LOCK_FREE
+#elif defined(__clang__)
+# define UATOMIC_HAS_ATOMIC_BYTE  __CLANG_ATOMIC_CHAR_LOCK_FREE
+# define UATOMIC_HAS_ATOMIC_SHORT __CLANG_ATOMIC_SHORT_LOCK_FREE
+#else
+/* #  define UATOMIC_HAS_ATOMIC_BYTE  */
+/* #  define UATOMIC_HAS_ATOMIC_SHORT */
+#endif
+
+#include <urcu/uatomic/builtins-generic.h>
+
+#endif	/* _URCU_UATOMIC_BUILTINS_H */
include/urcu/uatomic/generic.h +234 -0
index e31a19b..6b9c153 100644
@@ -33,10 +33,244 @@ extern "C" {
 #define uatomic_set(addr, v)	((void) CMM_STORE_SHARED(*(addr), (v)))
 #endif
 
+extern void abort(void);
+
+#define uatomic_load_store_return_op(op, addr, v, mo)			\
+	__extension__							\
+	({								\
+									\
+		switch (mo) {						\
+		case CMM_ACQUIRE:					\
+		case CMM_CONSUME:					\
+		case CMM_RELAXED:					\
+			break;						\
+		case CMM_RELEASE:					\
+		case CMM_ACQ_REL:					\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+									\
+		__typeof__((*addr)) _value = op(addr, v);		\
+									\
+		switch (mo) {						\
+		case CMM_CONSUME:					\
+			cmm_smp_read_barrier_depends();			\
+			break;						\
+		case CMM_ACQUIRE:					\
+		case CMM_ACQ_REL:					\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		case CMM_RELAXED:					\
+		case CMM_RELEASE:					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+		_value;							\
+	})
+
+#define uatomic_load_store_op(op, addr, v, mo)				\
+	({								\
+		switch (mo) {						\
+		case CMM_ACQUIRE:					\
+		case CMM_CONSUME:					\
+		case CMM_RELAXED:					\
+			break;						\
+		case CMM_RELEASE:					\
+		case CMM_ACQ_REL:					\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+									\
+		op(addr, v);						\
+									\
+		switch (mo) {						\
+		case CMM_CONSUME:					\
+			cmm_smp_read_barrier_depends();			\
+			break;						\
+		case CMM_ACQUIRE:					\
+		case CMM_ACQ_REL:					\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		case CMM_RELAXED:					\
+		case CMM_RELEASE:					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+	})
+
+#define uatomic_store(addr, v, mo)			\
+	({						\
+		switch (mo) {				\
+		case CMM_RELAXED:			\
+			break;				\
+		case CMM_RELEASE:			\
+		case CMM_SEQ_CST:			\
+		case CMM_SEQ_CST_FENCE:			\
+			cmm_smp_mb();			\
+			break;				\
+		default:				\
+			abort();			\
+		}					\
+							\
+		uatomic_set(addr, v);			\
+							\
+		switch (mo) {				\
+		case CMM_RELAXED:			\
+		case CMM_RELEASE:			\
+			break;				\
+		case CMM_SEQ_CST:			\
+		case CMM_SEQ_CST_FENCE:			\
+			cmm_smp_mb();			\
+			break;				\
+		default:				\
+			abort();			\
+		}					\
+	})
+
+#define uatomic_and_mo(addr, v, mo)				\
+	uatomic_load_store_op(uatomic_and, addr, v, mo)
+
+#define uatomic_or_mo(addr, v, mo)				\
+	uatomic_load_store_op(uatomic_or, addr, v, mo)
+
+#define uatomic_add_mo(addr, v, mo)				\
+	uatomic_load_store_op(uatomic_add, addr, v, mo)
+
+#define uatomic_sub_mo(addr, v, mo)				\
+	uatomic_load_store_op(uatomic_sub, addr, v, mo)
+
+#define uatomic_inc_mo(addr, mo)				\
+	uatomic_load_store_op(uatomic_add, addr, 1, mo)
+
+#define uatomic_dec_mo(addr, mo)				\
+	uatomic_load_store_op(uatomic_add, addr, -1, mo)
+/*
+ * NOTE: We can not just do switch (_value == (old) ? mos : mof) otherwise the
+ * compiler emit a -Wduplicated-cond warning.
+ */
+#define uatomic_cmpxchg_mo(addr, old, new, mos, mof)			\
+	__extension__							\
+	({								\
+		switch (mos) {						\
+		case CMM_ACQUIRE:					\
+		case CMM_CONSUME:					\
+		case CMM_RELAXED:					\
+			break;						\
+		case CMM_RELEASE:					\
+		case CMM_ACQ_REL:					\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+									\
+		__typeof__(*(addr)) _value = uatomic_cmpxchg(addr, old,	\
+							new);		\
+									\
+		if (_value == (old)) {					\
+			switch (mos) {					\
+			case CMM_CONSUME:				\
+				cmm_smp_read_barrier_depends();		\
+				break;					\
+			case CMM_ACQUIRE:				\
+			case CMM_ACQ_REL:				\
+			case CMM_SEQ_CST:				\
+			case CMM_SEQ_CST_FENCE:				\
+				cmm_smp_mb();				\
+				break;					\
+			case CMM_RELAXED:				\
+			case CMM_RELEASE:				\
+				break;					\
+			default:					\
+				abort();				\
+			}						\
+		} else {						\
+			switch (mof) {					\
+			case CMM_CONSUME:				\
+				cmm_smp_read_barrier_depends();		\
+				break;					\
+			case CMM_ACQUIRE:				\
+			case CMM_ACQ_REL:				\
+			case CMM_SEQ_CST:				\
+			case CMM_SEQ_CST_FENCE:				\
+				cmm_smp_mb();				\
+				break;					\
+			case CMM_RELAXED:				\
+			case CMM_RELEASE:				\
+				break;					\
+			default:					\
+				abort();				\
+			}						\
+		}							\
+		_value;							\
+	})
+
+#define uatomic_xchg_mo(addr, v, mo)				\
+	uatomic_load_store_return_op(uatomic_xchg, addr, v, mo)
+
+#define uatomic_add_return_mo(addr, v, mo)				\
+	uatomic_load_store_return_op(uatomic_add_return, addr, v)
+
+#define uatomic_sub_return_mo(addr, v, mo)				\
+	uatomic_load_store_return_op(uatomic_sub_return, addr, v)
+
+
 #ifndef uatomic_read
 #define uatomic_read(addr)	CMM_LOAD_SHARED(*(addr))
 #endif
 
+#define uatomic_load(addr, mo)						\
+	__extension__							\
+	({								\
+		switch (mo) {						\
+		case CMM_ACQUIRE:					\
+		case CMM_CONSUME:					\
+		case CMM_RELAXED:					\
+			break;						\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+									\
+		__typeof__(*(addr)) _rcu_value = uatomic_read(addr);	\
+									\
+		switch (mo) {						\
+		case CMM_RELAXED:					\
+			break;						\
+		case CMM_CONSUME:					\
+			cmm_smp_read_barrier_depends();			\
+			break;						\
+		case CMM_ACQUIRE:					\
+		case CMM_SEQ_CST:					\
+		case CMM_SEQ_CST_FENCE:					\
+			cmm_smp_mb();					\
+			break;						\
+		default:						\
+			abort();					\
+		}							\
+									\
+		_rcu_value;						\
+	})
+
 #if !defined __OPTIMIZE__  || defined UATOMIC_NO_LINK_ERROR
 #ifdef ILLEGAL_INSTR
 static inline __attribute__((always_inline))
src/urcu-pointer.c +3 -6
index d0854ac..cea8aeb 100644
@@ -39,19 +39,16 @@ void *rcu_dereference_sym(void *p)
 
 void *rcu_set_pointer_sym(void **p, void *v)
 {
-	cmm_wmb();
-	uatomic_set(p, v);
+	uatomic_store(p, v, CMM_RELEASE);
 	return v;
 }
 
 void *rcu_xchg_pointer_sym(void **p, void *v)
 {
-	cmm_wmb();
-	return uatomic_xchg(p, v);
+	return uatomic_xchg_mo(p, v, CMM_SEQ_CST);
 }
 
 void *rcu_cmpxchg_pointer_sym(void **p, void *old, void *_new)
 {
-	cmm_wmb();
-	return uatomic_cmpxchg(p, old, _new);
+	return uatomic_cmpxchg_mo(p, old, _new, CMM_SEQ_CST, CMM_SEQ_CST);
 }
Signature · 1 line
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#58 lore raw

Re: [lttng-dev] [PATCH v2 05/12] urcu/uatomic: Add CMM memory model

Quoted 12 lines · Paul E. McKenney

include/urcu/static/pointer.h extern "C" { · line 105

Quoted 7 lines · Olivier Dion
include/urcu/static/pointer.h +0 -7
-#ifdef __URCU_DEREFERENCE_USE_ATOMIC_CONSUME
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(__extension__ ({				\
-					__typeof__(p) __attribute__((unused)) _________p0 = { 0 }; \
-					_________p0;					\
-				})) _________p1;					\
-				__atomic_load(&(p), &_________p1, __ATOMIC_CONSUME);	\

There is talk of getting rid of memory_order_consume. But for the moment, it is what there is. Another alternative is to use a volatile load, similar to old-style CMM_LOAD_SHARED() or in-kernel READ_ONCE().

I think we can stick to __ATOMIC_CONSUME for now. Hopefully getting rid of it means it will be an alias for __ATOMIC_ACQUIRE for ever.

Signature · 7 lines
Olivier Dion
EfficiOS Inc.
https://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#59 lore raw

Re: [lttng-dev] [PATCH v2 05/12] urcu/uatomic: Add CMM memory model

Quoted 14 lines · Olivier Dion
Quoted 12 lines · Paul E. McKenney

include/urcu/static/pointer.h extern "C" { · line 105

Quoted 7 lines · Olivier Dion
include/urcu/static/pointer.h +0 -7
-#ifdef __URCU_DEREFERENCE_USE_ATOMIC_CONSUME
-# define _rcu_dereference(p) __extension__ ({						\
-				__typeof__(__extension__ ({				\
-					__typeof__(p) __attribute__((unused)) _________p0 = { 0 }; \
-					_________p0;					\
-				})) _________p1;					\
-				__atomic_load(&(p), &_________p1, __ATOMIC_CONSUME);	\

There is talk of getting rid of memory_order_consume. But for the moment, it is what there is. Another alternative is to use a volatile load, similar to old-style CMM_LOAD_SHARED() or in-kernel READ_ONCE().

Quoted 2 lines

I think we can stick to __ATOMIC_CONSUME for now. Hopefully getting rid of it means it will be an alias for __ATOMIC_ACQUIRE for ever.

That seems emininently reasonable to me!

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#60 lore raw

PATCH v2 6/12 urcu-wait: Fix wait state load/store

In reply to Olivier Dion via lttng-dev

The state of a wait node must be accessed atomically. Also, the action of busy loading until the teardown state is seen must follow a CMM_ACQUIRE semantic while storing the teardown must follow a CMM_RELEASE semantic.

Change-Id I9cd9cf4cd9ab2081551d7f33c0b1c23c3cf3942f
---
 src/urcu-wait.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
src/urcu-wait.h +3 -3
index ef5f7ed..4667a13 100644
@@ -135,7 +135,7 @@ void urcu_adaptative_wake_up(struct urcu_wait_node *wait)
 			urcu_die(errno);
 	}
 	/* Allow teardown of struct urcu_wait memory. */
-	uatomic_or(&wait->state, URCU_WAIT_TEARDOWN);
+	uatomic_or_mo(&wait->state, URCU_WAIT_TEARDOWN, CMM_RELEASE);
 }
 
 /*
@@ -193,7 +193,7 @@ skip_futex_wait:
 			break;
 		caa_cpu_relax();
 	}
-	while (!(uatomic_read(&wait->state) & URCU_WAIT_TEARDOWN))
+	while (!(uatomic_load(&wait->state, CMM_ACQUIRE) & URCU_WAIT_TEARDOWN))
 		poll(NULL, 0, 10);
 	urcu_posix_assert(uatomic_read(&wait->state) & URCU_WAIT_TEARDOWN);
 }
@@ -209,7 +209,7 @@ void urcu_wake_all_waiters(struct urcu_waiters *waiters)
 			caa_container_of(iter, struct urcu_wait_node, node);
 
 		/* Don't wake already running threads */
-		if (wait_node->state & URCU_WAIT_RUNNING)
+		if (uatomic_load(&wait_node->state, CMM_RELAXED) & URCU_WAIT_RUNNING)
 			continue;
 		urcu_adaptative_wake_up(wait_node);
 	}
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#61 lore raw

PATCH v2 7/12 tests: Use uatomic for accessing global states

In reply to Olivier Dion via lttng-dev

Global states accesses were protected via memory barriers. Use the uatomic API with the CMM memory model so that TSAN does not warns about none atomic concurrent accesses.

Also, the thread id map mutex must be unlocked after setting the new created thread id in the map. Otherwise, the new thread could observe an unset id.

Change-Id I1ecdc387b3f510621cbc116ad3b95c676f5d659a
---
 tests/common/api.h            |  12 ++--
 tests/regression/rcutorture.h | 106 +++++++++++++++++++++++-----------
 2 files changed, 80 insertions(+), 38 deletions(-)
tests/common/api.h +7 -5
index a260463..9d22b0f 100644
@@ -26,6 +26,7 @@
 
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
+#include <urcu/uatomic.h>
 
 /*
  * Machine parameters.
@@ -135,7 +136,7 @@ static int __smp_thread_id(void)
 	thread_id_t tid = pthread_self();
 
 	for (i = 0; i < NR_THREADS; i++) {
-		if (__thread_id_map[i] == tid) {
+		if (uatomic_read(&__thread_id_map[i]) == tid) {
 			long v = i + 1;  /* must be non-NULL. */
 
 			if (pthread_setspecific(thread_id_key, (void *)v) != 0) {
@@ -184,12 +185,13 @@ static thread_id_t create_thread(void *(*func)(void *), void *arg)
 		exit(-1);
 	}
 	__thread_id_map[i] = __THREAD_ID_MAP_WAITING;
-	spin_unlock(&__thread_id_map_mutex);
+
 	if (pthread_create(&tid, NULL, func, arg) != 0) {
 		perror("create_thread:pthread_create");
 		exit(-1);
 	}
-	__thread_id_map[i] = tid;
+	uatomic_set(&__thread_id_map[i], tid);
+	spin_unlock(&__thread_id_map_mutex);
 	return tid;
 }
 
@@ -199,7 +201,7 @@ static void *wait_thread(thread_id_t tid)
 	void *vp;
 
 	for (i = 0; i < NR_THREADS; i++) {
-		if (__thread_id_map[i] == tid)
+		if (uatomic_read(&__thread_id_map[i]) == tid)
 			break;
 	}
 	if (i >= NR_THREADS){
@@ -211,7 +213,7 @@ static void *wait_thread(thread_id_t tid)
 		perror("wait_thread:pthread_join");
 		exit(-1);
 	}
-	__thread_id_map[i] = __THREAD_ID_MAP_EMPTY;
+	uatomic_set(&__thread_id_map[i], __THREAD_ID_MAP_EMPTY);
 	return vp;
 }
 
tests/regression/rcutorture.h +73 -33

219 lines not on this page · raw message

Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#62 lore raw

Re: [lttng-dev] [PATCH v2 07/12] tests: Use uatomic for accessing global states

Quoted 2 lines · Olivier Dion

Global states accesses were protected via memory barriers. Use the uatomic API with the CMM memory model so that TSAN does not warns about

"does not warn", for whatever that is worth.

Quoted 123 lines

none atomic concurrent accesses.

Also, the thread id map mutex must be unlocked after setting the new created thread id in the map. Otherwise, the new thread could observe an unset id.

Change-Id I1ecdc387b3f510621cbc116ad3b95c676f5d659a
---
 tests/common/api.h            |  12 ++--
 tests/regression/rcutorture.h | 106 +++++++++++++++++++++++-----------
 2 files changed, 80 insertions(+), 38 deletions(-)
tests/common/api.h +7 -5
index a260463..9d22b0f 100644
@@ -26,6 +26,7 @@
 
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
+#include <urcu/uatomic.h>
 
 /*
  * Machine parameters.
@@ -135,7 +136,7 @@ static int __smp_thread_id(void)
 	thread_id_t tid = pthread_self();
 
 	for (i = 0; i < NR_THREADS; i++) {
-		if (__thread_id_map[i] == tid) {
+		if (uatomic_read(&__thread_id_map[i]) == tid) {
 			long v = i + 1;  /* must be non-NULL. */
 
 			if (pthread_setspecific(thread_id_key, (void *)v) != 0) {
@@ -184,12 +185,13 @@ static thread_id_t create_thread(void *(*func)(void *), void *arg)
 		exit(-1);
 	}
 	__thread_id_map[i] = __THREAD_ID_MAP_WAITING;
-	spin_unlock(&__thread_id_map_mutex);
+
 	if (pthread_create(&tid, NULL, func, arg) != 0) {
 		perror("create_thread:pthread_create");
 		exit(-1);
 	}
-	__thread_id_map[i] = tid;
+	uatomic_set(&__thread_id_map[i], tid);
+	spin_unlock(&__thread_id_map_mutex);
 	return tid;
 }
 
@@ -199,7 +201,7 @@ static void *wait_thread(thread_id_t tid)
 	void *vp;
 
 	for (i = 0; i < NR_THREADS; i++) {
-		if (__thread_id_map[i] == tid)
+		if (uatomic_read(&__thread_id_map[i]) == tid)
 			break;
 	}
 	if (i >= NR_THREADS){
@@ -211,7 +213,7 @@ static void *wait_thread(thread_id_t tid)
 		perror("wait_thread:pthread_join");
 		exit(-1);
 	}
-	__thread_id_map[i] = __THREAD_ID_MAP_EMPTY;
+	uatomic_set(&__thread_id_map[i], __THREAD_ID_MAP_EMPTY);
 	return vp;
 }
 
tests/regression/rcutorture.h +14 -4
index bc394f9..5835b8f 100644
@@ -44,6 +44,14 @@
  * data.  A correct RCU implementation will have all but the first two
  * numbers non-zero.
  *
+ * rcu_stress_count: Histogram of "ages" of structures seen by readers.  If any
+ * entries past the first two are non-zero, RCU is broken. The age of a newly
+ * allocated structure is zero, it becomes one when removed from reader
+ * visibility, and is incremented once per grace period subsequently -- and is
+ * freed after passing through (RCU_STRESS_PIPE_LEN-2) grace periods.  Since
+ * this tests only has one true writer (there are fake writers), only buckets at
+ * indexes 0 and 1 should be none-zero.
+ *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License as published by
  * the Free Software Foundation; either version 2 of the License, or
@@ -68,6 +76,8 @@
 #include <stdlib.h>
 #include "tap.h"
 
+#include <urcu/uatomic.h>
+
 #define NR_TESTS	1
 
 DEFINE_PER_THREAD(long long, n_reads_pt);
@@ -145,10 +155,10 @@ void *rcu_read_perf_test(void *arg)
 	run_on(me);
 	uatomic_inc(&nthreadsrunning);
 	put_thread_offline();
-	while (goflag == GOFLAG_INIT)
+	while (uatomic_read(&goflag) == GOFLAG_INIT)
 		(void) poll(NULL, 0, 1);
 	put_thread_online();
-	while (goflag == GOFLAG_RUN) {
+	while (uatomic_read(&goflag) == GOFLAG_RUN) {
 		for (i = 0; i < RCU_READ_RUN; i++) {
 			rcu_read_lock();
 			/* rcu_read_lock_nest(); */
@@ -180,9 +190,9 @@ void *rcu_update_perf_test(void *arg __attribute__((unused)))
 		}
 	}
 	uatomic_inc(&nthreadsrunning);
-	while (goflag == GOFLAG_INIT)
+	while (uatomic_read(&goflag) == GOFLAG_INIT)
 		(void) poll(NULL, 0, 1);
-	while (goflag == GOFLAG_RUN) {
+	while (uatomic_read(&goflag) == GOFLAG_RUN) {
 		synchronize_rcu();
 		n_updates_local++;
 	}

tests/regression/rcutorture.h int perftestrun(int nthreads, int nreaders, int nupdaters) · line 227

Quoted 10 lines
tests/regression/rcutorture.h +1 -3
@@ -211,15 +221,11 @@ int perftestrun(int nthreads, int nreaders, int nupdaters)
 	int t;
 	int duration = 1;
 
-	cmm_smp_mb();
 	while (uatomic_read(&nthreadsrunning) < nthreads)
 		(void) poll(NULL, 0, 1);
-	goflag = GOFLAG_RUN;
-	cmm_smp_mb();
+	uatomic_set(&goflag, GOFLAG_RUN);
 	sleep(duration);

The theory here being that the context switches implied by the sleep() make the memory barrier unnecesary? Not unreasonable, I guess. ;-)

Quoted 161 lines
tests/regression/rcutorture.h +56 -22
-	cmm_smp_mb();
-	goflag = GOFLAG_STOP;
-	cmm_smp_mb();
+	uatomic_set(&goflag, GOFLAG_STOP);
 	wait_all_threads();
 	for_each_thread(t) {
 		n_reads += per_thread(n_reads_pt, t);
@@ -300,6 +306,13 @@ struct rcu_stress rcu_stress_array[RCU_STRESS_PIPE_LEN] = { { 0, 0 } };
 struct rcu_stress *rcu_stress_current;
 int rcu_stress_idx = 0;
 
+/*
+ * How many time a reader has seen something that should not be visible. It is
+ * an error if this value is different than zero at the end of the stress test.
+ *
+ * Here, the something that should not be visibile is an old pipe that has been
+ * freed (mbtest = 0).
+ */
 int n_mberror = 0;
 DEFINE_PER_THREAD(long long [RCU_STRESS_PIPE_LEN + 1], rcu_stress_count);
 
@@ -315,19 +328,25 @@ void *rcu_read_stress_test(void *arg __attribute__((unused)))
 
 	rcu_register_thread();
 	put_thread_offline();
-	while (goflag == GOFLAG_INIT)
+	while (uatomic_read(&goflag) == GOFLAG_INIT)
 		(void) poll(NULL, 0, 1);
 	put_thread_online();
-	while (goflag == GOFLAG_RUN) {
+	while (uatomic_read(&goflag) == GOFLAG_RUN) {
 		rcu_read_lock();
 		p = rcu_dereference(rcu_stress_current);
 		if (p->mbtest == 0)
-			n_mberror++;
+			uatomic_inc_mo(&n_mberror, CMM_RELAXED);
 		rcu_read_lock_nest();
+		/*
+		 * The value of garbage is nothing important. This is
+		 * essentially a busy loop. The atomic operation -- while not
+		 * important here -- helps tools such as TSAN to not flag this
+		 * as a race condition.
+		 */
 		for (i = 0; i < 100; i++)
-			garbage++;
+			uatomic_inc(&garbage);
 		rcu_read_unlock_nest();
-		pc = p->pipe_count;
+		pc = uatomic_read(&p->pipe_count);
 		rcu_read_unlock();
 		if ((pc > RCU_STRESS_PIPE_LEN) || (pc < 0))
 			pc = RCU_STRESS_PIPE_LEN;
@@ -397,26 +416,47 @@ static
 void *rcu_update_stress_test(void *arg __attribute__((unused)))
 {
 	int i;
-	struct rcu_stress *p;
+	struct rcu_stress *p, *old_p;
 	struct rcu_head rh;
 	enum writer_state writer_state = WRITER_STATE_SYNC_RCU;
 
-	while (goflag == GOFLAG_INIT)
+	rcu_register_thread();
+
+	put_thread_offline();
+	while (uatomic_read(&goflag) == GOFLAG_INIT)
 		(void) poll(NULL, 0, 1);
-	while (goflag == GOFLAG_RUN) {
+
+	put_thread_online();
+	while (uatomic_read(&goflag) == GOFLAG_RUN) {
 		i = rcu_stress_idx + 1;
 		if (i >= RCU_STRESS_PIPE_LEN)
 			i = 0;
+		/*
+		 * Get old pipe that we free after a synchronize_rcu().
+		 */
+		rcu_read_lock();
+		old_p = rcu_dereference(rcu_stress_current);
+		rcu_read_unlock();
+
+		/*
+		 * Allocate a new pipe.
+		 */
 		p = &rcu_stress_array[i];
-		p->mbtest = 0;
-		cmm_smp_mb();
 		p->pipe_count = 0;
 		p->mbtest = 1;
+
 		rcu_assign_pointer(rcu_stress_current, p);
 		rcu_stress_idx = i;
+
+		/*
+		 * Increment every pipe except the freshly allocated one. A
+		 * reader should only see either the old pipe or the new
+		 * pipe. This is reflected in the rcu_stress_count histogram.
+		 */
 		for (i = 0; i < RCU_STRESS_PIPE_LEN; i++)
 			if (i != rcu_stress_idx)
-				rcu_stress_array[i].pipe_count++;
+				uatomic_inc(&rcu_stress_array[i].pipe_count);
+
 		switch (writer_state) {
 		case WRITER_STATE_SYNC_RCU:
 			synchronize_rcu();
@@ -432,9 +472,7 @@ void *rcu_update_stress_test(void *arg __attribute__((unused)))
 					strerror(errno));
 				abort();
 			}
-			rcu_register_thread();
 			call_rcu(&rh, rcu_update_stress_test_rcu);
-			rcu_unregister_thread();
 			/*
 			 * Our MacOS X test machine with the following
 			 * config:
@@ -470,18 +508,24 @@ void *rcu_update_stress_test(void *arg __attribute__((unused)))
 		{
 			struct urcu_gp_poll_state poll_state;
 
-			rcu_register_thread();
 			poll_state = start_poll_synchronize_rcu();
-			rcu_unregister_thread();
 			while (!poll_state_synchronize_rcu(poll_state))
 				(void) poll(NULL, 0, 1);	/* Wait for 1ms */
 			break;
 		}
 		}
+		/*
+		 * No readers should see that old pipe now. Setting mbtest to 0
+		 * to mark it as "freed".
+		 */
+		old_p->mbtest = 0;
 		n_updates++;
 		advance_writer_state(&writer_state);
 	}
 
+	put_thread_offline();
+	rcu_unregister_thread();
+
 	return NULL;
 }
 
@@ -497,9 +541,9 @@ void *rcu_fake_update_stress_test(void *arg __attribute__((unused)))
 			set_thread_call_rcu_data(crdp);
 		}
 	}
-	while (goflag == GOFLAG_INIT)
+	while (uatomic_read(&goflag) == GOFLAG_INIT)
 		(void) poll(NULL, 0, 1);
-	while (goflag == GOFLAG_RUN) {
+	while (uatomic_read(&goflag) == GOFLAG_RUN) {
 		synchronize_rcu();
 		(void) poll(NULL, 0, 1);
 	}
@@ -535,13 +579,9 @@ int stresstest(int nreaders)
 	create_thread(rcu_update_stress_test, NULL);
 	for (i = 0; i < 5; i++)
 		create_thread(rcu_fake_update_stress_test, NULL);
-	cmm_smp_mb();
-	goflag = GOFLAG_RUN;

tests/regression/rcutorture.h int stresstest(int nreaders) · line 587

Quoted 10 lines
tests/regression/rcutorture.h +2 -4
-	cmm_smp_mb();
+	uatomic_set(&goflag, GOFLAG_RUN);
 	sleep(10);
-	cmm_smp_mb();
-	goflag = GOFLAG_STOP;
-	cmm_smp_mb();
+	uatomic_set(&goflag, GOFLAG_STOP);
 	wait_all_threads();
 	for_each_thread(t)
 		n_reads += per_thread(n_reads_pt, t);

Looks plausible!

Thanx, Paul
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#63 lore raw

PATCH v2 8/12 benchmark: Use uatomic for accessing global states

In reply to Olivier Dion via lttng-dev

Global states accesses were protected via memory barriers. Use the uatomic API with the CMM memory model so that TSAN can understand the ordering imposed by the synchronization flags.

Change-Id I1bf5702c5ac470f308c478effe39e424a3158060
---
 tests/benchmark/Makefile.am             | 91 +++++++++++++------------
 tests/benchmark/common-states.c         |  1 +
 tests/benchmark/common-states.h         | 51 ++++++++++++++
 tests/benchmark/test_mutex.c            | 32 +--------
 tests/benchmark/test_perthreadlock.c    | 32 +--------
 tests/benchmark/test_rwlock.c           | 32 +--------
 tests/benchmark/test_urcu.c             | 33 +--------
 tests/benchmark/test_urcu_assign.c      | 33 +--------
 tests/benchmark/test_urcu_bp.c          | 33 +--------
 tests/benchmark/test_urcu_defer.c       | 33 +--------
 tests/benchmark/test_urcu_gc.c          | 34 ++-------
 tests/benchmark/test_urcu_hash.c        |  6 +-
 tests/benchmark/test_urcu_hash.h        | 15 ----
 tests/benchmark/test_urcu_hash_rw.c     | 10 +--
 tests/benchmark/test_urcu_hash_unique.c | 10 +--
 tests/benchmark/test_urcu_lfq.c         | 20 ++----
 tests/benchmark/test_urcu_lfs.c         | 20 ++----
 tests/benchmark/test_urcu_lfs_rcu.c     | 20 ++----
 tests/benchmark/test_urcu_qsbr.c        | 33 +--------
 tests/benchmark/test_urcu_qsbr_gc.c     | 34 ++-------
 tests/benchmark/test_urcu_wfcq.c        | 22 +++---
 tests/benchmark/test_urcu_wfq.c         | 20 ++----
 tests/benchmark/test_urcu_wfs.c         | 22 +++---
 23 files changed, 177 insertions(+), 460 deletions(-)
 create mode 100644 tests/benchmark/common-states.c
 create mode 100644 tests/benchmark/common-states.h
tests/benchmark/Makefile.am +47 -44
index c53e025..a7f91c2 100644
@@ -1,4 +1,5 @@
 AM_CPPFLAGS += -I$(top_srcdir)/src -I$(top_srcdir)/tests/common
+AM_CPPFLAGS += -include $(top_srcdir)/tests/benchmark/common-states.h
 
 TEST_EXTENSIONS = .tap
 TAP_LOG_DRIVER_FLAGS = --merge --comments
@@ -7,6 +8,8 @@ TAP_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' \
 	URCU_TESTS_BUILDDIR='$(abs_top_builddir)/tests' \
 	$(SHELL) $(top_srcdir)/tests/utils/tap-driver.sh
 
+noinst_HEADERS = common-states.h
+
 SCRIPT_LIST = \
 	runpaul-phase1.sh \
 	runpaul-phase2.sh \
@@ -61,163 +64,163 @@ URCU_CDS_LIB=$(top_builddir)/src/liburcu-cds.la
 
 DEBUG_YIELD_LIB=$(builddir)/../common/libdebug-yield.la
 
-test_urcu_SOURCES = test_urcu.c
+test_urcu_SOURCES = test_urcu.c common-states.c
 test_urcu_LDADD = $(URCU_LIB)
 
-test_urcu_dynamic_link_SOURCES = test_urcu.c
+test_urcu_dynamic_link_SOURCES = test_urcu.c common-states.c
 test_urcu_dynamic_link_LDADD = $(URCU_LIB)
 test_urcu_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_timing_SOURCES = test_urcu_timing.c
+test_urcu_timing_SOURCES = test_urcu_timing.c common-states.c
 test_urcu_timing_LDADD = $(URCU_LIB)
 
-test_urcu_yield_SOURCES = test_urcu.c
+test_urcu_yield_SOURCES = test_urcu.c common-states.c
 test_urcu_yield_LDADD = $(URCU_LIB) $(DEBUG_YIELD_LIB)
 test_urcu_yield_CFLAGS = -DDEBUG_YIELD $(AM_CFLAGS)
 
 
-test_urcu_qsbr_SOURCES = test_urcu_qsbr.c
+test_urcu_qsbr_SOURCES = test_urcu_qsbr.c common-states.c
 test_urcu_qsbr_LDADD = $(URCU_QSBR_LIB)
 
-test_urcu_qsbr_timing_SOURCES = test_urcu_qsbr_timing.c
+test_urcu_qsbr_timing_SOURCES = test_urcu_qsbr_timing.c common-states.c
 test_urcu_qsbr_timing_LDADD = $(URCU_QSBR_LIB)
 
 
-test_urcu_mb_SOURCES = test_urcu.c
+test_urcu_mb_SOURCES = test_urcu.c common-states.c
 test_urcu_mb_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_CFLAGS = -DRCU_MB $(AM_CFLAGS)
 
 
-test_urcu_signal_SOURCES = test_urcu.c
+test_urcu_signal_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_CFLAGS = -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_signal_dynamic_link_SOURCES = test_urcu.c
+test_urcu_signal_dynamic_link_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_dynamic_link_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_dynamic_link_CFLAGS = -DRCU_SIGNAL -DDYNAMIC_LINK_TEST \
 					$(AM_CFLAGS)
 
-test_urcu_signal_timing_SOURCES = test_urcu_timing.c
+test_urcu_signal_timing_SOURCES = test_urcu_timing.c common-states.c
 test_urcu_signal_timing_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_timing_CFLAGS= -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_signal_yield_SOURCES = test_urcu.c
+test_urcu_signal_yield_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_yield_LDADD = $(URCU_SIGNAL_LIB) $(DEBUG_YIELD_LIB)
 test_urcu_signal_yield_CFLAGS = -DRCU_SIGNAL -DDEBUG_YIELD $(AM_CFLAGS)
 
-test_rwlock_timing_SOURCES = test_rwlock_timing.c
+test_rwlock_timing_SOURCES = test_rwlock_timing.c common-states.c
 test_rwlock_timing_LDADD = $(URCU_SIGNAL_LIB)
 
-test_rwlock_SOURCES = test_rwlock.c
+test_rwlock_SOURCES = test_rwlock.c common-states.c
 test_rwlock_LDADD = $(URCU_SIGNAL_LIB)
 
-test_perthreadlock_timing_SOURCES = test_perthreadlock_timing.c
+test_perthreadlock_timing_SOURCES = test_perthreadlock_timing.c common-states.c
 test_perthreadlock_timing_LDADD = $(URCU_SIGNAL_LIB)
 
-test_perthreadlock_SOURCES = test_perthreadlock.c
+test_perthreadlock_SOURCES = test_perthreadlock.c common-states.c
 test_perthreadlock_LDADD = $(URCU_SIGNAL_LIB)
 
-test_mutex_SOURCES = test_mutex.c
+test_mutex_SOURCES = test_mutex.c common-states.c
 
-test_looplen_SOURCES = test_looplen.c
+test_looplen_SOURCES = test_looplen.c common-states.c
 
-test_urcu_gc_SOURCES = test_urcu_gc.c
+test_urcu_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_gc_LDADD = $(URCU_LIB)
 
-test_urcu_signal_gc_SOURCES = test_urcu_gc.c
+test_urcu_signal_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_signal_gc_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_gc_CFLAGS = -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_mb_gc_SOURCES = test_urcu_gc.c
+test_urcu_mb_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_mb_gc_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_gc_CFLAGS = -DRCU_MB $(AM_CFLAGS)
 
-test_urcu_qsbr_gc_SOURCES = test_urcu_qsbr_gc.c
+test_urcu_qsbr_gc_SOURCES = test_urcu_qsbr_gc.c common-states.c
 test_urcu_qsbr_gc_LDADD = $(URCU_QSBR_LIB)
 
-test_urcu_qsbr_lgc_SOURCES = test_urcu_qsbr_gc.c
+test_urcu_qsbr_lgc_SOURCES = test_urcu_qsbr_gc.c common-states.c
 test_urcu_qsbr_lgc_LDADD = $(URCU_QSBR_LIB)
 test_urcu_qsbr_lgc_CFLAGS = -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_lgc_SOURCES = test_urcu_gc.c
+test_urcu_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_lgc_LDADD = $(URCU_LIB)
 test_urcu_lgc_CFLAGS = -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_signal_lgc_SOURCES = test_urcu_gc.c
+test_urcu_signal_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_signal_lgc_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_lgc_CFLAGS = -DRCU_SIGNAL -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_mb_lgc_SOURCES = test_urcu_gc.c
+test_urcu_mb_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_mb_lgc_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_lgc_CFLAGS = -DTEST_LOCAL_GC -DRCU_MB $(AM_CFLAGS)
 
-test_urcu_qsbr_dynamic_link_SOURCES = test_urcu_qsbr.c
+test_urcu_qsbr_dynamic_link_SOURCES = test_urcu_qsbr.c common-states.c
 test_urcu_qsbr_dynamic_link_LDADD = $(URCU_QSBR_LIB)
 test_urcu_qsbr_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_defer_SOURCES = test_urcu_defer.c
+test_urcu_defer_SOURCES = test_urcu_defer.c common-states.c
 test_urcu_defer_LDADD = $(URCU_LIB)
 
 test_cycles_per_loop_SOURCES = test_cycles_per_loop.c
 
-test_urcu_assign_SOURCES = test_urcu_assign.c
+test_urcu_assign_SOURCES = test_urcu_assign.c common-states.c
 test_urcu_assign_LDADD = $(URCU_LIB)
 
-test_urcu_assign_dynamic_link_SOURCES = test_urcu_assign.c
+test_urcu_assign_dynamic_link_SOURCES = test_urcu_assign.c common-states.c
 test_urcu_assign_dynamic_link_LDADD = $(URCU_LIB)
 test_urcu_assign_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_bp_SOURCES = test_urcu_bp.c
+test_urcu_bp_SOURCES = test_urcu_bp.c common-states.c
 test_urcu_bp_LDADD = $(URCU_BP_LIB)
 
-test_urcu_bp_dynamic_link_SOURCES = test_urcu_bp.c
+test_urcu_bp_dynamic_link_SOURCES = test_urcu_bp.c common-states.c
 test_urcu_bp_dynamic_link_LDADD = $(URCU_BP_LIB)
 test_urcu_bp_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_lfq_SOURCES = test_urcu_lfq.c
+test_urcu_lfq_SOURCES = test_urcu_lfq.c common-states.c
 test_urcu_lfq_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfq_dynlink_SOURCES = test_urcu_lfq.c
+test_urcu_lfq_dynlink_SOURCES = test_urcu_lfq.c common-states.c
 test_urcu_lfq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfq_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_wfq_SOURCES = test_urcu_wfq.c
+test_urcu_wfq_SOURCES = test_urcu_wfq.c common-states.c
 test_urcu_wfq_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfq_dynlink_SOURCES = test_urcu_wfq.c
+test_urcu_wfq_dynlink_SOURCES = test_urcu_wfq.c common-states.c
 test_urcu_wfq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfq_dynlink_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfcq_SOURCES = test_urcu_wfcq.c
+test_urcu_wfcq_SOURCES = test_urcu_wfcq.c common-states.c
 test_urcu_wfcq_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfcq_dynlink_SOURCES = test_urcu_wfcq.c
+test_urcu_wfcq_dynlink_SOURCES = test_urcu_wfcq.c common-states.c
 test_urcu_wfcq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfcq_dynlink_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_lfs_SOURCES = test_urcu_lfs.c
+test_urcu_lfs_SOURCES = test_urcu_lfs.c common-states.c
 test_urcu_lfs_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_rcu_SOURCES = test_urcu_lfs_rcu.c
+test_urcu_lfs_rcu_SOURCES = test_urcu_lfs_rcu.c common-states.c
 test_urcu_lfs_rcu_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_dynlink_SOURCES = test_urcu_lfs.c
+test_urcu_lfs_dynlink_SOURCES = test_urcu_lfs.c common-states.c
 test_urcu_lfs_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfs_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_rcu_dynlink_SOURCES = test_urcu_lfs_rcu.c
+test_urcu_lfs_rcu_dynlink_SOURCES = test_urcu_lfs_rcu.c common-states.c
 test_urcu_lfs_rcu_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfs_rcu_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_wfs_SOURCES = test_urcu_wfs.c
+test_urcu_wfs_SOURCES = test_urcu_wfs.c common-states.c
 test_urcu_wfs_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfs_dynlink_SOURCES = test_urcu_wfs.c
+test_urcu_wfs_dynlink_SOURCES = test_urcu_wfs.c common-states.c
 test_urcu_wfs_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfs_dynlink_LDADD = $(URCU_COMMON_LIB)
 
 test_urcu_hash_SOURCES = test_urcu_hash.c test_urcu_hash.h \
-		test_urcu_hash_rw.c test_urcu_hash_unique.c
+		test_urcu_hash_rw.c test_urcu_hash_unique.c common-states.c
 test_urcu_hash_CFLAGS = -DRCU_QSBR $(AM_CFLAGS)
 test_urcu_hash_LDADD = $(URCU_QSBR_LIB) $(URCU_COMMON_LIB) $(URCU_CDS_LIB)
 
tests/benchmark/common-states.c new file +1 -0
new file mode 100644
index 0000000..6e70351
@@ -0,0 +1 @@
+volatile int _test_go = 0, _test_stop = 0;
tests/benchmark/common-states.h new file +51 -0

51 lines not on this page · raw message

tests/benchmark/test_mutex.c +3 -29
index 55f7c38..145139c 100644
@@ -49,8 +49,6 @@ struct test_array {
 
 static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -111,19 +109,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -147,9 +132,7 @@ void *thr_reader(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int v;
@@ -182,10 +165,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		pthread_mutex_lock(&lock);
@@ -325,13 +305,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_perthreadlock.c +3 -29
index 47a512c..bf468eb 100644
@@ -53,8 +53,6 @@ struct per_thread_lock {
 
 static struct per_thread_lock *per_thread_lock;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -117,19 +115,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -175,9 +160,7 @@ void *thr_reader(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int v;
@@ -211,10 +194,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		for (tidx = 0; tidx < (long)nr_readers; tidx++) {
@@ -359,13 +339,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_rwlock.c +3 -29
index 6908ea4..f5099e8 100644
@@ -53,8 +53,6 @@ struct test_array {
  */
 pthread_rwlock_t lock;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -116,19 +114,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -147,9 +132,7 @@ void *thr_reader(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int a, ret;
@@ -194,10 +177,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		int ret;
@@ -355,13 +335,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu.c +3 -30
index ea849fa..b89513b 100644
@@ -44,8 +44,6 @@
 #endif
 #include <urcu.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -107,19 +105,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -142,10 +127,7 @@ void *thr_reader(void *_count)
 	rcu_register_thread();
 	urcu_posix_assert(!rcu_read_ongoing());
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -186,10 +168,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -337,13 +316,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_assign.c +3 -30
index 88889a8..e83b05e 100644
@@ -48,8 +48,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -111,19 +109,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -201,10 +186,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -240,10 +222,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_copy_mutex_lock();
@@ -394,13 +373,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_bp.c +3 -30
index 6f8c59d..c3b00f1 100644
@@ -44,8 +44,6 @@
 #endif
 #include <urcu-bp.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -107,19 +105,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -142,10 +127,7 @@ void *thr_reader(void *_count)
 	rcu_register_thread();
 	urcu_posix_assert(!rcu_read_ongoing());
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -182,10 +164,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -332,13 +311,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_defer.c +3 -30
index e948ebf..c501f60 100644
@@ -49,8 +49,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -112,19 +110,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -149,10 +134,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -203,10 +185,7 @@ void *thr_writer(void *data)
 		exit(-1);
 	}
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(*new));
@@ -359,13 +338,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_gc.c +4 -30
index f14f728..1cbee44 100644
@@ -33,6 +33,7 @@
 #include <urcu/arch.h>
 #include <urcu/assert.h>
 #include <urcu/tls-compat.h>
+#include <urcu/uatomic.h>
 #include "thread-id.h"
 #include "../common/debug-yield.h"
 
@@ -48,8 +49,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -120,19 +119,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -157,10 +143,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -231,10 +214,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 #ifndef TEST_LOCAL_GC
@@ -399,13 +379,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_hash.c +2 -4
index 3574b4c..1a3087e 100644
@@ -96,8 +96,6 @@ DEFINE_URCU_TLS(unsigned long, lookup_ok);
 
 struct cds_lfht *test_ht;
 
-volatile int test_go, test_stop;
-
 unsigned long wdelay;
 
 unsigned long duration;
@@ -649,14 +647,14 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	remain = duration;
 	do {
 		remain = sleep(remain);
 	} while (remain > 0);
 
-	test_stop = 1;
+	end_test();
 
 end_pthread_join:
 	for (i_thr = 0; i_thr < nr_readers_created; i_thr++) {
tests/benchmark/test_urcu_hash.h +0 -15
index 47b2ae3..73a0a6d 100644
@@ -125,8 +125,6 @@ cds_lfht_iter_get_test_node(struct cds_lfht_iter *iter)
 	return to_test_node(cds_lfht_iter_get_node(iter));
 }
 
-extern volatile int test_go, test_stop;
-
 extern unsigned long wdelay;
 
 extern unsigned long duration;
@@ -174,19 +172,6 @@ extern pthread_mutex_t affinity_mutex;
 
 void set_affinity(void);
 
-/*
- * returns 0 if test should end.
- */
-static inline int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static inline int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 extern DECLARE_URCU_TLS(unsigned long long, nr_writes);
 extern DECLARE_URCU_TLS(unsigned long long, nr_reads);
 
tests/benchmark/test_urcu_hash_rw.c +2 -8
index 862a6f0..087e869 100644
@@ -73,10 +73,7 @@ void *test_hash_rw_thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -133,10 +130,7 @@ void *test_hash_rw_thr_writer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfht_node *ret_node = NULL;
tests/benchmark/test_urcu_hash_unique.c +2 -8
index de7c427..90c0e19 100644
@@ -71,10 +71,7 @@ void *test_hash_unique_thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct lfht_test_node *node;
@@ -136,10 +133,7 @@ void *test_hash_unique_thr_writer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		/*
tests/benchmark/test_urcu_lfq.c +6 -14
index 490e8b0..50c4211 100644
@@ -47,8 +47,6 @@
 #include <urcu.h>
 #include <urcu/cds.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -110,12 +108,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -146,10 +144,7 @@ void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -202,10 +197,7 @@ void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfq_node_rcu *qnode;
@@ -375,7 +367,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -385,7 +377,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_lfs.c +6 -14
index 52239e0..48b2b23 100644
@@ -59,8 +59,6 @@ enum test_sync {
 
 static enum test_sync test_sync;
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -124,12 +122,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -159,10 +157,7 @@ static void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -261,10 +256,7 @@ static void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	urcu_posix_assert(test_pop || test_pop_all);
 
@@ -459,7 +451,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -469,7 +461,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_lfs_rcu.c +6 -14
index 7975faf..ae3dff4 100644
@@ -51,8 +51,6 @@
 
 #include <urcu/cds.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -114,12 +112,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -150,10 +148,7 @@ void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -205,10 +200,7 @@ void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfs_node_rcu *snode;
@@ -377,7 +369,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -387,7 +379,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_qsbr.c +3 -30
index 1ea369c..295e9db 100644
@@ -44,8 +44,6 @@
 #endif
 #include "urcu-qsbr.h"
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -106,19 +104,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -145,10 +130,7 @@ void *thr_reader(void *_count)
 	urcu_posix_assert(!rcu_read_ongoing());
 	rcu_thread_online();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -192,10 +174,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -343,13 +322,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_qsbr_gc.c +4 -30
index 8877a82..163405d 100644
@@ -33,6 +33,7 @@
 #include <urcu/arch.h>
 #include <urcu/assert.h>
 #include <urcu/tls-compat.h>
+#include <urcu/uatomic.h>
 #include "thread-id.h"
 #include "../common/debug-yield.h"
 
@@ -46,8 +47,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -118,19 +117,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -154,10 +140,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		_rcu_read_lock();
@@ -231,10 +214,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 #ifndef TEST_LOCAL_GC
@@ -399,13 +379,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_wfcq.c +8 -14
index 2c6e0fd..542a13a 100644
@@ -56,7 +56,7 @@ static enum test_sync test_sync;
 
 static int test_force_sync;
 
-static volatile int test_go, test_stop_enqueue, test_stop_dequeue;
+static volatile int test_stop_enqueue, test_stop_dequeue;
 
 static unsigned long rduration;
 
@@ -122,12 +122,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop_dequeue;
+	return !uatomic_load(&test_stop_dequeue, CMM_RELAXED);
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop_enqueue;
+	return !uatomic_load(&test_stop_enqueue, CMM_RELAXED);
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -155,10 +155,7 @@ static void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfcq_node *node = malloc(sizeof(*node));
@@ -266,10 +263,7 @@ static void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		if (test_dequeue && test_splice) {
@@ -482,7 +476,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -492,7 +486,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_enqueue = 1;
+	uatomic_store(&test_stop_enqueue, 1, CMM_RELEASE);
 
 	if (test_wait_empty) {
 		while (nr_enqueuers != uatomic_read(&test_enqueue_stopped)) {
@@ -503,7 +497,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_dequeue = 1;
+	uatomic_store(&test_stop_dequeue, 1, CMM_RELAXED);
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_wfq.c +6 -14
index 8381160..2d8de87 100644
@@ -51,8 +51,6 @@
 #include <urcu.h>
 #include <urcu/wfqueue.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -114,12 +112,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -143,10 +141,7 @@ void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfq_node *node = malloc(sizeof(*node));
@@ -185,10 +180,7 @@ void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfq_node *node = cds_wfq_dequeue_blocking(&q);
@@ -343,7 +335,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -353,7 +345,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_wfs.c +8 -14
index c285feb..d1a4afc 100644
@@ -59,7 +59,7 @@ static enum test_sync test_sync;
 
 static int test_force_sync;
 
-static volatile int test_go, test_stop_enqueue, test_stop_dequeue;
+static volatile int test_stop_enqueue, test_stop_dequeue;
 
 static unsigned long rduration;
 
@@ -125,12 +125,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop_dequeue;
+	return !uatomic_load(&test_stop_dequeue, CMM_RELAXED);
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop_enqueue;
+	return !uatomic_load(&test_stop_enqueue, CMM_RELAXED);
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -157,10 +157,7 @@ static void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfs_node *node = malloc(sizeof(*node));
@@ -250,10 +247,7 @@ static void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	urcu_posix_assert(test_pop || test_pop_all);
 
@@ -469,7 +463,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -479,7 +473,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_enqueue = 1;
+	uatomic_store(&test_stop_enqueue, 1, CMM_RELEASE);
 
 	if (test_wait_empty) {
 		while (nr_enqueuers != uatomic_read(&test_enqueue_stopped)) {
@@ -490,7 +484,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_dequeue = 1;
+	uatomic_store(&test_stop_dequeue, 1, CMM_RELAXED);
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#64 lore raw

Re: [lttng-dev] [PATCH v2 08/12] benchmark: Use uatomic for accessing global states

Quoted 7 lines · Olivier Dion

Global states accesses were protected via memory barriers. Use the uatomic API with the CMM memory model so that TSAN can understand the ordering imposed by the synchronization flags.

Change-Id I1bf5702c5ac470f308c478effe39e424a3158060

This does look more organized!

Thanx, Paul

Quoted 1,612 lines
---
 tests/benchmark/Makefile.am             | 91 +++++++++++++------------
 tests/benchmark/common-states.c         |  1 +
 tests/benchmark/common-states.h         | 51 ++++++++++++++
 tests/benchmark/test_mutex.c            | 32 +--------
 tests/benchmark/test_perthreadlock.c    | 32 +--------
 tests/benchmark/test_rwlock.c           | 32 +--------
 tests/benchmark/test_urcu.c             | 33 +--------
 tests/benchmark/test_urcu_assign.c      | 33 +--------
 tests/benchmark/test_urcu_bp.c          | 33 +--------
 tests/benchmark/test_urcu_defer.c       | 33 +--------
 tests/benchmark/test_urcu_gc.c          | 34 ++-------
 tests/benchmark/test_urcu_hash.c        |  6 +-
 tests/benchmark/test_urcu_hash.h        | 15 ----
 tests/benchmark/test_urcu_hash_rw.c     | 10 +--
 tests/benchmark/test_urcu_hash_unique.c | 10 +--
 tests/benchmark/test_urcu_lfq.c         | 20 ++----
 tests/benchmark/test_urcu_lfs.c         | 20 ++----
 tests/benchmark/test_urcu_lfs_rcu.c     | 20 ++----
 tests/benchmark/test_urcu_qsbr.c        | 33 +--------
 tests/benchmark/test_urcu_qsbr_gc.c     | 34 ++-------
 tests/benchmark/test_urcu_wfcq.c        | 22 +++---
 tests/benchmark/test_urcu_wfq.c         | 20 ++----
 tests/benchmark/test_urcu_wfs.c         | 22 +++---
 23 files changed, 177 insertions(+), 460 deletions(-)
 create mode 100644 tests/benchmark/common-states.c
 create mode 100644 tests/benchmark/common-states.h
tests/benchmark/Makefile.am +47 -44
index c53e025..a7f91c2 100644
@@ -1,4 +1,5 @@
 AM_CPPFLAGS += -I$(top_srcdir)/src -I$(top_srcdir)/tests/common
+AM_CPPFLAGS += -include $(top_srcdir)/tests/benchmark/common-states.h
 
 TEST_EXTENSIONS = .tap
 TAP_LOG_DRIVER_FLAGS = --merge --comments
@@ -7,6 +8,8 @@ TAP_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' \
 	URCU_TESTS_BUILDDIR='$(abs_top_builddir)/tests' \
 	$(SHELL) $(top_srcdir)/tests/utils/tap-driver.sh
 
+noinst_HEADERS = common-states.h
+
 SCRIPT_LIST = \
 	runpaul-phase1.sh \
 	runpaul-phase2.sh \
@@ -61,163 +64,163 @@ URCU_CDS_LIB=$(top_builddir)/src/liburcu-cds.la
 
 DEBUG_YIELD_LIB=$(builddir)/../common/libdebug-yield.la
 
-test_urcu_SOURCES = test_urcu.c
+test_urcu_SOURCES = test_urcu.c common-states.c
 test_urcu_LDADD = $(URCU_LIB)
 
-test_urcu_dynamic_link_SOURCES = test_urcu.c
+test_urcu_dynamic_link_SOURCES = test_urcu.c common-states.c
 test_urcu_dynamic_link_LDADD = $(URCU_LIB)
 test_urcu_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_timing_SOURCES = test_urcu_timing.c
+test_urcu_timing_SOURCES = test_urcu_timing.c common-states.c
 test_urcu_timing_LDADD = $(URCU_LIB)
 
-test_urcu_yield_SOURCES = test_urcu.c
+test_urcu_yield_SOURCES = test_urcu.c common-states.c
 test_urcu_yield_LDADD = $(URCU_LIB) $(DEBUG_YIELD_LIB)
 test_urcu_yield_CFLAGS = -DDEBUG_YIELD $(AM_CFLAGS)
 
 
-test_urcu_qsbr_SOURCES = test_urcu_qsbr.c
+test_urcu_qsbr_SOURCES = test_urcu_qsbr.c common-states.c
 test_urcu_qsbr_LDADD = $(URCU_QSBR_LIB)
 
-test_urcu_qsbr_timing_SOURCES = test_urcu_qsbr_timing.c
+test_urcu_qsbr_timing_SOURCES = test_urcu_qsbr_timing.c common-states.c
 test_urcu_qsbr_timing_LDADD = $(URCU_QSBR_LIB)
 
 
-test_urcu_mb_SOURCES = test_urcu.c
+test_urcu_mb_SOURCES = test_urcu.c common-states.c
 test_urcu_mb_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_CFLAGS = -DRCU_MB $(AM_CFLAGS)
 
 
-test_urcu_signal_SOURCES = test_urcu.c
+test_urcu_signal_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_CFLAGS = -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_signal_dynamic_link_SOURCES = test_urcu.c
+test_urcu_signal_dynamic_link_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_dynamic_link_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_dynamic_link_CFLAGS = -DRCU_SIGNAL -DDYNAMIC_LINK_TEST \
 					$(AM_CFLAGS)
 
-test_urcu_signal_timing_SOURCES = test_urcu_timing.c
+test_urcu_signal_timing_SOURCES = test_urcu_timing.c common-states.c
 test_urcu_signal_timing_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_timing_CFLAGS= -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_signal_yield_SOURCES = test_urcu.c
+test_urcu_signal_yield_SOURCES = test_urcu.c common-states.c
 test_urcu_signal_yield_LDADD = $(URCU_SIGNAL_LIB) $(DEBUG_YIELD_LIB)
 test_urcu_signal_yield_CFLAGS = -DRCU_SIGNAL -DDEBUG_YIELD $(AM_CFLAGS)
 
-test_rwlock_timing_SOURCES = test_rwlock_timing.c
+test_rwlock_timing_SOURCES = test_rwlock_timing.c common-states.c
 test_rwlock_timing_LDADD = $(URCU_SIGNAL_LIB)
 
-test_rwlock_SOURCES = test_rwlock.c
+test_rwlock_SOURCES = test_rwlock.c common-states.c
 test_rwlock_LDADD = $(URCU_SIGNAL_LIB)
 
-test_perthreadlock_timing_SOURCES = test_perthreadlock_timing.c
+test_perthreadlock_timing_SOURCES = test_perthreadlock_timing.c common-states.c
 test_perthreadlock_timing_LDADD = $(URCU_SIGNAL_LIB)
 
-test_perthreadlock_SOURCES = test_perthreadlock.c
+test_perthreadlock_SOURCES = test_perthreadlock.c common-states.c
 test_perthreadlock_LDADD = $(URCU_SIGNAL_LIB)
 
-test_mutex_SOURCES = test_mutex.c
+test_mutex_SOURCES = test_mutex.c common-states.c
 
-test_looplen_SOURCES = test_looplen.c
+test_looplen_SOURCES = test_looplen.c common-states.c
 
-test_urcu_gc_SOURCES = test_urcu_gc.c
+test_urcu_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_gc_LDADD = $(URCU_LIB)
 
-test_urcu_signal_gc_SOURCES = test_urcu_gc.c
+test_urcu_signal_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_signal_gc_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_gc_CFLAGS = -DRCU_SIGNAL $(AM_CFLAGS)
 
-test_urcu_mb_gc_SOURCES = test_urcu_gc.c
+test_urcu_mb_gc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_mb_gc_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_gc_CFLAGS = -DRCU_MB $(AM_CFLAGS)
 
-test_urcu_qsbr_gc_SOURCES = test_urcu_qsbr_gc.c
+test_urcu_qsbr_gc_SOURCES = test_urcu_qsbr_gc.c common-states.c
 test_urcu_qsbr_gc_LDADD = $(URCU_QSBR_LIB)
 
-test_urcu_qsbr_lgc_SOURCES = test_urcu_qsbr_gc.c
+test_urcu_qsbr_lgc_SOURCES = test_urcu_qsbr_gc.c common-states.c
 test_urcu_qsbr_lgc_LDADD = $(URCU_QSBR_LIB)
 test_urcu_qsbr_lgc_CFLAGS = -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_lgc_SOURCES = test_urcu_gc.c
+test_urcu_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_lgc_LDADD = $(URCU_LIB)
 test_urcu_lgc_CFLAGS = -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_signal_lgc_SOURCES = test_urcu_gc.c
+test_urcu_signal_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_signal_lgc_LDADD = $(URCU_SIGNAL_LIB)
 test_urcu_signal_lgc_CFLAGS = -DRCU_SIGNAL -DTEST_LOCAL_GC $(AM_CFLAGS)
 
-test_urcu_mb_lgc_SOURCES = test_urcu_gc.c
+test_urcu_mb_lgc_SOURCES = test_urcu_gc.c common-states.c
 test_urcu_mb_lgc_LDADD = $(URCU_MB_LIB)
 test_urcu_mb_lgc_CFLAGS = -DTEST_LOCAL_GC -DRCU_MB $(AM_CFLAGS)
 
-test_urcu_qsbr_dynamic_link_SOURCES = test_urcu_qsbr.c
+test_urcu_qsbr_dynamic_link_SOURCES = test_urcu_qsbr.c common-states.c
 test_urcu_qsbr_dynamic_link_LDADD = $(URCU_QSBR_LIB)
 test_urcu_qsbr_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_defer_SOURCES = test_urcu_defer.c
+test_urcu_defer_SOURCES = test_urcu_defer.c common-states.c
 test_urcu_defer_LDADD = $(URCU_LIB)
 
 test_cycles_per_loop_SOURCES = test_cycles_per_loop.c
 
-test_urcu_assign_SOURCES = test_urcu_assign.c
+test_urcu_assign_SOURCES = test_urcu_assign.c common-states.c
 test_urcu_assign_LDADD = $(URCU_LIB)
 
-test_urcu_assign_dynamic_link_SOURCES = test_urcu_assign.c
+test_urcu_assign_dynamic_link_SOURCES = test_urcu_assign.c common-states.c
 test_urcu_assign_dynamic_link_LDADD = $(URCU_LIB)
 test_urcu_assign_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_bp_SOURCES = test_urcu_bp.c
+test_urcu_bp_SOURCES = test_urcu_bp.c common-states.c
 test_urcu_bp_LDADD = $(URCU_BP_LIB)
 
-test_urcu_bp_dynamic_link_SOURCES = test_urcu_bp.c
+test_urcu_bp_dynamic_link_SOURCES = test_urcu_bp.c common-states.c
 test_urcu_bp_dynamic_link_LDADD = $(URCU_BP_LIB)
 test_urcu_bp_dynamic_link_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 
-test_urcu_lfq_SOURCES = test_urcu_lfq.c
+test_urcu_lfq_SOURCES = test_urcu_lfq.c common-states.c
 test_urcu_lfq_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfq_dynlink_SOURCES = test_urcu_lfq.c
+test_urcu_lfq_dynlink_SOURCES = test_urcu_lfq.c common-states.c
 test_urcu_lfq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfq_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_wfq_SOURCES = test_urcu_wfq.c
+test_urcu_wfq_SOURCES = test_urcu_wfq.c common-states.c
 test_urcu_wfq_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfq_dynlink_SOURCES = test_urcu_wfq.c
+test_urcu_wfq_dynlink_SOURCES = test_urcu_wfq.c common-states.c
 test_urcu_wfq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfq_dynlink_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfcq_SOURCES = test_urcu_wfcq.c
+test_urcu_wfcq_SOURCES = test_urcu_wfcq.c common-states.c
 test_urcu_wfcq_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfcq_dynlink_SOURCES = test_urcu_wfcq.c
+test_urcu_wfcq_dynlink_SOURCES = test_urcu_wfcq.c common-states.c
 test_urcu_wfcq_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfcq_dynlink_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_lfs_SOURCES = test_urcu_lfs.c
+test_urcu_lfs_SOURCES = test_urcu_lfs.c common-states.c
 test_urcu_lfs_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_rcu_SOURCES = test_urcu_lfs_rcu.c
+test_urcu_lfs_rcu_SOURCES = test_urcu_lfs_rcu.c common-states.c
 test_urcu_lfs_rcu_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_dynlink_SOURCES = test_urcu_lfs.c
+test_urcu_lfs_dynlink_SOURCES = test_urcu_lfs.c common-states.c
 test_urcu_lfs_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfs_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_lfs_rcu_dynlink_SOURCES = test_urcu_lfs_rcu.c
+test_urcu_lfs_rcu_dynlink_SOURCES = test_urcu_lfs_rcu.c common-states.c
 test_urcu_lfs_rcu_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_lfs_rcu_dynlink_LDADD = $(URCU_LIB) $(URCU_CDS_LIB)
 
-test_urcu_wfs_SOURCES = test_urcu_wfs.c
+test_urcu_wfs_SOURCES = test_urcu_wfs.c common-states.c
 test_urcu_wfs_LDADD = $(URCU_COMMON_LIB)
 
-test_urcu_wfs_dynlink_SOURCES = test_urcu_wfs.c
+test_urcu_wfs_dynlink_SOURCES = test_urcu_wfs.c common-states.c
 test_urcu_wfs_dynlink_CFLAGS = -DDYNAMIC_LINK_TEST $(AM_CFLAGS)
 test_urcu_wfs_dynlink_LDADD = $(URCU_COMMON_LIB)
 
 test_urcu_hash_SOURCES = test_urcu_hash.c test_urcu_hash.h \
-		test_urcu_hash_rw.c test_urcu_hash_unique.c
+		test_urcu_hash_rw.c test_urcu_hash_unique.c common-states.c
 test_urcu_hash_CFLAGS = -DRCU_QSBR $(AM_CFLAGS)
 test_urcu_hash_LDADD = $(URCU_QSBR_LIB) $(URCU_COMMON_LIB) $(URCU_CDS_LIB)
 
tests/benchmark/common-states.c new file +1 -0
new file mode 100644
index 0000000..6e70351
@@ -0,0 +1 @@
+volatile int _test_go = 0, _test_stop = 0;
tests/benchmark/common-states.h new file +51 -0
new file mode 100644
index 0000000..dfbbfe5
@@ -0,0 +1,51 @@
+/* Common states for benchmarks. */
+
+#include <unistd.h>
+
+#include <urcu/uatomic.h>
+
+extern volatile int _test_go, _test_stop;
+
+static inline void complete_sleep(unsigned int seconds)
+{
+	while (seconds != 0) {
+		seconds = sleep(seconds);
+	}
+}
+
+static inline void begin_test(void)
+{
+	uatomic_store(&_test_go, 1, CMM_RELEASE);
+}
+
+static inline void end_test(void)
+{
+	uatomic_store(&_test_stop, 1, CMM_RELAXED);
+}
+
+static inline void test_for(unsigned int duration)
+{
+	begin_test();
+	complete_sleep(duration);
+	end_test();
+}
+
+static inline void wait_until_go(void)
+{
+	while (!uatomic_load(&_test_go, CMM_ACQUIRE))
+	{
+	}
+}
+
+/*
+ * returns 0 if test should end.
+ */
+static inline int test_duration_write(void)
+{
+	return !uatomic_load(&_test_stop, CMM_RELAXED);
+}
+
+static inline int test_duration_read(void)
+{
+	return !uatomic_load(&_test_stop, CMM_RELAXED);
+}
tests/benchmark/test_mutex.c +3 -29
index 55f7c38..145139c 100644
@@ -49,8 +49,6 @@ struct test_array {
 
 static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -111,19 +109,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -147,9 +132,7 @@ void *thr_reader(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int v;
@@ -182,10 +165,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		pthread_mutex_lock(&lock);
@@ -325,13 +305,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_perthreadlock.c +3 -29
index 47a512c..bf468eb 100644
@@ -53,8 +53,6 @@ struct per_thread_lock {
 
 static struct per_thread_lock *per_thread_lock;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -117,19 +115,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -175,9 +160,7 @@ void *thr_reader(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int v;
@@ -211,10 +194,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		for (tidx = 0; tidx < (long)nr_readers; tidx++) {
@@ -359,13 +339,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_rwlock.c +3 -29
index 6908ea4..f5099e8 100644
@@ -53,8 +53,6 @@ struct test_array {
  */
 pthread_rwlock_t lock;
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static volatile struct test_array test_array = { 8 };
@@ -116,19 +114,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -147,9 +132,7 @@ void *thr_reader(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
+	wait_until_go();
 
 	for (;;) {
 		int a, ret;
@@ -194,10 +177,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		int ret;
@@ -355,13 +335,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu.c +3 -30
index ea849fa..b89513b 100644
@@ -44,8 +44,6 @@
 #endif
 #include <urcu.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -107,19 +105,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -142,10 +127,7 @@ void *thr_reader(void *_count)
 	rcu_register_thread();
 	urcu_posix_assert(!rcu_read_ongoing());
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -186,10 +168,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -337,13 +316,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_assign.c +3 -30
index 88889a8..e83b05e 100644
@@ -48,8 +48,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -111,19 +109,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -201,10 +186,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -240,10 +222,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_copy_mutex_lock();
@@ -394,13 +373,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_bp.c +3 -30
index 6f8c59d..c3b00f1 100644
@@ -44,8 +44,6 @@
 #endif
 #include <urcu-bp.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -107,19 +105,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -142,10 +127,7 @@ void *thr_reader(void *_count)
 	rcu_register_thread();
 	urcu_posix_assert(!rcu_read_ongoing());
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -182,10 +164,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -332,13 +311,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_defer.c +3 -30
index e948ebf..c501f60 100644
@@ -49,8 +49,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -112,19 +110,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -149,10 +134,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -203,10 +185,7 @@ void *thr_writer(void *data)
 		exit(-1);
 	}
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(*new));
@@ -359,13 +338,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_gc.c +4 -30
index f14f728..1cbee44 100644
@@ -33,6 +33,7 @@
 #include <urcu/arch.h>
 #include <urcu/assert.h>
 #include <urcu/tls-compat.h>
+#include <urcu/uatomic.h>
 #include "thread-id.h"
 #include "../common/debug-yield.h"
 
@@ -48,8 +49,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -120,19 +119,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -157,10 +143,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -231,10 +214,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 #ifndef TEST_LOCAL_GC
@@ -399,13 +379,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_hash.c +2 -4
index 3574b4c..1a3087e 100644
@@ -96,8 +96,6 @@ DEFINE_URCU_TLS(unsigned long, lookup_ok);
 
 struct cds_lfht *test_ht;
 
-volatile int test_go, test_stop;
-
 unsigned long wdelay;
 
 unsigned long duration;
@@ -649,14 +647,14 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	remain = duration;
 	do {
 		remain = sleep(remain);
 	} while (remain > 0);
 
-	test_stop = 1;
+	end_test();
 
 end_pthread_join:
 	for (i_thr = 0; i_thr < nr_readers_created; i_thr++) {
tests/benchmark/test_urcu_hash.h +0 -15
index 47b2ae3..73a0a6d 100644
@@ -125,8 +125,6 @@ cds_lfht_iter_get_test_node(struct cds_lfht_iter *iter)
 	return to_test_node(cds_lfht_iter_get_node(iter));
 }
 
-extern volatile int test_go, test_stop;
-
 extern unsigned long wdelay;
 
 extern unsigned long duration;
@@ -174,19 +172,6 @@ extern pthread_mutex_t affinity_mutex;
 
 void set_affinity(void);
 
-/*
- * returns 0 if test should end.
- */
-static inline int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static inline int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 extern DECLARE_URCU_TLS(unsigned long long, nr_writes);
 extern DECLARE_URCU_TLS(unsigned long long, nr_reads);
 
tests/benchmark/test_urcu_hash_rw.c +2 -8
index 862a6f0..087e869 100644
@@ -73,10 +73,7 @@ void *test_hash_rw_thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -133,10 +130,7 @@ void *test_hash_rw_thr_writer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfht_node *ret_node = NULL;
tests/benchmark/test_urcu_hash_unique.c +2 -8
index de7c427..90c0e19 100644
@@ -71,10 +71,7 @@ void *test_hash_unique_thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct lfht_test_node *node;
@@ -136,10 +133,7 @@ void *test_hash_unique_thr_writer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		/*
tests/benchmark/test_urcu_lfq.c +6 -14
index 490e8b0..50c4211 100644
@@ -47,8 +47,6 @@
 #include <urcu.h>
 #include <urcu/cds.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -110,12 +108,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -146,10 +144,7 @@ void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -202,10 +197,7 @@ void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfq_node_rcu *qnode;
@@ -375,7 +367,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -385,7 +377,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_lfs.c +6 -14
index 52239e0..48b2b23 100644
@@ -59,8 +59,6 @@ enum test_sync {
 
 static enum test_sync test_sync;
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -124,12 +122,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -159,10 +157,7 @@ static void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -261,10 +256,7 @@ static void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	urcu_posix_assert(test_pop || test_pop_all);
 
@@ -459,7 +451,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -469,7 +461,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_lfs_rcu.c +6 -14
index 7975faf..ae3dff4 100644
@@ -51,8 +51,6 @@
 
 #include <urcu/cds.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -114,12 +112,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -150,10 +148,7 @@ void *thr_enqueuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct test *node = malloc(sizeof(*node));
@@ -205,10 +200,7 @@ void *thr_dequeuer(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_lfs_node_rcu *snode;
@@ -377,7 +369,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -387,7 +379,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_qsbr.c +3 -30
index 1ea369c..295e9db 100644
@@ -44,8 +44,6 @@
 #endif
 #include "urcu-qsbr.h"
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static int *test_rcu_pointer;
@@ -106,19 +104,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -145,10 +130,7 @@ void *thr_reader(void *_count)
 	urcu_posix_assert(!rcu_read_ongoing());
 	rcu_thread_online();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		rcu_read_lock();
@@ -192,10 +174,7 @@ void *thr_writer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		new = malloc(sizeof(int));
@@ -343,13 +322,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_qsbr_gc.c +4 -30
index 8877a82..163405d 100644
@@ -33,6 +33,7 @@
 #include <urcu/arch.h>
 #include <urcu/assert.h>
 #include <urcu/tls-compat.h>
+#include <urcu/uatomic.h>
 #include "thread-id.h"
 #include "../common/debug-yield.h"
 
@@ -46,8 +47,6 @@ struct test_array {
 	int a;
 };
 
-static volatile int test_go, test_stop;
-
 static unsigned long wdelay;
 
 static struct test_array *test_rcu_pointer;
@@ -118,19 +117,6 @@ static void set_affinity(void)
 #endif /* HAVE_SCHED_SETAFFINITY */
 }
 
-/*
- * returns 0 if test should end.
- */
-static int test_duration_write(void)
-{
-	return !test_stop;
-}
-
-static int test_duration_read(void)
-{
-	return !test_stop;
-}
-
 static DEFINE_URCU_TLS(unsigned long long, nr_writes);
 static DEFINE_URCU_TLS(unsigned long long, nr_reads);
 
@@ -154,10 +140,7 @@ void *thr_reader(void *_count)
 
 	rcu_register_thread();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		_rcu_read_lock();
@@ -231,10 +214,7 @@ void *thr_writer(void *data)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 #ifndef TEST_LOCAL_GC
@@ -399,13 +379,7 @@ int main(int argc, char **argv)
 			exit(1);
 	}
 
-	cmm_smp_mb();
-
-	test_go = 1;
-
-	sleep(duration);
-
-	test_stop = 1;
+	test_for(duration);
 
 	for (i_thr = 0; i_thr < nr_readers; i_thr++) {
 		err = pthread_join(tid_reader[i_thr], &tret);
tests/benchmark/test_urcu_wfcq.c +8 -14
index 2c6e0fd..542a13a 100644
@@ -56,7 +56,7 @@ static enum test_sync test_sync;
 
 static int test_force_sync;
 
-static volatile int test_go, test_stop_enqueue, test_stop_dequeue;
+static volatile int test_stop_enqueue, test_stop_dequeue;
 
 static unsigned long rduration;
 
@@ -122,12 +122,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop_dequeue;
+	return !uatomic_load(&test_stop_dequeue, CMM_RELAXED);
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop_enqueue;
+	return !uatomic_load(&test_stop_enqueue, CMM_RELAXED);
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -155,10 +155,7 @@ static void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfcq_node *node = malloc(sizeof(*node));
@@ -266,10 +263,7 @@ static void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		if (test_dequeue && test_splice) {
@@ -482,7 +476,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -492,7 +486,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_enqueue = 1;
+	uatomic_store(&test_stop_enqueue, 1, CMM_RELEASE);
 
 	if (test_wait_empty) {
 		while (nr_enqueuers != uatomic_read(&test_enqueue_stopped)) {
@@ -503,7 +497,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_dequeue = 1;
+	uatomic_store(&test_stop_dequeue, 1, CMM_RELAXED);
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_wfq.c +6 -14
index 8381160..2d8de87 100644
@@ -51,8 +51,6 @@
 #include <urcu.h>
 #include <urcu/wfqueue.h>
 
-static volatile int test_go, test_stop;
-
 static unsigned long rduration;
 
 static unsigned long duration;
@@ -114,12 +112,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop;
+	return test_duration_read();
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop;
+	return test_duration_write();
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -143,10 +141,7 @@ void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfq_node *node = malloc(sizeof(*node));
@@ -185,10 +180,7 @@ void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfq_node *node = cds_wfq_dequeue_blocking(&q);
@@ -343,7 +335,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -353,7 +345,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop = 1;
+	end_test();
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
tests/benchmark/test_urcu_wfs.c +8 -14
index c285feb..d1a4afc 100644
@@ -59,7 +59,7 @@ static enum test_sync test_sync;
 
 static int test_force_sync;
 
-static volatile int test_go, test_stop_enqueue, test_stop_dequeue;
+static volatile int test_stop_enqueue, test_stop_dequeue;
 
 static unsigned long rduration;
 
@@ -125,12 +125,12 @@ static void set_affinity(void)
  */
 static int test_duration_dequeue(void)
 {
-	return !test_stop_dequeue;
+	return !uatomic_load(&test_stop_dequeue, CMM_RELAXED);
 }
 
 static int test_duration_enqueue(void)
 {
-	return !test_stop_enqueue;
+	return !uatomic_load(&test_stop_enqueue, CMM_RELAXED);
 }
 
 static DEFINE_URCU_TLS(unsigned long long, nr_dequeues);
@@ -157,10 +157,7 @@ static void *thr_enqueuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	for (;;) {
 		struct cds_wfs_node *node = malloc(sizeof(*node));
@@ -250,10 +247,7 @@ static void *thr_dequeuer(void *_count)
 
 	set_affinity();
 
-	while (!test_go)
-	{
-	}
-	cmm_smp_mb();
+	wait_until_go();
 
 	urcu_posix_assert(test_pop || test_pop_all);
 
@@ -469,7 +463,7 @@ int main(int argc, char **argv)
 
 	cmm_smp_mb();
 
-	test_go = 1;
+	begin_test();
 
 	for (i_thr = 0; i_thr < duration; i_thr++) {
 		sleep(1);
@@ -479,7 +473,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_enqueue = 1;
+	uatomic_store(&test_stop_enqueue, 1, CMM_RELEASE);
 
 	if (test_wait_empty) {
 		while (nr_enqueuers != uatomic_read(&test_enqueue_stopped)) {
@@ -490,7 +484,7 @@ int main(int argc, char **argv)
 		}
 	}
 
-	test_stop_dequeue = 1;
+	uatomic_store(&test_stop_dequeue, 1, CMM_RELAXED);
 
 	for (i_thr = 0; i_thr < nr_enqueuers; i_thr++) {
 		err = pthread_join(tid_enqueuer[i_thr], &tret);
Signature · 1 line
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#65 lore raw

PATCH v2 9/12 tests/unit/test_build: Quiet unused return value

In reply to Olivier Dion via lttng-dev
Change-Id Ie5a18e0ccc4b1b5ee85c5bd140561cc2ff9e2fbc
---
 tests/unit/test_build.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
tests/unit/test_build.c +4 -4
index f6b667c..702c1ef 100644
@@ -129,10 +129,10 @@ void test_build_rcu_dereference(void)
 	static struct a_clear_struct *clear = NULL;
 	static struct a_clear_struct *const clear_const = NULL;
 
-	rcu_dereference(opaque);
-	rcu_dereference(opaque_const);
-	rcu_dereference(clear);
-	rcu_dereference(clear_const);
+	(void) rcu_dereference(opaque);
+	(void) rcu_dereference(opaque_const);
+	(void) rcu_dereference(clear);
+	(void) rcu_dereference(clear_const);
 }
 
 int main(void)
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#66 lore raw

PATCH v2 10/12 urcu/annotate: Add CMM annotation

In reply to Olivier Dion via lttng-dev

The CMM annotation is highly experimental and not meant to be used by user for now, even though it is exposed in the public API since some parts of the liburcu public API require those annotations.

The main primitive is the cmm_annotate_t which denotes a group of memory operations associated with a memory barrier. A group follows a state machine, starting from the `CMM_ANNOTATE_VOID' state. The following are the only valid transitions:

CMM_ANNOTATE_VOID -> CMM_ANNOTATE_MB (acquire & release MB) CMM_ANNOTATE_VOID -> CMM_ANNOTATE_LOAD (acquire memory) CMM_ANNOTATE_LOAD -> CMM_ANNOTATE_MB (acquire MB)

The macro cmm_annotate_define(name)' can be used to create an annotation object on the stack. The rest of the cmm_annotate_*' macros can be used to change the state of the group after validating that the transition is allowed. Some of these macros also inject TSAN annotations to help it understand the flow of events in the program since it does not currently support thread fence.

Sometime, a single memory access does not need to be associated with a group. In the case, the acquire/release macros variant without the `group' infix can be used to annotate memory accesses.

Note that TSAN can not be used on the liburcu-signal flavor. This is because TSAN hijacks calls to sigaction(3) and places its own handler that will deliver the signal to the application at a synchronization point.

Thus, the usage of TSAN on the signal flavor is undefined behavior. However, there's at least one known behavior which is a deadlock between readers that want to unregister them-self by locking the `rcu_registry_lock' while a synchronize RCU is made on the writer side which has already locked that mutex until all the registered readers execute a memory barrier in a signal handler defined by liburcu-signal. However, TSAN will not call the registered handler while waiting on the mutex. Therefore, the writer spin infinitely on pthread_kill(3p) because the reader simply never complete the handshake.

See the deadlock minimal reproducer below.

Deadlock reproducer:

```
#include <poll.h>
#include <signal.h>

#include <pthread.h>

#define SIGURCU SIGUSR1

static pthread_mutex_t rcu_registry_lock = PTHREAD_MUTEX_INITIALIZER;
static int need_mb = 0;

static void *reader_side(void *nil)
{
	(void) nil;

	pthread_mutex_lock(&rcu_registry_lock);
	pthread_mutex_unlock(&rcu_registry_lock);

	return NULL;
}

static void writer_side(pthread_t reader)
{
	__atomic_store_n(&need_mb, 1, __ATOMIC_RELEASE);
	while (__atomic_load_n(&need_mb, __ATOMIC_ACQUIRE)) {
		pthread_kill(reader, SIGURCU);
		(void) poll(NULL, 0, 1);
	}
	pthread_mutex_unlock(&rcu_registry_lock);

	pthread_join(reader, NULL);
}

static void sigrcu_handler(int signo, siginfo_t *siginfo, void *context)
{
	(void) signo;
	(void) siginfo;
	(void) context;

	__atomic_store_n(&need_mb, 0, __ATOMIC_SEQ_CST);
}

static void install_signal(void)
{
	struct sigaction act;

	act.sa_sigaction = sigrcu_handler;
	act.sa_flags     = SA_SIGINFO | SA_RESTART;

	sigemptyset(&act.sa_mask);

	(void) sigaction(SIGURCU, &act, NULL);
}

int main(void)
{
	pthread_t th;

	install_signal();

	pthread_mutex_lock(&rcu_registry_lock);
	pthread_create(&th, NULL, reader_side, NULL);

	writer_side(th);

	return 0;
}
```
Change-Id I9c234bb311cc0f82ea9dbefdf4fee07047ab93f9
---
 include/Makefile.am               |   1 +
 include/urcu/annotate.h           | 174 ++++++++++++++++++++++++++++++
 include/urcu/arch/generic.h       |  33 +++++-
 include/urcu/compiler.h           |  14 +++
 include/urcu/static/urcu-bp.h     |  12 ++-
 include/urcu/static/urcu-common.h |   8 +-
 include/urcu/static/urcu-mb.h     |  11 +-
 include/urcu/static/urcu-memb.h   |  26 +++--
 include/urcu/static/urcu-qsbr.h   |  29 +++--
 src/rculfhash.c                   |  92 ++++++++++------
 src/urcu-bp.c                     |  17 ++-
 src/urcu-qsbr.c                   |  31 ++++--
 src/urcu-wait.h                   |   9 +-
 src/urcu.c                        |  24 +++--
 14 files changed, 392 insertions(+), 89 deletions(-)
 create mode 100644 include/urcu/annotate.h
include/Makefile.am +1 -0
index b20e56d..24151c1 100644
@@ -1,4 +1,5 @@
 nobase_include_HEADERS = \
+	urcu/annotate.h \
 	urcu/arch/aarch64.h \
 	urcu/arch/alpha.h \
 	urcu/arch/arm.h \
include/urcu/annotate.h new file +174 -0

174 lines not on this page · raw message

include/urcu/arch/generic.h +31 -2
index e292c70..65dedf2 100644
@@ -45,9 +45,38 @@ extern "C" {
 
 #ifdef CONFIG_RCU_USE_ATOMIC_BUILTINS
 
+# ifdef CMM_SANITIZE_THREAD
+/*
+ * This makes TSAN quiet about unsupported thread fence.
+ */
+static inline void _cmm_thread_fence_wrapper(void)
+{
+#   if defined(__clang__)
+#    pragma clang diagnostic push
+#    pragma clang diagnostic ignored "-Wpragmas"
+#    pragma clang diagnostic ignored "-Wunknown-warning-option"
+#    pragma clang diagnostic ignored "-Wtsan"
+#   elif defined(__GNUC__)
+#    pragma GCC diagnostic push
+#    pragma GCC diagnostic ignored "-Wpragmas"
+#    pragma GCC diagnostic ignored "-Wtsan"
+#   endif
+	__atomic_thread_fence(__ATOMIC_SEQ_CST);
+#   if defined(__clang__)
+#    pragma clang diagnostic pop
+#   elif defined(__GNUC__)
+#    pragma GCC diagnostic pop
+#   endif
+}
+# endif	 /* CMM_SANITIZE_THREAD */
+
 # ifndef cmm_smp_mb
-#  define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
-# endif
+#  ifdef CMM_SANITIZE_THREAD
+#   define cmm_smp_mb() _cmm_thread_fence_wrapper()
+#  else
+#   define cmm_smp_mb() __atomic_thread_fence(__ATOMIC_SEQ_CST)
+#  endif /* CMM_SANITIZE_THREAD */
+# endif /* !cmm_smp_mb */
 
 #endif	/* CONFIG_RCU_USE_ATOMIC_BUILTINS */
 
include/urcu/compiler.h +14 -0
index 3604488..b0c7850 100644
@@ -129,4 +129,18 @@
 				+ __GNUC_PATCHLEVEL__)
 #endif
 
+/*
+ * Allow user to manually define CMM_SANITIZE_THREAD if their toolchain is not
+ * supported by this check.
+ */
+#ifndef CMM_SANITIZE_THREAD
+# if defined(__GNUC__) && defined(__SANITIZE_THREAD__)
+#  define CMM_SANITIZE_THREAD
+# elif defined(__clang__) && defined(__has_feature)
+#  if __has_feature(thread_sanitizer)
+#   define CMM_SANITIZE_THREAD
+#  endif
+# endif
+#endif	/* !CMM_SANITIZE_THREAD */
+
 #endif /* _URCU_COMPILER_H */
include/urcu/static/urcu-bp.h +9 -3
index 8ba3830..3e14ef7 100644
@@ -33,6 +33,7 @@
 #include <pthread.h>
 #include <unistd.h>
 
+#include <urcu/annotate.h>
 #include <urcu/debug.h>
 #include <urcu/config.h>
 #include <urcu/compiler.h>
@@ -117,7 +118,8 @@ static inline void urcu_bp_smp_mb_slave(void)
 		cmm_smp_mb();
 }
 
-static inline enum urcu_bp_state urcu_bp_reader_state(unsigned long *ctr)
+static inline enum urcu_bp_state urcu_bp_reader_state(unsigned long *ctr,
+						cmm_annotate_t *group)
 {
 	unsigned long v;
 
@@ -127,7 +129,9 @@ static inline enum urcu_bp_state urcu_bp_reader_state(unsigned long *ctr)
 	 * Make sure both tests below are done on the same version of *value
 	 * to insure consistency.
 	 */
-	v = CMM_LOAD_SHARED(*ctr);
+	v = uatomic_load(ctr, CMM_RELAXED);
+	cmm_annotate_group_mem_acquire(group, ctr);
+
 	if (!(v & URCU_BP_GP_CTR_NEST_MASK))
 		return URCU_BP_READER_INACTIVE;
 	if (!((v ^ urcu_bp_gp.ctr) & URCU_BP_GP_CTR_PHASE))
@@ -181,12 +185,14 @@ static inline void _urcu_bp_read_lock(void)
 static inline void _urcu_bp_read_unlock(void)
 {
 	unsigned long tmp;
+	unsigned long *ctr = &URCU_TLS(urcu_bp_reader)->ctr;
 
 	tmp = URCU_TLS(urcu_bp_reader)->ctr;
 	urcu_assert_debug(tmp & URCU_BP_GP_CTR_NEST_MASK);
 	/* Finish using rcu before decrementing the pointer. */
 	urcu_bp_smp_mb_slave();
-	_CMM_STORE_SHARED(URCU_TLS(urcu_bp_reader)->ctr, tmp - URCU_BP_GP_COUNT);
+	cmm_annotate_mem_release(ctr);
+	uatomic_store(ctr, tmp - URCU_BP_GP_COUNT, CMM_RELAXED);
 	cmm_barrier();	/* Ensure the compiler does not reorder us with mutex */
 }
 
include/urcu/static/urcu-common.h +6 -2
index 60ea8b8..32cb834 100644
@@ -34,6 +34,7 @@
 #include <unistd.h>
 #include <stdint.h>
 
+#include <urcu/annotate.h>
 #include <urcu/config.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
@@ -105,7 +106,8 @@ static inline void urcu_common_wake_up_gp(struct urcu_gp *gp)
 }
 
 static inline enum urcu_state urcu_common_reader_state(struct urcu_gp *gp,
-		unsigned long *ctr)
+						unsigned long *ctr,
+						cmm_annotate_t *group)
 {
 	unsigned long v;
 
@@ -113,7 +115,9 @@ static inline enum urcu_state urcu_common_reader_state(struct urcu_gp *gp,
 	 * Make sure both tests below are done on the same version of *value
 	 * to insure consistency.
 	 */
-	v = CMM_LOAD_SHARED(*ctr);
+	v = uatomic_load(ctr, CMM_RELAXED);
+	cmm_annotate_group_mem_acquire(group, ctr);
+
 	if (!(v & URCU_GP_CTR_NEST_MASK))
 		return URCU_READER_INACTIVE;
 	if (!((v ^ gp->ctr) & URCU_GP_CTR_PHASE))
include/urcu/static/urcu-mb.h +6 -5
index b97e42a..5bf7933 100644
@@ -108,13 +108,14 @@ static inline void _urcu_mb_read_lock(void)
  */
 static inline void _urcu_mb_read_unlock_update_and_wakeup(unsigned long tmp)
 {
+	unsigned long *ctr = &URCU_TLS(urcu_mb_reader).ctr;
+
 	if (caa_likely((tmp & URCU_GP_CTR_NEST_MASK) == URCU_GP_COUNT)) {
-		cmm_smp_mb();
-		_CMM_STORE_SHARED(URCU_TLS(urcu_mb_reader).ctr, tmp - URCU_GP_COUNT);
-		cmm_smp_mb();
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_SEQ_CST);
 		urcu_common_wake_up_gp(&urcu_mb_gp);
-	} else
-		_CMM_STORE_SHARED(URCU_TLS(urcu_mb_reader).ctr, tmp - URCU_GP_COUNT);
+	} else {
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_RELAXED);
+	}
 }
 
 /*
include/urcu/static/urcu-memb.h +20 -6
index c8d102f..8191ccc 100644
@@ -34,6 +34,7 @@
 #include <unistd.h>
 #include <stdint.h>
 
+#include <urcu/annotate.h>
 #include <urcu/debug.h>
 #include <urcu/config.h>
 #include <urcu/compiler.h>
@@ -93,11 +94,20 @@ extern DECLARE_URCU_TLS(struct urcu_reader, urcu_memb_reader);
  */
 static inline void _urcu_memb_read_lock_update(unsigned long tmp)
 {
+	unsigned long *ctr = &URCU_TLS(urcu_memb_reader).ctr;
+
 	if (caa_likely(!(tmp & URCU_GP_CTR_NEST_MASK))) {
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, _CMM_LOAD_SHARED(urcu_memb_gp.ctr));
+		unsigned long *pgctr = &urcu_memb_gp.ctr;
+		unsigned long gctr = uatomic_load(pgctr, CMM_RELAXED);
+
+		/* Paired with following mb slave. */
+		cmm_annotate_mem_acquire(pgctr);
+		uatomic_store(ctr, gctr, CMM_RELAXED);
+
 		urcu_memb_smp_mb_slave();
-	} else
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, tmp + URCU_GP_COUNT);
+	} else {
+		uatomic_store(ctr, tmp + URCU_GP_COUNT, CMM_RELAXED);
+	}
 }
 
 /*
@@ -131,13 +141,17 @@ static inline void _urcu_memb_read_lock(void)
  */
 static inline void _urcu_memb_read_unlock_update_and_wakeup(unsigned long tmp)
 {
+	unsigned long *ctr = &URCU_TLS(urcu_memb_reader).ctr;
+
 	if (caa_likely((tmp & URCU_GP_CTR_NEST_MASK) == URCU_GP_COUNT)) {
 		urcu_memb_smp_mb_slave();
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, tmp - URCU_GP_COUNT);
+		cmm_annotate_mem_release(ctr);
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_RELAXED);
 		urcu_memb_smp_mb_slave();
 		urcu_common_wake_up_gp(&urcu_memb_gp);
-	} else
-		_CMM_STORE_SHARED(URCU_TLS(urcu_memb_reader).ctr, tmp - URCU_GP_COUNT);
+	} else {
+		uatomic_store(ctr, tmp - URCU_GP_COUNT, CMM_RELAXED);
+	}
 }
 
 /*
include/urcu/static/urcu-qsbr.h +19 -10
index b878877..864cbcf 100644
@@ -35,6 +35,7 @@
 #include <unistd.h>
 #include <stdint.h>
 
+#include <urcu/annotate.h>
 #include <urcu/debug.h>
 #include <urcu/compiler.h>
 #include <urcu/arch.h>
@@ -96,11 +97,14 @@ static inline void urcu_qsbr_wake_up_gp(void)
 	}
 }
 
-static inline enum urcu_state urcu_qsbr_reader_state(unsigned long *ctr)
+static inline enum urcu_state urcu_qsbr_reader_state(unsigned long *ctr,
+						cmm_annotate_t *group)
 {
 	unsigned long v;
 
-	v = CMM_LOAD_SHARED(*ctr);
+	v = uatomic_load(ctr, CMM_RELAXED);
+	cmm_annotate_group_mem_acquire(group, ctr);
+
 	if (!v)
 		return URCU_READER_INACTIVE;
 	if (v == urcu_qsbr_gp.ctr)
@@ -155,9 +159,9 @@ static inline int _urcu_qsbr_read_ongoing(void)
  */
 static inline void _urcu_qsbr_quiescent_state_update_and_wakeup(unsigned long gp_ctr)
 {
-	cmm_smp_mb();
-	_CMM_STORE_SHARED(URCU_TLS(urcu_qsbr_reader).ctr, gp_ctr);
-	cmm_smp_mb();	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
+	uatomic_store(&URCU_TLS(urcu_qsbr_reader).ctr, gp_ctr, CMM_SEQ_CST);
+
+	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
 	urcu_qsbr_wake_up_gp();
 	cmm_smp_mb();
 }
@@ -179,7 +183,8 @@ static inline void _urcu_qsbr_quiescent_state(void)
 	unsigned long gp_ctr;
 
 	urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
-	if ((gp_ctr = CMM_LOAD_SHARED(urcu_qsbr_gp.ctr)) == URCU_TLS(urcu_qsbr_reader).ctr)
+	gp_ctr = uatomic_load(&urcu_qsbr_gp.ctr, CMM_RELAXED);
+	if (gp_ctr == URCU_TLS(urcu_qsbr_reader).ctr)
 		return;
 	_urcu_qsbr_quiescent_state_update_and_wakeup(gp_ctr);
 }
@@ -195,9 +200,8 @@ static inline void _urcu_qsbr_quiescent_state(void)
 static inline void _urcu_qsbr_thread_offline(void)
 {
 	urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
-	cmm_smp_mb();
-	CMM_STORE_SHARED(URCU_TLS(urcu_qsbr_reader).ctr, 0);
-	cmm_smp_mb();	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
+	uatomic_store(&URCU_TLS(urcu_qsbr_reader).ctr, 0, CMM_SEQ_CST);
+	/* write URCU_TLS(urcu_qsbr_reader).ctr before read futex */
 	urcu_qsbr_wake_up_gp();
 	cmm_barrier();	/* Ensure the compiler does not reorder us with mutex */
 }
@@ -212,9 +216,14 @@ static inline void _urcu_qsbr_thread_offline(void)
  */
 static inline void _urcu_qsbr_thread_online(void)
 {
+	unsigned long *pctr = &URCU_TLS(urcu_qsbr_reader).ctr;
+	unsigned long ctr;
+
 	urcu_assert_debug(URCU_TLS(urcu_qsbr_reader).registered);
 	cmm_barrier();	/* Ensure the compiler does not reorder us with mutex */
-	_CMM_STORE_SHARED(URCU_TLS(urcu_qsbr_reader).ctr, CMM_LOAD_SHARED(urcu_qsbr_gp.ctr));
+	ctr = uatomic_load(&urcu_qsbr_gp.ctr, CMM_RELAXED);
+	cmm_annotate_mem_acquire(&urcu_qsbr_gp.ctr);
+	uatomic_store(pctr, ctr, CMM_RELAXED);
 	cmm_smp_mb();
 }
 
src/rculfhash.c +58 -34

227 lines not on this page · raw message

src/urcu-bp.c +12 -5
index 47fad8e..08aaa88 100644
@@ -36,6 +36,7 @@
 #include <stdbool.h>
 #include <sys/mman.h>
 
+#include <urcu/annotate.h>
 #include <urcu/assert.h>
 #include <urcu/config.h>
 #include <urcu/arch.h>
@@ -220,7 +221,8 @@ static void smp_mb_master(void)
  */
 static void wait_for_readers(struct cds_list_head *input_readers,
 			struct cds_list_head *cur_snap_readers,
-			struct cds_list_head *qsreaders)
+			struct cds_list_head *qsreaders,
+			cmm_annotate_t *group)
 {
 	unsigned int wait_loops = 0;
 	struct urcu_bp_reader *index, *tmp;
@@ -235,7 +237,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 			wait_loops++;
 
 		cds_list_for_each_entry_safe(index, tmp, input_readers, node) {
-			switch (urcu_bp_reader_state(&index->ctr)) {
+			switch (urcu_bp_reader_state(&index->ctr, group)) {
 			case URCU_BP_READER_ACTIVE_CURRENT:
 				if (cur_snap_readers) {
 					cds_list_move(&index->node,
@@ -274,6 +276,8 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 
 void urcu_bp_synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(cur_snap_readers);
 	CDS_LIST_HEAD(qsreaders);
 	sigset_t newmask, oldmask;
@@ -295,13 +299,14 @@ void urcu_bp_synchronize_rcu(void)
 	 * where new ptr points to. */
 	/* Write new ptr before changing the qparity */
 	smp_mb_master();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Wait for readers to observe original parity or be quiescent.
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, &cur_snap_readers, &qsreaders);
+	wait_for_readers(&registry, &cur_snap_readers, &qsreaders, &acquire_group);
 
 	/*
 	 * Adding a cmm_smp_mb() which is _not_ formally required, but makes the
@@ -311,7 +316,8 @@ void urcu_bp_synchronize_rcu(void)
 	cmm_smp_mb();
 
 	/* Switch parity: 0 -> 1, 1 -> 0 */
-	CMM_STORE_SHARED(rcu_gp.ctr, rcu_gp.ctr ^ URCU_BP_GP_CTR_PHASE);
+	cmm_annotate_group_mem_release(&release_group, &rcu_gp.ctr);
+	uatomic_store(&rcu_gp.ctr, rcu_gp.ctr ^ URCU_BP_GP_CTR_PHASE, CMM_RELAXED);
 
 	/*
 	 * Must commit qparity update to memory before waiting for other parity
@@ -332,7 +338,7 @@ void urcu_bp_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&cur_snap_readers, NULL, &qsreaders);
+	wait_for_readers(&cur_snap_readers, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -344,6 +350,7 @@ void urcu_bp_synchronize_rcu(void)
 	 * freed.
 	 */
 	smp_mb_master();
+	cmm_annotate_group_mb_acquire(&acquire_group);
 out:
 	mutex_unlock(&rcu_registry_lock);
 	mutex_unlock(&rcu_gp_lock);
src/urcu-qsbr.c +22 -9
index 318ab29..fd50e80 100644
@@ -34,6 +34,7 @@
 #include <errno.h>
 #include <poll.h>
 
+#include <urcu/annotate.h>
 #include <urcu/assert.h>
 #include <urcu/wfcqueue.h>
 #include <urcu/map/urcu-qsbr.h>
@@ -156,7 +157,8 @@ static void wait_gp(void)
  */
 static void wait_for_readers(struct cds_list_head *input_readers,
 			struct cds_list_head *cur_snap_readers,
-			struct cds_list_head *qsreaders)
+			struct cds_list_head *qsreaders,
+			cmm_annotate_t *group)
 {
 	unsigned int wait_loops = 0;
 	struct urcu_qsbr_reader *index, *tmp;
@@ -183,7 +185,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 			cmm_smp_mb();
 		}
 		cds_list_for_each_entry_safe(index, tmp, input_readers, node) {
-			switch (urcu_qsbr_reader_state(&index->ctr)) {
+			switch (urcu_qsbr_reader_state(&index->ctr, group)) {
 			case URCU_READER_ACTIVE_CURRENT:
 				if (cur_snap_readers) {
 					cds_list_move(&index->node,
@@ -208,8 +210,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 		if (cds_list_empty(input_readers)) {
 			if (wait_loops >= RCU_QS_ACTIVE_ATTEMPTS) {
 				/* Read reader_gp before write futex */
-				cmm_smp_mb();
-				uatomic_set(&urcu_qsbr_gp.futex, 0);
+				uatomic_store(&urcu_qsbr_gp.futex, 0, CMM_RELEASE);
 			}
 			break;
 		} else {
@@ -238,6 +239,8 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 #if (CAA_BITS_PER_LONG < 64)
 void urcu_qsbr_synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(cur_snap_readers);
 	CDS_LIST_HEAD(qsreaders);
 	unsigned long was_online;
@@ -258,6 +261,7 @@ void urcu_qsbr_synchronize_rcu(void)
 		urcu_qsbr_thread_offline();
 	else
 		cmm_smp_mb();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Add ourself to gp_waiters queue of threads awaiting to wait
@@ -289,7 +293,7 @@ void urcu_qsbr_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, &cur_snap_readers, &qsreaders);
+	wait_for_readers(&registry, &cur_snap_readers, &qsreaders, &acquire_group);
 
 	/*
 	 * Must finish waiting for quiescent state for original parity
@@ -309,7 +313,8 @@ void urcu_qsbr_synchronize_rcu(void)
 	cmm_smp_mb();
 
 	/* Switch parity: 0 -> 1, 1 -> 0 */
-	CMM_STORE_SHARED(urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr ^ URCU_QSBR_GP_CTR);
+	cmm_annotate_group_mem_release(&release_group, &urcu_qsbr_gp.ctr);
+	uatomic_store(&urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr ^ URCU_QSBR_GP_CTR, CMM_RELAXED);
 
 	/*
 	 * Must commit urcu_qsbr_gp.ctr update to memory before waiting for
@@ -332,7 +337,7 @@ void urcu_qsbr_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&cur_snap_readers, NULL, &qsreaders);
+	wait_for_readers(&cur_snap_readers, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -347,6 +352,8 @@ gp_end:
 	 * Finish waiting for reader threads before letting the old ptr being
 	 * freed.
 	 */
+	cmm_annotate_group_mb_acquire(&acquire_group);
+
 	if (was_online)
 		urcu_qsbr_thread_online();
 	else
@@ -355,6 +362,8 @@ gp_end:
 #else /* !(CAA_BITS_PER_LONG < 64) */
 void urcu_qsbr_synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(qsreaders);
 	unsigned long was_online;
 	DEFINE_URCU_WAIT_NODE(wait, URCU_WAIT_WAITING);
@@ -371,6 +380,7 @@ void urcu_qsbr_synchronize_rcu(void)
 		urcu_qsbr_thread_offline();
 	else
 		cmm_smp_mb();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Add ourself to gp_waiters queue of threads awaiting to wait
@@ -398,7 +408,8 @@ void urcu_qsbr_synchronize_rcu(void)
 		goto out;
 
 	/* Increment current G.P. */
-	CMM_STORE_SHARED(urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr + URCU_QSBR_GP_CTR);
+	cmm_annotate_group_mem_release(&release_group, &urcu_qsbr_gp.ctr);
+	uatomic_store(&urcu_qsbr_gp.ctr, urcu_qsbr_gp.ctr + URCU_QSBR_GP_CTR, CMM_RELAXED);
 
 	/*
 	 * Must commit urcu_qsbr_gp.ctr update to memory before waiting for
@@ -421,7 +432,7 @@ void urcu_qsbr_synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, NULL, &qsreaders);
+	wait_for_readers(&registry, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -436,6 +447,8 @@ gp_end:
 		urcu_qsbr_thread_online();
 	else
 		cmm_smp_mb();
+
+	cmm_annotate_group_mb_acquire(&acquire_group);
 }
 #endif  /* !(CAA_BITS_PER_LONG < 64) */
 
src/urcu-wait.h +4 -5
index 4667a13..1ffced4 100644
@@ -126,9 +126,8 @@ void urcu_wait_node_init(struct urcu_wait_node *node,
 static inline
 void urcu_adaptative_wake_up(struct urcu_wait_node *wait)
 {
-	cmm_smp_mb();
 	urcu_posix_assert(uatomic_read(&wait->state) == URCU_WAIT_WAITING);
-	uatomic_set(&wait->state, URCU_WAIT_WAKEUP);
+	uatomic_store(&wait->state, URCU_WAIT_WAKEUP, CMM_RELEASE);
 	if (!(uatomic_read(&wait->state) & URCU_WAIT_RUNNING)) {
 		if (futex_noasync(&wait->state, FUTEX_WAKE, 1,
 				NULL, NULL, 0) < 0)
@@ -150,11 +149,11 @@ void urcu_adaptative_busy_wait(struct urcu_wait_node *wait)
 	/* Load and test condition before read state */
 	cmm_smp_rmb();
 	for (i = 0; i < URCU_WAIT_ATTEMPTS; i++) {
-		if (uatomic_read(&wait->state) != URCU_WAIT_WAITING)
+		if (uatomic_load(&wait->state, CMM_ACQUIRE) != URCU_WAIT_WAITING)
 			goto skip_futex_wait;
 		caa_cpu_relax();
 	}
-	while (uatomic_read(&wait->state) == URCU_WAIT_WAITING) {
+	while (uatomic_load(&wait->state, CMM_ACQUIRE) == URCU_WAIT_WAITING) {
 		if (!futex_noasync(&wait->state, FUTEX_WAIT, URCU_WAIT_WAITING, NULL, NULL, 0)) {
 			/*
 			 * Prior queued wakeups queued by unrelated code
@@ -189,7 +188,7 @@ skip_futex_wait:
 	 * memory allocated for struct urcu_wait.
 	 */
 	for (i = 0; i < URCU_WAIT_ATTEMPTS; i++) {
-		if (uatomic_read(&wait->state) & URCU_WAIT_TEARDOWN)
+		if (uatomic_load(&wait->state, CMM_RELAXED) & URCU_WAIT_TEARDOWN)
 			break;
 		caa_cpu_relax();
 	}
src/urcu.c +16 -8
index c60307e..353e9bb 100644
@@ -38,6 +38,7 @@
 #include <poll.h>
 
 #include <urcu/config.h>
+#include <urcu/annotate.h>
 #include <urcu/assert.h>
 #include <urcu/arch.h>
 #include <urcu/wfcqueue.h>
@@ -300,7 +301,8 @@ end:
  */
 static void wait_for_readers(struct cds_list_head *input_readers,
 			struct cds_list_head *cur_snap_readers,
-			struct cds_list_head *qsreaders)
+			struct cds_list_head *qsreaders,
+			cmm_annotate_t *group)
 {
 	unsigned int wait_loops = 0;
 	struct urcu_reader *index, *tmp;
@@ -323,7 +325,7 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 		}
 
 		cds_list_for_each_entry_safe(index, tmp, input_readers, node) {
-			switch (urcu_common_reader_state(&rcu_gp, &index->ctr)) {
+			switch (urcu_common_reader_state(&rcu_gp, &index->ctr, group)) {
 			case URCU_READER_ACTIVE_CURRENT:
 				if (cur_snap_readers) {
 					cds_list_move(&index->node,
@@ -407,6 +409,8 @@ static void wait_for_readers(struct cds_list_head *input_readers,
 
 void synchronize_rcu(void)
 {
+	cmm_annotate_define(acquire_group);
+	cmm_annotate_define(release_group);
 	CDS_LIST_HEAD(cur_snap_readers);
 	CDS_LIST_HEAD(qsreaders);
 	DEFINE_URCU_WAIT_NODE(wait, URCU_WAIT_WAITING);
@@ -421,10 +425,11 @@ void synchronize_rcu(void)
 	 * queue before their insertion into the wait queue.
 	 */
 	if (urcu_wait_add(&gp_waiters, &wait) != 0) {
-		/* Not first in queue: will be awakened by another thread. */
+		/*
+		 * Not first in queue: will be awakened by another thread.
+		 * Implies a memory barrier after grace period.
+		 */
 		urcu_adaptative_busy_wait(&wait);
-		/* Order following memory accesses after grace period. */
-		cmm_smp_mb();
 		return;
 	}
 	/* We won't need to wake ourself up */
@@ -449,13 +454,14 @@ void synchronize_rcu(void)
 	 */
 	/* Write new ptr before changing the qparity */
 	smp_mb_master();
+	cmm_annotate_group_mb_release(&release_group);
 
 	/*
 	 * Wait for readers to observe original parity or be quiescent.
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&registry, &cur_snap_readers, &qsreaders);
+	wait_for_readers(&registry, &cur_snap_readers, &qsreaders, &acquire_group);
 
 	/*
 	 * Must finish waiting for quiescent state for original parity before
@@ -474,7 +480,8 @@ void synchronize_rcu(void)
 	cmm_smp_mb();
 
 	/* Switch parity: 0 -> 1, 1 -> 0 */
-	CMM_STORE_SHARED(rcu_gp.ctr, rcu_gp.ctr ^ URCU_GP_CTR_PHASE);
+	cmm_annotate_group_mem_release(&release_group, &rcu_gp.ctr);
+	uatomic_store(&rcu_gp.ctr, rcu_gp.ctr ^ URCU_GP_CTR_PHASE, CMM_RELAXED);
 
 	/*
 	 * Must commit rcu_gp.ctr update to memory before waiting for quiescent
@@ -497,7 +504,7 @@ void synchronize_rcu(void)
 	 * wait_for_readers() can release and grab again rcu_registry_lock
 	 * internally.
 	 */
-	wait_for_readers(&cur_snap_readers, NULL, &qsreaders);
+	wait_for_readers(&cur_snap_readers, NULL, &qsreaders, &acquire_group);
 
 	/*
 	 * Put quiescent reader list back into registry.
@@ -510,6 +517,7 @@ void synchronize_rcu(void)
 	 * iterates on reader threads.
 	 */
 	smp_mb_master();
+	cmm_annotate_group_mb_acquire(&acquire_group);
 out:
 	mutex_unlock(&rcu_registry_lock);
 	mutex_unlock(&rcu_gp_lock);
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#67 lore raw

PATCH v2 11/12 Add cmm_emit_legacy_smp_mb()

In reply to Olivier Dion via lttng-dev

Some public APIs stipulate implicit memory barriers on operations. These were coherent with the memory model used at that time. However, with the migration to a memory model closer to the C11 memory model, these memory barriers are not strictly emitted by the atomic operations in the new memory model.

Therefore, introducing the `--disable-legacy-mb' configuration option. By default, liburcu is configured to emit these legacy memory barriers, thus keeping backward compatibility at the expense of slower performances. However, users can opt-out by disabling the legacy memory barriers.

This options is publicly exported in the system configuration header file and can be overrode manually on a compilation unit basis by defining `CONFIG_RCU_EMIT_LEGACY_MB' before including any liburcu files.

The usage of this macro requires to re-write atomic operations in term of the CMM memory model. This is done for the queue and stack APIs.

Change-Id Ia5ce3b3d8cd1955556ce96fa4408a63aa098a1a6
---
 configure.ac                     | 13 ++++++
 include/urcu/arch.h              |  6 +++
 include/urcu/config.h.in         |  3 ++
 include/urcu/static/lfstack.h    | 25 ++++++++----
 include/urcu/static/rculfqueue.h | 14 ++++---
 include/urcu/static/rculfstack.h |  8 +++-
 include/urcu/static/wfcqueue.h   | 68 +++++++++++++++++---------------
 include/urcu/static/wfqueue.h    |  9 +++--
 include/urcu/static/wfstack.h    | 24 +++++++----
 9 files changed, 114 insertions(+), 56 deletions(-)
configure.ac +13 -0
index 4450a31..ca51b5b 100644
@@ -235,6 +235,11 @@ AE_FEATURE([cds-lfht-iter-debug], [Enable extra debugging checks for lock-free h
 AE_FEATURE_DEFAULT_ENABLE
 AE_FEATURE([atomic-builtins], [Disable the usage of toolchain atomic builtins.])
 
+# emit legacy memory barriers
+# Enable by default
+AE_FEATURE_DEFAULT_ENABLE
+AE_FEATURE([legacy-mb], [Disable legacy memory barriers.])
+
 # When given, add -Werror to WARN_CFLAGS and WARN_CXXFLAGS.
 # Disabled by default
 AE_FEATURE_DEFAULT_DISABLE
@@ -282,6 +287,10 @@ AE_IF_FEATURE_ENABLED([atomic-builtins], [
 	[AE_FEATURE_DISABLE(atomic-builtins)])
 ])
 
+AE_IF_FEATURE_ENABLED([legacy-mb], [
+  AC_DEFINE([CONFIG_RCU_EMIT_LEGACY_MB], [1], [Emit legacy memory barriers that were documented in the APIs.])
+])
+
 ##                                                                          ##
 ## Set automake variables for optional feature conditionnals in Makefile.am ##
 ##                                                                          ##
@@ -387,6 +396,10 @@ PPRINT_PROP_BOOL([Lock-free HT iterator debugging], $value)
 AE_IS_FEATURE_ENABLED([atomic-builtins]) && value=1 || value=0
 PPRINT_PROP_BOOL([Use toolchain atomic builtins], $value)
 
+# legacy memory barriers
+AE_IS_FEATURE_ENABLED([legacy-mb]) && value=1 || value=0
+PPRINT_PROP_BOOL([Emit legacy memory barriers], $value)
+
 PPRINT_PROP_BOOL([Multi-flavor support], 1)
 
 report_bindir="`eval eval echo $bindir`"
include/urcu/arch.h +6 -0
index d3914da..377a0ec 100644
@@ -171,5 +171,11 @@
 #error "Cannot build: unrecognized architecture, see <urcu/arch.h>."
 #endif
 
+#ifdef CONFIG_RCU_EMIT_LEGACY_MB
+# define cmm_emit_legacy_smp_mb() cmm_smp_mb()
+#else
+# define cmm_emit_legacy_smp_mb() do { } while (0)
+#endif
+
 
 #endif /* _URCU_ARCH_H */
include/urcu/config.h.in +3 -0
index 1daaa7e..d2f6c8c 100644
@@ -22,6 +22,9 @@
 /* Uatomic API uses atomic builtins? */
 #undef CONFIG_RCU_USE_ATOMIC_BUILTINS
 
+/* Emit legacy memory barriers? */
+#undef CONFIG_RCU_EMIT_LEGACY_MB
+
 /* Expose multi-flavor support */
 #define CONFIG_RCU_HAVE_MULTIFLAVOR 1
 
include/urcu/static/lfstack.h +18 -7
index a05acb4..07604db 100644
@@ -114,7 +114,7 @@ bool ___cds_lfs_empty_head(struct cds_lfs_head *head)
 static inline
 bool _cds_lfs_empty(cds_lfs_stack_ptr_t s)
 {
-	return ___cds_lfs_empty_head(CMM_LOAD_SHARED(s._s->head));
+	return ___cds_lfs_empty_head(uatomic_load(&s._s->head, CMM_RELAXED));
 }
 
 /*
@@ -122,6 +122,8 @@ bool _cds_lfs_empty(cds_lfs_stack_ptr_t s)
  *
  * Does not require any synchronization with other push nor pop.
  *
+ * Operations before push are consistent when observed after associated pop.
+ *
  * Lock-free stack push is not subject to ABA problem, so no need to
  * take the RCU read-side lock. Even if "head" changes between two
  * uatomic_cmpxchg() invocations here (being popped, and then pushed
@@ -167,7 +169,9 @@ bool _cds_lfs_push(cds_lfs_stack_ptr_t u_s,
 		 * uatomic_cmpxchg() implicit memory barrier orders earlier
 		 * stores to node before publication.
 		 */
-		head = uatomic_cmpxchg(&s->head, old_head, new_head);
+		cmm_emit_legacy_smp_mb();
+		head = uatomic_cmpxchg_mo(&s->head, old_head, new_head,
+					CMM_SEQ_CST, CMM_SEQ_CST);
 		if (old_head == head)
 			break;
 	}
@@ -179,6 +183,8 @@ bool _cds_lfs_push(cds_lfs_stack_ptr_t u_s,
  *
  * Returns NULL if stack is empty.
  *
+ * Operations after pop are consistent when observed before associated push.
+ *
  * __cds_lfs_pop needs to be synchronized using one of the following
  * techniques:
  *
@@ -203,7 +209,7 @@ struct cds_lfs_node *___cds_lfs_pop(cds_lfs_stack_ptr_t u_s)
 		struct cds_lfs_head *head, *next_head;
 		struct cds_lfs_node *next;
 
-		head = _CMM_LOAD_SHARED(s->head);
+		head = uatomic_load(&s->head, CMM_CONSUME);
 		if (___cds_lfs_empty_head(head))
 			return NULL;	/* Empty stack */
 
@@ -212,12 +218,14 @@ struct cds_lfs_node *___cds_lfs_pop(cds_lfs_stack_ptr_t u_s)
 		 * memory barrier before uatomic_cmpxchg() in
 		 * cds_lfs_push.
 		 */
-		cmm_smp_read_barrier_depends();
-		next = _CMM_LOAD_SHARED(head->node.next);
+		next = uatomic_load(&head->node.next, CMM_RELAXED);
 		next_head = caa_container_of(next,
 				struct cds_lfs_head, node);
-		if (uatomic_cmpxchg(&s->head, head, next_head) == head)
+		if (uatomic_cmpxchg_mo(&s->head, head, next_head,
+					CMM_SEQ_CST, CMM_SEQ_CST) == head){
+			cmm_emit_legacy_smp_mb();
 			return &head->node;
+		}
 		/* busy-loop if head changed under us */
 	}
 }
@@ -245,6 +253,7 @@ static inline
 struct cds_lfs_head *___cds_lfs_pop_all(cds_lfs_stack_ptr_t u_s)
 {
 	struct __cds_lfs_stack *s = u_s._s;
+	struct cds_lfs_head *head;
 
 	/*
 	 * Implicit memory barrier after uatomic_xchg() matches implicit
@@ -256,7 +265,9 @@ struct cds_lfs_head *___cds_lfs_pop_all(cds_lfs_stack_ptr_t u_s)
 	 * taking care to order writes to each node prior to the full
 	 * memory barrier after this uatomic_xchg().
 	 */
-	return uatomic_xchg(&s->head, NULL);
+	head = uatomic_xchg_mo(&s->head, NULL, CMM_SEQ_CST);
+	cmm_emit_legacy_smp_mb();
+	return head;
 }
 
 /*
include/urcu/static/rculfqueue.h +9 -5
index ad73454..25a4ec8 100644
@@ -148,26 +148,29 @@ void _cds_lfq_enqueue_rcu(struct cds_lfq_queue_rcu *q,
 	 * uatomic_cmpxchg() implicit memory barrier orders earlier stores to
 	 * node before publication.
 	 */
-
 	for (;;) {
 		struct cds_lfq_node_rcu *tail, *next;
 
 		tail = rcu_dereference(q->tail);
-		next = uatomic_cmpxchg(&tail->next, NULL, node);
+		cmm_emit_legacy_smp_mb();
+		next = uatomic_cmpxchg_mo(&tail->next, NULL, node,
+					CMM_SEQ_CST, CMM_SEQ_CST);
 		if (next == NULL) {
 			/*
 			 * Tail was at the end of queue, we successfully
 			 * appended to it. Now move tail (another
 			 * enqueue might beat us to it, that's fine).
 			 */
-			(void) uatomic_cmpxchg(&q->tail, tail, node);
+			(void) uatomic_cmpxchg_mo(&q->tail, tail, node,
+						CMM_SEQ_CST, CMM_SEQ_CST);
 			return;
 		} else {
 			/*
 			 * Failure to append to current tail.
 			 * Help moving tail further and retry.
 			 */
-			(void) uatomic_cmpxchg(&q->tail, tail, next);
+			(void) uatomic_cmpxchg_mo(&q->tail, tail, next,
+						CMM_SEQ_CST, CMM_SEQ_CST);
 			continue;
 		}
 	}
@@ -211,7 +214,8 @@ struct cds_lfq_node_rcu *_cds_lfq_dequeue_rcu(struct cds_lfq_queue_rcu *q)
 			enqueue_dummy(q);
 			next = rcu_dereference(head->next);
 		}
-		if (uatomic_cmpxchg(&q->head, head, next) != head)
+		if (uatomic_cmpxchg_mo(&q->head, head, next,
+					CMM_SEQ_CST, CMM_SEQ_CST) != head)
 			continue;	/* Concurrently pushed. */
 		if (head->dummy) {
 			/* Free dummy after grace period. */
include/urcu/static/rculfstack.h +6 -2
index 54ff377..2befb2a 100644
@@ -83,7 +83,9 @@ int _cds_lfs_push_rcu(struct cds_lfs_stack_rcu *s,
 		 * uatomic_cmpxchg() implicit memory barrier orders earlier
 		 * stores to node before publication.
 		 */
-		head = uatomic_cmpxchg(&s->head, old_head, node);
+		cmm_emit_legacy_smp_mb();
+		head = uatomic_cmpxchg_mo(&s->head, old_head, node,
+					CMM_SEQ_CST, CMM_SEQ_CST);
 		if (old_head == head)
 			break;
 	}
@@ -108,7 +110,9 @@ _cds_lfs_pop_rcu(struct cds_lfs_stack_rcu *s)
 		if (head) {
 			struct cds_lfs_node_rcu *next = rcu_dereference(head->next);
 
-			if (uatomic_cmpxchg(&s->head, head, next) == head) {
+			if (uatomic_cmpxchg_mo(&s->head, head, next,
+						CMM_SEQ_CST, CMM_SEQ_CST) == head) {
+				cmm_emit_legacy_smp_mb();
 				return head;
 			} else {
 				/* Concurrent modification. Retry. */
include/urcu/static/wfcqueue.h +37 -31

175 lines not on this page · raw message

include/urcu/static/wfqueue.h +5 -4
index d04f66f..290fe0a 100644
@@ -81,13 +81,14 @@ static inline void _cds_wfq_enqueue(struct cds_wfq_queue *q,
 	 * structure containing node and setting node->next to NULL before
 	 * publication.
 	 */
-	old_tail = uatomic_xchg(&q->tail, &node->next);
+	cmm_emit_legacy_smp_mb();
+	old_tail = uatomic_xchg_mo(&q->tail, &node->next, CMM_SEQ_CST);
 	/*
 	 * At this point, dequeuers see a NULL old_tail->next, which indicates
 	 * that the queue is being appended to. The following store will append
 	 * "node" to the queue from a dequeuer perspective.
 	 */
-	CMM_STORE_SHARED(*old_tail, node);
+	uatomic_store(old_tail, node, CMM_RELEASE);
 }
 
 /*
@@ -102,7 +103,7 @@ ___cds_wfq_node_sync_next(struct cds_wfq_node *node)
 	/*
 	 * Adaptative busy-looping waiting for enqueuer to complete enqueue.
 	 */
-	while ((next = CMM_LOAD_SHARED(node->next)) == NULL) {
+	while ((next = uatomic_load(&node->next, CMM_CONSUME)) == NULL) {
 		if (++attempt >= WFQ_ADAPT_ATTEMPTS) {
 			(void) poll(NULL, 0, WFQ_WAIT);	/* Wait for 10ms */
 			attempt = 0;
@@ -129,7 +130,7 @@ ___cds_wfq_dequeue_blocking(struct cds_wfq_queue *q)
 	/*
 	 * Queue is empty if it only contains the dummy node.
 	 */
-	if (q->head == &q->dummy && CMM_LOAD_SHARED(q->tail) == &q->dummy.next)
+	if (q->head == &q->dummy && uatomic_load(&q->tail, CMM_CONSUME) == &q->dummy.next)
 		return NULL;
 	node = q->head;
 
include/urcu/static/wfstack.h +17 -7
index 088e6e3..cfaf675 100644
@@ -124,7 +124,7 @@ static inline bool _cds_wfs_empty(cds_wfs_stack_ptr_t u_stack)
 {
 	struct __cds_wfs_stack *s = u_stack._s;
 
-	return ___cds_wfs_end(CMM_LOAD_SHARED(s->head));
+	return ___cds_wfs_end(uatomic_load(&s->head, CMM_RELAXED));
 }
 
 /*
@@ -133,6 +133,8 @@ static inline bool _cds_wfs_empty(cds_wfs_stack_ptr_t u_stack)
  * Issues a full memory barrier before push. No mutual exclusion is
  * required.
  *
+ * Operations before push are consistent when observed after associated pop.
+ *
  * Returns 0 if the stack was empty prior to adding the node.
  * Returns non-zero otherwise.
  */
@@ -148,12 +150,13 @@ int _cds_wfs_push(cds_wfs_stack_ptr_t u_stack, struct cds_wfs_node *node)
 	 * uatomic_xchg() implicit memory barrier orders earlier stores
 	 * to node (setting it to NULL) before publication.
 	 */
-	old_head = uatomic_xchg(&s->head, new_head);
+	cmm_emit_legacy_smp_mb();
+	old_head = uatomic_xchg_mo(&s->head, new_head, CMM_SEQ_CST);
 	/*
 	 * At this point, dequeuers see a NULL node->next, they should
 	 * busy-wait until node->next is set to old_head.
 	 */
-	CMM_STORE_SHARED(node->next, &old_head->node);
+	uatomic_store(&node->next, &old_head->node, CMM_RELEASE);
 	return !___cds_wfs_end(old_head);
 }
 
@@ -169,7 +172,7 @@ ___cds_wfs_node_sync_next(struct cds_wfs_node *node, int blocking)
 	/*
 	 * Adaptative busy-looping waiting for push to complete.
 	 */
-	while ((next = CMM_LOAD_SHARED(node->next)) == NULL) {
+	while ((next = uatomic_load(&node->next, CMM_CONSUME)) == NULL) {
 		if (!blocking)
 			return CDS_WFS_WOULDBLOCK;
 		if (++attempt >= CDS_WFS_ADAPT_ATTEMPTS) {
@@ -194,7 +197,7 @@ ___cds_wfs_pop(cds_wfs_stack_ptr_t u_stack, int *state, int blocking)
 	if (state)
 		*state = 0;
 	for (;;) {
-		head = CMM_LOAD_SHARED(s->head);
+		head = uatomic_load(&s->head, CMM_CONSUME);
 		if (___cds_wfs_end(head)) {
 			return NULL;
 		}
@@ -203,9 +206,11 @@ ___cds_wfs_pop(cds_wfs_stack_ptr_t u_stack, int *state, int blocking)
 			return CDS_WFS_WOULDBLOCK;
 		}
 		new_head = caa_container_of(next, struct cds_wfs_head, node);
-		if (uatomic_cmpxchg(&s->head, head, new_head) == head) {
+		if (uatomic_cmpxchg_mo(&s->head, head, new_head,
+					CMM_SEQ_CST, CMM_SEQ_CST) == head) {
 			if (state && ___cds_wfs_end(new_head))
 				*state |= CDS_WFS_STATE_LAST;
+			cmm_emit_legacy_smp_mb();
 			return &head->node;
 		}
 		if (!blocking) {
@@ -220,6 +225,8 @@ ___cds_wfs_pop(cds_wfs_stack_ptr_t u_stack, int *state, int blocking)
  *
  * Returns NULL if stack is empty.
  *
+ * Operations after pop push are consistent when observed before associated push.
+ *
  * __cds_wfs_pop_blocking needs to be synchronized using one of the
  * following techniques:
  *
@@ -278,6 +285,8 @@ ___cds_wfs_pop_nonblocking(cds_wfs_stack_ptr_t u_stack)
 /*
  * __cds_wfs_pop_all: pop all nodes from a stack.
  *
+ * Operations after pop push are consistent when observed before associated push.
+ *
  * __cds_wfs_pop_all does not require any synchronization with other
  * push, nor with other __cds_wfs_pop_all, but requires synchronization
  * matching the technique used to synchronize __cds_wfs_pop_blocking:
@@ -309,7 +318,8 @@ ___cds_wfs_pop_all(cds_wfs_stack_ptr_t u_stack)
 	 * taking care to order writes to each node prior to the full
 	 * memory barrier after this uatomic_xchg().
 	 */
-	head = uatomic_xchg(&s->head, CDS_WFS_END);
+	head = uatomic_xchg_mo(&s->head, CDS_WFS_END, CMM_SEQ_CST);
+	cmm_emit_legacy_smp_mb();
 	if (___cds_wfs_end(head))
 		return NULL;
 	return head;
Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#68 lore raw

PATCH v2 12/12 tests: Add tests for checking race conditions

In reply to Olivier Dion via lttng-dev

These tests do nothing useful except of stress testing a single-consumer, multiple-producers program on various data structures.

These tests are only meaningful when compiling liburcu with TSAN.

Change-Id If22b27ed0fb95bf890947fc4e75f923edb5ada8f
---
 tests/unit/test_lfstack.c  |  90 ++++++++++++++++++++++++++++
 tests/unit/test_wfcqueue.c | 119 +++++++++++++++++++++++++++++++++++++
 tests/unit/test_wfqueue.c  |  91 ++++++++++++++++++++++++++++
 tests/unit/test_wfstack.c  |  90 ++++++++++++++++++++++++++++
 4 files changed, 390 insertions(+)
 create mode 100644 tests/unit/test_lfstack.c
 create mode 100644 tests/unit/test_wfcqueue.c
 create mode 100644 tests/unit/test_wfqueue.c
 create mode 100644 tests/unit/test_wfstack.c
tests/unit/test_lfstack.c new file +90 -0
new file mode 100644
index 0000000..a1f99f0
@@ -0,0 +1,90 @@
+/*
+ * test_lfstack.c
+ *
+ * Userspace RCU library - test wftack race conditions
+ *
+ * Copyright 2023 - Olivier Dion <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+
+#define _LGPL_SOURCE
+
+#include <stdlib.h>
+
+#include <pthread.h>
+
+#include <urcu/lfstack.h>
+
+#include "tap.h"
+
+#define NR_TESTS 1
+#define NR_PRODUCERS 4
+#define LOOP 100
+
+static void async_run(struct cds_lfs_stack *queue)
+{
+	struct cds_lfs_node *node = malloc(sizeof(*node));
+
+	cds_lfs_node_init(node);
+
+	cds_lfs_push(queue, node);
+}
+
+static void *async_loop(void *queue)
+{
+	size_t k = 0;
+
+	while (k < LOOP * NR_PRODUCERS) {
+		free(cds_lfs_pop_blocking(queue));
+		++k;
+	}
+
+	return NULL;
+}
+
+static void *spawn_jobs(void *queue)
+{
+	for (size_t k = 0; k < LOOP; ++k) {
+		async_run(queue);
+	}
+
+	return 0;
+}
+
+int main(void)
+{
+	pthread_t consumer;
+	pthread_t producers[NR_PRODUCERS];
+	struct cds_lfs_stack queue;
+
+	plan_tests(NR_TESTS);
+
+	cds_lfs_init(&queue);
+	pthread_create(&consumer, NULL, async_loop, &queue);
+
+	for (size_t k = 0; k < NR_PRODUCERS; ++k) {
+		pthread_create(&producers[k], NULL, spawn_jobs, &queue);
+	}
+
+	pthread_join(consumer, NULL);
+	for (size_t k = 0; k < NR_PRODUCERS; ++k) {
+		pthread_join(producers[k], NULL);
+	}
+
+	ok1("No race conditions");
+
+	return exit_status();
+}
tests/unit/test_wfcqueue.c new file +119 -0

119 lines not on this page · raw message

tests/unit/test_wfqueue.c new file +91 -0

91 lines not on this page · raw message

tests/unit/test_wfstack.c new file +90 -0

90 lines not on this page · raw message

Signature · 6 lines
2.40.1

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev