Skip to contentSkip to conversation

locking/atomic: restructuring + kerneldoc

PATCH Applied Mark Rutland 5 participants 63 messages Jun 5, 2023 – Jul 15, 2023 Page 1 of 2

Subsystems (28)
lore
Share & tools
lore

Mark Rutland

author reviewer
#0 lore raw

v2 cover locking/atomic: restructuring + kerneldoc

Hi all,

These patches restructure the generated atomic headers, and add kerneldoc comments for all of the generic atomic{,64,_long}_t operations.

The core headers now generate raw_atomic*() operations as the fundamental instrumentation-safe atomics, with the arch_atomic*() functions being an implementation detail that shouldn't be used directly.

Each raw_atomic*() op is given a single definition with all related ifdeffery inside, e.g.

| /**
|  * raw_atomic_inc_return_acquire() - atomic increment with acquire ordering
|  * @v: pointer to atomic_t
|  *
|  * Atomically updates @v to (@v + 1) with acquire ordering.
|  *
|  * Safe to use in noinstr code; prefer atomic_inc_return_acquire() elsewhere.
|  *
|  * Return: the updated value of @v.
|  */
| static __always_inline int
| raw_atomic_inc_return_acquire(atomic_t *v)
| {
| #if defined(arch_atomic_inc_return_acquire)
| 	return arch_atomic_inc_return_acquire(v);
| #elif defined(arch_atomic_inc_return_relaxed)
| 	int ret = arch_atomic_inc_return_relaxed(v);
| 	__atomic_acquire_fence();
| 	return ret;
| #elif defined(arch_atomic_inc_return)
| 	return arch_atomic_inc_return(v);
| #else
| 	return raw_atomic_add_return_acquire(1, v);
| #endif
| }

Similarly, the regular atomic*() ops (which already have a single definition) are given kerneldoc comments, e.g.

| /**
|  * atomic_inc_return_acquire() - atomic increment with acquire ordering
|  * @v: pointer to atomic_t
|  *
|  * Atomically updates @v to (@v + 1) with acquire ordering.
|  *
|  * Unsafe to use in noinstr code; use raw_atomic_inc_return_acquire() there.
|  *
|  * Return: the updated value of @v.
|  */
| static __always_inline int
| atomic_inc_return_acquire(atomic_t *v)
| {
| 	instrument_atomic_read_write(v, sizeof(*v));
| 	return raw_atomic_inc_return_acquire(v);
| }

The kerneldoc comments themselves are built from templates as with the fallbacks, which should allow them to be extended in future if necessary.

I've compile-tested this for a number of architectures and configurations, but as usual this probably needs to see some testing by build robots.

The patches are based on Peter Zijlstra's queued locking/core branch, specifically commit:

bb6e9a06cba6b850 ("s390/cpum_sf: Convert to cmpxchg128()")

Which can be found in the git tree at:

https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/

Since v1 [1]:
* Add kernel-doc handling of "~@v"
* Add atomic-instrumented.h to included documentation headers
* Fix typos and punctuation
* Clarify kerneldoc wording

[1] https://lore.kernel.org/lkml/[email protected]/

Thanks,
Mark.

Mark Rutland (26):
locking/atomic: arm: fix sync ops
locking/atomic: remove fallback comments locking/atomic: hexagon: remove redundant arch_atomic_cmpxchg locking/atomic: make atomic*_{cmp,}xchg optional locking/atomic: arc: add preprocessor symbols locking/atomic: arm: add preprocessor symbols locking/atomic: hexagon: add preprocessor symbols locking/atomic: m68k: add preprocessor symbols locking/atomic: parisc: add preprocessor symbols locking/atomic: sh: add preprocessor symbols locking/atomic: sparc: add preprocessor symbols locking/atomic: x86: add preprocessor symbols locking/atomic: xtensa: add preprocessor symbols locking/atomic: scripts: remove bogus order parameter locking/atomic: scripts: remove leftover "${mult}" locking/atomic: scripts: factor out order template generation locking/atomic: scripts: add trivial raw_atomic*_<op>() locking/atomic: treewide: use raw_atomic*_<op>() locking/atomic: scripts: build raw_atomic_long*() directly locking/atomic: scripts: restructure fallback ifdeffery locking/atomic: scripts: split pfx/name/sfx/order locking/atomic: scripts: simplify raw_atomic_long*() definitions locking/atomic: scripts: simplify raw_atomic*() definitions
docs: scripts: kernel-doc: accept bitwise negation like ~@var locking/atomic: scripts: generate kerneldoc comments locking/atomic: treewide: delete arch_atomic_*() kerneldoc

Paul E. McKenney (1):
locking/atomic: docs: Add atomic operations to the driver basic API
documentation

 Documentation/driver-api/basics.rst          |    8 +-
 arch/alpha/include/asm/atomic.h              |   35 -
 arch/arc/include/asm/atomic-spinlock.h       |    9 +
 arch/arc/include/asm/atomic.h                |   24 -
 arch/arc/include/asm/atomic64-arcv2.h        |   19 +-
 arch/arm/include/asm/assembler.h             |   17 +
 arch/arm/include/asm/atomic.h                |   15 +-
 arch/arm/include/asm/sync_bitops.h           |   29 +-
 arch/arm/lib/bitops.h                        |   14 +-
 arch/arm/lib/testchangebit.S                 |    4 +
 arch/arm/lib/testclearbit.S                  |    4 +
 arch/arm/lib/testsetbit.S                    |    4 +
 arch/arm64/include/asm/atomic.h              |   28 -
 arch/csky/include/asm/atomic.h               |   35 -
 arch/hexagon/include/asm/atomic.h            |   69 +-
 arch/ia64/include/asm/atomic.h               |    7 -
 arch/loongarch/include/asm/atomic.h          |   56 -
 arch/m68k/include/asm/atomic.h               |   18 +-
 arch/mips/include/asm/atomic.h               |   11 -
 arch/openrisc/include/asm/atomic.h           |    3 -
 arch/parisc/include/asm/atomic.h             |   27 +-
 arch/powerpc/include/asm/atomic.h            |   24 -
 arch/powerpc/kernel/smp.c                    |   12 +-
 arch/riscv/include/asm/atomic.h              |   72 -
 arch/sh/include/asm/atomic-grb.h             |    9 +
 arch/sh/include/asm/atomic-irq.h             |    9 +
 arch/sh/include/asm/atomic-llsc.h            |    9 +
 arch/sh/include/asm/atomic.h                 |    3 -
 arch/sparc/include/asm/atomic_32.h           |   18 +-
 arch/sparc/include/asm/atomic_64.h           |   29 +-
 arch/x86/include/asm/atomic.h                |   87 -
 arch/x86/include/asm/atomic64_32.h           |   76 -
 arch/x86/include/asm/atomic64_64.h           |   81 -
 arch/x86/include/asm/cmpxchg_64.h            |    4 +
 arch/x86/kernel/alternative.c                |    4 +-
 arch/x86/kernel/cpu/mce/core.c               |   16 +-
 arch/x86/kernel/nmi.c                        |    2 +-
 arch/x86/kernel/pvclock.c                    |    4 +-
 arch/x86/kvm/x86.c                           |    2 +-
 arch/xtensa/include/asm/atomic.h             |   12 +-
 include/asm-generic/atomic.h                 |    3 -
 include/asm-generic/bitops/atomic.h          |   12 +-
 include/asm-generic/bitops/lock.h            |    8 +-
 include/linux/atomic/atomic-arch-fallback.h  | 5200 ++++++++++++------
 include/linux/atomic/atomic-instrumented.h   | 3484 ++++++++++--
 include/linux/atomic/atomic-long.h           | 2122 ++++---
 include/linux/context_tracking.h             |    4 +-
 include/linux/context_tracking_state.h       |    2 +-
 include/linux/cpumask.h                      |    2 +-
 include/linux/jump_label.h                   |    2 +-
 kernel/context_tracking.c                    |   12 +-
 kernel/sched/clock.c                         |    2 +-
 scripts/atomic/atomic-tbl.sh                 |  112 +-
 scripts/atomic/atomics.tbl                   |    2 +-
 scripts/atomic/fallbacks/acquire             |    4 -
 scripts/atomic/fallbacks/add_negative        |   14 +-
 scripts/atomic/fallbacks/add_unless          |   15 +-
 scripts/atomic/fallbacks/andnot              |    6 +-
 scripts/atomic/fallbacks/cmpxchg             |    3 +
 scripts/atomic/fallbacks/dec                 |    6 +-
 scripts/atomic/fallbacks/dec_and_test        |   14 +-
 scripts/atomic/fallbacks/dec_if_positive     |    8 +-
 scripts/atomic/fallbacks/dec_unless_positive |    8 +-
 scripts/atomic/fallbacks/fence               |    4 -
 scripts/atomic/fallbacks/fetch_add_unless    |   17 +-
 scripts/atomic/fallbacks/inc                 |    6 +-
 scripts/atomic/fallbacks/inc_and_test        |   14 +-
 scripts/atomic/fallbacks/inc_not_zero        |   13 +-
 scripts/atomic/fallbacks/inc_unless_negative |    8 +-
 scripts/atomic/fallbacks/read_acquire        |    6 +-
 scripts/atomic/fallbacks/release             |    4 -
 scripts/atomic/fallbacks/set_release         |    6 +-
 scripts/atomic/fallbacks/sub_and_test        |   15 +-
 scripts/atomic/fallbacks/try_cmpxchg         |    6 +-
 scripts/atomic/fallbacks/xchg                |    3 +
 scripts/atomic/gen-atomic-fallback.sh        |  264 +-
 scripts/atomic/gen-atomic-instrumented.sh    |   23 +-
 scripts/atomic/gen-atomic-long.sh            |   38 +-
 scripts/atomic/kerneldoc/add                 |   13 +
 scripts/atomic/kerneldoc/add_negative        |   13 +
 scripts/atomic/kerneldoc/add_unless          |   18 +
 scripts/atomic/kerneldoc/and                 |   13 +
 scripts/atomic/kerneldoc/andnot              |   13 +
 scripts/atomic/kerneldoc/cmpxchg             |   14 +
 scripts/atomic/kerneldoc/dec                 |   12 +
 scripts/atomic/kerneldoc/dec_and_test        |   12 +
 scripts/atomic/kerneldoc/dec_if_positive     |   12 +
 scripts/atomic/kerneldoc/dec_unless_positive |   12 +
 scripts/atomic/kerneldoc/inc                 |   12 +
 scripts/atomic/kerneldoc/inc_and_test        |   12 +
 scripts/atomic/kerneldoc/inc_not_zero        |   12 +
 scripts/atomic/kerneldoc/inc_unless_negative |   12 +
 scripts/atomic/kerneldoc/or                  |   13 +
 scripts/atomic/kerneldoc/read                |   12 +
 scripts/atomic/kerneldoc/set                 |   13 +
 scripts/atomic/kerneldoc/sub                 |   13 +
 scripts/atomic/kerneldoc/sub_and_test        |   13 +
 scripts/atomic/kerneldoc/try_cmpxchg         |   15 +
 scripts/atomic/kerneldoc/xchg                |   13 +
 scripts/atomic/kerneldoc/xor                 |   13 +
 scripts/kernel-doc                           |    2 +-
 101 files changed, 8979 insertions(+), 3689 deletions(-)
 create mode 100755 scripts/atomic/fallbacks/cmpxchg
 create mode 100755 scripts/atomic/fallbacks/xchg
 create mode 100644 scripts/atomic/kerneldoc/add
 create mode 100644 scripts/atomic/kerneldoc/add_negative
 create mode 100644 scripts/atomic/kerneldoc/add_unless
 create mode 100644 scripts/atomic/kerneldoc/and
 create mode 100644 scripts/atomic/kerneldoc/andnot
 create mode 100644 scripts/atomic/kerneldoc/cmpxchg
 create mode 100644 scripts/atomic/kerneldoc/dec
 create mode 100644 scripts/atomic/kerneldoc/dec_and_test
 create mode 100644 scripts/atomic/kerneldoc/dec_if_positive
 create mode 100644 scripts/atomic/kerneldoc/dec_unless_positive
 create mode 100644 scripts/atomic/kerneldoc/inc
 create mode 100644 scripts/atomic/kerneldoc/inc_and_test
 create mode 100644 scripts/atomic/kerneldoc/inc_not_zero
 create mode 100644 scripts/atomic/kerneldoc/inc_unless_negative
 create mode 100644 scripts/atomic/kerneldoc/or
 create mode 100644 scripts/atomic/kerneldoc/read
 create mode 100644 scripts/atomic/kerneldoc/set
 create mode 100644 scripts/atomic/kerneldoc/sub
 create mode 100644 scripts/atomic/kerneldoc/sub_and_test
 create mode 100644 scripts/atomic/kerneldoc/try_cmpxchg
 create mode 100644 scripts/atomic/kerneldoc/xchg
 create mode 100644 scripts/atomic/kerneldoc/xor
Signature · 1 line
2.30.2
author reviewer
#1 lore raw

PATCH v2 1/27 locking/atomic: arm: fix sync ops

The sync_*() ops on arch/arm are defined in terms of the regular bitops with no special handling. This is not correct, as UP kernels elide barriers for the fully-ordered operations, and so the required ordering is lost when such UP kernels are run under a hypervsior on an SMP system.

Fix this by defining sync ops with the required barriers.

Note: On 32-bit arm, the sync_*() ops are currently only used by Xen, which requires ARMv7, but the semantics can be implemented for ARMv6+.

Fixes e54d2f61528165bb ("xen/arm: sync_bitops")
Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Russell King <[email protected]>
Cc Stefano Stabellini <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/assembler.h   | 17 +++++++++++++++++
 arch/arm/include/asm/sync_bitops.h | 29 +++++++++++++++++++++++++----
 arch/arm/lib/bitops.h              | 14 +++++++++++---
 arch/arm/lib/testchangebit.S       |  4 ++++
 arch/arm/lib/testclearbit.S        |  4 ++++
 arch/arm/lib/testsetbit.S          |  4 ++++
 6 files changed, 65 insertions(+), 7 deletions(-)
arch/arm/include/asm/assembler.h +17 -0
index 505a306e0271a..aebe2c8f6a686 100644
@@ -394,6 +394,23 @@ ALT_UP_B(.L0_\@)
 #endif
 	.endm
 
+/*
+ * Raw SMP data memory barrier
+ */
+	.macro	__smp_dmb mode
+#if __LINUX_ARM_ARCH__ >= 7
+	.ifeqs "\mode","arm"
+	dmb	ish
+	.else
+	W(dmb)	ish
+	.endif
+#elif __LINUX_ARM_ARCH__ == 6
+	mcr	p15, 0, r0, c7, c10, 5	@ dmb
+#else
+	.error "Incompatible SMP platform"
+#endif
+	.endm
+
 #if defined(CONFIG_CPU_V7M)
 	/*
 	 * setmode is used to assert to be in svc mode during boot. For v7-M
arch/arm/include/asm/sync_bitops.h +25 -4
index 6f5d627c44a3c..f46b3c570f92e 100644
@@ -14,14 +14,35 @@
  * ops which are SMP safe even on a UP kernel.
  */
 
+/*
+ * Unordered
+ */
+
 #define sync_set_bit(nr, p)		_set_bit(nr, p)
 #define sync_clear_bit(nr, p)		_clear_bit(nr, p)
 #define sync_change_bit(nr, p)		_change_bit(nr, p)
-#define sync_test_and_set_bit(nr, p)	_test_and_set_bit(nr, p)
-#define sync_test_and_clear_bit(nr, p)	_test_and_clear_bit(nr, p)
-#define sync_test_and_change_bit(nr, p)	_test_and_change_bit(nr, p)
 #define sync_test_bit(nr, addr)		test_bit(nr, addr)
-#define arch_sync_cmpxchg		arch_cmpxchg
 
+/*
+ * Fully ordered
+ */
+
+int _sync_test_and_set_bit(int nr, volatile unsigned long * p);
+#define sync_test_and_set_bit(nr, p)	_sync_test_and_set_bit(nr, p)
+
+int _sync_test_and_clear_bit(int nr, volatile unsigned long * p);
+#define sync_test_and_clear_bit(nr, p)	_sync_test_and_clear_bit(nr, p)
+
+int _sync_test_and_change_bit(int nr, volatile unsigned long * p);
+#define sync_test_and_change_bit(nr, p)	_sync_test_and_change_bit(nr, p)
+
+#define arch_sync_cmpxchg(ptr, old, new)				\
+({									\
+	__typeof__(*(ptr)) __ret;					\
+	__smp_mb__before_atomic();					\
+	__ret = arch_cmpxchg_relaxed((ptr), (old), (new));		\
+	__smp_mb__after_atomic();					\
+	__ret;								\
+})
 
 #endif
arch/arm/lib/bitops.h +11 -3
index 95bd359912889..f069d1b2318e6 100644
@@ -28,7 +28,7 @@ UNWIND(	.fnend		)
 ENDPROC(\name		)
 	.endm
 
-	.macro	testop, name, instr, store
+	.macro	__testop, name, instr, store, barrier
 ENTRY(	\name		)
 UNWIND(	.fnstart	)
 	ands	ip, r1, #3
@@ -38,7 +38,7 @@ UNWIND(	.fnstart	)
 	mov	r0, r0, lsr #5
 	add	r1, r1, r0, lsl #2	@ Get word offset
 	mov	r3, r2, lsl r3		@ create mask
-	smp_dmb
+	\barrier
 #if __LINUX_ARM_ARCH__ >= 7 && defined(CONFIG_SMP)
 	.arch_extension	mp
 	ALT_SMP(W(pldw)	[r1])
@@ -50,13 +50,21 @@ UNWIND(	.fnstart	)
 	strex	ip, r2, [r1]
 	cmp	ip, #0
 	bne	1b
-	smp_dmb
+	\barrier
 	cmp	r0, #0
 	movne	r0, #1
 2:	bx	lr
 UNWIND(	.fnend		)
 ENDPROC(\name		)
 	.endm
+
+	.macro	testop, name, instr, store
+	__testop \name, \instr, \store, smp_dmb
+	.endm
+
+	.macro	sync_testop, name, instr, store
+	__testop \name, \instr, \store, __smp_dmb
+	.endm
 #else
 	.macro	bitop, name, instr
 ENTRY(	\name		)
arch/arm/lib/testchangebit.S +4 -0
index 4ebecc67e6e04..f13fe9bc2399a 100644
@@ -10,3 +10,7 @@
                 .text
 
 testop	_test_and_change_bit, eor, str
+
+#if __LINUX_ARM_ARCH__ >= 6
+sync_testop	_sync_test_and_change_bit, eor, str
+#endif
arch/arm/lib/testclearbit.S +4 -0
index 009afa0f5b4a7..4d2c5ca620ebf 100644
@@ -10,3 +10,7 @@
                 .text
 
 testop	_test_and_clear_bit, bicne, strne
+
+#if __LINUX_ARM_ARCH__ >= 6
+sync_testop	_sync_test_and_clear_bit, bicne, strne
+#endif
arch/arm/lib/testsetbit.S +4 -0
index f3192e55acc87..649dbab65d8d0 100644
@@ -10,3 +10,7 @@
                 .text
 
 testop	_test_and_set_bit, orreq, streq
+
+#if __LINUX_ARM_ARCH__ >= 6
+sync_testop	_sync_test_and_set_bit, orreq, streq
+#endif
Signature · 1 line
2.30.2
merge bot Applied
#2 lore raw

[tip: locking/core] locking/atomic: arm: fix sync ops

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     dda5f312bb09e56e7a1c3e3851f2000eb2e9c879
Gitweb:        https://git.kernel.org/tip/dda5f312bb09e56e7a1c3e3851f2000eb2e9c879
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:00:58 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:13 +02:00

locking/atomic: arm: fix sync ops

The sync_*() ops on arch/arm are defined in terms of the regular bitops with no special handling. This is not correct, as UP kernels elide barriers for the fully-ordered operations, and so the required ordering is lost when such UP kernels are run under a hypervsior on an SMP system.

Fix this by defining sync ops with the required barriers.

Note: On 32-bit arm, the sync_*() ops are currently only used by Xen, which requires ARMv7, but the semantics can be implemented for ARMv6+.

---
 arch/arm/include/asm/assembler.h   | 17 +++++++++++++++++-
 arch/arm/include/asm/sync_bitops.h | 29 +++++++++++++++++++++++++----
 arch/arm/lib/bitops.h              | 14 +++++++++++---
 arch/arm/lib/testchangebit.S       |  4 ++++-
 arch/arm/lib/testclearbit.S        |  4 ++++-
 arch/arm/lib/testsetbit.S          |  4 ++++-
 6 files changed, 65 insertions(+), 7 deletions(-)
arch/arm/include/asm/assembler.h +17 -0
index 505a306..aebe2c8 100644
@@ -394,6 +394,23 @@ ALT_UP_B(.L0_\@)
 #endif
 	.endm
 
+/*
+ * Raw SMP data memory barrier
+ */
+	.macro	__smp_dmb mode
+#if __LINUX_ARM_ARCH__ >= 7
+	.ifeqs "\mode","arm"
+	dmb	ish
+	.else
+	W(dmb)	ish
+	.endif
+#elif __LINUX_ARM_ARCH__ == 6
+	mcr	p15, 0, r0, c7, c10, 5	@ dmb
+#else
+	.error "Incompatible SMP platform"
+#endif
+	.endm
+
 #if defined(CONFIG_CPU_V7M)
 	/*
 	 * setmode is used to assert to be in svc mode during boot. For v7-M
arch/arm/include/asm/sync_bitops.h +25 -4
index 6f5d627..f46b3c5 100644
@@ -14,14 +14,35 @@
  * ops which are SMP safe even on a UP kernel.
  */
 
+/*
+ * Unordered
+ */
+
 #define sync_set_bit(nr, p)		_set_bit(nr, p)
 #define sync_clear_bit(nr, p)		_clear_bit(nr, p)
 #define sync_change_bit(nr, p)		_change_bit(nr, p)
-#define sync_test_and_set_bit(nr, p)	_test_and_set_bit(nr, p)
-#define sync_test_and_clear_bit(nr, p)	_test_and_clear_bit(nr, p)
-#define sync_test_and_change_bit(nr, p)	_test_and_change_bit(nr, p)
 #define sync_test_bit(nr, addr)		test_bit(nr, addr)
-#define arch_sync_cmpxchg		arch_cmpxchg
 
+/*
+ * Fully ordered
+ */
+
+int _sync_test_and_set_bit(int nr, volatile unsigned long * p);
+#define sync_test_and_set_bit(nr, p)	_sync_test_and_set_bit(nr, p)
+
+int _sync_test_and_clear_bit(int nr, volatile unsigned long * p);
+#define sync_test_and_clear_bit(nr, p)	_sync_test_and_clear_bit(nr, p)
+
+int _sync_test_and_change_bit(int nr, volatile unsigned long * p);
+#define sync_test_and_change_bit(nr, p)	_sync_test_and_change_bit(nr, p)
+
+#define arch_sync_cmpxchg(ptr, old, new)				\
+({									\
+	__typeof__(*(ptr)) __ret;					\
+	__smp_mb__before_atomic();					\
+	__ret = arch_cmpxchg_relaxed((ptr), (old), (new));		\
+	__smp_mb__after_atomic();					\
+	__ret;								\
+})
 
 #endif
arch/arm/lib/bitops.h +11 -3
index 95bd359..f069d1b 100644
@@ -28,7 +28,7 @@ UNWIND(	.fnend		)
 ENDPROC(\name		)
 	.endm
 
-	.macro	testop, name, instr, store
+	.macro	__testop, name, instr, store, barrier
 ENTRY(	\name		)
 UNWIND(	.fnstart	)
 	ands	ip, r1, #3
@@ -38,7 +38,7 @@ UNWIND(	.fnstart	)
 	mov	r0, r0, lsr #5
 	add	r1, r1, r0, lsl #2	@ Get word offset
 	mov	r3, r2, lsl r3		@ create mask
-	smp_dmb
+	\barrier
 #if __LINUX_ARM_ARCH__ >= 7 && defined(CONFIG_SMP)
 	.arch_extension	mp
 	ALT_SMP(W(pldw)	[r1])
@@ -50,13 +50,21 @@ UNWIND(	.fnstart	)
 	strex	ip, r2, [r1]
 	cmp	ip, #0
 	bne	1b
-	smp_dmb
+	\barrier
 	cmp	r0, #0
 	movne	r0, #1
 2:	bx	lr
 UNWIND(	.fnend		)
 ENDPROC(\name		)
 	.endm
+
+	.macro	testop, name, instr, store
+	__testop \name, \instr, \store, smp_dmb
+	.endm
+
+	.macro	sync_testop, name, instr, store
+	__testop \name, \instr, \store, __smp_dmb
+	.endm
 #else
 	.macro	bitop, name, instr
 ENTRY(	\name		)
arch/arm/lib/testchangebit.S +4 -0
index 4ebecc6..f13fe9b 100644
@@ -10,3 +10,7 @@
                 .text
 
 testop	_test_and_change_bit, eor, str
+
+#if __LINUX_ARM_ARCH__ >= 6
+sync_testop	_sync_test_and_change_bit, eor, str
+#endif
arch/arm/lib/testclearbit.S +4 -0
index 009afa0..4d2c5ca 100644
@@ -10,3 +10,7 @@
                 .text
 
 testop	_test_and_clear_bit, bicne, strne
+
+#if __LINUX_ARM_ARCH__ >= 6
+sync_testop	_sync_test_and_clear_bit, bicne, strne
+#endif
arch/arm/lib/testsetbit.S +4 -0
index f3192e5..649dbab 100644
@@ -10,3 +10,7 @@
                 .text
 
 testop	_test_and_set_bit, orreq, streq
+
+#if __LINUX_ARM_ARCH__ >= 6
+sync_testop	_sync_test_and_set_bit, orreq, streq
+#endif
author reviewer
#3 lore raw

PATCH v2 2/27 locking/atomic: remove fallback comments

In reply to Mark Rutland

Currently a subset of the fallback templates have kerneldoc comments, resulting in a haphazard set of generated kerneldoc comments as only some operations have fallback templates to begin with.

We'd like to generate more consistent kerneldoc comments, and to do so we'll need to restructure the way the fallback code is generated.

To minimize churn and to make it easier to restructure the fallback code, this patch removes the existing kerneldoc comments from the fallback templates. We can add new kerneldoc comments in subsequent patches.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 include/linux/atomic/atomic-arch-fallback.h | 166 +-------------------
 scripts/atomic/fallbacks/add_negative       |   8 -
 scripts/atomic/fallbacks/add_unless         |   9 --
 scripts/atomic/fallbacks/dec_and_test       |   8 -
 scripts/atomic/fallbacks/fetch_add_unless   |   9 --
 scripts/atomic/fallbacks/inc_and_test       |   8 -
 scripts/atomic/fallbacks/inc_not_zero       |   7 -
 scripts/atomic/fallbacks/sub_and_test       |   9 --
 8 files changed, 1 insertion(+), 223 deletions(-)
include/linux/atomic/atomic-arch-fallback.h +1 -165
index 1722ddb6f17e0..3ce4cb5e790c5 100644
@@ -1272,15 +1272,6 @@ arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
 #endif /* arch_atomic_try_cmpxchg_relaxed */
 
 #ifndef arch_atomic_sub_and_test
-/**
- * arch_atomic_sub_and_test - subtract value from variable and test result
- * @i: integer value to subtract
- * @v: pointer of type atomic_t
- *
- * Atomically subtracts @i from @v and returns
- * true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic_sub_and_test(int i, atomic_t *v)
 {
@@ -1290,14 +1281,6 @@ arch_atomic_sub_and_test(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_dec_and_test
-/**
- * arch_atomic_dec_and_test - decrement and test
- * @v: pointer of type atomic_t
- *
- * Atomically decrements @v by 1 and
- * returns true if the result is 0, or false for all other
- * cases.
- */
 static __always_inline bool
 arch_atomic_dec_and_test(atomic_t *v)
 {
@@ -1307,14 +1290,6 @@ arch_atomic_dec_and_test(atomic_t *v)
 #endif
 
 #ifndef arch_atomic_inc_and_test
-/**
- * arch_atomic_inc_and_test - increment and test
- * @v: pointer of type atomic_t
- *
- * Atomically increments @v by 1
- * and returns true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic_inc_and_test(atomic_t *v)
 {
@@ -1331,14 +1306,6 @@ arch_atomic_inc_and_test(atomic_t *v)
 #endif /* arch_atomic_add_negative */
 
 #ifndef arch_atomic_add_negative
-/**
- * arch_atomic_add_negative - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative(int i, atomic_t *v)
 {
@@ -1348,14 +1315,6 @@ arch_atomic_add_negative(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_add_negative_acquire
-/**
- * arch_atomic_add_negative_acquire - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative_acquire(int i, atomic_t *v)
 {
@@ -1365,14 +1324,6 @@ arch_atomic_add_negative_acquire(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_add_negative_release
-/**
- * arch_atomic_add_negative_release - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative_release(int i, atomic_t *v)
 {
@@ -1382,14 +1333,6 @@ arch_atomic_add_negative_release(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_add_negative_relaxed
-/**
- * arch_atomic_add_negative_relaxed - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative_relaxed(int i, atomic_t *v)
 {
@@ -1437,15 +1380,6 @@ arch_atomic_add_negative(int i, atomic_t *v)
 #endif /* arch_atomic_add_negative_relaxed */
 
 #ifndef arch_atomic_fetch_add_unless
-/**
- * arch_atomic_fetch_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as @v was not already @u.
- * Returns original value of @v
- */
 static __always_inline int
 arch_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 {
@@ -1462,15 +1396,6 @@ arch_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 #endif
 
 #ifndef arch_atomic_add_unless
-/**
- * arch_atomic_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, if @v was not already @u.
- * Returns true if the addition was done.
- */
 static __always_inline bool
 arch_atomic_add_unless(atomic_t *v, int a, int u)
 {
@@ -1480,13 +1405,6 @@ arch_atomic_add_unless(atomic_t *v, int a, int u)
 #endif
 
 #ifndef arch_atomic_inc_not_zero
-/**
- * arch_atomic_inc_not_zero - increment unless the number is zero
- * @v: pointer of type atomic_t
- *
- * Atomically increments @v by 1, if @v is non-zero.
- * Returns true if the increment was done.
- */
 static __always_inline bool
 arch_atomic_inc_not_zero(atomic_t *v)
 {
@@ -2488,15 +2406,6 @@ arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
 #endif /* arch_atomic64_try_cmpxchg_relaxed */
 
 #ifndef arch_atomic64_sub_and_test
-/**
- * arch_atomic64_sub_and_test - subtract value from variable and test result
- * @i: integer value to subtract
- * @v: pointer of type atomic64_t
- *
- * Atomically subtracts @i from @v and returns
- * true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic64_sub_and_test(s64 i, atomic64_t *v)
 {
@@ -2506,14 +2415,6 @@ arch_atomic64_sub_and_test(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_dec_and_test
-/**
- * arch_atomic64_dec_and_test - decrement and test
- * @v: pointer of type atomic64_t
- *
- * Atomically decrements @v by 1 and
- * returns true if the result is 0, or false for all other
- * cases.
- */
 static __always_inline bool
 arch_atomic64_dec_and_test(atomic64_t *v)
 {
@@ -2523,14 +2424,6 @@ arch_atomic64_dec_and_test(atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_inc_and_test
-/**
- * arch_atomic64_inc_and_test - increment and test
- * @v: pointer of type atomic64_t
- *
- * Atomically increments @v by 1
- * and returns true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic64_inc_and_test(atomic64_t *v)
 {
@@ -2547,14 +2440,6 @@ arch_atomic64_inc_and_test(atomic64_t *v)
 #endif /* arch_atomic64_add_negative */
 
 #ifndef arch_atomic64_add_negative
-/**
- * arch_atomic64_add_negative - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative(s64 i, atomic64_t *v)
 {
@@ -2564,14 +2449,6 @@ arch_atomic64_add_negative(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_add_negative_acquire
-/**
- * arch_atomic64_add_negative_acquire - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative_acquire(s64 i, atomic64_t *v)
 {
@@ -2581,14 +2458,6 @@ arch_atomic64_add_negative_acquire(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_add_negative_release
-/**
- * arch_atomic64_add_negative_release - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative_release(s64 i, atomic64_t *v)
 {
@@ -2598,14 +2467,6 @@ arch_atomic64_add_negative_release(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_add_negative_relaxed
-/**
- * arch_atomic64_add_negative_relaxed - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative_relaxed(s64 i, atomic64_t *v)
 {
@@ -2653,15 +2514,6 @@ arch_atomic64_add_negative(s64 i, atomic64_t *v)
 #endif /* arch_atomic64_add_negative_relaxed */
 
 #ifndef arch_atomic64_fetch_add_unless
-/**
- * arch_atomic64_fetch_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic64_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as @v was not already @u.
- * Returns original value of @v
- */
 static __always_inline s64
 arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
 {
@@ -2678,15 +2530,6 @@ arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
 #endif
 
 #ifndef arch_atomic64_add_unless
-/**
- * arch_atomic64_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic64_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, if @v was not already @u.
- * Returns true if the addition was done.
- */
 static __always_inline bool
 arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
 {
@@ -2696,13 +2539,6 @@ arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
 #endif
 
 #ifndef arch_atomic64_inc_not_zero
-/**
- * arch_atomic64_inc_not_zero - increment unless the number is zero
- * @v: pointer of type atomic64_t
- *
- * Atomically increments @v by 1, if @v is non-zero.
- * Returns true if the increment was done.
- */
 static __always_inline bool
 arch_atomic64_inc_not_zero(atomic64_t *v)
 {
@@ -2761,4 +2597,4 @@ arch_atomic64_dec_if_positive(atomic64_t *v)
 #endif
 
 #endif /* _LINUX_ATOMIC_FALLBACK_H */
-// 52dfc6fe4a2e7234bbd2aa3e16a377c1db793a53
+// 9f0fd6ed53267c6ec64e36cd18e6fd8df57ea277
scripts/atomic/fallbacks/add_negative +0 -8
index e5980abf5904e..d0bd2dfbb244c 100755
@@ -1,12 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_add_negative${order} - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type ${atomic}_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_${atomic}_add_negative${order}(${int} i, ${atomic}_t *v)
 {
scripts/atomic/fallbacks/add_unless +0 -9
index 9e5159c2ccfc8..cf79b9da38dbb 100755
@@ -1,13 +1,4 @@
 cat << EOF
-/**
- * arch_${atomic}_add_unless - add unless the number is already a given value
- * @v: pointer of type ${atomic}_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, if @v was not already @u.
- * Returns true if the addition was done.
- */
 static __always_inline bool
 arch_${atomic}_add_unless(${atomic}_t *v, ${int} a, ${int} u)
 {
scripts/atomic/fallbacks/dec_and_test +0 -8
index 8549f359bd0ef..3f6b6a8b47733 100755
@@ -1,12 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_dec_and_test - decrement and test
- * @v: pointer of type ${atomic}_t
- *
- * Atomically decrements @v by 1 and
- * returns true if the result is 0, or false for all other
- * cases.
- */
 static __always_inline bool
 arch_${atomic}_dec_and_test(${atomic}_t *v)
 {
scripts/atomic/fallbacks/fetch_add_unless +0 -9
index 68ce13c8b9dad..81d2834f03d23 100755
@@ -1,13 +1,4 @@
 cat << EOF
-/**
- * arch_${atomic}_fetch_add_unless - add unless the number is already a given value
- * @v: pointer of type ${atomic}_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as @v was not already @u.
- * Returns original value of @v
- */
 static __always_inline ${int}
 arch_${atomic}_fetch_add_unless(${atomic}_t *v, ${int} a, ${int} u)
 {
scripts/atomic/fallbacks/inc_and_test +0 -8
index 0cf23fe1efb85..c726a6d0634d3 100755
@@ -1,12 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_inc_and_test - increment and test
- * @v: pointer of type ${atomic}_t
- *
- * Atomically increments @v by 1
- * and returns true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_${atomic}_inc_and_test(${atomic}_t *v)
 {
scripts/atomic/fallbacks/inc_not_zero +0 -7
index ed8a1f5626675..97603591aac2a 100755
@@ -1,11 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_inc_not_zero - increment unless the number is zero
- * @v: pointer of type ${atomic}_t
- *
- * Atomically increments @v by 1, if @v is non-zero.
- * Returns true if the increment was done.
- */
 static __always_inline bool
 arch_${atomic}_inc_not_zero(${atomic}_t *v)
 {
scripts/atomic/fallbacks/sub_and_test +0 -9
index 260f37341c888..da8a049c9b02b 100755
@@ -1,13 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_sub_and_test - subtract value from variable and test result
- * @i: integer value to subtract
- * @v: pointer of type ${atomic}_t
- *
- * Atomically subtracts @i from @v and returns
- * true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_${atomic}_sub_and_test(${int} i, ${atomic}_t *v)
 {
Signature · 1 line
2.30.2
merge bot Applied
#4 lore raw

[tip: locking/core] locking/atomic: remove fallback comments

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     14d72d4b6f0e88b5f683c1a5b7a876a55055852d
Gitweb:        https://git.kernel.org/tip/14d72d4b6f0e88b5f683c1a5b7a876a55055852d
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:00:59 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:13 +02:00

locking/atomic: remove fallback comments

Currently a subset of the fallback templates have kerneldoc comments, resulting in a haphazard set of generated kerneldoc comments as only some operations have fallback templates to begin with.

We'd like to generate more consistent kerneldoc comments, and to do so we'll need to restructure the way the fallback code is generated.

To minimize churn and to make it easier to restructure the fallback code, this patch removes the existing kerneldoc comments from the fallback templates. We can add new kerneldoc comments in subsequent patches.

There should be no functional change as a result of this patch.

---
 include/linux/atomic/atomic-arch-fallback.h | 166 +-------------------
 scripts/atomic/fallbacks/add_negative       |   8 +-
 scripts/atomic/fallbacks/add_unless         |   9 +-
 scripts/atomic/fallbacks/dec_and_test       |   8 +-
 scripts/atomic/fallbacks/fetch_add_unless   |   9 +-
 scripts/atomic/fallbacks/inc_and_test       |   8 +-
 scripts/atomic/fallbacks/inc_not_zero       |   7 +-
 scripts/atomic/fallbacks/sub_and_test       |   9 +-
 8 files changed, 1 insertion(+), 223 deletions(-)
include/linux/atomic/atomic-arch-fallback.h +1 -165
index 1722ddb..3ce4cb5 100644
@@ -1272,15 +1272,6 @@ arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
 #endif /* arch_atomic_try_cmpxchg_relaxed */
 
 #ifndef arch_atomic_sub_and_test
-/**
- * arch_atomic_sub_and_test - subtract value from variable and test result
- * @i: integer value to subtract
- * @v: pointer of type atomic_t
- *
- * Atomically subtracts @i from @v and returns
- * true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic_sub_and_test(int i, atomic_t *v)
 {
@@ -1290,14 +1281,6 @@ arch_atomic_sub_and_test(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_dec_and_test
-/**
- * arch_atomic_dec_and_test - decrement and test
- * @v: pointer of type atomic_t
- *
- * Atomically decrements @v by 1 and
- * returns true if the result is 0, or false for all other
- * cases.
- */
 static __always_inline bool
 arch_atomic_dec_and_test(atomic_t *v)
 {
@@ -1307,14 +1290,6 @@ arch_atomic_dec_and_test(atomic_t *v)
 #endif
 
 #ifndef arch_atomic_inc_and_test
-/**
- * arch_atomic_inc_and_test - increment and test
- * @v: pointer of type atomic_t
- *
- * Atomically increments @v by 1
- * and returns true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic_inc_and_test(atomic_t *v)
 {
@@ -1331,14 +1306,6 @@ arch_atomic_inc_and_test(atomic_t *v)
 #endif /* arch_atomic_add_negative */
 
 #ifndef arch_atomic_add_negative
-/**
- * arch_atomic_add_negative - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative(int i, atomic_t *v)
 {
@@ -1348,14 +1315,6 @@ arch_atomic_add_negative(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_add_negative_acquire
-/**
- * arch_atomic_add_negative_acquire - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative_acquire(int i, atomic_t *v)
 {
@@ -1365,14 +1324,6 @@ arch_atomic_add_negative_acquire(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_add_negative_release
-/**
- * arch_atomic_add_negative_release - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative_release(int i, atomic_t *v)
 {
@@ -1382,14 +1333,6 @@ arch_atomic_add_negative_release(int i, atomic_t *v)
 #endif
 
 #ifndef arch_atomic_add_negative_relaxed
-/**
- * arch_atomic_add_negative_relaxed - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic_add_negative_relaxed(int i, atomic_t *v)
 {
@@ -1437,15 +1380,6 @@ arch_atomic_add_negative(int i, atomic_t *v)
 #endif /* arch_atomic_add_negative_relaxed */
 
 #ifndef arch_atomic_fetch_add_unless
-/**
- * arch_atomic_fetch_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as @v was not already @u.
- * Returns original value of @v
- */
 static __always_inline int
 arch_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 {
@@ -1462,15 +1396,6 @@ arch_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 #endif
 
 #ifndef arch_atomic_add_unless
-/**
- * arch_atomic_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, if @v was not already @u.
- * Returns true if the addition was done.
- */
 static __always_inline bool
 arch_atomic_add_unless(atomic_t *v, int a, int u)
 {
@@ -1480,13 +1405,6 @@ arch_atomic_add_unless(atomic_t *v, int a, int u)
 #endif
 
 #ifndef arch_atomic_inc_not_zero
-/**
- * arch_atomic_inc_not_zero - increment unless the number is zero
- * @v: pointer of type atomic_t
- *
- * Atomically increments @v by 1, if @v is non-zero.
- * Returns true if the increment was done.
- */
 static __always_inline bool
 arch_atomic_inc_not_zero(atomic_t *v)
 {
@@ -2488,15 +2406,6 @@ arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
 #endif /* arch_atomic64_try_cmpxchg_relaxed */
 
 #ifndef arch_atomic64_sub_and_test
-/**
- * arch_atomic64_sub_and_test - subtract value from variable and test result
- * @i: integer value to subtract
- * @v: pointer of type atomic64_t
- *
- * Atomically subtracts @i from @v and returns
- * true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic64_sub_and_test(s64 i, atomic64_t *v)
 {
@@ -2506,14 +2415,6 @@ arch_atomic64_sub_and_test(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_dec_and_test
-/**
- * arch_atomic64_dec_and_test - decrement and test
- * @v: pointer of type atomic64_t
- *
- * Atomically decrements @v by 1 and
- * returns true if the result is 0, or false for all other
- * cases.
- */
 static __always_inline bool
 arch_atomic64_dec_and_test(atomic64_t *v)
 {
@@ -2523,14 +2424,6 @@ arch_atomic64_dec_and_test(atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_inc_and_test
-/**
- * arch_atomic64_inc_and_test - increment and test
- * @v: pointer of type atomic64_t
- *
- * Atomically increments @v by 1
- * and returns true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_atomic64_inc_and_test(atomic64_t *v)
 {
@@ -2547,14 +2440,6 @@ arch_atomic64_inc_and_test(atomic64_t *v)
 #endif /* arch_atomic64_add_negative */
 
 #ifndef arch_atomic64_add_negative
-/**
- * arch_atomic64_add_negative - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative(s64 i, atomic64_t *v)
 {
@@ -2564,14 +2449,6 @@ arch_atomic64_add_negative(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_add_negative_acquire
-/**
- * arch_atomic64_add_negative_acquire - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative_acquire(s64 i, atomic64_t *v)
 {
@@ -2581,14 +2458,6 @@ arch_atomic64_add_negative_acquire(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_add_negative_release
-/**
- * arch_atomic64_add_negative_release - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative_release(s64 i, atomic64_t *v)
 {
@@ -2598,14 +2467,6 @@ arch_atomic64_add_negative_release(s64 i, atomic64_t *v)
 #endif
 
 #ifndef arch_atomic64_add_negative_relaxed
-/**
- * arch_atomic64_add_negative_relaxed - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type atomic64_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_atomic64_add_negative_relaxed(s64 i, atomic64_t *v)
 {
@@ -2653,15 +2514,6 @@ arch_atomic64_add_negative(s64 i, atomic64_t *v)
 #endif /* arch_atomic64_add_negative_relaxed */
 
 #ifndef arch_atomic64_fetch_add_unless
-/**
- * arch_atomic64_fetch_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic64_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as @v was not already @u.
- * Returns original value of @v
- */
 static __always_inline s64
 arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
 {
@@ -2678,15 +2530,6 @@ arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
 #endif
 
 #ifndef arch_atomic64_add_unless
-/**
- * arch_atomic64_add_unless - add unless the number is already a given value
- * @v: pointer of type atomic64_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, if @v was not already @u.
- * Returns true if the addition was done.
- */
 static __always_inline bool
 arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
 {
@@ -2696,13 +2539,6 @@ arch_atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
 #endif
 
 #ifndef arch_atomic64_inc_not_zero
-/**
- * arch_atomic64_inc_not_zero - increment unless the number is zero
- * @v: pointer of type atomic64_t
- *
- * Atomically increments @v by 1, if @v is non-zero.
- * Returns true if the increment was done.
- */
 static __always_inline bool
 arch_atomic64_inc_not_zero(atomic64_t *v)
 {
@@ -2761,4 +2597,4 @@ arch_atomic64_dec_if_positive(atomic64_t *v)
 #endif
 
 #endif /* _LINUX_ATOMIC_FALLBACK_H */
-// 52dfc6fe4a2e7234bbd2aa3e16a377c1db793a53
+// 9f0fd6ed53267c6ec64e36cd18e6fd8df57ea277
scripts/atomic/fallbacks/add_negative +0 -8
index e5980ab..d0bd2df 100755
@@ -1,12 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_add_negative${order} - Add and test if negative
- * @i: integer value to add
- * @v: pointer of type ${atomic}_t
- *
- * Atomically adds @i to @v and returns true if the result is negative,
- * or false when the result is greater than or equal to zero.
- */
 static __always_inline bool
 arch_${atomic}_add_negative${order}(${int} i, ${atomic}_t *v)
 {
scripts/atomic/fallbacks/add_unless +0 -9
index 9e5159c..cf79b9d 100755
@@ -1,13 +1,4 @@
 cat << EOF
-/**
- * arch_${atomic}_add_unless - add unless the number is already a given value
- * @v: pointer of type ${atomic}_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, if @v was not already @u.
- * Returns true if the addition was done.
- */
 static __always_inline bool
 arch_${atomic}_add_unless(${atomic}_t *v, ${int} a, ${int} u)
 {
scripts/atomic/fallbacks/dec_and_test +0 -8
index 8549f35..3f6b6a8 100755
@@ -1,12 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_dec_and_test - decrement and test
- * @v: pointer of type ${atomic}_t
- *
- * Atomically decrements @v by 1 and
- * returns true if the result is 0, or false for all other
- * cases.
- */
 static __always_inline bool
 arch_${atomic}_dec_and_test(${atomic}_t *v)
 {
scripts/atomic/fallbacks/fetch_add_unless +0 -9
index 68ce13c..81d2834 100755
@@ -1,13 +1,4 @@
 cat << EOF
-/**
- * arch_${atomic}_fetch_add_unless - add unless the number is already a given value
- * @v: pointer of type ${atomic}_t
- * @a: the amount to add to v...
- * @u: ...unless v is equal to u.
- *
- * Atomically adds @a to @v, so long as @v was not already @u.
- * Returns original value of @v
- */
 static __always_inline ${int}
 arch_${atomic}_fetch_add_unless(${atomic}_t *v, ${int} a, ${int} u)
 {
scripts/atomic/fallbacks/inc_and_test +0 -8
index 0cf23fe..c726a6d 100755
@@ -1,12 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_inc_and_test - increment and test
- * @v: pointer of type ${atomic}_t
- *
- * Atomically increments @v by 1
- * and returns true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_${atomic}_inc_and_test(${atomic}_t *v)
 {
scripts/atomic/fallbacks/inc_not_zero +0 -7
index ed8a1f5..9760359 100755
@@ -1,11 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_inc_not_zero - increment unless the number is zero
- * @v: pointer of type ${atomic}_t
- *
- * Atomically increments @v by 1, if @v is non-zero.
- * Returns true if the increment was done.
- */
 static __always_inline bool
 arch_${atomic}_inc_not_zero(${atomic}_t *v)
 {
scripts/atomic/fallbacks/sub_and_test +0 -9
index 260f373..da8a049 100755
@@ -1,13 +1,4 @@
 cat <<EOF
-/**
- * arch_${atomic}_sub_and_test - subtract value from variable and test result
- * @i: integer value to subtract
- * @v: pointer of type ${atomic}_t
- *
- * Atomically subtracts @i from @v and returns
- * true if the result is zero, or false for all
- * other cases.
- */
 static __always_inline bool
 arch_${atomic}_sub_and_test(${int} i, ${atomic}_t *v)
 {
author reviewer
#5 lore raw

PATCH v2 3/27 locking/atomic: hexagon: remove redundant arch_atomic_cmpxchg

In reply to Mark Rutland

Hexagon's implementation of arch_atomic_cmpxchg() is identical to its implementation of arch_cmpxchg(). Have it define arch_atomic_cmpxchg() in terms of arch_cmpxchg(), matching what it does for arch_atomic_xchg() and arch_xchg().

At the same time, remove the kerneldoc comments for hexagon's arch_atomic_xchg() and arch_atomic_cmpxchg(). The arch_atomic_*() namespace is shared by all architectures and the API should be documented centrally, and the comments aren't all that helpful as-is.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/hexagon/include/asm/atomic.h | 46 +++----------------------------
 1 file changed, 4 insertions(+), 42 deletions(-)
arch/hexagon/include/asm/atomic.h +4 -42
index 6e94f8d04146f..738857e10d6ec 100644
@@ -36,49 +36,11 @@ static inline void arch_atomic_set(atomic_t *v, int new)
  */
 #define arch_atomic_read(v)		READ_ONCE((v)->counter)
 
-/**
- * arch_atomic_xchg - atomic
- * @v: pointer to memory to change
- * @new: new value (technically passed in a register -- see xchg)
- */
-#define arch_atomic_xchg(v, new)	(arch_xchg(&((v)->counter), (new)))
-
-
-/**
- * arch_atomic_cmpxchg - atomic compare-and-exchange values
- * @v: pointer to value to change
- * @old:  desired old value to match
- * @new:  new value to put in
- *
- * Parameters are then pointer, value-in-register, value-in-register,
- * and the output is the old value.
- *
- * Apparently this is complicated for archs that don't support
- * the memw_locked like we do (or it's broken or whatever).
- *
- * Kind of the lynchpin of the rest of the generically defined routines.
- * Remember V2 had that bug with dotnew predicate set by memw_locked.
- *
- * "old" is "expected" old val, __oldval is actual old value
- */
-static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
-{
-	int __oldval;
+#define arch_atomic_xchg(v, new)					\
+	(arch_xchg(&((v)->counter), (new)))
 
-	asm volatile(
-		"1:	%0 = memw_locked(%1);\n"
-		"	{ P0 = cmp.eq(%0,%2);\n"
-		"	  if (!P0.new) jump:nt 2f; }\n"
-		"	memw_locked(%1,P0) = %3;\n"
-		"	if (!P0) jump 1b;\n"
-		"2:\n"
-		: "=&r" (__oldval)
-		: "r" (&v->counter), "r" (old), "r" (new)
-		: "memory", "p0"
-	);
-
-	return __oldval;
-}
+#define arch_atomic_cmpxchg(v, old, new)				\
+	(arch_cmpxchg(&((v)->counter), (old), (new)))
 
 #define ATOMIC_OP(op)							\
 static inline void arch_atomic_##op(int i, atomic_t *v)			\
Signature · 1 line
2.30.2
merge bot Applied
#6 lore raw

[tip: locking/core] locking/atomic: hexagon: remove redundant arch_atomic_cmpxchg

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     a7bafa7969da1c0e9c342c792d8224078d1c491c
Gitweb:        https://git.kernel.org/tip/a7bafa7969da1c0e9c342c792d8224078d1c491c
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:00 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:13 +02:00

locking/atomic: hexagon: remove redundant arch_atomic_cmpxchg

Hexagon's implementation of arch_atomic_cmpxchg() is identical to its implementation of arch_cmpxchg(). Have it define arch_atomic_cmpxchg() in terms of arch_cmpxchg(), matching what it does for arch_atomic_xchg() and arch_xchg().

At the same time, remove the kerneldoc comments for hexagon's arch_atomic_xchg() and arch_atomic_cmpxchg(). The arch_atomic_*() namespace is shared by all architectures and the API should be documented centrally, and the comments aren't all that helpful as-is.

There should be no functional change as a result of this patch.

---
 arch/hexagon/include/asm/atomic.h | 46 ++----------------------------
 1 file changed, 4 insertions(+), 42 deletions(-)
arch/hexagon/include/asm/atomic.h +4 -42
index 6e94f8d..738857e 100644
@@ -36,49 +36,11 @@ static inline void arch_atomic_set(atomic_t *v, int new)
  */
 #define arch_atomic_read(v)		READ_ONCE((v)->counter)
 
-/**
- * arch_atomic_xchg - atomic
- * @v: pointer to memory to change
- * @new: new value (technically passed in a register -- see xchg)
- */
-#define arch_atomic_xchg(v, new)	(arch_xchg(&((v)->counter), (new)))
-
-
-/**
- * arch_atomic_cmpxchg - atomic compare-and-exchange values
- * @v: pointer to value to change
- * @old:  desired old value to match
- * @new:  new value to put in
- *
- * Parameters are then pointer, value-in-register, value-in-register,
- * and the output is the old value.
- *
- * Apparently this is complicated for archs that don't support
- * the memw_locked like we do (or it's broken or whatever).
- *
- * Kind of the lynchpin of the rest of the generically defined routines.
- * Remember V2 had that bug with dotnew predicate set by memw_locked.
- *
- * "old" is "expected" old val, __oldval is actual old value
- */
-static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
-{
-	int __oldval;
+#define arch_atomic_xchg(v, new)					\
+	(arch_xchg(&((v)->counter), (new)))
 
-	asm volatile(
-		"1:	%0 = memw_locked(%1);\n"
-		"	{ P0 = cmp.eq(%0,%2);\n"
-		"	  if (!P0.new) jump:nt 2f; }\n"
-		"	memw_locked(%1,P0) = %3;\n"
-		"	if (!P0) jump 1b;\n"
-		"2:\n"
-		: "=&r" (__oldval)
-		: "r" (&v->counter), "r" (old), "r" (new)
-		: "memory", "p0"
-	);
-
-	return __oldval;
-}
+#define arch_atomic_cmpxchg(v, old, new)				\
+	(arch_cmpxchg(&((v)->counter), (old), (new)))
 
 #define ATOMIC_OP(op)							\
 static inline void arch_atomic_##op(int i, atomic_t *v)			\
author reviewer
#7 lore raw

PATCH v2 4/27 locking/atomic: make atomic*_{cmp,}xchg optional

In reply to Mark Rutland

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/alpha/include/asm/atomic.h             |  10 --
 arch/arc/include/asm/atomic.h               |  24 ---
 arch/arc/include/asm/atomic64-arcv2.h       |   2 +
 arch/arm/include/asm/atomic.h               |   3 +-
 arch/arm64/include/asm/atomic.h             |  28 ----
 arch/csky/include/asm/atomic.h              |  35 -----
 arch/hexagon/include/asm/atomic.h           |   6 -
 arch/ia64/include/asm/atomic.h              |   7 -
 arch/loongarch/include/asm/atomic.h         |   7 -
 arch/m68k/include/asm/atomic.h              |   9 +-
 arch/mips/include/asm/atomic.h              |  11 --
 arch/openrisc/include/asm/atomic.h          |   3 -
 arch/parisc/include/asm/atomic.h            |   9 --
 arch/powerpc/include/asm/atomic.h           |  24 ---
 arch/riscv/include/asm/atomic.h             |  72 ---------
 arch/sh/include/asm/atomic.h                |   3 -
 arch/sparc/include/asm/atomic_32.h          |   2 +
 arch/sparc/include/asm/atomic_64.h          |  11 --
 arch/xtensa/include/asm/atomic.h            |   3 -
 include/asm-generic/atomic.h                |   3 -
 include/linux/atomic/atomic-arch-fallback.h | 158 +++++++++++++++++++-
 scripts/atomic/fallbacks/cmpxchg            |   7 +
 scripts/atomic/fallbacks/xchg               |   7 +
 23 files changed, 179 insertions(+), 265 deletions(-)
 create mode 100755 scripts/atomic/fallbacks/cmpxchg
 create mode 100755 scripts/atomic/fallbacks/xchg
arch/alpha/include/asm/atomic.h +0 -10
index f2861a43a61ef..ec8ab552c527a 100644
@@ -200,16 +200,6 @@ ATOMIC_OPS(xor, xor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic64_cmpxchg(v, old, new) \
-	(arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic64_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-
-#define arch_atomic_cmpxchg(v, old, new) \
-	(arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-
 /**
  * arch_atomic_fetch_add_unless - add unless the number is a given value
  * @v: pointer of type atomic_t
arch/arc/include/asm/atomic.h +0 -24
index 52ee51e1ff7c2..592d7fffc223c 100644
@@ -22,30 +22,6 @@
 #include <asm/atomic-spinlock.h>
 #endif
 
-#define arch_atomic_cmpxchg(v, o, n)					\
-({									\
-	arch_cmpxchg(&((v)->counter), (o), (n));			\
-})
-
-#ifdef arch_cmpxchg_relaxed
-#define arch_atomic_cmpxchg_relaxed(v, o, n)				\
-({									\
-	arch_cmpxchg_relaxed(&((v)->counter), (o), (n));		\
-})
-#endif
-
-#define arch_atomic_xchg(v, n)						\
-({									\
-	arch_xchg(&((v)->counter), (n));				\
-})
-
-#ifdef arch_xchg_relaxed
-#define arch_atomic_xchg_relaxed(v, n)					\
-({									\
-	arch_xchg_relaxed(&((v)->counter), (n));			\
-})
-#endif
-
 /*
  * 64-bit atomics
  */
arch/arc/include/asm/atomic64-arcv2.h +2 -0
index c5a8010fdc97d..2b7c9e61a2947 100644
@@ -159,6 +159,7 @@ arch_atomic64_cmpxchg(atomic64_t *ptr, s64 expected, s64 new)
 
 	return prev;
 }
+#define arch_atomic64_cmpxchg arch_atomic64_cmpxchg
 
 static inline s64 arch_atomic64_xchg(atomic64_t *ptr, s64 new)
 {
@@ -179,6 +180,7 @@ static inline s64 arch_atomic64_xchg(atomic64_t *ptr, s64 new)
 
 	return prev;
 }
+#define arch_atomic64_xchg arch_atomic64_xchg
 
 /**
  * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
arch/arm/include/asm/atomic.h +1 -2
index db8512d9a918d..9458d47ff209c 100644
@@ -210,6 +210,7 @@ static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 
 	return ret;
 }
+#define arch_atomic_cmpxchg arch_atomic_cmpxchg
 
 #define arch_atomic_fetch_andnot		arch_atomic_fetch_andnot
 
@@ -240,8 +241,6 @@ ATOMIC_OPS(xor, ^=, eor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #ifndef CONFIG_GENERIC_ATOMIC64
 typedef struct {
 	s64 counter;
arch/arm64/include/asm/atomic.h +0 -28
index c9979273d3898..400d279e0f8d0 100644
@@ -142,24 +142,6 @@ static __always_inline long arch_atomic64_dec_if_positive(atomic64_t *v)
 #define arch_atomic_fetch_xor_release		arch_atomic_fetch_xor_release
 #define arch_atomic_fetch_xor			arch_atomic_fetch_xor
 
-#define arch_atomic_xchg_relaxed(v, new) \
-	arch_xchg_relaxed(&((v)->counter), (new))
-#define arch_atomic_xchg_acquire(v, new) \
-	arch_xchg_acquire(&((v)->counter), (new))
-#define arch_atomic_xchg_release(v, new) \
-	arch_xchg_release(&((v)->counter), (new))
-#define arch_atomic_xchg(v, new) \
-	arch_xchg(&((v)->counter), (new))
-
-#define arch_atomic_cmpxchg_relaxed(v, old, new) \
-	arch_cmpxchg_relaxed(&((v)->counter), (old), (new))
-#define arch_atomic_cmpxchg_acquire(v, old, new) \
-	arch_cmpxchg_acquire(&((v)->counter), (old), (new))
-#define arch_atomic_cmpxchg_release(v, old, new) \
-	arch_cmpxchg_release(&((v)->counter), (old), (new))
-#define arch_atomic_cmpxchg(v, old, new) \
-	arch_cmpxchg(&((v)->counter), (old), (new))
-
 #define arch_atomic_andnot			arch_atomic_andnot
 
 /*
@@ -209,16 +191,6 @@ static __always_inline long arch_atomic64_dec_if_positive(atomic64_t *v)
 #define arch_atomic64_fetch_xor_release		arch_atomic64_fetch_xor_release
 #define arch_atomic64_fetch_xor			arch_atomic64_fetch_xor
 
-#define arch_atomic64_xchg_relaxed		arch_atomic_xchg_relaxed
-#define arch_atomic64_xchg_acquire		arch_atomic_xchg_acquire
-#define arch_atomic64_xchg_release		arch_atomic_xchg_release
-#define arch_atomic64_xchg			arch_atomic_xchg
-
-#define arch_atomic64_cmpxchg_relaxed		arch_atomic_cmpxchg_relaxed
-#define arch_atomic64_cmpxchg_acquire		arch_atomic_cmpxchg_acquire
-#define arch_atomic64_cmpxchg_release		arch_atomic_cmpxchg_release
-#define arch_atomic64_cmpxchg			arch_atomic_cmpxchg
-
 #define arch_atomic64_andnot			arch_atomic64_andnot
 
 #define arch_atomic64_dec_if_positive		arch_atomic64_dec_if_positive
arch/csky/include/asm/atomic.h +0 -35
index 60406ef9c2bbc..4dab44f6143a5 100644
@@ -195,41 +195,6 @@ arch_atomic_dec_if_positive(atomic_t *v)
 }
 #define arch_atomic_dec_if_positive arch_atomic_dec_if_positive
 
-#define ATOMIC_OP()							\
-static __always_inline							\
-int arch_atomic_xchg_relaxed(atomic_t *v, int n)			\
-{									\
-	return __xchg_relaxed(n, &(v->counter), 4);			\
-}									\
-static __always_inline							\
-int arch_atomic_cmpxchg_relaxed(atomic_t *v, int o, int n)		\
-{									\
-	return __cmpxchg_relaxed(&(v->counter), o, n, 4);		\
-}									\
-static __always_inline							\
-int arch_atomic_cmpxchg_acquire(atomic_t *v, int o, int n)		\
-{									\
-	return __cmpxchg_acquire(&(v->counter), o, n, 4);		\
-}									\
-static __always_inline							\
-int arch_atomic_cmpxchg(atomic_t *v, int o, int n)			\
-{									\
-	return __cmpxchg(&(v->counter), o, n, 4);			\
-}
-
-#define ATOMIC_OPS()							\
-	ATOMIC_OP()
-
-ATOMIC_OPS()
-
-#define arch_atomic_xchg_relaxed	arch_atomic_xchg_relaxed
-#define arch_atomic_cmpxchg_relaxed	arch_atomic_cmpxchg_relaxed
-#define arch_atomic_cmpxchg_acquire	arch_atomic_cmpxchg_acquire
-#define arch_atomic_cmpxchg		arch_atomic_cmpxchg
-
-#undef ATOMIC_OPS
-#undef ATOMIC_OP
-
 #else
 #include <asm-generic/atomic.h>
 #endif
arch/hexagon/include/asm/atomic.h +0 -6
index 738857e10d6ec..ad6c111e9c10f 100644
@@ -36,12 +36,6 @@ static inline void arch_atomic_set(atomic_t *v, int new)
  */
 #define arch_atomic_read(v)		READ_ONCE((v)->counter)
 
-#define arch_atomic_xchg(v, new)					\
-	(arch_xchg(&((v)->counter), (new)))
-
-#define arch_atomic_cmpxchg(v, old, new)				\
-	(arch_cmpxchg(&((v)->counter), (old), (new)))
-
 #define ATOMIC_OP(op)							\
 static inline void arch_atomic_##op(int i, atomic_t *v)			\
 {									\
arch/ia64/include/asm/atomic.h +0 -7
index 266c429b91372..6540a628d2573 100644
@@ -207,13 +207,6 @@ ATOMIC64_FETCH_OP(xor, ^)
 #undef ATOMIC64_FETCH_OP
 #undef ATOMIC64_OP
 
-#define arch_atomic_cmpxchg(v, old, new) (arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
-#define arch_atomic64_cmpxchg(v, old, new) \
-	(arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #define arch_atomic_add(i,v)		(void)arch_atomic_add_return((i), (v))
 #define arch_atomic_sub(i,v)		(void)arch_atomic_sub_return((i), (v))
 
arch/loongarch/include/asm/atomic.h +0 -7
index 6b9aca9ab6e9f..8d73c85911b08 100644
@@ -181,9 +181,6 @@ static inline int arch_atomic_sub_if_positive(int i, atomic_t *v)
 	return result;
 }
 
-#define arch_atomic_cmpxchg(v, o, n) (arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), (new)))
-
 /*
  * arch_atomic_dec_if_positive - decrement by 1 if old value positive
  * @v: pointer of type atomic_t
@@ -342,10 +339,6 @@ static inline long arch_atomic64_sub_if_positive(long i, atomic64_t *v)
 	return result;
 }
 
-#define arch_atomic64_cmpxchg(v, o, n) \
-	((__typeof__((v)->counter))arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), (new)))
-
 /*
  * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
  * @v: pointer of type atomic64_t
arch/m68k/include/asm/atomic.h +3 -6
index cfba83d230fde..190a032f19be7 100644
@@ -158,12 +158,7 @@ static inline int arch_atomic_inc_and_test(atomic_t *v)
 }
 #define arch_atomic_inc_and_test arch_atomic_inc_and_test
 
-#ifdef CONFIG_RMW_INSNS
-
-#define arch_atomic_cmpxchg(v, o, n) ((int)arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
-#else /* !CONFIG_RMW_INSNS */
+#ifndef CONFIG_RMW_INSNS
 
 static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 {
@@ -177,6 +172,7 @@ static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 	local_irq_restore(flags);
 	return prev;
 }
+#define arch_atomic_cmpxchg arch_atomic_cmpxchg
 
 static inline int arch_atomic_xchg(atomic_t *v, int new)
 {
@@ -189,6 +185,7 @@ static inline int arch_atomic_xchg(atomic_t *v, int new)
 	local_irq_restore(flags);
 	return prev;
 }
+#define arch_atomic_xchg arch_atomic_xchg
 
 #endif /* !CONFIG_RMW_INSNS */
 
arch/mips/include/asm/atomic.h +0 -11
index 712fb5a6a5682..ba188e77768b2 100644
@@ -33,17 +33,6 @@ static __always_inline void arch_##pfx##_set(pfx##_t *v, type i)	\
 {									\
 	WRITE_ONCE(v->counter, i);					\
 }									\
-									\
-static __always_inline type						\
-arch_##pfx##_cmpxchg(pfx##_t *v, type o, type n)			\
-{									\
-	return arch_cmpxchg(&v->counter, o, n);				\
-}									\
-									\
-static __always_inline type arch_##pfx##_xchg(pfx##_t *v, type n)	\
-{									\
-	return arch_xchg(&v->counter, n);				\
-}
 
 ATOMIC_OPS(atomic, int)
 
arch/openrisc/include/asm/atomic.h +0 -3
index 326167e4783a9..8ce67ec7c9a30 100644
@@ -130,7 +130,4 @@ static inline int arch_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 
 #include <asm/cmpxchg.h>
 
-#define arch_atomic_xchg(ptr, v)		(arch_xchg(&(ptr)->counter, (v)))
-#define arch_atomic_cmpxchg(v, old, new)	(arch_cmpxchg(&((v)->counter), (old), (new)))
-
 #endif /* __ASM_OPENRISC_ATOMIC_H */
arch/parisc/include/asm/atomic.h +0 -9
index dd5a299ada695..0b3f64c92e3c0 100644
@@ -73,10 +73,6 @@ static __inline__ int arch_atomic_read(const atomic_t *v)
 	return READ_ONCE((v)->counter);
 }
 
-/* exported interface */
-#define arch_atomic_cmpxchg(v, o, n)	(arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new)	(arch_xchg(&((v)->counter), new))
-
 #define ATOMIC_OP(op, c_op)						\
 static __inline__ void arch_atomic_##op(int i, atomic_t *v)		\
 {									\
@@ -218,11 +214,6 @@ arch_atomic64_read(const atomic64_t *v)
 	return READ_ONCE((v)->counter);
 }
 
-/* exported interface */
-#define arch_atomic64_cmpxchg(v, o, n) \
-	((__typeof__((v)->counter))arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #endif /* !CONFIG_64BIT */
 
 
arch/powerpc/include/asm/atomic.h +0 -24
index 47228b1774781..5bf6a4d49268c 100644
@@ -126,18 +126,6 @@ ATOMIC_OPS(xor, xor, "", K)
 #undef ATOMIC_OP_RETURN_RELAXED
 #undef ATOMIC_OP
 
-#define arch_atomic_cmpxchg(v, o, n) \
-	(arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_cmpxchg_relaxed(v, o, n) \
-	arch_cmpxchg_relaxed(&((v)->counter), (o), (n))
-#define arch_atomic_cmpxchg_acquire(v, o, n) \
-	arch_cmpxchg_acquire(&((v)->counter), (o), (n))
-
-#define arch_atomic_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-#define arch_atomic_xchg_relaxed(v, new) \
-	arch_xchg_relaxed(&((v)->counter), (new))
-
 /**
  * atomic_fetch_add_unless - add unless the number is a given value
  * @v: pointer of type atomic_t
@@ -396,18 +384,6 @@ static __inline__ s64 arch_atomic64_dec_if_positive(atomic64_t *v)
 }
 #define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive
 
-#define arch_atomic64_cmpxchg(v, o, n) \
-	(arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_cmpxchg_relaxed(v, o, n) \
-	arch_cmpxchg_relaxed(&((v)->counter), (o), (n))
-#define arch_atomic64_cmpxchg_acquire(v, o, n) \
-	arch_cmpxchg_acquire(&((v)->counter), (o), (n))
-
-#define arch_atomic64_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-#define arch_atomic64_xchg_relaxed(v, new) \
-	arch_xchg_relaxed(&((v)->counter), (new))
-
 /**
  * atomic64_fetch_add_unless - add unless the number is a given value
  * @v: pointer of type atomic64_t
arch/riscv/include/asm/atomic.h +0 -72

78 lines not on this page · raw message

arch/sh/include/asm/atomic.h +0 -3
index 528bfeda78f56..7a18cb2a1c1ac 100644
@@ -30,9 +30,6 @@
 #include <asm/atomic-irq.h>
 #endif
 
-#define arch_atomic_xchg(v, new)	(arch_xchg(&((v)->counter), new))
-#define arch_atomic_cmpxchg(v, o, n)	(arch_cmpxchg(&((v)->counter), (o), (n)))
-
 #endif /* CONFIG_CPU_J2 */
 
 #endif /* __ASM_SH_ATOMIC_H */
arch/sparc/include/asm/atomic_32.h +2 -0
index d775daa83d129..1c9e6c7366e41 100644
@@ -24,7 +24,9 @@ int arch_atomic_fetch_and(int, atomic_t *);
 int arch_atomic_fetch_or(int, atomic_t *);
 int arch_atomic_fetch_xor(int, atomic_t *);
 int arch_atomic_cmpxchg(atomic_t *, int, int);
+#define arch_atomic_cmpxchg arch_atomic_cmpxchg
 int arch_atomic_xchg(atomic_t *, int);
+#define arch_atomic_xchg arch_atomic_xchg
 int arch_atomic_fetch_add_unless(atomic_t *, int, int);
 void arch_atomic_set(atomic_t *, int);
 
arch/sparc/include/asm/atomic_64.h +0 -11
index 077891686715a..df6a8b07d7e63 100644
@@ -49,17 +49,6 @@ ATOMIC_OPS(xor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic_cmpxchg(v, o, n) (arch_cmpxchg(&((v)->counter), (o), (n)))
-
-static inline int arch_atomic_xchg(atomic_t *v, int new)
-{
-	return arch_xchg(&v->counter, new);
-}
-
-#define arch_atomic64_cmpxchg(v, o, n) \
-	((__typeof__((v)->counter))arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 s64 arch_atomic64_dec_if_positive(atomic64_t *v);
 #define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive
 
arch/xtensa/include/asm/atomic.h +0 -3
index 52da614f953ce..1d323a864002c 100644
@@ -257,7 +257,4 @@ ATOMIC_OPS(xor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic_cmpxchg(v, o, n) ((int)arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #endif /* _XTENSA_ATOMIC_H */
include/asm-generic/atomic.h +0 -3
index e271d6708c876..22142c71d35a1 100644
@@ -130,7 +130,4 @@ ATOMIC_OP(xor, ^)
 #define arch_atomic_read(v)			READ_ONCE((v)->counter)
 #define arch_atomic_set(v, i)			WRITE_ONCE(((v)->counter), (i))
 
-#define arch_atomic_xchg(ptr, v)		(arch_xchg(&(ptr)->counter, (u32)(v)))
-#define arch_atomic_cmpxchg(v, old, new)	(arch_cmpxchg(&((v)->counter), (u32)(old), (u32)(new)))
-
 #endif /* __ASM_GENERIC_ATOMIC_H */
include/linux/atomic/atomic-arch-fallback.h +157 -1

197 lines not on this page · raw message

scripts/atomic/fallbacks/cmpxchg new file +7 -0
new file mode 100755
index 0000000000000..87cd010f98d58
@@ -0,0 +1,7 @@
+cat <<EOF
+static __always_inline ${int}
+arch_${atomic}_cmpxchg${order}(${atomic}_t *v, ${int} old, ${int} new)
+{
+	return arch_cmpxchg${order}(&v->counter, old, new);
+}
+EOF
scripts/atomic/fallbacks/xchg new file +7 -0
new file mode 100755
index 0000000000000..733b8980b2f3b
@@ -0,0 +1,7 @@
+cat <<EOF
+static __always_inline ${int}
+arch_${atomic}_xchg${order}(${atomic}_t *v, ${int} new)
+{
+	return arch_xchg${order}(&v->counter, new);
+}
+EOF
Signature · 1 line
2.30.2
merge bot Applied
#8 lore raw

[tip: locking/core] locking/atomic: make atomic*_{cmp,}xchg optional

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     d12157efc8e083c77d054675fcdd594f54cc7e2b
Gitweb:        https://git.kernel.org/tip/d12157efc8e083c77d054675fcdd594f54cc7e2b
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:01 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:14 +02:00

locking/atomic: make atomic*_{cmp,}xchg optional

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

---
 arch/alpha/include/asm/atomic.h             |  10 +-
 arch/arc/include/asm/atomic.h               |  24 +---
 arch/arc/include/asm/atomic64-arcv2.h       |   2 +-
 arch/arm/include/asm/atomic.h               |   3 +-
 arch/arm64/include/asm/atomic.h             |  28 +---
 arch/csky/include/asm/atomic.h              |  35 +----
 arch/hexagon/include/asm/atomic.h           |   6 +-
 arch/ia64/include/asm/atomic.h              |   7 +-
 arch/loongarch/include/asm/atomic.h         |   7 +-
 arch/m68k/include/asm/atomic.h              |   9 +-
 arch/mips/include/asm/atomic.h              |  11 +-
 arch/openrisc/include/asm/atomic.h          |   3 +-
 arch/parisc/include/asm/atomic.h            |   9 +-
 arch/powerpc/include/asm/atomic.h           |  24 +---
 arch/riscv/include/asm/atomic.h             |  72 +---------
 arch/sh/include/asm/atomic.h                |   3 +-
 arch/sparc/include/asm/atomic_32.h          |   2 +-
 arch/sparc/include/asm/atomic_64.h          |  11 +-
 arch/xtensa/include/asm/atomic.h            |   3 +-
 include/asm-generic/atomic.h                |   3 +-
 include/linux/atomic/atomic-arch-fallback.h | 158 ++++++++++++++++++-
 scripts/atomic/fallbacks/cmpxchg            |   7 +-
 scripts/atomic/fallbacks/xchg               |   7 +-
 23 files changed, 179 insertions(+), 265 deletions(-)
 create mode 100644 scripts/atomic/fallbacks/cmpxchg
 create mode 100644 scripts/atomic/fallbacks/xchg
arch/alpha/include/asm/atomic.h +0 -10
index f2861a4..ec8ab55 100644
@@ -200,16 +200,6 @@ ATOMIC_OPS(xor, xor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic64_cmpxchg(v, old, new) \
-	(arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic64_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-
-#define arch_atomic_cmpxchg(v, old, new) \
-	(arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-
 /**
  * arch_atomic_fetch_add_unless - add unless the number is a given value
  * @v: pointer of type atomic_t
arch/arc/include/asm/atomic.h +0 -24
index 52ee51e..592d7ff 100644
@@ -22,30 +22,6 @@
 #include <asm/atomic-spinlock.h>
 #endif
 
-#define arch_atomic_cmpxchg(v, o, n)					\
-({									\
-	arch_cmpxchg(&((v)->counter), (o), (n));			\
-})
-
-#ifdef arch_cmpxchg_relaxed
-#define arch_atomic_cmpxchg_relaxed(v, o, n)				\
-({									\
-	arch_cmpxchg_relaxed(&((v)->counter), (o), (n));		\
-})
-#endif
-
-#define arch_atomic_xchg(v, n)						\
-({									\
-	arch_xchg(&((v)->counter), (n));				\
-})
-
-#ifdef arch_xchg_relaxed
-#define arch_atomic_xchg_relaxed(v, n)					\
-({									\
-	arch_xchg_relaxed(&((v)->counter), (n));			\
-})
-#endif
-
 /*
  * 64-bit atomics
  */
arch/arc/include/asm/atomic64-arcv2.h +2 -0
index c5a8010..2b7c9e6 100644
@@ -159,6 +159,7 @@ arch_atomic64_cmpxchg(atomic64_t *ptr, s64 expected, s64 new)
 
 	return prev;
 }
+#define arch_atomic64_cmpxchg arch_atomic64_cmpxchg
 
 static inline s64 arch_atomic64_xchg(atomic64_t *ptr, s64 new)
 {
@@ -179,6 +180,7 @@ static inline s64 arch_atomic64_xchg(atomic64_t *ptr, s64 new)
 
 	return prev;
 }
+#define arch_atomic64_xchg arch_atomic64_xchg
 
 /**
  * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
arch/arm/include/asm/atomic.h +1 -2
index db8512d..9458d47 100644
@@ -210,6 +210,7 @@ static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 
 	return ret;
 }
+#define arch_atomic_cmpxchg arch_atomic_cmpxchg
 
 #define arch_atomic_fetch_andnot		arch_atomic_fetch_andnot
 
@@ -240,8 +241,6 @@ ATOMIC_OPS(xor, ^=, eor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #ifndef CONFIG_GENERIC_ATOMIC64
 typedef struct {
 	s64 counter;
arch/arm64/include/asm/atomic.h +0 -28
index c997927..400d279 100644
@@ -142,24 +142,6 @@ static __always_inline long arch_atomic64_dec_if_positive(atomic64_t *v)
 #define arch_atomic_fetch_xor_release		arch_atomic_fetch_xor_release
 #define arch_atomic_fetch_xor			arch_atomic_fetch_xor
 
-#define arch_atomic_xchg_relaxed(v, new) \
-	arch_xchg_relaxed(&((v)->counter), (new))
-#define arch_atomic_xchg_acquire(v, new) \
-	arch_xchg_acquire(&((v)->counter), (new))
-#define arch_atomic_xchg_release(v, new) \
-	arch_xchg_release(&((v)->counter), (new))
-#define arch_atomic_xchg(v, new) \
-	arch_xchg(&((v)->counter), (new))
-
-#define arch_atomic_cmpxchg_relaxed(v, old, new) \
-	arch_cmpxchg_relaxed(&((v)->counter), (old), (new))
-#define arch_atomic_cmpxchg_acquire(v, old, new) \
-	arch_cmpxchg_acquire(&((v)->counter), (old), (new))
-#define arch_atomic_cmpxchg_release(v, old, new) \
-	arch_cmpxchg_release(&((v)->counter), (old), (new))
-#define arch_atomic_cmpxchg(v, old, new) \
-	arch_cmpxchg(&((v)->counter), (old), (new))
-
 #define arch_atomic_andnot			arch_atomic_andnot
 
 /*
@@ -209,16 +191,6 @@ static __always_inline long arch_atomic64_dec_if_positive(atomic64_t *v)
 #define arch_atomic64_fetch_xor_release		arch_atomic64_fetch_xor_release
 #define arch_atomic64_fetch_xor			arch_atomic64_fetch_xor
 
-#define arch_atomic64_xchg_relaxed		arch_atomic_xchg_relaxed
-#define arch_atomic64_xchg_acquire		arch_atomic_xchg_acquire
-#define arch_atomic64_xchg_release		arch_atomic_xchg_release
-#define arch_atomic64_xchg			arch_atomic_xchg
-
-#define arch_atomic64_cmpxchg_relaxed		arch_atomic_cmpxchg_relaxed
-#define arch_atomic64_cmpxchg_acquire		arch_atomic_cmpxchg_acquire
-#define arch_atomic64_cmpxchg_release		arch_atomic_cmpxchg_release
-#define arch_atomic64_cmpxchg			arch_atomic_cmpxchg
-
 #define arch_atomic64_andnot			arch_atomic64_andnot
 
 #define arch_atomic64_dec_if_positive		arch_atomic64_dec_if_positive
arch/csky/include/asm/atomic.h +0 -35
index 60406ef..4dab44f 100644
@@ -195,41 +195,6 @@ arch_atomic_dec_if_positive(atomic_t *v)
 }
 #define arch_atomic_dec_if_positive arch_atomic_dec_if_positive
 
-#define ATOMIC_OP()							\
-static __always_inline							\
-int arch_atomic_xchg_relaxed(atomic_t *v, int n)			\
-{									\
-	return __xchg_relaxed(n, &(v->counter), 4);			\
-}									\
-static __always_inline							\
-int arch_atomic_cmpxchg_relaxed(atomic_t *v, int o, int n)		\
-{									\
-	return __cmpxchg_relaxed(&(v->counter), o, n, 4);		\
-}									\
-static __always_inline							\
-int arch_atomic_cmpxchg_acquire(atomic_t *v, int o, int n)		\
-{									\
-	return __cmpxchg_acquire(&(v->counter), o, n, 4);		\
-}									\
-static __always_inline							\
-int arch_atomic_cmpxchg(atomic_t *v, int o, int n)			\
-{									\
-	return __cmpxchg(&(v->counter), o, n, 4);			\
-}
-
-#define ATOMIC_OPS()							\
-	ATOMIC_OP()
-
-ATOMIC_OPS()
-
-#define arch_atomic_xchg_relaxed	arch_atomic_xchg_relaxed
-#define arch_atomic_cmpxchg_relaxed	arch_atomic_cmpxchg_relaxed
-#define arch_atomic_cmpxchg_acquire	arch_atomic_cmpxchg_acquire
-#define arch_atomic_cmpxchg		arch_atomic_cmpxchg
-
-#undef ATOMIC_OPS
-#undef ATOMIC_OP
-
 #else
 #include <asm-generic/atomic.h>
 #endif
arch/hexagon/include/asm/atomic.h +0 -6
index 738857e..ad6c111 100644
@@ -36,12 +36,6 @@ static inline void arch_atomic_set(atomic_t *v, int new)
  */
 #define arch_atomic_read(v)		READ_ONCE((v)->counter)
 
-#define arch_atomic_xchg(v, new)					\
-	(arch_xchg(&((v)->counter), (new)))
-
-#define arch_atomic_cmpxchg(v, old, new)				\
-	(arch_cmpxchg(&((v)->counter), (old), (new)))
-
 #define ATOMIC_OP(op)							\
 static inline void arch_atomic_##op(int i, atomic_t *v)			\
 {									\
arch/ia64/include/asm/atomic.h +0 -7
index 266c429..6540a62 100644
@@ -207,13 +207,6 @@ ATOMIC64_FETCH_OP(xor, ^)
 #undef ATOMIC64_FETCH_OP
 #undef ATOMIC64_OP
 
-#define arch_atomic_cmpxchg(v, old, new) (arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
-#define arch_atomic64_cmpxchg(v, old, new) \
-	(arch_cmpxchg(&((v)->counter), old, new))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #define arch_atomic_add(i,v)		(void)arch_atomic_add_return((i), (v))
 #define arch_atomic_sub(i,v)		(void)arch_atomic_sub_return((i), (v))
 
arch/loongarch/include/asm/atomic.h +0 -7
index 6b9aca9..8d73c85 100644
@@ -181,9 +181,6 @@ static inline int arch_atomic_sub_if_positive(int i, atomic_t *v)
 	return result;
 }
 
-#define arch_atomic_cmpxchg(v, o, n) (arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), (new)))
-
 /*
  * arch_atomic_dec_if_positive - decrement by 1 if old value positive
  * @v: pointer of type atomic_t
@@ -342,10 +339,6 @@ static inline long arch_atomic64_sub_if_positive(long i, atomic64_t *v)
 	return result;
 }
 
-#define arch_atomic64_cmpxchg(v, o, n) \
-	((__typeof__((v)->counter))arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), (new)))
-
 /*
  * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
  * @v: pointer of type atomic64_t
arch/m68k/include/asm/atomic.h +3 -6
index cfba83d..190a032 100644
@@ -158,12 +158,7 @@ static inline int arch_atomic_inc_and_test(atomic_t *v)
 }
 #define arch_atomic_inc_and_test arch_atomic_inc_and_test
 
-#ifdef CONFIG_RMW_INSNS
-
-#define arch_atomic_cmpxchg(v, o, n) ((int)arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
-#else /* !CONFIG_RMW_INSNS */
+#ifndef CONFIG_RMW_INSNS
 
 static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 {
@@ -177,6 +172,7 @@ static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 	local_irq_restore(flags);
 	return prev;
 }
+#define arch_atomic_cmpxchg arch_atomic_cmpxchg
 
 static inline int arch_atomic_xchg(atomic_t *v, int new)
 {
@@ -189,6 +185,7 @@ static inline int arch_atomic_xchg(atomic_t *v, int new)
 	local_irq_restore(flags);
 	return prev;
 }
+#define arch_atomic_xchg arch_atomic_xchg
 
 #endif /* !CONFIG_RMW_INSNS */
 
arch/mips/include/asm/atomic.h +0 -11
index 712fb5a..ba188e7 100644
@@ -33,17 +33,6 @@ static __always_inline void arch_##pfx##_set(pfx##_t *v, type i)	\
 {									\
 	WRITE_ONCE(v->counter, i);					\
 }									\
-									\
-static __always_inline type						\
-arch_##pfx##_cmpxchg(pfx##_t *v, type o, type n)			\
-{									\
-	return arch_cmpxchg(&v->counter, o, n);				\
-}									\
-									\
-static __always_inline type arch_##pfx##_xchg(pfx##_t *v, type n)	\
-{									\
-	return arch_xchg(&v->counter, n);				\
-}
 
 ATOMIC_OPS(atomic, int)
 
arch/openrisc/include/asm/atomic.h +0 -3
index 326167e..8ce67ec 100644
@@ -130,7 +130,4 @@ static inline int arch_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 
 #include <asm/cmpxchg.h>
 
-#define arch_atomic_xchg(ptr, v)		(arch_xchg(&(ptr)->counter, (v)))
-#define arch_atomic_cmpxchg(v, old, new)	(arch_cmpxchg(&((v)->counter), (old), (new)))
-
 #endif /* __ASM_OPENRISC_ATOMIC_H */
arch/parisc/include/asm/atomic.h +0 -9
index dd5a299..0b3f64c 100644
@@ -73,10 +73,6 @@ static __inline__ int arch_atomic_read(const atomic_t *v)
 	return READ_ONCE((v)->counter);
 }
 
-/* exported interface */
-#define arch_atomic_cmpxchg(v, o, n)	(arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new)	(arch_xchg(&((v)->counter), new))
-
 #define ATOMIC_OP(op, c_op)						\
 static __inline__ void arch_atomic_##op(int i, atomic_t *v)		\
 {									\
@@ -218,11 +214,6 @@ arch_atomic64_read(const atomic64_t *v)
 	return READ_ONCE((v)->counter);
 }
 
-/* exported interface */
-#define arch_atomic64_cmpxchg(v, o, n) \
-	((__typeof__((v)->counter))arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #endif /* !CONFIG_64BIT */
 
 
arch/powerpc/include/asm/atomic.h +0 -24
index 47228b1..5bf6a4d 100644
@@ -126,18 +126,6 @@ ATOMIC_OPS(xor, xor, "", K)
 #undef ATOMIC_OP_RETURN_RELAXED
 #undef ATOMIC_OP
 
-#define arch_atomic_cmpxchg(v, o, n) \
-	(arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_cmpxchg_relaxed(v, o, n) \
-	arch_cmpxchg_relaxed(&((v)->counter), (o), (n))
-#define arch_atomic_cmpxchg_acquire(v, o, n) \
-	arch_cmpxchg_acquire(&((v)->counter), (o), (n))
-
-#define arch_atomic_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-#define arch_atomic_xchg_relaxed(v, new) \
-	arch_xchg_relaxed(&((v)->counter), (new))
-
 /**
  * atomic_fetch_add_unless - add unless the number is a given value
  * @v: pointer of type atomic_t
@@ -396,18 +384,6 @@ static __inline__ s64 arch_atomic64_dec_if_positive(atomic64_t *v)
 }
 #define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive
 
-#define arch_atomic64_cmpxchg(v, o, n) \
-	(arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_cmpxchg_relaxed(v, o, n) \
-	arch_cmpxchg_relaxed(&((v)->counter), (o), (n))
-#define arch_atomic64_cmpxchg_acquire(v, o, n) \
-	arch_cmpxchg_acquire(&((v)->counter), (o), (n))
-
-#define arch_atomic64_xchg(v, new) \
-	(arch_xchg(&((v)->counter), new))
-#define arch_atomic64_xchg_relaxed(v, new) \
-	arch_xchg_relaxed(&((v)->counter), (new))
-
 /**
  * atomic64_fetch_add_unless - add unless the number is a given value
  * @v: pointer of type atomic64_t
arch/riscv/include/asm/atomic.h +0 -72

78 lines not on this page · raw message

arch/sh/include/asm/atomic.h +0 -3
index 528bfed..7a18cb2 100644
@@ -30,9 +30,6 @@
 #include <asm/atomic-irq.h>
 #endif
 
-#define arch_atomic_xchg(v, new)	(arch_xchg(&((v)->counter), new))
-#define arch_atomic_cmpxchg(v, o, n)	(arch_cmpxchg(&((v)->counter), (o), (n)))
-
 #endif /* CONFIG_CPU_J2 */
 
 #endif /* __ASM_SH_ATOMIC_H */
arch/sparc/include/asm/atomic_32.h +2 -0
index d775daa..1c9e6c7 100644
@@ -24,7 +24,9 @@ int arch_atomic_fetch_and(int, atomic_t *);
 int arch_atomic_fetch_or(int, atomic_t *);
 int arch_atomic_fetch_xor(int, atomic_t *);
 int arch_atomic_cmpxchg(atomic_t *, int, int);
+#define arch_atomic_cmpxchg arch_atomic_cmpxchg
 int arch_atomic_xchg(atomic_t *, int);
+#define arch_atomic_xchg arch_atomic_xchg
 int arch_atomic_fetch_add_unless(atomic_t *, int, int);
 void arch_atomic_set(atomic_t *, int);
 
arch/sparc/include/asm/atomic_64.h +0 -11
index 0778916..df6a8b0 100644
@@ -49,17 +49,6 @@ ATOMIC_OPS(xor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic_cmpxchg(v, o, n) (arch_cmpxchg(&((v)->counter), (o), (n)))
-
-static inline int arch_atomic_xchg(atomic_t *v, int new)
-{
-	return arch_xchg(&v->counter, new);
-}
-
-#define arch_atomic64_cmpxchg(v, o, n) \
-	((__typeof__((v)->counter))arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic64_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 s64 arch_atomic64_dec_if_positive(atomic64_t *v);
 #define arch_atomic64_dec_if_positive arch_atomic64_dec_if_positive
 
arch/xtensa/include/asm/atomic.h +0 -3
index 52da614..1d323a8 100644
@@ -257,7 +257,4 @@ ATOMIC_OPS(xor)
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define arch_atomic_cmpxchg(v, o, n) ((int)arch_cmpxchg(&((v)->counter), (o), (n)))
-#define arch_atomic_xchg(v, new) (arch_xchg(&((v)->counter), new))
-
 #endif /* _XTENSA_ATOMIC_H */
include/asm-generic/atomic.h +0 -3
index e271d67..22142c7 100644
@@ -130,7 +130,4 @@ ATOMIC_OP(xor, ^)
 #define arch_atomic_read(v)			READ_ONCE((v)->counter)
 #define arch_atomic_set(v, i)			WRITE_ONCE(((v)->counter), (i))
 
-#define arch_atomic_xchg(ptr, v)		(arch_xchg(&(ptr)->counter, (u32)(v)))
-#define arch_atomic_cmpxchg(v, old, new)	(arch_cmpxchg(&((v)->counter), (u32)(old), (u32)(new)))
-
 #endif /* __ASM_GENERIC_ATOMIC_H */
include/linux/atomic/atomic-arch-fallback.h +157 -1

197 lines not on this page · raw message

scripts/atomic/fallbacks/cmpxchg new file +7 -0
new file mode 100644
index 0000000..87cd010
@@ -0,0 +1,7 @@
+cat <<EOF
+static __always_inline ${int}
+arch_${atomic}_cmpxchg${order}(${atomic}_t *v, ${int} old, ${int} new)
+{
+	return arch_cmpxchg${order}(&v->counter, old, new);
+}
+EOF
scripts/atomic/fallbacks/xchg new file +7 -0
new file mode 100644
index 0000000..733b898
@@ -0,0 +1,7 @@
+cat <<EOF
+static __always_inline ${int}
+arch_${atomic}_xchg${order}(${atomic}_t *v, ${int} new)
+{
+	return arch_xchg${order}(&v->counter, new);
+}
+EOF
#9 lore raw

Re: [PATCH v2 04/27] locking/atomic: make atomic*_{cmp,}xchg optional

In reply to Mark Rutland

Hi,

Quoted 7 lines · Mark Rutland

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

Quoted 6 lines
Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>

This patch results in:

ERROR: modpost: "__xchg_called_with_bad_pointer" [lib/atomic64_test.ko] undefined!

when trying to build sparc64:allmodconfig.

Guenter

---
bisect log:

# bad: [60e7c4a25da68cd826719b685babbd23e73b85b0] Add linux-next specific files for 20230626
# good: [45a3e24f65e90a047bef86f927ebdc4c710edaa1] Linux 6.4-rc7
git bisect start 'HEAD' 'v6.4-rc7'
# good: [1fc7b1b3c9c3211898874f51919fcb1cf6f1cc79] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect good 1fc7b1b3c9c3211898874f51919fcb1cf6f1cc79
# good: [4fce1fc9cf89412590fb681fa480cde0b23b3381] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git
git bisect good 4fce1fc9cf89412590fb681fa480cde0b23b3381
# bad: [cf1a0283badf6d0bfb91876583c24ef535a3c04c] Merge branch 'driver-core-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git
git bisect bad cf1a0283badf6d0bfb91876583c24ef535a3c04c
# bad: [3c5388e722ea98022b4d557ab33acca2eb16c4f0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect bad 3c5388e722ea98022b4d557ab33acca2eb16c4f0
# good: [997730bdbf14f352ab03e42461f500aafdabc03e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git
git bisect good 997730bdbf14f352ab03e42461f500aafdabc03e
# bad: [6fd8266556af196763b9f876ed682873e605469b] Merge branch into tip/master: 'ras/core'
git bisect bad 6fd8266556af196763b9f876ed682873e605469b
# good: [37380ea71463658934c2d3167d559d4034ea1c5b] Merge branch into tip/master: 'irq/core'
git bisect good 37380ea71463658934c2d3167d559d4034ea1c5b
# bad: [a967852939f864c35f155a2f431292ad6fc3fed9] Merge branch into tip/master: 'locking/core'
git bisect bad a967852939f864c35f155a2f431292ad6fc3fed9
# bad: [e50f06ce2d876c740993b5e3d01e203520391ccd] locking/atomic: m68k: add preprocessor symbols
git bisect bad e50f06ce2d876c740993b5e3d01e203520391ccd
# good: [b1fe7f2cda2a003afe316ce8dfe8d3645694a67e] x86,intel_iommu: Replace cmpxchg_double()
git bisect good b1fe7f2cda2a003afe316ce8dfe8d3645694a67e
# good: [14d72d4b6f0e88b5f683c1a5b7a876a55055852d] locking/atomic: remove fallback comments
git bisect good 14d72d4b6f0e88b5f683c1a5b7a876a55055852d
# bad: [f739287ef57bc01155e556033462e9a6ff020c97] locking/atomic: arc: add preprocessor symbols
git bisect bad f739287ef57bc01155e556033462e9a6ff020c97
# bad: [d12157efc8e083c77d054675fcdd594f54cc7e2b] locking/atomic: make atomic*_{cmp,}xchg optional
git bisect bad d12157efc8e083c77d054675fcdd594f54cc7e2b
# good: [a7bafa7969da1c0e9c342c792d8224078d1c491c] locking/atomic: hexagon: remove redundant arch_atomic_cmpxchg
git bisect good a7bafa7969da1c0e9c342c792d8224078d1c491c

# first bad commit: [d12157efc8e083c77d054675fcdd594f54cc7e2b] locking/atomic: make atomic*_{cmp,}xchg optional
author reviewer
#10 lore raw

Re: [PATCH v2 04/27] locking/atomic: make atomic*_{cmp,}xchg optional

Quoted 1 line · Guenter Roeck

Hi,

Hi Guenter,

Quoted 15 lines · Mark Rutland
Quoted 7 lines

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

Quoted 6 lines
Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
Quoted 5 lines

This patch results in:

ERROR: modpost: "__xchg_called_with_bad_pointer" [lib/atomic64_test.ko] undefined!

when trying to build sparc64:allmodconfig.

Hmm... it seems that in that configuration, the compiler decides to place __arch_xchg() out-of-line, and hence can't remove the call to __xchg_called_with_bad_pointer() via dead code elimination.

So this is due to tickling the compiler into making a different inlining decision rather than due to a semantic issue in the patch.

Marking __arch_xchg() as __always_inline solves that in local testing, and we should do likewise for the other bits used under the arch_ atomics.

I'll try to spin a patch for that soon, unless someone beats me to it.

Thanks,
Mark.

Quoted 39 lines

Guenter

---
bisect log:

# bad: [60e7c4a25da68cd826719b685babbd23e73b85b0] Add linux-next specific files for 20230626
# good: [45a3e24f65e90a047bef86f927ebdc4c710edaa1] Linux 6.4-rc7
git bisect start 'HEAD' 'v6.4-rc7'
# good: [1fc7b1b3c9c3211898874f51919fcb1cf6f1cc79] Merge branch 'main' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect good 1fc7b1b3c9c3211898874f51919fcb1cf6f1cc79
# good: [4fce1fc9cf89412590fb681fa480cde0b23b3381] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git
git bisect good 4fce1fc9cf89412590fb681fa480cde0b23b3381
# bad: [cf1a0283badf6d0bfb91876583c24ef535a3c04c] Merge branch 'driver-core-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git
git bisect bad cf1a0283badf6d0bfb91876583c24ef535a3c04c
# bad: [3c5388e722ea98022b4d557ab33acca2eb16c4f0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect bad 3c5388e722ea98022b4d557ab33acca2eb16c4f0
# good: [997730bdbf14f352ab03e42461f500aafdabc03e] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi.git
git bisect good 997730bdbf14f352ab03e42461f500aafdabc03e
# bad: [6fd8266556af196763b9f876ed682873e605469b] Merge branch into tip/master: 'ras/core'
git bisect bad 6fd8266556af196763b9f876ed682873e605469b
# good: [37380ea71463658934c2d3167d559d4034ea1c5b] Merge branch into tip/master: 'irq/core'
git bisect good 37380ea71463658934c2d3167d559d4034ea1c5b
# bad: [a967852939f864c35f155a2f431292ad6fc3fed9] Merge branch into tip/master: 'locking/core'
git bisect bad a967852939f864c35f155a2f431292ad6fc3fed9
# bad: [e50f06ce2d876c740993b5e3d01e203520391ccd] locking/atomic: m68k: add preprocessor symbols
git bisect bad e50f06ce2d876c740993b5e3d01e203520391ccd
# good: [b1fe7f2cda2a003afe316ce8dfe8d3645694a67e] x86,intel_iommu: Replace cmpxchg_double()
git bisect good b1fe7f2cda2a003afe316ce8dfe8d3645694a67e
# good: [14d72d4b6f0e88b5f683c1a5b7a876a55055852d] locking/atomic: remove fallback comments
git bisect good 14d72d4b6f0e88b5f683c1a5b7a876a55055852d
# bad: [f739287ef57bc01155e556033462e9a6ff020c97] locking/atomic: arc: add preprocessor symbols
git bisect bad f739287ef57bc01155e556033462e9a6ff020c97
# bad: [d12157efc8e083c77d054675fcdd594f54cc7e2b] locking/atomic: make atomic*_{cmp,}xchg optional
git bisect bad d12157efc8e083c77d054675fcdd594f54cc7e2b
# good: [a7bafa7969da1c0e9c342c792d8224078d1c491c] locking/atomic: hexagon: remove redundant arch_atomic_cmpxchg
git bisect good a7bafa7969da1c0e9c342c792d8224078d1c491c

# first bad commit: [d12157efc8e083c77d054675fcdd594f54cc7e2b] locking/atomic: make atomic*_{cmp,}xchg optional
#11 lore raw

Re: [PATCH v2 04/27] locking/atomic: make atomic*_{cmp,}xchg optional

In reply to Guenter Roeck
Quoted 8 lines · Guenter Roeck
Quoted 7 lines · Mark Rutland

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

Quoted 8 lines
Quoted 1 line

[...]

This patch results in:

ERROR: modpost: "__xchg_called_with_bad_pointer" [lib/atomic64_test.ko] undefined!

when trying to build sparc64:allmodconfig.

Guenter, please correct me if I'm wrong:

This is fixed by Arnd's patch "sparc: mark __arch_xchg() as __always_inline", but that afaics sadly hasn't even reached -next yet.
https://lore.kernel.org/all/[email protected]/

Hence adding it to the tracking now that the end of the merge window is near:

#regzbot ^introduced d12157efc8e083c7
#regzbot title locking/atomic: build error on sparc64:allmodconfig #regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

#12 lore raw

Re: [PATCH v2 04/27] locking/atomic: make atomic*_{cmp,}xchg optional

Quoted 24 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 8 lines · Guenter Roeck
Quoted 7 lines · Mark Rutland

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

Quoted 8 lines
Quoted 1 line

[...]

This patch results in:

ERROR: modpost: "__xchg_called_with_bad_pointer" [lib/atomic64_test.ko] undefined!

when trying to build sparc64:allmodconfig.

Guenter, please correct me if I'm wrong:

This is fixed by Arnd's patch "sparc: mark __arch_xchg() as __always_inline", but that afaics sadly hasn't even reached -next yet.
https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

Hence adding it to the tracking now that the end of the merge window is near:

#regzbot ^introduced d12157efc8e083c7
#regzbot title locking/atomic: build error on sparc64:allmodconfig #regzbot ignore-activity

Yes, this is correct.

Guenter

#13 lore raw

Re: [PATCH v2 04/27] locking/atomic: make atomic*_{cmp,}xchg optional

Quoted 26 lines · Guenter Roeck
Quoted 25 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 9 lines · Guenter Roeck
Quoted 8 lines · Mark Rutland

Most architectures define the atomic/atomic64 xchg and cmpxchg operations in terms of arch_xchg and arch_cmpxchg respectfully.

Add fallbacks for these cases and remove the trivial cases from arch code. On some architectures the existing definitions are kept as these are used to build other arch_atomic*() operations.

Quoted 8 lines

[...]

This patch results in:

ERROR: modpost: "__xchg_called_with_bad_pointer"
[lib/atomic64_test.ko] undefined!

when trying to build sparc64:allmodconfig.

Guenter, please correct me if I'm wrong:

This is fixed by Arnd's patch "sparc: mark __arch_xchg() as __always_inline", but that afaics sadly hasn't even reached -next yet.
https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 6 lines

Hence adding it to the tracking now that the end of the merge window is near:

#regzbot ^introduced d12157efc8e083c7
#regzbot title locking/atomic: build error on sparc64:allmodconfig #regzbot ignore-activity

Yes, this is correct.

Thx for confirming (and also for adding the other regression to the tracking). Let me use this opportunity to tell regzbot about the patch for this regression, which I forgot to do earlier (sorry!):

#regzbot monitor:
https://lore.kernel.org/all/[email protected]/

Also CCing Arnd, maybe the fix fell through the cracks on his side.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#14 lore raw

Re: [PATCH v2 04/27] locking/atomic: make atomic*_{cmp,}xchg optional

[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]

Quoted 14 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 13 lines · Guenter Roeck
Quoted 12 lines · Linux regression tracking (Thorsten Leemhuis)

On 27.06.23 19:07, Guenter Roeck wrote:
Guenter, please correct me if I'm wrong:

This is fixed by Arnd's patch "sparc: mark __arch_xchg() as __always_inline", but that afaics sadly hasn't even reached -next yet.
https://lore.kernel.org/all/[email protected]/

Hence adding it to the tracking now that the end of the merge window is near:
#regzbot ^introduced d12157efc8e083c7
#regzbot title locking/atomic: build error on sparc64:allmodconfig #regzbot ignore-activity

Quoted 6 lines
Quoted 1 line

Yes, this is correct.

Thx for confirming (and also for adding the other regression to the tracking). Let me use this opportunity to tell regzbot about the patch for this regression, which I forgot to do earlier (sorry!):
#regzbot monitor:
https://lore.kernel.org/all/[email protected]/

Kees applied this: https://git.kernel.org/kees/c/ec7633de404e

#regzbot fix: sparc: mark __arch_xchg() as __always_inline #regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

author reviewer
#15 lore raw

PATCH v2 5/27 locking/atomic: arc: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/arc.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arc/include/asm/atomic-spinlock.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/arc/include/asm/atomic-spinlock.h +9 -0
index 2c830347bfb4e..89d12a60f84c0 100644
@@ -81,6 +81,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add, +=, add)
 ATOMIC_OPS(sub, -=, sub)
 
+#define arch_atomic_fetch_add		arch_atomic_fetch_add
+#define arch_atomic_fetch_sub		arch_atomic_fetch_sub
+#define arch_atomic_add_return		arch_atomic_add_return
+#define arch_atomic_sub_return		arch_atomic_sub_return
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op, asm_op)					\
 	ATOMIC_OP(op, c_op, asm_op)					\
@@ -92,7 +97,11 @@ ATOMIC_OPS(or, |=, or)
 ATOMIC_OPS(xor, ^=, xor)
 
 #define arch_atomic_andnot		arch_atomic_andnot
+
+#define arch_atomic_fetch_and		arch_atomic_fetch_and
 #define arch_atomic_fetch_andnot	arch_atomic_fetch_andnot
+#define arch_atomic_fetch_or		arch_atomic_fetch_or
+#define arch_atomic_fetch_xor		arch_atomic_fetch_xor
 
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
Signature · 1 line
2.30.2
merge bot Applied
#16 lore raw

[tip: locking/core] locking/atomic: arc: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     f739287ef57bc01155e556033462e9a6ff020c97
Gitweb:        https://git.kernel.org/tip/f739287ef57bc01155e556033462e9a6ff020c97
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:02 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:14 +02:00

locking/atomic: arc: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/arc.

---
 arch/arc/include/asm/atomic-spinlock.h |  9 +++++++++
 1 file changed, 9 insertions(+)
arch/arc/include/asm/atomic-spinlock.h +9 -0
index 2c83034..89d12a6 100644
@@ -81,6 +81,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add, +=, add)
 ATOMIC_OPS(sub, -=, sub)
 
+#define arch_atomic_fetch_add		arch_atomic_fetch_add
+#define arch_atomic_fetch_sub		arch_atomic_fetch_sub
+#define arch_atomic_add_return		arch_atomic_add_return
+#define arch_atomic_sub_return		arch_atomic_sub_return
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op, asm_op)					\
 	ATOMIC_OP(op, c_op, asm_op)					\
@@ -92,7 +97,11 @@ ATOMIC_OPS(or, |=, or)
 ATOMIC_OPS(xor, ^=, xor)
 
 #define arch_atomic_andnot		arch_atomic_andnot
+
+#define arch_atomic_fetch_and		arch_atomic_fetch_and
 #define arch_atomic_fetch_andnot	arch_atomic_fetch_andnot
+#define arch_atomic_fetch_or		arch_atomic_fetch_or
+#define arch_atomic_fetch_xor		arch_atomic_fetch_xor
 
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
author reviewer
#17 lore raw

PATCH v2 6/27 locking/atomic: arm: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/arm.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/arm/include/asm/atomic.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
arch/arm/include/asm/atomic.h +10 -2
index 9458d47ff209c..f0e3b01afa746 100644
@@ -197,6 +197,16 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 	return val;							\
 }
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_andnot		arch_atomic_fetch_andnot
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 {
 	int ret;
@@ -212,8 +222,6 @@ static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 }
 #define arch_atomic_cmpxchg arch_atomic_cmpxchg
 
-#define arch_atomic_fetch_andnot		arch_atomic_fetch_andnot
-
 #endif /* __LINUX_ARM_ARCH__ */
 
 #define ATOMIC_OPS(op, c_op, asm_op)					\
Signature · 1 line
2.30.2
merge bot Applied
#18 lore raw

[tip: locking/core] locking/atomic: arm: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     d6cd3664806fbe8313b8e04b042d40e8135ca459
Gitweb:        https://git.kernel.org/tip/d6cd3664806fbe8313b8e04b042d40e8135ca459
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:03 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:15 +02:00

locking/atomic: arm: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/arm.

---
 arch/arm/include/asm/atomic.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
arch/arm/include/asm/atomic.h +10 -2
index 9458d47..f0e3b01 100644
@@ -197,6 +197,16 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 	return val;							\
 }
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_andnot		arch_atomic_fetch_andnot
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 {
 	int ret;
@@ -212,8 +222,6 @@ static inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
 }
 #define arch_atomic_cmpxchg arch_atomic_cmpxchg
 
-#define arch_atomic_fetch_andnot		arch_atomic_fetch_andnot
-
 #endif /* __LINUX_ARM_ARCH__ */
 
 #define ATOMIC_OPS(op, c_op, asm_op)					\
author reviewer
#19 lore raw

PATCH v2 7/27 locking/atomic: hexagon: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/hexagon.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/hexagon/include/asm/atomic.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/hexagon/include/asm/atomic.h +9 -0
index ad6c111e9c10f..5c8440016c762 100644
@@ -91,6 +91,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -98,6 +103,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
Signature · 1 line
2.30.2
merge bot Applied
#20 lore raw

[tip: locking/core] locking/atomic: hexagon: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     8ad17f2183fd7e37ceafddbdff334a3e2608cc84
Gitweb:        https://git.kernel.org/tip/8ad17f2183fd7e37ceafddbdff334a3e2608cc84
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:04 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:15 +02:00

locking/atomic: hexagon: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/hexagon.

---
 arch/hexagon/include/asm/atomic.h |  9 +++++++++
 1 file changed, 9 insertions(+)
arch/hexagon/include/asm/atomic.h +9 -0
index ad6c111..5c84400 100644
@@ -91,6 +91,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -98,6 +103,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
author reviewer
#21 lore raw

PATCH v2 8/27 locking/atomic: m68k: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/m68k.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/m68k/include/asm/atomic.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/m68k/include/asm/atomic.h +9 -0
index 190a032f19be7..4bfbc25f6ecf4 100644
@@ -106,6 +106,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t * v)		\
 ATOMIC_OPS(add, +=, add)
 ATOMIC_OPS(sub, -=, sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op, asm_op)					\
 	ATOMIC_OP(op, c_op, asm_op)					\
@@ -115,6 +120,10 @@ ATOMIC_OPS(and, &=, and)
 ATOMIC_OPS(or, |=, or)
 ATOMIC_OPS(xor, ^=, eor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
Signature · 1 line
2.30.2
merge bot Applied
#22 lore raw

[tip: locking/core] locking/atomic: m68k: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     e50f06ce2d876c740993b5e3d01e203520391ccd
Gitweb:        https://git.kernel.org/tip/e50f06ce2d876c740993b5e3d01e203520391ccd
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:05 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:16 +02:00

locking/atomic: m68k: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/m68k.

---
 arch/m68k/include/asm/atomic.h |  9 +++++++++
 1 file changed, 9 insertions(+)
arch/m68k/include/asm/atomic.h +9 -0
index 190a032..4bfbc25 100644
@@ -106,6 +106,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t * v)		\
 ATOMIC_OPS(add, +=, add)
 ATOMIC_OPS(sub, -=, sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op, asm_op)					\
 	ATOMIC_OP(op, c_op, asm_op)					\
@@ -115,6 +120,10 @@ ATOMIC_OPS(and, &=, and)
 ATOMIC_OPS(or, |=, or)
 ATOMIC_OPS(xor, ^=, eor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
author reviewer
#23 lore raw

PATCH v2 9/27 locking/atomic: parisc: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/parisc.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/parisc/include/asm/atomic.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/parisc/include/asm/atomic.h +18 -0
index 0b3f64c92e3c0..d4f023887ff87 100644
@@ -118,6 +118,11 @@ static __inline__ int arch_atomic_fetch_##op(int i, atomic_t *v)	\
 ATOMIC_OPS(add, +=)
 ATOMIC_OPS(sub, -=)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op)						\
 	ATOMIC_OP(op, c_op)						\
@@ -127,6 +132,10 @@ ATOMIC_OPS(and, &=)
 ATOMIC_OPS(or, |=)
 ATOMIC_OPS(xor, ^=)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
@@ -181,6 +190,11 @@ static __inline__ s64 arch_atomic64_fetch_##op(s64 i, atomic64_t *v)	\
 ATOMIC64_OPS(add, +=)
 ATOMIC64_OPS(sub, -=)
 
+#define arch_atomic64_add_return	arch_atomic64_add_return
+#define arch_atomic64_sub_return	arch_atomic64_sub_return
+#define arch_atomic64_fetch_add		arch_atomic64_fetch_add
+#define arch_atomic64_fetch_sub		arch_atomic64_fetch_sub
+
 #undef ATOMIC64_OPS
 #define ATOMIC64_OPS(op, c_op)						\
 	ATOMIC64_OP(op, c_op)						\
@@ -190,6 +204,10 @@ ATOMIC64_OPS(and, &=)
 ATOMIC64_OPS(or, |=)
 ATOMIC64_OPS(xor, ^=)
 
+#define arch_atomic64_fetch_and		arch_atomic64_fetch_and
+#define arch_atomic64_fetch_or		arch_atomic64_fetch_or
+#define arch_atomic64_fetch_xor		arch_atomic64_fetch_xor
+
 #undef ATOMIC64_OPS
 #undef ATOMIC64_FETCH_OP
 #undef ATOMIC64_OP_RETURN
Signature · 1 line
2.30.2
merge bot Applied
#24 lore raw

[tip: locking/core] locking/atomic: parisc: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     07bf3dcbe0e199422598f12918021c516161fd12
Gitweb:        https://git.kernel.org/tip/07bf3dcbe0e199422598f12918021c516161fd12
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:06 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:16 +02:00

locking/atomic: parisc: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/parisc.

---
 arch/parisc/include/asm/atomic.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/parisc/include/asm/atomic.h +18 -0
index 0b3f64c..d4f0238 100644
@@ -118,6 +118,11 @@ static __inline__ int arch_atomic_fetch_##op(int i, atomic_t *v)	\
 ATOMIC_OPS(add, +=)
 ATOMIC_OPS(sub, -=)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op)						\
 	ATOMIC_OP(op, c_op)						\
@@ -127,6 +132,10 @@ ATOMIC_OPS(and, &=)
 ATOMIC_OPS(or, |=)
 ATOMIC_OPS(xor, ^=)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
@@ -181,6 +190,11 @@ static __inline__ s64 arch_atomic64_fetch_##op(s64 i, atomic64_t *v)	\
 ATOMIC64_OPS(add, +=)
 ATOMIC64_OPS(sub, -=)
 
+#define arch_atomic64_add_return	arch_atomic64_add_return
+#define arch_atomic64_sub_return	arch_atomic64_sub_return
+#define arch_atomic64_fetch_add		arch_atomic64_fetch_add
+#define arch_atomic64_fetch_sub		arch_atomic64_fetch_sub
+
 #undef ATOMIC64_OPS
 #define ATOMIC64_OPS(op, c_op)						\
 	ATOMIC64_OP(op, c_op)						\
@@ -190,6 +204,10 @@ ATOMIC64_OPS(and, &=)
 ATOMIC64_OPS(or, |=)
 ATOMIC64_OPS(xor, ^=)
 
+#define arch_atomic64_fetch_and		arch_atomic64_fetch_and
+#define arch_atomic64_fetch_or		arch_atomic64_fetch_or
+#define arch_atomic64_fetch_xor		arch_atomic64_fetch_xor
+
 #undef ATOMIC64_OPS
 #undef ATOMIC64_FETCH_OP
 #undef ATOMIC64_OP_RETURN
author reviewer
#25 lore raw

PATCH v2 10/27 locking/atomic: sh: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/sh.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/sh/include/asm/atomic-grb.h  | 9 +++++++++
 arch/sh/include/asm/atomic-irq.h  | 9 +++++++++
 arch/sh/include/asm/atomic-llsc.h | 9 +++++++++
 3 files changed, 27 insertions(+)
arch/sh/include/asm/atomic-grb.h +9 -0
index 059791fd394fc..cf1c10f15528b 100644
@@ -71,6 +71,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -78,6 +83,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
arch/sh/include/asm/atomic-irq.h +9 -0
index 7665de9d00d0d..b4090cc354935 100644
@@ -55,6 +55,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add, +=)
 ATOMIC_OPS(sub, -=)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op)						\
 	ATOMIC_OP(op, c_op)						\
@@ -64,6 +69,10 @@ ATOMIC_OPS(and, &=)
 ATOMIC_OPS(or, |=)
 ATOMIC_OPS(xor, ^=)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
arch/sh/include/asm/atomic-llsc.h +9 -0
index b63dcfbfa14ef..9ef1fb1dd12ee 100644
@@ -73,6 +73,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -80,6 +85,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
Signature · 1 line
2.30.2
merge bot Applied
#26 lore raw

[tip: locking/core] locking/atomic: sh: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     770345adc38485c688e5d832d82306a4c2da828c
Gitweb:        https://git.kernel.org/tip/770345adc38485c688e5d832d82306a4c2da828c
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:07 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:16 +02:00

locking/atomic: sh: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/sh.

---
 arch/sh/include/asm/atomic-grb.h  |  9 +++++++++
 arch/sh/include/asm/atomic-irq.h  |  9 +++++++++
 arch/sh/include/asm/atomic-llsc.h |  9 +++++++++
 3 files changed, 27 insertions(+)
arch/sh/include/asm/atomic-grb.h +9 -0
index 059791f..cf1c10f 100644
@@ -71,6 +71,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -78,6 +83,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
arch/sh/include/asm/atomic-irq.h +9 -0
index 7665de9..b4090cc 100644
@@ -55,6 +55,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add, +=)
 ATOMIC_OPS(sub, -=)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op, c_op)						\
 	ATOMIC_OP(op, c_op)						\
@@ -64,6 +69,10 @@ ATOMIC_OPS(and, &=)
 ATOMIC_OPS(or, |=)
 ATOMIC_OPS(xor, ^=)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
arch/sh/include/asm/atomic-llsc.h +9 -0
index b63dcfb..9ef1fb1 100644
@@ -73,6 +73,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t *v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return	arch_atomic_add_return
+#define arch_atomic_sub_return	arch_atomic_sub_return
+#define arch_atomic_fetch_add	arch_atomic_fetch_add
+#define arch_atomic_fetch_sub	arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -80,6 +85,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and	arch_atomic_fetch_and
+#define arch_atomic_fetch_or	arch_atomic_fetch_or
+#define arch_atomic_fetch_xor	arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
author reviewer
#27 lore raw

PATCH v2 11/27 locking/atomic: sparc: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/sparc.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/sparc/include/asm/atomic_32.h | 16 ++++++++++++++--
 arch/sparc/include/asm/atomic_64.h | 18 ++++++++++++++++++
 2 files changed, 32 insertions(+), 2 deletions(-)
arch/sparc/include/asm/atomic_32.h +14 -2
index 1c9e6c7366e41..60ce2fe57fcd7 100644
@@ -19,19 +19,31 @@
 #include <asm-generic/atomic64.h>
 
 int arch_atomic_add_return(int, atomic_t *);
+#define arch_atomic_add_return arch_atomic_add_return
+
 int arch_atomic_fetch_add(int, atomic_t *);
+#define arch_atomic_fetch_add arch_atomic_fetch_add
+
 int arch_atomic_fetch_and(int, atomic_t *);
+#define arch_atomic_fetch_and arch_atomic_fetch_and
+
 int arch_atomic_fetch_or(int, atomic_t *);
+#define arch_atomic_fetch_or arch_atomic_fetch_or
+
 int arch_atomic_fetch_xor(int, atomic_t *);
+#define arch_atomic_fetch_xor arch_atomic_fetch_xor
+
 int arch_atomic_cmpxchg(atomic_t *, int, int);
 #define arch_atomic_cmpxchg arch_atomic_cmpxchg
+
 int arch_atomic_xchg(atomic_t *, int);
 #define arch_atomic_xchg arch_atomic_xchg
-int arch_atomic_fetch_add_unless(atomic_t *, int, int);
-void arch_atomic_set(atomic_t *, int);
 
+int arch_atomic_fetch_add_unless(atomic_t *, int, int);
 #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless
 
+void arch_atomic_set(atomic_t *, int);
+
 #define arch_atomic_set_release(v, i)	arch_atomic_set((v), (i))
 
 #define arch_atomic_read(v)		READ_ONCE((v)->counter)
arch/sparc/include/asm/atomic_64.h +18 -0
index df6a8b07d7e63..a5e9c37605a70 100644
@@ -37,6 +37,16 @@ s64 arch_atomic64_fetch_##op(s64, atomic64_t *);
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
+#define arch_atomic64_add_return		arch_atomic64_add_return
+#define arch_atomic64_sub_return		arch_atomic64_sub_return
+#define arch_atomic64_fetch_add			arch_atomic64_fetch_add
+#define arch_atomic64_fetch_sub			arch_atomic64_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -44,6 +54,14 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
+#define arch_atomic64_fetch_and			arch_atomic64_fetch_and
+#define arch_atomic64_fetch_or			arch_atomic64_fetch_or
+#define arch_atomic64_fetch_xor			arch_atomic64_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
Signature · 1 line
2.30.2
merge bot Applied
#28 lore raw

[tip: locking/core] locking/atomic: sparc: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     358c449afa662b1120d43738d2b0400ed2cc97df
Gitweb:        https://git.kernel.org/tip/358c449afa662b1120d43738d2b0400ed2cc97df
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:08 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:17 +02:00

locking/atomic: sparc: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/sparc.

---
 arch/sparc/include/asm/atomic_32.h | 16 ++++++++++++++--
 arch/sparc/include/asm/atomic_64.h | 18 ++++++++++++++++++
 2 files changed, 32 insertions(+), 2 deletions(-)
arch/sparc/include/asm/atomic_32.h +14 -2
index 1c9e6c7..60ce2fe 100644
@@ -19,19 +19,31 @@
 #include <asm-generic/atomic64.h>
 
 int arch_atomic_add_return(int, atomic_t *);
+#define arch_atomic_add_return arch_atomic_add_return
+
 int arch_atomic_fetch_add(int, atomic_t *);
+#define arch_atomic_fetch_add arch_atomic_fetch_add
+
 int arch_atomic_fetch_and(int, atomic_t *);
+#define arch_atomic_fetch_and arch_atomic_fetch_and
+
 int arch_atomic_fetch_or(int, atomic_t *);
+#define arch_atomic_fetch_or arch_atomic_fetch_or
+
 int arch_atomic_fetch_xor(int, atomic_t *);
+#define arch_atomic_fetch_xor arch_atomic_fetch_xor
+
 int arch_atomic_cmpxchg(atomic_t *, int, int);
 #define arch_atomic_cmpxchg arch_atomic_cmpxchg
+
 int arch_atomic_xchg(atomic_t *, int);
 #define arch_atomic_xchg arch_atomic_xchg
-int arch_atomic_fetch_add_unless(atomic_t *, int, int);
-void arch_atomic_set(atomic_t *, int);
 
+int arch_atomic_fetch_add_unless(atomic_t *, int, int);
 #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless
 
+void arch_atomic_set(atomic_t *, int);
+
 #define arch_atomic_set_release(v, i)	arch_atomic_set((v), (i))
 
 #define arch_atomic_read(v)		READ_ONCE((v)->counter)
arch/sparc/include/asm/atomic_64.h +18 -0
index df6a8b0..a5e9c37 100644
@@ -37,6 +37,16 @@ s64 arch_atomic64_fetch_##op(s64, atomic64_t *);
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
+#define arch_atomic64_add_return		arch_atomic64_add_return
+#define arch_atomic64_sub_return		arch_atomic64_sub_return
+#define arch_atomic64_fetch_add			arch_atomic64_fetch_add
+#define arch_atomic64_fetch_sub			arch_atomic64_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -44,6 +54,14 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
+#define arch_atomic64_fetch_and			arch_atomic64_fetch_and
+#define arch_atomic64_fetch_or			arch_atomic64_fetch_or
+#define arch_atomic64_fetch_xor			arch_atomic64_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
author reviewer
#29 lore raw

PATCH v2 12/27 locking/atomic: x86: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/x86.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/x86/include/asm/cmpxchg_64.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/include/asm/cmpxchg_64.h +4 -0
index 3e6e3eef701b3..44b08b53ab32f 100644
@@ -45,11 +45,13 @@ static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 n
 {
 	return __arch_cmpxchg128(ptr, old, new, LOCK_PREFIX);
 }
+#define arch_cmpxchg128 arch_cmpxchg128
 
 static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new)
 {
 	return __arch_cmpxchg128(ptr, old, new,);
 }
+#define arch_cmpxchg128_local arch_cmpxchg128_local
 
 #define __arch_try_cmpxchg128(_ptr, _oldp, _new, _lock)			\
 ({									\
@@ -75,11 +77,13 @@ static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *oldp,
 {
 	return __arch_try_cmpxchg128(ptr, oldp, new, LOCK_PREFIX);
 }
+#define arch_try_cmpxchg128 arch_try_cmpxchg128
 
 static __always_inline bool arch_try_cmpxchg128_local(volatile u128 *ptr, u128 *oldp, u128 new)
 {
 	return __arch_try_cmpxchg128(ptr, oldp, new,);
 }
+#define arch_try_cmpxchg128_local arch_try_cmpxchg128_local
 
 #define system_has_cmpxchg128()		boot_cpu_has(X86_FEATURE_CX16)
 
Signature · 1 line
2.30.2
merge bot Applied
#30 lore raw

[tip: locking/core] locking/atomic: x86: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     5bef003538ae8621c95ac6ebfd37324373fae37d
Gitweb:        https://git.kernel.org/tip/5bef003538ae8621c95ac6ebfd37324373fae37d
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:09 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:17 +02:00

locking/atomic: x86: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/x86.

---
 arch/x86/include/asm/cmpxchg_64.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/include/asm/cmpxchg_64.h +4 -0
index 3e6e3ee..44b08b5 100644
@@ -45,11 +45,13 @@ static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 n
 {
 	return __arch_cmpxchg128(ptr, old, new, LOCK_PREFIX);
 }
+#define arch_cmpxchg128 arch_cmpxchg128
 
 static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new)
 {
 	return __arch_cmpxchg128(ptr, old, new,);
 }
+#define arch_cmpxchg128_local arch_cmpxchg128_local
 
 #define __arch_try_cmpxchg128(_ptr, _oldp, _new, _lock)			\
 ({									\
@@ -75,11 +77,13 @@ static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *oldp, 
 {
 	return __arch_try_cmpxchg128(ptr, oldp, new, LOCK_PREFIX);
 }
+#define arch_try_cmpxchg128 arch_try_cmpxchg128
 
 static __always_inline bool arch_try_cmpxchg128_local(volatile u128 *ptr, u128 *oldp, u128 new)
 {
 	return __arch_try_cmpxchg128(ptr, oldp, new,);
 }
+#define arch_try_cmpxchg128_local arch_try_cmpxchg128_local
 
 #define system_has_cmpxchg128()		boot_cpu_has(X86_FEATURE_CX16)
 
author reviewer
#31 lore raw

PATCH v2 13/27 locking/atomic: xtensa: add preprocessor symbols

In reply to Mark Rutland

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/xtensa.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/xtensa/include/asm/atomic.h | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/xtensa/include/asm/atomic.h +9 -0
index 1d323a864002c..7308b7f777d79 100644
@@ -245,6 +245,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t * v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -252,6 +257,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
Signature · 1 line
2.30.2
merge bot Applied
#32 lore raw

[tip: locking/core] locking/atomic: xtensa: add preprocessor symbols

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     7c7084f3ba4031a9c2858afed696a577fcfe41d2
Gitweb:        https://git.kernel.org/tip/7c7084f3ba4031a9c2858afed696a577fcfe41d2
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:10 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:18 +02:00

locking/atomic: xtensa: add preprocessor symbols

Some atomics can be implemented in several different ways, e.g. FULL/ACQUIRE/RELEASE ordered atomics can be implemented in terms of RELAXED atomics, and ACQUIRE/RELEASE/RELAXED can be implemented in terms of FULL ordered atomics. Other atomics are optional, and don't exist in some configurations (e.g. not all architectures implement the 128-bit cmpxchg ops).

Subsequent patches will require that architectures define a preprocessor symbol for any atomic (or ordering variant) which is optional. This will make the fallback ifdeffery more robust, and simplify future changes.

Add the required definitions to arch/xtensa.

---
 arch/xtensa/include/asm/atomic.h |  9 +++++++++
 1 file changed, 9 insertions(+)
arch/xtensa/include/asm/atomic.h +9 -0
index 1d323a8..7308b7f 100644
@@ -245,6 +245,11 @@ static inline int arch_atomic_fetch_##op(int i, atomic_t * v)		\
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
+#define arch_atomic_add_return			arch_atomic_add_return
+#define arch_atomic_sub_return			arch_atomic_sub_return
+#define arch_atomic_fetch_add			arch_atomic_fetch_add
+#define arch_atomic_fetch_sub			arch_atomic_fetch_sub
+
 #undef ATOMIC_OPS
 #define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
 
@@ -252,6 +257,10 @@ ATOMIC_OPS(and)
 ATOMIC_OPS(or)
 ATOMIC_OPS(xor)
 
+#define arch_atomic_fetch_and			arch_atomic_fetch_and
+#define arch_atomic_fetch_or			arch_atomic_fetch_or
+#define arch_atomic_fetch_xor			arch_atomic_fetch_xor
+
 #undef ATOMIC_OPS
 #undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
author reviewer
#33 lore raw

PATCH v2 14/27 locking/atomic: scripts: remove bogus order parameter

In reply to Mark Rutland

At the start of gen_proto_order_variants(), the ${order} variable is not yet defined, and will be substituted with an empty string.

Replace the current bogus use of ${order} with an empty string instead.

This results in no change to the generated headers.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 scripts/atomic/gen-atomic-fallback.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/atomic/gen-atomic-fallback.sh +1 -1
index a70acd548fcd8..7a6bcea8f565b 100755
@@ -81,7 +81,7 @@ gen_proto_order_variants()
 
 	local basename="arch_${atomic}_${pfx}${name}${sfx}"
 
-	local template="$(find_fallback_template "${pfx}" "${name}" "${sfx}" "${order}")"
+	local template="$(find_fallback_template "${pfx}" "${name}" "${sfx}" "")"
 
 	# If we don't have relaxed atomics, then we don't bother with ordering fallbacks
 	# read_acquire and set_release need to be templated, though
Signature · 1 line
2.30.2
merge bot Applied
#34 lore raw

[tip: locking/core] locking/atomic: scripts: remove bogus order parameter

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     a083ecc9333c62237551ad93f42e86a42a3c7cc2
Gitweb:        https://git.kernel.org/tip/a083ecc9333c62237551ad93f42e86a42a3c7cc2
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:11 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:18 +02:00

locking/atomic: scripts: remove bogus order parameter

At the start of gen_proto_order_variants(), the ${order} variable is not yet defined, and will be substituted with an empty string.

Replace the current bogus use of ${order} with an empty string instead.

This results in no change to the generated headers.

There should be no functional change as a result of this patch.

---
 scripts/atomic/gen-atomic-fallback.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/atomic/gen-atomic-fallback.sh +1 -1
index a70acd5..7a6bcea 100755
@@ -81,7 +81,7 @@ gen_proto_order_variants()
 
 	local basename="arch_${atomic}_${pfx}${name}${sfx}"
 
-	local template="$(find_fallback_template "${pfx}" "${name}" "${sfx}" "${order}")"
+	local template="$(find_fallback_template "${pfx}" "${name}" "${sfx}" "")"
 
 	# If we don't have relaxed atomics, then we don't bother with ordering fallbacks
 	# read_acquire and set_release need to be templated, though
author reviewer
#35 lore raw

PATCH v2 15/27 locking/atomic: scripts: remove leftover "${mult}"

In reply to Mark Rutland

We removed cmpxchg_double() and variants in commit:

b4cf83b2d1da40b2 ("arch: Remove cmpxchg_double")

Which removed the need for "${mult}" in the instrumentation logic. Unfortunately we missed an instance of "${mult}".

There is no change to the generated header.
There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 scripts/atomic/gen-atomic-instrumented.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/atomic/gen-atomic-instrumented.sh +1 -1
index a2ef735be8ca9..68557bfbbdc5e 100755
@@ -118,7 +118,7 @@ cat <<EOF
 EOF
 [ -n "$kcsan_barrier" ] && printf "\t${kcsan_barrier}; \\\\\n"
 cat <<EOF
-	instrument_atomic_read_write(__ai_ptr, ${mult}sizeof(*__ai_ptr)); \\
+	instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \\
 	arch_${xchg}${order}(__ai_ptr, __VA_ARGS__); \\
 })
 EOF
Signature · 1 line
2.30.2
merge bot Applied
#36 lore raw

[tip: locking/core] locking/atomic: scripts: remove leftover "${mult}"

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     e40e5298e692bb6b5a200b3f0f55e6e5adf0e5ad
Gitweb:        https://git.kernel.org/tip/e40e5298e692bb6b5a200b3f0f55e6e5adf0e5ad
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:12 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:18 +02:00

locking/atomic: scripts: remove leftover "${mult}"

We removed cmpxchg_double() and variants in commit:

b4cf83b2d1da40b2 ("arch: Remove cmpxchg_double")

Which removed the need for "${mult}" in the instrumentation logic. Unfortunately we missed an instance of "${mult}".

There is no change to the generated header.
There should be no functional change as a result of this patch.

---
 scripts/atomic/gen-atomic-instrumented.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/atomic/gen-atomic-instrumented.sh +1 -1
index a2ef735..68557bf 100755
@@ -118,7 +118,7 @@ cat <<EOF
 EOF
 [ -n "$kcsan_barrier" ] && printf "\t${kcsan_barrier}; \\\\\n"
 cat <<EOF
-	instrument_atomic_read_write(__ai_ptr, ${mult}sizeof(*__ai_ptr)); \\
+	instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \\
 	arch_${xchg}${order}(__ai_ptr, __VA_ARGS__); \\
 })
 EOF
author reviewer
#37 lore raw

PATCH v2 16/27 locking/atomic: scripts: factor out order template generation

In reply to Mark Rutland

Currently gen_proto_order_variants() hard codes the path for the templates used for order fallbacks. Factor this out into a helper so that it can be reused elsewhere.

This results in no change to the generated headers, so there should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 scripts/atomic/gen-atomic-fallback.sh | 34 +++++++++++++--------------
 1 file changed, 17 insertions(+), 17 deletions(-)
scripts/atomic/gen-atomic-fallback.sh +17 -17
index 7a6bcea8f565b..337330865fa2e 100755
@@ -32,6 +32,20 @@ gen_template_fallback()
 	fi
 }
 
+#gen_order_fallback(meta, pfx, name, sfx, order, atomic, int, args...)
+gen_order_fallback()
+{
+	local meta="$1"; shift
+	local pfx="$1"; shift
+	local name="$1"; shift
+	local sfx="$1"; shift
+	local order="$1"; shift
+
+	local tmpl_order=${order#_}
+	local tmpl="${ATOMICDIR}/fallbacks/${tmpl_order:-fence}"
+	gen_template_fallback "${tmpl}" "${meta}" "${pfx}" "${name}" "${sfx}" "${order}" "$@"
+}
+
 #gen_proto_fallback(meta, pfx, name, sfx, order, atomic, int, args...)
 gen_proto_fallback()
 {
@@ -56,20 +70,6 @@ cat << EOF
 EOF
 }
 
-gen_proto_order_variant()
-{
-	local meta="$1"; shift
-	local pfx="$1"; shift
-	local name="$1"; shift
-	local sfx="$1"; shift
-	local order="$1"; shift
-	local atomic="$1"
-
-	local basename="arch_${atomic}_${pfx}${name}${sfx}"
-
-	printf "#define ${basename}${order} ${basename}${order}\n"
-}
-
 #gen_proto_order_variants(meta, pfx, name, sfx, atomic, int, args...)
 gen_proto_order_variants()
 {
@@ -117,9 +117,9 @@ gen_proto_order_variants()
 
 	printf "#else /* ${basename}_relaxed */\n\n"
 
-	gen_template_fallback "${ATOMICDIR}/fallbacks/acquire"  "${meta}" "${pfx}" "${name}" "${sfx}" "_acquire" "$@"
-	gen_template_fallback "${ATOMICDIR}/fallbacks/release"  "${meta}" "${pfx}" "${name}" "${sfx}" "_release" "$@"
-	gen_template_fallback "${ATOMICDIR}/fallbacks/fence"  "${meta}" "${pfx}" "${name}" "${sfx}" "" "$@"
+	gen_order_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_acquire" "$@"
+	gen_order_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_release" "$@"
+	gen_order_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "" "$@"
 
 	printf "#endif /* ${basename}_relaxed */\n\n"
 }
Signature · 1 line
2.30.2
merge bot Applied
#38 lore raw

[tip: locking/core] locking/atomic: scripts: factor out order template generation

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     7ed7a1564090fdd265f49d1ad94ee92845b14c76
Gitweb:        https://git.kernel.org/tip/7ed7a1564090fdd265f49d1ad94ee92845b14c76
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:13 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:19 +02:00

locking/atomic: scripts: factor out order template generation

Currently gen_proto_order_variants() hard codes the path for the templates used for order fallbacks. Factor this out into a helper so that it can be reused elsewhere.

This results in no change to the generated headers, so there should be no functional change as a result of this patch.

---
 scripts/atomic/gen-atomic-fallback.sh | 34 +++++++++++++-------------
 1 file changed, 17 insertions(+), 17 deletions(-)
scripts/atomic/gen-atomic-fallback.sh +17 -17
index 7a6bcea..3373308 100755
@@ -32,6 +32,20 @@ gen_template_fallback()
 	fi
 }
 
+#gen_order_fallback(meta, pfx, name, sfx, order, atomic, int, args...)
+gen_order_fallback()
+{
+	local meta="$1"; shift
+	local pfx="$1"; shift
+	local name="$1"; shift
+	local sfx="$1"; shift
+	local order="$1"; shift
+
+	local tmpl_order=${order#_}
+	local tmpl="${ATOMICDIR}/fallbacks/${tmpl_order:-fence}"
+	gen_template_fallback "${tmpl}" "${meta}" "${pfx}" "${name}" "${sfx}" "${order}" "$@"
+}
+
 #gen_proto_fallback(meta, pfx, name, sfx, order, atomic, int, args...)
 gen_proto_fallback()
 {
@@ -56,20 +70,6 @@ cat << EOF
 EOF
 }
 
-gen_proto_order_variant()
-{
-	local meta="$1"; shift
-	local pfx="$1"; shift
-	local name="$1"; shift
-	local sfx="$1"; shift
-	local order="$1"; shift
-	local atomic="$1"
-
-	local basename="arch_${atomic}_${pfx}${name}${sfx}"
-
-	printf "#define ${basename}${order} ${basename}${order}\n"
-}
-
 #gen_proto_order_variants(meta, pfx, name, sfx, atomic, int, args...)
 gen_proto_order_variants()
 {
@@ -117,9 +117,9 @@ gen_proto_order_variants()
 
 	printf "#else /* ${basename}_relaxed */\n\n"
 
-	gen_template_fallback "${ATOMICDIR}/fallbacks/acquire"  "${meta}" "${pfx}" "${name}" "${sfx}" "_acquire" "$@"
-	gen_template_fallback "${ATOMICDIR}/fallbacks/release"  "${meta}" "${pfx}" "${name}" "${sfx}" "_release" "$@"
-	gen_template_fallback "${ATOMICDIR}/fallbacks/fence"  "${meta}" "${pfx}" "${name}" "${sfx}" "" "$@"
+	gen_order_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_acquire" "$@"
+	gen_order_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_release" "$@"
+	gen_order_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "" "$@"
 
 	printf "#endif /* ${basename}_relaxed */\n\n"
 }
author reviewer
#39 lore raw

PATCH v2 17/27 locking/atomic: scripts: add trivial raw_atomic*_<op>()

In reply to Mark Rutland

Currently a number of arch_atomic*_<op>() functions are optional, and where an arch does not provide a given arch_atomic*_<op>() we will define an implementation of arch_atomic*_<op>() in atomic-arch-fallback.h.

Filling in the missing ops requires special care as we want to select the optimal definition of each op (e.g. preferentially defining ops in terms of their relaxed form rather than their fully-ordered form). The ifdeffery necessary for this requires us to group ordering variants together, which can be a bit painful to read, and is painful for kerneldoc generation.

It would be easier to handle this if we generated ops into a separate namespace, as this would remove the need to take special care with the ifdeffery, and allow each ordering variant to be generated separately.

This patch adds a new set of raw_atomic_<op>() definitions, which are currently trivial wrappers of their arch_atomic_<op>() equivalent. This will allow us to move treewide users of arch_atomic_<op>() over to raw atomic op before we rework the fallback generation to generate raw_atomic_<op> directly.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 include/linux/atomic.h                     |    1 +
 include/linux/atomic/atomic-instrumented.h |  595 ++++---
 include/linux/atomic/atomic-raw.h          | 1645 ++++++++++++++++++++
 scripts/atomic/gen-atomic-instrumented.sh  |   19 +-
 scripts/atomic/gen-atomic-raw.sh           |   84 +
 scripts/atomic/gen-atomics.sh              |    1 +
 6 files changed, 2033 insertions(+), 312 deletions(-)
 create mode 100644 include/linux/atomic/atomic-raw.h
 create mode 100755 scripts/atomic/gen-atomic-raw.sh
include/linux/atomic.h +1 -0
index 8dd57c3a99e9b..127f5dc63a7df 100644
@@ -79,6 +79,7 @@
 
 #include <linux/atomic/atomic-arch-fallback.h>
 #include <linux/atomic/atomic-long.h>
+#include <linux/atomic/atomic-raw.h>
 #include <linux/atomic/atomic-instrumented.h>
 
 #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-instrumented.h +295 -300

2,342 lines not on this page · raw message

include/linux/atomic/atomic-raw.h new file +1645 -0

651 lines not on this page · raw message

scripts/atomic/gen-atomic-instrumented.sh +7 -12
index 68557bfbbdc5e..93c949aa9e544 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/gen-atomic-raw.sh new file +84 -0

84 lines not on this page · raw message

scripts/atomic/gen-atomics.sh +1 -0
index 5b98a83076932..631d351f9f1f3 100755

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.30.2

Part of the diff is not shown here. View the raw message.

merge bot Applied
#40 lore raw

[tip: locking/core] locking/atomic: scripts: add trivial raw_atomic*_<op>()

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     c9268ac615f9f6dded7801df5993374598934377
Gitweb:        https://git.kernel.org/tip/c9268ac615f9f6dded7801df5993374598934377
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:14 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:19 +02:00

locking/atomic: scripts: add trivial raw_atomic*_<op>()

Currently a number of arch_atomic*_<op>() functions are optional, and where an arch does not provide a given arch_atomic*_<op>() we will define an implementation of arch_atomic*_<op>() in atomic-arch-fallback.h.

Filling in the missing ops requires special care as we want to select the optimal definition of each op (e.g. preferentially defining ops in terms of their relaxed form rather than their fully-ordered form). The ifdeffery necessary for this requires us to group ordering variants together, which can be a bit painful to read, and is painful for kerneldoc generation.

It would be easier to handle this if we generated ops into a separate namespace, as this would remove the need to take special care with the ifdeffery, and allow each ordering variant to be generated separately.

This patch adds a new set of raw_atomic_<op>() definitions, which are currently trivial wrappers of their arch_atomic_<op>() equivalent. This will allow us to move treewide users of arch_atomic_<op>() over to raw atomic op before we rework the fallback generation to generate raw_atomic_<op> directly.

There should be no functional change as a result of this patch.

---
 include/linux/atomic.h                     |    1 +-
 include/linux/atomic/atomic-instrumented.h |  595 +++----
 include/linux/atomic/atomic-raw.h          | 1645 +++++++++++++++++++-
 scripts/atomic/gen-atomic-instrumented.sh  |   19 +-
 scripts/atomic/gen-atomic-raw.sh           |   84 +-
 scripts/atomic/gen-atomics.sh              |    1 +-
 6 files changed, 2033 insertions(+), 312 deletions(-)
 create mode 100644 include/linux/atomic/atomic-raw.h
 create mode 100644 scripts/atomic/gen-atomic-raw.sh
include/linux/atomic.h +1 -0
index 8dd57c3..127f5dc 100644
@@ -79,6 +79,7 @@
 
 #include <linux/atomic/atomic-arch-fallback.h>
 #include <linux/atomic/atomic-long.h>
+#include <linux/atomic/atomic-raw.h>
 #include <linux/atomic/atomic-instrumented.h>
 
 #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-instrumented.h +295 -300

2,342 lines not on this page · raw message

include/linux/atomic/atomic-raw.h new file +1645 -0

651 lines not on this page · raw message

scripts/atomic/gen-atomic-instrumented.sh +7 -12
index 68557bf..93c949a 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/gen-atomic-raw.sh new file +84 -0

84 lines not on this page · raw message

scripts/atomic/gen-atomics.sh +1 -0
index 5b98a83..631d351 100755

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

author reviewer
#41 lore raw

PATCH v2 18/27 locking/atomic: treewide: use raw_atomic*_<op>()

In reply to Mark Rutland

Now that we have raw_atomic*_<op>() definitions, there's no need to use arch_atomic*_<op>() definitions outside of the low-level atomic definitions.

Move treewide users of arch_atomic*_<op>() over to the equivalent raw_atomic*_<op>().

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 arch/powerpc/kernel/smp.c              | 12 ++++++------
 arch/x86/kernel/alternative.c          |  4 ++--
 arch/x86/kernel/cpu/mce/core.c         | 16 ++++++++--------
 arch/x86/kernel/nmi.c                  |  2 +-
 arch/x86/kernel/pvclock.c              |  4 ++--
 arch/x86/kvm/x86.c                     |  2 +-
 include/asm-generic/bitops/atomic.h    | 12 ++++++------
 include/asm-generic/bitops/lock.h      |  8 ++++----
 include/linux/context_tracking.h       |  4 ++--
 include/linux/context_tracking_state.h |  2 +-
 include/linux/cpumask.h                |  2 +-
 include/linux/jump_label.h             |  2 +-
 kernel/context_tracking.c              | 12 ++++++------
 kernel/sched/clock.c                   |  2 +-
 14 files changed, 42 insertions(+), 42 deletions(-)
arch/powerpc/kernel/smp.c +6 -6
index 265801a3e94cf..e8965f18686f0 100644
@@ -417,9 +417,9 @@ noinstr static void nmi_ipi_lock_start(unsigned long *flags)
 {
 	raw_local_irq_save(*flags);
 	hard_irq_disable();
-	while (arch_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1) {
+	while (raw_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1) {
 		raw_local_irq_restore(*flags);
-		spin_until_cond(arch_atomic_read(&__nmi_ipi_lock) == 0);
+		spin_until_cond(raw_atomic_read(&__nmi_ipi_lock) == 0);
 		raw_local_irq_save(*flags);
 		hard_irq_disable();
 	}
@@ -427,15 +427,15 @@ noinstr static void nmi_ipi_lock_start(unsigned long *flags)
 
 noinstr static void nmi_ipi_lock(void)
 {
-	while (arch_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1)
-		spin_until_cond(arch_atomic_read(&__nmi_ipi_lock) == 0);
+	while (raw_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1)
+		spin_until_cond(raw_atomic_read(&__nmi_ipi_lock) == 0);
 }
 
 noinstr static void nmi_ipi_unlock(void)
 {
 	smp_mb();
-	WARN_ON(arch_atomic_read(&__nmi_ipi_lock) != 1);
-	arch_atomic_set(&__nmi_ipi_lock, 0);
+	WARN_ON(raw_atomic_read(&__nmi_ipi_lock) != 1);
+	raw_atomic_set(&__nmi_ipi_lock, 0);
 }
 
 noinstr static void nmi_ipi_unlock_end(unsigned long *flags)
arch/x86/kernel/alternative.c +2 -2
index f615e0cb6d932..18f16e93838fe 100644
@@ -1799,7 +1799,7 @@ struct bp_patching_desc *try_get_desc(void)
 {
 	struct bp_patching_desc *desc = &bp_desc;
 
-	if (!arch_atomic_inc_not_zero(&desc->refs))
+	if (!raw_atomic_inc_not_zero(&desc->refs))
 		return NULL;
 
 	return desc;
@@ -1810,7 +1810,7 @@ static __always_inline void put_desc(void)
 	struct bp_patching_desc *desc = &bp_desc;
 
 	smp_mb__before_atomic();
-	arch_atomic_dec(&desc->refs);
+	raw_atomic_dec(&desc->refs);
 }
 
 static __always_inline void *text_poke_addr(struct text_poke_loc *tp)
arch/x86/kernel/cpu/mce/core.c +8 -8
index 2eec60f50057a..ab156e6e71208 100644
@@ -1022,12 +1022,12 @@ static noinstr int mce_start(int *no_way_out)
 	if (!timeout)
 		return ret;
 
-	arch_atomic_add(*no_way_out, &global_nwo);
+	raw_atomic_add(*no_way_out, &global_nwo);
 	/*
 	 * Rely on the implied barrier below, such that global_nwo
 	 * is updated before mce_callin.
 	 */
-	order = arch_atomic_inc_return(&mce_callin);
+	order = raw_atomic_inc_return(&mce_callin);
 	arch_cpumask_clear_cpu(smp_processor_id(), &mce_missing_cpus);
 
 	/* Enable instrumentation around calls to external facilities */
@@ -1036,10 +1036,10 @@ static noinstr int mce_start(int *no_way_out)
 	/*
 	 * Wait for everyone.
 	 */
-	while (arch_atomic_read(&mce_callin) != num_online_cpus()) {
+	while (raw_atomic_read(&mce_callin) != num_online_cpus()) {
 		if (mce_timed_out(&timeout,
 				  "Timeout: Not all CPUs entered broadcast exception handler")) {
-			arch_atomic_set(&global_nwo, 0);
+			raw_atomic_set(&global_nwo, 0);
 			goto out;
 		}
 		ndelay(SPINUNIT);
@@ -1054,7 +1054,7 @@ static noinstr int mce_start(int *no_way_out)
 		/*
 		 * Monarch: Starts executing now, the others wait.
 		 */
-		arch_atomic_set(&mce_executing, 1);
+		raw_atomic_set(&mce_executing, 1);
 	} else {
 		/*
 		 * Subject: Now start the scanning loop one by one in
@@ -1062,10 +1062,10 @@ static noinstr int mce_start(int *no_way_out)
 		 * This way when there are any shared banks it will be
 		 * only seen by one CPU before cleared, avoiding duplicates.
 		 */
-		while (arch_atomic_read(&mce_executing) < order) {
+		while (raw_atomic_read(&mce_executing) < order) {
 			if (mce_timed_out(&timeout,
 					  "Timeout: Subject CPUs unable to finish machine check processing")) {
-				arch_atomic_set(&global_nwo, 0);
+				raw_atomic_set(&global_nwo, 0);
 				goto out;
 			}
 			ndelay(SPINUNIT);
@@ -1075,7 +1075,7 @@ static noinstr int mce_start(int *no_way_out)
 	/*
 	 * Cache the global no_way_out state.
 	 */
-	*no_way_out = arch_atomic_read(&global_nwo);
+	*no_way_out = raw_atomic_read(&global_nwo);
 
 	ret = order;
 
arch/x86/kernel/nmi.c +1 -1
index 776f4b1e395b5..a0c551846b35f 100644
@@ -496,7 +496,7 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
 	 */
 	sev_es_nmi_complete();
 	if (IS_ENABLED(CONFIG_NMI_CHECK_CPU))
-		arch_atomic_long_inc(&nsp->idt_calls);
+		raw_atomic_long_inc(&nsp->idt_calls);
 
 	if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id()))
 		return;
arch/x86/kernel/pvclock.c +2 -2
index 56acf53a782ad..b3f81379c2fc0 100644
@@ -101,11 +101,11 @@ u64 __pvclock_clocksource_read(struct pvclock_vcpu_time_info *src, bool dowd)
 	 * updating at the same time, and one of them could be slightly behind,
 	 * making the assumption that last_value always go forward fail to hold.
 	 */
-	last = arch_atomic64_read(&last_value);
+	last = raw_atomic64_read(&last_value);
 	do {
 		if (ret <= last)
 			return last;
-	} while (!arch_atomic64_try_cmpxchg(&last_value, &last, ret));
+	} while (!raw_atomic64_try_cmpxchg(&last_value, &last, ret));
 
 	return ret;
 }
arch/x86/kvm/x86.c +1 -1
index ceb7c5e9cf9e9..ac6f609068106 100644
@@ -13155,7 +13155,7 @@ EXPORT_SYMBOL_GPL(kvm_arch_end_assignment);
 
 bool noinstr kvm_arch_has_assigned_device(struct kvm *kvm)
 {
-	return arch_atomic_read(&kvm->arch.assigned_device_count);
+	return raw_atomic_read(&kvm->arch.assigned_device_count);
 }
 EXPORT_SYMBOL_GPL(kvm_arch_has_assigned_device);
 
include/asm-generic/bitops/atomic.h +6 -6
index 71ab4ba9c25d1..e076e079f6b2e 100644
@@ -15,21 +15,21 @@ static __always_inline void
 arch_set_bit(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_or(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_or(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 static __always_inline void
 arch_clear_bit(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_andnot(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_andnot(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 static __always_inline void
 arch_change_bit(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_xor(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_xor(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 static __always_inline int
@@ -39,7 +39,7 @@ arch_test_and_set_bit(unsigned int nr, volatile unsigned long *p)
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_or(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_or(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
@@ -50,7 +50,7 @@ arch_test_and_clear_bit(unsigned int nr, volatile unsigned long *p)
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_andnot(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_andnot(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
@@ -61,7 +61,7 @@ arch_test_and_change_bit(unsigned int nr, volatile unsigned long *p)
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_xor(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_xor(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
include/asm-generic/bitops/lock.h +4 -4
index 630f2f6b95956..40913516e654c 100644
@@ -25,7 +25,7 @@ arch_test_and_set_bit_lock(unsigned int nr, volatile unsigned long *p)
 	if (READ_ONCE(*p) & mask)
 		return 1;
 
-	old = arch_atomic_long_fetch_or_acquire(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_or_acquire(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
@@ -41,7 +41,7 @@ static __always_inline void
 arch_clear_bit_unlock(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_fetch_andnot_release(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_fetch_andnot_release(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 /**
@@ -63,7 +63,7 @@ arch___clear_bit_unlock(unsigned int nr, volatile unsigned long *p)
 	p += BIT_WORD(nr);
 	old = READ_ONCE(*p);
 	old &= ~BIT_MASK(nr);
-	arch_atomic_long_set_release((atomic_long_t *)p, old);
+	raw_atomic_long_set_release((atomic_long_t *)p, old);
 }
 
 /**
@@ -83,7 +83,7 @@ static inline bool arch_clear_bit_unlock_is_negative_byte(unsigned int nr,
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
 	return !!(old & BIT(7));
 }
 #define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte
include/linux/context_tracking.h +2 -2
index d3cbb6c16babf..6e76b9dba00e7 100644
@@ -119,7 +119,7 @@ extern void ct_idle_exit(void);
  */
 static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void)
 {
-	return !(arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & RCU_DYNTICKS_IDX);
+	return !(raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & RCU_DYNTICKS_IDX);
 }
 
 /*
@@ -128,7 +128,7 @@ static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void)
  */
 static __always_inline unsigned long ct_state_inc(int incby)
 {
-	return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state));
+	return raw_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state));
 }
 
 static __always_inline bool warn_rcu_enter(void)
include/linux/context_tracking_state.h +1 -1
index fdd537ea513ff..bbff5f7f88030 100644
@@ -51,7 +51,7 @@ DECLARE_PER_CPU(struct context_tracking, context_tracking);
 #ifdef CONFIG_CONTEXT_TRACKING_USER
 static __always_inline int __ct_state(void)
 {
-	return arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK;
+	return raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK;
 }
 #endif
 
include/linux/cpumask.h +1 -1
index ca736b05ec7b0..0d2e2a38b92d0 100644
@@ -1071,7 +1071,7 @@ static inline const struct cpumask *get_cpu_mask(unsigned int cpu)
  */
 static __always_inline unsigned int num_online_cpus(void)
 {
-	return arch_atomic_read(&__num_online_cpus);
+	return raw_atomic_read(&__num_online_cpus);
 }
 #define num_possible_cpus()	cpumask_weight(cpu_possible_mask)
 #define num_present_cpus()	cpumask_weight(cpu_present_mask)
include/linux/jump_label.h +1 -1
index 4e968ebadce60..f0a949b7c9733 100644
@@ -257,7 +257,7 @@ extern enum jump_label_type jump_label_init_type(struct jump_entry *entry);
 
 static __always_inline int static_key_count(struct static_key *key)
 {
-	return arch_atomic_read(&key->enabled);
+	return raw_atomic_read(&key->enabled);
 }
 
 static __always_inline void jump_label_init(void)
kernel/context_tracking.c +6 -6
index a09f1c19336ae..6ef0b35fc28c5 100644
@@ -510,7 +510,7 @@ void noinstr __ct_user_enter(enum ctx_state state)
 			 * In this we case we don't care about any concurrency/ordering.
 			 */
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE))
-				arch_atomic_set(&ct->state, state);
+				raw_atomic_set(&ct->state, state);
 		} else {
 			/*
 			 * Even if context tracking is disabled on this CPU, because it's outside
@@ -527,7 +527,7 @@ void noinstr __ct_user_enter(enum ctx_state state)
 			 */
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) {
 				/* Tracking for vtime only, no concurrent RCU EQS accounting */
-				arch_atomic_set(&ct->state, state);
+				raw_atomic_set(&ct->state, state);
 			} else {
 				/*
 				 * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -535,7 +535,7 @@ void noinstr __ct_user_enter(enum ctx_state state)
 				 * RCU only requires RCU_DYNTICKS_IDX increments to be fully
 				 * ordered.
 				 */
-				arch_atomic_add(state, &ct->state);
+				raw_atomic_add(state, &ct->state);
 			}
 		}
 	}
@@ -630,12 +630,12 @@ void noinstr __ct_user_exit(enum ctx_state state)
 			 * In this we case we don't care about any concurrency/ordering.
 			 */
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE))
-				arch_atomic_set(&ct->state, CONTEXT_KERNEL);
+				raw_atomic_set(&ct->state, CONTEXT_KERNEL);
 
 		} else {
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) {
 				/* Tracking for vtime only, no concurrent RCU EQS accounting */
-				arch_atomic_set(&ct->state, CONTEXT_KERNEL);
+				raw_atomic_set(&ct->state, CONTEXT_KERNEL);
 			} else {
 				/*
 				 * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -643,7 +643,7 @@ void noinstr __ct_user_exit(enum ctx_state state)
 				 * RCU only requires RCU_DYNTICKS_IDX increments to be fully
 				 * ordered.
 				 */
-				arch_atomic_sub(state, &ct->state);
+				raw_atomic_sub(state, &ct->state);
 			}
 		}
 	}
kernel/sched/clock.c +1 -1
index b5cc2b53464de..71443cff31f0d 100644
@@ -287,7 +287,7 @@ static __always_inline u64 sched_clock_local(struct sched_clock_data *scd)
 	clock = wrap_max(clock, min_clock);
 	clock = wrap_min(clock, max_clock);
 
-	if (!arch_try_cmpxchg64(&scd->clock, &old_clock, clock))
+	if (!raw_try_cmpxchg64(&scd->clock, &old_clock, clock))
 		goto again;
 
 	return clock;
Signature · 1 line
2.30.2
merge bot Applied
#42 lore raw

[tip: locking/core] locking/atomic: treewide: use raw_atomic*_<op>()

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     0f613bfa8268a89be25f2b6b58fc6fe8ccd9a2ba
Gitweb:        https://git.kernel.org/tip/0f613bfa8268a89be25f2b6b58fc6fe8ccd9a2ba
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:15 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:20 +02:00

locking/atomic: treewide: use raw_atomic*_<op>()

Now that we have raw_atomic*_<op>() definitions, there's no need to use arch_atomic*_<op>() definitions outside of the low-level atomic definitions.

Move treewide users of arch_atomic*_<op>() over to the equivalent raw_atomic*_<op>().

There should be no functional change as a result of this patch.

---
 arch/powerpc/kernel/smp.c              | 12 ++++++------
 arch/x86/kernel/alternative.c          |  4 ++--
 arch/x86/kernel/cpu/mce/core.c         | 16 ++++++++--------
 arch/x86/kernel/nmi.c                  |  2 +-
 arch/x86/kernel/pvclock.c              |  4 ++--
 arch/x86/kvm/x86.c                     |  2 +-
 include/asm-generic/bitops/atomic.h    | 12 ++++++------
 include/asm-generic/bitops/lock.h      |  8 ++++----
 include/linux/context_tracking.h       |  4 ++--
 include/linux/context_tracking_state.h |  2 +-
 include/linux/cpumask.h                |  2 +-
 include/linux/jump_label.h             |  2 +-
 kernel/context_tracking.c              | 12 ++++++------
 kernel/sched/clock.c                   |  2 +-
 14 files changed, 42 insertions(+), 42 deletions(-)
arch/powerpc/kernel/smp.c +6 -6
index 265801a..e8965f1 100644
@@ -417,9 +417,9 @@ noinstr static void nmi_ipi_lock_start(unsigned long *flags)
 {
 	raw_local_irq_save(*flags);
 	hard_irq_disable();
-	while (arch_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1) {
+	while (raw_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1) {
 		raw_local_irq_restore(*flags);
-		spin_until_cond(arch_atomic_read(&__nmi_ipi_lock) == 0);
+		spin_until_cond(raw_atomic_read(&__nmi_ipi_lock) == 0);
 		raw_local_irq_save(*flags);
 		hard_irq_disable();
 	}
@@ -427,15 +427,15 @@ noinstr static void nmi_ipi_lock_start(unsigned long *flags)
 
 noinstr static void nmi_ipi_lock(void)
 {
-	while (arch_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1)
-		spin_until_cond(arch_atomic_read(&__nmi_ipi_lock) == 0);
+	while (raw_atomic_cmpxchg(&__nmi_ipi_lock, 0, 1) == 1)
+		spin_until_cond(raw_atomic_read(&__nmi_ipi_lock) == 0);
 }
 
 noinstr static void nmi_ipi_unlock(void)
 {
 	smp_mb();
-	WARN_ON(arch_atomic_read(&__nmi_ipi_lock) != 1);
-	arch_atomic_set(&__nmi_ipi_lock, 0);
+	WARN_ON(raw_atomic_read(&__nmi_ipi_lock) != 1);
+	raw_atomic_set(&__nmi_ipi_lock, 0);
 }
 
 noinstr static void nmi_ipi_unlock_end(unsigned long *flags)
arch/x86/kernel/alternative.c +2 -2
index f615e0c..18f16e9 100644
@@ -1799,7 +1799,7 @@ struct bp_patching_desc *try_get_desc(void)
 {
 	struct bp_patching_desc *desc = &bp_desc;
 
-	if (!arch_atomic_inc_not_zero(&desc->refs))
+	if (!raw_atomic_inc_not_zero(&desc->refs))
 		return NULL;
 
 	return desc;
@@ -1810,7 +1810,7 @@ static __always_inline void put_desc(void)
 	struct bp_patching_desc *desc = &bp_desc;
 
 	smp_mb__before_atomic();
-	arch_atomic_dec(&desc->refs);
+	raw_atomic_dec(&desc->refs);
 }
 
 static __always_inline void *text_poke_addr(struct text_poke_loc *tp)
arch/x86/kernel/cpu/mce/core.c +8 -8
index 2eec60f..ab156e6 100644
@@ -1022,12 +1022,12 @@ static noinstr int mce_start(int *no_way_out)
 	if (!timeout)
 		return ret;
 
-	arch_atomic_add(*no_way_out, &global_nwo);
+	raw_atomic_add(*no_way_out, &global_nwo);
 	/*
 	 * Rely on the implied barrier below, such that global_nwo
 	 * is updated before mce_callin.
 	 */
-	order = arch_atomic_inc_return(&mce_callin);
+	order = raw_atomic_inc_return(&mce_callin);
 	arch_cpumask_clear_cpu(smp_processor_id(), &mce_missing_cpus);
 
 	/* Enable instrumentation around calls to external facilities */
@@ -1036,10 +1036,10 @@ static noinstr int mce_start(int *no_way_out)
 	/*
 	 * Wait for everyone.
 	 */
-	while (arch_atomic_read(&mce_callin) != num_online_cpus()) {
+	while (raw_atomic_read(&mce_callin) != num_online_cpus()) {
 		if (mce_timed_out(&timeout,
 				  "Timeout: Not all CPUs entered broadcast exception handler")) {
-			arch_atomic_set(&global_nwo, 0);
+			raw_atomic_set(&global_nwo, 0);
 			goto out;
 		}
 		ndelay(SPINUNIT);
@@ -1054,7 +1054,7 @@ static noinstr int mce_start(int *no_way_out)
 		/*
 		 * Monarch: Starts executing now, the others wait.
 		 */
-		arch_atomic_set(&mce_executing, 1);
+		raw_atomic_set(&mce_executing, 1);
 	} else {
 		/*
 		 * Subject: Now start the scanning loop one by one in
@@ -1062,10 +1062,10 @@ static noinstr int mce_start(int *no_way_out)
 		 * This way when there are any shared banks it will be
 		 * only seen by one CPU before cleared, avoiding duplicates.
 		 */
-		while (arch_atomic_read(&mce_executing) < order) {
+		while (raw_atomic_read(&mce_executing) < order) {
 			if (mce_timed_out(&timeout,
 					  "Timeout: Subject CPUs unable to finish machine check processing")) {
-				arch_atomic_set(&global_nwo, 0);
+				raw_atomic_set(&global_nwo, 0);
 				goto out;
 			}
 			ndelay(SPINUNIT);
@@ -1075,7 +1075,7 @@ static noinstr int mce_start(int *no_way_out)
 	/*
 	 * Cache the global no_way_out state.
 	 */
-	*no_way_out = arch_atomic_read(&global_nwo);
+	*no_way_out = raw_atomic_read(&global_nwo);
 
 	ret = order;
 
arch/x86/kernel/nmi.c +1 -1
index 776f4b1..a0c5518 100644
@@ -496,7 +496,7 @@ DEFINE_IDTENTRY_RAW(exc_nmi)
 	 */
 	sev_es_nmi_complete();
 	if (IS_ENABLED(CONFIG_NMI_CHECK_CPU))
-		arch_atomic_long_inc(&nsp->idt_calls);
+		raw_atomic_long_inc(&nsp->idt_calls);
 
 	if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id()))
 		return;
arch/x86/kernel/pvclock.c +2 -2
index 56acf53..b3f8137 100644
@@ -101,11 +101,11 @@ u64 __pvclock_clocksource_read(struct pvclock_vcpu_time_info *src, bool dowd)
 	 * updating at the same time, and one of them could be slightly behind,
 	 * making the assumption that last_value always go forward fail to hold.
 	 */
-	last = arch_atomic64_read(&last_value);
+	last = raw_atomic64_read(&last_value);
 	do {
 		if (ret <= last)
 			return last;
-	} while (!arch_atomic64_try_cmpxchg(&last_value, &last, ret));
+	} while (!raw_atomic64_try_cmpxchg(&last_value, &last, ret));
 
 	return ret;
 }
arch/x86/kvm/x86.c +1 -1
index ceb7c5e..ac6f609 100644
@@ -13155,7 +13155,7 @@ EXPORT_SYMBOL_GPL(kvm_arch_end_assignment);
 
 bool noinstr kvm_arch_has_assigned_device(struct kvm *kvm)
 {
-	return arch_atomic_read(&kvm->arch.assigned_device_count);
+	return raw_atomic_read(&kvm->arch.assigned_device_count);
 }
 EXPORT_SYMBOL_GPL(kvm_arch_has_assigned_device);
 
include/asm-generic/bitops/atomic.h +6 -6
index 71ab4ba..e076e07 100644
@@ -15,21 +15,21 @@ static __always_inline void
 arch_set_bit(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_or(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_or(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 static __always_inline void
 arch_clear_bit(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_andnot(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_andnot(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 static __always_inline void
 arch_change_bit(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_xor(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_xor(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 static __always_inline int
@@ -39,7 +39,7 @@ arch_test_and_set_bit(unsigned int nr, volatile unsigned long *p)
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_or(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_or(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
@@ -50,7 +50,7 @@ arch_test_and_clear_bit(unsigned int nr, volatile unsigned long *p)
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_andnot(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_andnot(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
@@ -61,7 +61,7 @@ arch_test_and_change_bit(unsigned int nr, volatile unsigned long *p)
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_xor(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_xor(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
include/asm-generic/bitops/lock.h +4 -4
index 630f2f6..4091351 100644
@@ -25,7 +25,7 @@ arch_test_and_set_bit_lock(unsigned int nr, volatile unsigned long *p)
 	if (READ_ONCE(*p) & mask)
 		return 1;
 
-	old = arch_atomic_long_fetch_or_acquire(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_or_acquire(mask, (atomic_long_t *)p);
 	return !!(old & mask);
 }
 
@@ -41,7 +41,7 @@ static __always_inline void
 arch_clear_bit_unlock(unsigned int nr, volatile unsigned long *p)
 {
 	p += BIT_WORD(nr);
-	arch_atomic_long_fetch_andnot_release(BIT_MASK(nr), (atomic_long_t *)p);
+	raw_atomic_long_fetch_andnot_release(BIT_MASK(nr), (atomic_long_t *)p);
 }
 
 /**
@@ -63,7 +63,7 @@ arch___clear_bit_unlock(unsigned int nr, volatile unsigned long *p)
 	p += BIT_WORD(nr);
 	old = READ_ONCE(*p);
 	old &= ~BIT_MASK(nr);
-	arch_atomic_long_set_release((atomic_long_t *)p, old);
+	raw_atomic_long_set_release((atomic_long_t *)p, old);
 }
 
 /**
@@ -83,7 +83,7 @@ static inline bool arch_clear_bit_unlock_is_negative_byte(unsigned int nr,
 	unsigned long mask = BIT_MASK(nr);
 
 	p += BIT_WORD(nr);
-	old = arch_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
+	old = raw_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p);
 	return !!(old & BIT(7));
 }
 #define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte
include/linux/context_tracking.h +2 -2
index d3cbb6c..6e76b9d 100644
@@ -119,7 +119,7 @@ extern void ct_idle_exit(void);
  */
 static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void)
 {
-	return !(arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & RCU_DYNTICKS_IDX);
+	return !(raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & RCU_DYNTICKS_IDX);
 }
 
 /*
@@ -128,7 +128,7 @@ static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void)
  */
 static __always_inline unsigned long ct_state_inc(int incby)
 {
-	return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state));
+	return raw_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state));
 }
 
 static __always_inline bool warn_rcu_enter(void)
include/linux/context_tracking_state.h +1 -1
index fdd537e..bbff5f7 100644
@@ -51,7 +51,7 @@ DECLARE_PER_CPU(struct context_tracking, context_tracking);
 #ifdef CONFIG_CONTEXT_TRACKING_USER
 static __always_inline int __ct_state(void)
 {
-	return arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK;
+	return raw_atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK;
 }
 #endif
 
include/linux/cpumask.h +1 -1
index ca736b0..0d2e2a3 100644
@@ -1071,7 +1071,7 @@ static inline const struct cpumask *get_cpu_mask(unsigned int cpu)
  */
 static __always_inline unsigned int num_online_cpus(void)
 {
-	return arch_atomic_read(&__num_online_cpus);
+	return raw_atomic_read(&__num_online_cpus);
 }
 #define num_possible_cpus()	cpumask_weight(cpu_possible_mask)
 #define num_present_cpus()	cpumask_weight(cpu_present_mask)
include/linux/jump_label.h +1 -1
index 4e968eb..f0a949b 100644
@@ -257,7 +257,7 @@ extern enum jump_label_type jump_label_init_type(struct jump_entry *entry);
 
 static __always_inline int static_key_count(struct static_key *key)
 {
-	return arch_atomic_read(&key->enabled);
+	return raw_atomic_read(&key->enabled);
 }
 
 static __always_inline void jump_label_init(void)
kernel/context_tracking.c +6 -6
index a09f1c1..6ef0b35 100644
@@ -510,7 +510,7 @@ void noinstr __ct_user_enter(enum ctx_state state)
 			 * In this we case we don't care about any concurrency/ordering.
 			 */
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE))
-				arch_atomic_set(&ct->state, state);
+				raw_atomic_set(&ct->state, state);
 		} else {
 			/*
 			 * Even if context tracking is disabled on this CPU, because it's outside
@@ -527,7 +527,7 @@ void noinstr __ct_user_enter(enum ctx_state state)
 			 */
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) {
 				/* Tracking for vtime only, no concurrent RCU EQS accounting */
-				arch_atomic_set(&ct->state, state);
+				raw_atomic_set(&ct->state, state);
 			} else {
 				/*
 				 * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -535,7 +535,7 @@ void noinstr __ct_user_enter(enum ctx_state state)
 				 * RCU only requires RCU_DYNTICKS_IDX increments to be fully
 				 * ordered.
 				 */
-				arch_atomic_add(state, &ct->state);
+				raw_atomic_add(state, &ct->state);
 			}
 		}
 	}
@@ -630,12 +630,12 @@ void noinstr __ct_user_exit(enum ctx_state state)
 			 * In this we case we don't care about any concurrency/ordering.
 			 */
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE))
-				arch_atomic_set(&ct->state, CONTEXT_KERNEL);
+				raw_atomic_set(&ct->state, CONTEXT_KERNEL);
 
 		} else {
 			if (!IS_ENABLED(CONFIG_CONTEXT_TRACKING_IDLE)) {
 				/* Tracking for vtime only, no concurrent RCU EQS accounting */
-				arch_atomic_set(&ct->state, CONTEXT_KERNEL);
+				raw_atomic_set(&ct->state, CONTEXT_KERNEL);
 			} else {
 				/*
 				 * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -643,7 +643,7 @@ void noinstr __ct_user_exit(enum ctx_state state)
 				 * RCU only requires RCU_DYNTICKS_IDX increments to be fully
 				 * ordered.
 				 */
-				arch_atomic_sub(state, &ct->state);
+				raw_atomic_sub(state, &ct->state);
 			}
 		}
 	}
kernel/sched/clock.c +1 -1
index b5cc2b5..71443cf 100644
@@ -287,7 +287,7 @@ again:
 	clock = wrap_max(clock, min_clock);
 	clock = wrap_min(clock, max_clock);
 
-	if (!arch_try_cmpxchg64(&scd->clock, &old_clock, clock))
+	if (!raw_try_cmpxchg64(&scd->clock, &old_clock, clock))
 		goto again;
 
 	return clock;
author reviewer
#43 lore raw

PATCH v2 19/27 locking/atomic: scripts: build raw_atomic_long*() directly

In reply to Mark Rutland

Now that arch_atomic*() usage is limited to the atomic headers, we no longer have any users of arch_atomic_long_*(), and can generate raw_atomic_long_*() directly.

Generate the raw_atomic_long_*() ops directly.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 include/linux/atomic.h             |   2 +-
 include/linux/atomic/atomic-long.h | 682 ++++++++++++++---------------
 include/linux/atomic/atomic-raw.h  | 512 +---------------------
 scripts/atomic/gen-atomic-long.sh  |   4 +-
 scripts/atomic/gen-atomic-raw.sh   |   4 -
 5 files changed, 345 insertions(+), 859 deletions(-)
include/linux/atomic.h +1 -1
index 127f5dc63a7df..296cfae0389fe 100644
@@ -78,8 +78,8 @@
 })
 
 #include <linux/atomic/atomic-arch-fallback.h>
-#include <linux/atomic/atomic-long.h>
 #include <linux/atomic/atomic-raw.h>
+#include <linux/atomic/atomic-long.h>
 #include <linux/atomic/atomic-instrumented.h>
 
 #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-long.h +341 -341

1,368 lines not on this page · raw message

include/linux/atomic/atomic-raw.h +1 -511

521 lines not on this page · raw message

scripts/atomic/gen-atomic-long.sh +2 -2
index eda89cea6e1d1..75e91d6da30d3 100755
@@ -47,9 +47,9 @@ gen_proto_order_variant()
 
 cat <<EOF
 static __always_inline ${ret}
-arch_atomic_long_${name}(${params})
+raw_atomic_long_${name}(${params})
 {
-	${retstmt}arch_${atomic}_${name}(${argscast});
+	${retstmt}raw_${atomic}_${name}(${argscast});
 }
 
 EOF
scripts/atomic/gen-atomic-raw.sh +0 -4
index ba8d136f30e4c..c7e3c52b49279 100755
@@ -63,10 +63,6 @@ grep '^[a-z]' "$1" | while read name meta args; do
 	gen_proto "${meta}" "${name}" "atomic64" "s64" ${args}
 done
 
-grep '^[a-z]' "$1" | while read name meta args; do
-	gen_proto "${meta}" "${name}" "atomic_long" "long" ${args}
-done
-
 for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do
 	for order in "" "_acquire" "_release" "_relaxed"; do
 		gen_xchg "${xchg}" "${order}"
Signature · 1 line
2.30.2
merge bot Applied
#44 lore raw

[tip: locking/core] locking/atomic: scripts: build raw_atomic_long*() directly

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     1815da1718aa4c062b94cf3fc09432f552e25768
Gitweb:        https://git.kernel.org/tip/1815da1718aa4c062b94cf3fc09432f552e25768
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:16 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:20 +02:00

locking/atomic: scripts: build raw_atomic_long*() directly

Now that arch_atomic*() usage is limited to the atomic headers, we no longer have any users of arch_atomic_long_*(), and can generate raw_atomic_long_*() directly.

Generate the raw_atomic_long_*() ops directly.

There should be no functional change as a result of this patch.

---
 include/linux/atomic.h             |   2 +-
 include/linux/atomic/atomic-long.h | 682 ++++++++++++++--------------
 include/linux/atomic/atomic-raw.h  | 512 +---------------------
 scripts/atomic/gen-atomic-long.sh  |   4 +-
 scripts/atomic/gen-atomic-raw.sh   |   4 +-
 5 files changed, 345 insertions(+), 859 deletions(-)
include/linux/atomic.h +1 -1
index 127f5dc..296cfae 100644
@@ -78,8 +78,8 @@
 })
 
 #include <linux/atomic/atomic-arch-fallback.h>
-#include <linux/atomic/atomic-long.h>
 #include <linux/atomic/atomic-raw.h>
+#include <linux/atomic/atomic-long.h>
 #include <linux/atomic/atomic-instrumented.h>
 
 #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-long.h +341 -341

1,368 lines not on this page · raw message

include/linux/atomic/atomic-raw.h +1 -511

521 lines not on this page · raw message

scripts/atomic/gen-atomic-long.sh +2 -2
index eda89ce..75e91d6 100755
@@ -47,9 +47,9 @@ gen_proto_order_variant()
 
 cat <<EOF
 static __always_inline ${ret}
-arch_atomic_long_${name}(${params})
+raw_atomic_long_${name}(${params})
 {
-	${retstmt}arch_${atomic}_${name}(${argscast});
+	${retstmt}raw_${atomic}_${name}(${argscast});
 }
 
 EOF
scripts/atomic/gen-atomic-raw.sh +0 -4
index ba8d136..c7e3c52 100644
@@ -63,10 +63,6 @@ grep '^[a-z]' "$1" | while read name meta args; do
 	gen_proto "${meta}" "${name}" "atomic64" "s64" ${args}
 done
 
-grep '^[a-z]' "$1" | while read name meta args; do
-	gen_proto "${meta}" "${name}" "atomic_long" "long" ${args}
-done
-
 for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do
 	for order in "" "_acquire" "_release" "_relaxed"; do
 		gen_xchg "${xchg}" "${order}"
author reviewer
#45 lore raw

PATCH v2 20/27 locking/atomic: scripts: restructure fallback ifdeffery

In reply to Mark Rutland

Currently the various ordering variants of an atomic operation are defined in groups of full/acquire/release/relaxed ordering variants with some shared ifdeffery and several potential definitions of each ordering variant in different branches of the shared ifdeffery.

As an ordering variant can have several potential definitions down different branches of the shared ifdeffery, it can be painful for a human to find a relevant definition, and we don't have a good location to place anything common to all definitions of an ordering variant (e.g. kerneldoc).

Historically the grouping of full/acquire/release/relaxed ordering variants was necessary as we filled in the missing atomics in the same namespace as the architecture used. It would be easy to accidentally define one ordering fallback in terms of another ordering fallback with redundant barriers, and avoiding that would otherwise require a lot of baroque ifdeffery.

With recent changes we no longer need to fill in the missing atomics in the arch_atomic*_<op>() namespace, and only need to fill in the raw_atomic*_<op>() namespace. Due to this, there's no risk of a namespace collision, and we can define each raw_atomic*_<op> ordering variant with its own ifdeffery checking for the arch_atomic*_<op> ordering variants.

Restructure the fallbacks in this way, with each ordering variant having its own ifdeffery of the form:

| #if defined(arch_atomic_fetch_andnot_acquire)
| #define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot_acquire
| #elif defined(arch_atomic_fetch_andnot_relaxed)
| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
| 	int ret = arch_atomic_fetch_andnot_relaxed(i, v);
| 	__atomic_acquire_fence();
| 	return ret;
| }
| #elif defined(arch_atomic_fetch_andnot)
| #define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot
| #else
| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
| 	return raw_atomic_fetch_and_acquire(~i, v);
| }
| #endif

Note that where there's no relevant arch_atomic*_<op>() ordering variant, we'll define the operation in terms of a distinct raw_atomic*_<otherop>(), as this itself might have been filled in with a fallback.

As we now generate the raw_atomic*_<op>() implementations directly, we no longer need the trivial wrappers, so they are removed.

This makes the ifdeffery easier to follow, and will allow for further improvements in subsequent patches.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 include/linux/atomic.h                       |    1 -
 include/linux/atomic/atomic-arch-fallback.h  | 3178 +++++++++---------
 include/linux/atomic/atomic-raw.h            | 1135 -------
 scripts/atomic/fallbacks/acquire             |    2 +-
 scripts/atomic/fallbacks/add_negative        |    4 +-
 scripts/atomic/fallbacks/add_unless          |    4 +-
 scripts/atomic/fallbacks/andnot              |    4 +-
 scripts/atomic/fallbacks/cmpxchg             |    4 +-
 scripts/atomic/fallbacks/dec                 |    4 +-
 scripts/atomic/fallbacks/dec_and_test        |    4 +-
 scripts/atomic/fallbacks/dec_if_positive     |    6 +-
 scripts/atomic/fallbacks/dec_unless_positive |    6 +-
 scripts/atomic/fallbacks/fence               |    2 +-
 scripts/atomic/fallbacks/fetch_add_unless    |    6 +-
 scripts/atomic/fallbacks/inc                 |    4 +-
 scripts/atomic/fallbacks/inc_and_test        |    4 +-
 scripts/atomic/fallbacks/inc_not_zero        |    4 +-
 scripts/atomic/fallbacks/inc_unless_negative |    6 +-
 scripts/atomic/fallbacks/read_acquire        |    4 +-
 scripts/atomic/fallbacks/release             |    2 +-
 scripts/atomic/fallbacks/set_release         |    4 +-
 scripts/atomic/fallbacks/sub_and_test        |    4 +-
 scripts/atomic/fallbacks/try_cmpxchg         |    4 +-
 scripts/atomic/fallbacks/xchg                |    4 +-
 scripts/atomic/gen-atomic-fallback.sh        |  236 +-
 scripts/atomic/gen-atomic-raw.sh             |   80 -
 scripts/atomic/gen-atomics.sh                |    1 -
 27 files changed, 1866 insertions(+), 2851 deletions(-)
 delete mode 100644 include/linux/atomic/atomic-raw.h
 delete mode 100755 scripts/atomic/gen-atomic-raw.sh
include/linux/atomic.h +0 -1
index 296cfae0389fe..8dd57c3a99e9b 100644
@@ -78,7 +78,6 @@
 })
 
 #include <linux/atomic/atomic-arch-fallback.h>
-#include <linux/atomic/atomic-raw.h>
 #include <linux/atomic/atomic-long.h>
 #include <linux/atomic/atomic-instrumented.h>
 
include/linux/atomic/atomic-arch-fallback.h +1670 -1508

2,993 lines not on this page · raw message

include/linux/atomic/atomic-raw.h deleted +0 -1135

1,135 lines not on this page · raw message

scripts/atomic/fallbacks/acquire +1 -1
index ef764085c79aa..b0f732a5c46ef 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/add_negative +2 -2
index d0bd2dfbb244c..16876118019ec 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/add_unless +2 -2
index cf79b9da38dbb..88593e28b1637 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/andnot +2 -2
index 5a42f54a35950..5b83bb63f7284 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/cmpxchg +2 -2
index 87cd010f98d58..312ee67f1743e 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec +2 -2
index 8c144c818e9ed..a660ac65994bd 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_and_test +2 -2
index 3f6b6a8b47733..521dfcae03f24 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_if_positive +3 -3
index 86bdced3428d6..7acb205e6ce35 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_unless_positive +3 -3
index c531d5afecc47..bcb4f27945eaa 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/fence +1 -1
index 07757d8e338ef..067eea553f5e0 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/fetch_add_unless +3 -3
index 81d2834f03d23..c18b940153dfd 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc +2 -2
index 3c2c3739169e5..7d838f0b66391 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_and_test +2 -2
index c726a6d0634d3..de25aebee715d 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_not_zero +2 -2
index 97603591aac2a..e02206d017f62 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_unless_negative +3 -3
index 95d8ce48233ff..7b85cc5b00d2b 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/read_acquire +2 -2
index a0ea1d26e6b2e..26d15ad92d043 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/release +1 -1
index b46feb56d69ca..cbbff708129b8 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/set_release +2 -2
index 05cdb7f42477a..104693bc3c660 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/sub_and_test +2 -2
index da8a049c9b02b..8975a496d495c 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/try_cmpxchg +2 -2
index 890f850ede378..4c911a6cced94 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/xchg +2 -2
index 733b8980b2f3b..bdd788aa575ff 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/gen-atomic-fallback.sh +153 -83

236 lines not on this page · raw message

scripts/atomic/gen-atomic-raw.sh deleted +0 -80

80 lines not on this page · raw message

scripts/atomic/gen-atomics.sh +0 -1
index 631d351f9f1f3..5b98a83076932 100755

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.30.2

Part of the diff is not shown here. View the raw message.

merge bot Applied
#46 lore raw

[tip: locking/core] locking/atomic: scripts: restructure fallback ifdeffery

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     9257959a6e5b4fca6fc8e985790bff62c2046f20
Gitweb:        https://git.kernel.org/tip/9257959a6e5b4fca6fc8e985790bff62c2046f20
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:17 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:21 +02:00

locking/atomic: scripts: restructure fallback ifdeffery

Currently the various ordering variants of an atomic operation are defined in groups of full/acquire/release/relaxed ordering variants with some shared ifdeffery and several potential definitions of each ordering variant in different branches of the shared ifdeffery.

As an ordering variant can have several potential definitions down different branches of the shared ifdeffery, it can be painful for a human to find a relevant definition, and we don't have a good location to place anything common to all definitions of an ordering variant (e.g. kerneldoc).

Historically the grouping of full/acquire/release/relaxed ordering variants was necessary as we filled in the missing atomics in the same namespace as the architecture used. It would be easy to accidentally define one ordering fallback in terms of another ordering fallback with redundant barriers, and avoiding that would otherwise require a lot of baroque ifdeffery.

With recent changes we no longer need to fill in the missing atomics in the arch_atomic*_<op>() namespace, and only need to fill in the raw_atomic*_<op>() namespace. Due to this, there's no risk of a namespace collision, and we can define each raw_atomic*_<op> ordering variant with its own ifdeffery checking for the arch_atomic*_<op> ordering variants.

Restructure the fallbacks in this way, with each ordering variant having its own ifdeffery of the form:

| #if defined(arch_atomic_fetch_andnot_acquire)
| #define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot_acquire
| #elif defined(arch_atomic_fetch_andnot_relaxed)
| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
| 	int ret = arch_atomic_fetch_andnot_relaxed(i, v);
| 	__atomic_acquire_fence();
| 	return ret;
| }
| #elif defined(arch_atomic_fetch_andnot)
| #define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot
| #else
| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
| 	return raw_atomic_fetch_and_acquire(~i, v);
| }
| #endif

Note that where there's no relevant arch_atomic*_<op>() ordering variant, we'll define the operation in terms of a distinct raw_atomic*_<otherop>(), as this itself might have been filled in with a fallback.

As we now generate the raw_atomic*_<op>() implementations directly, we no longer need the trivial wrappers, so they are removed.

This makes the ifdeffery easier to follow, and will allow for further improvements in subsequent patches.

There should be no functional change as a result of this patch.

---
 include/linux/atomic.h                       |    1 +-
 include/linux/atomic/atomic-arch-fallback.h  | 3178 ++++++++---------
 include/linux/atomic/atomic-raw.h            | 1135 +------
 scripts/atomic/fallbacks/acquire             |    2 +-
 scripts/atomic/fallbacks/add_negative        |    4 +-
 scripts/atomic/fallbacks/add_unless          |    4 +-
 scripts/atomic/fallbacks/andnot              |    4 +-
 scripts/atomic/fallbacks/cmpxchg             |    4 +-
 scripts/atomic/fallbacks/dec                 |    4 +-
 scripts/atomic/fallbacks/dec_and_test        |    4 +-
 scripts/atomic/fallbacks/dec_if_positive     |    6 +-
 scripts/atomic/fallbacks/dec_unless_positive |    6 +-
 scripts/atomic/fallbacks/fence               |    2 +-
 scripts/atomic/fallbacks/fetch_add_unless    |    6 +-
 scripts/atomic/fallbacks/inc                 |    4 +-
 scripts/atomic/fallbacks/inc_and_test        |    4 +-
 scripts/atomic/fallbacks/inc_not_zero        |    4 +-
 scripts/atomic/fallbacks/inc_unless_negative |    6 +-
 scripts/atomic/fallbacks/read_acquire        |    4 +-
 scripts/atomic/fallbacks/release             |    2 +-
 scripts/atomic/fallbacks/set_release         |    4 +-
 scripts/atomic/fallbacks/sub_and_test        |    4 +-
 scripts/atomic/fallbacks/try_cmpxchg         |    4 +-
 scripts/atomic/fallbacks/xchg                |    4 +-
 scripts/atomic/gen-atomic-fallback.sh        |  236 +-
 scripts/atomic/gen-atomic-raw.sh             |   80 +-
 scripts/atomic/gen-atomics.sh                |    1 +-
 27 files changed, 1866 insertions(+), 2851 deletions(-)
 delete mode 100644 include/linux/atomic/atomic-raw.h
 delete mode 100644 scripts/atomic/gen-atomic-raw.sh
include/linux/atomic.h +0 -1
index 296cfae..8dd57c3 100644
@@ -78,7 +78,6 @@
 })
 
 #include <linux/atomic/atomic-arch-fallback.h>
-#include <linux/atomic/atomic-raw.h>
 #include <linux/atomic/atomic-long.h>
 #include <linux/atomic/atomic-instrumented.h>
 
include/linux/atomic/atomic-arch-fallback.h +1670 -1508

2,993 lines not on this page · raw message

include/linux/atomic/atomic-raw.h deleted +0 -1135

1,135 lines not on this page · raw message

scripts/atomic/fallbacks/acquire +1 -1
index ef76408..b0f732a 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/add_negative +2 -2
index d0bd2df..1687611 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/add_unless +2 -2
index cf79b9d..88593e2 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/andnot +2 -2
index 5a42f54..5b83bb6 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/cmpxchg +2 -2
index 87cd010..312ee67 100644

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec +2 -2
index 8c144c8..a660ac6 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_and_test +2 -2
index 3f6b6a8..521dfca 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_if_positive +3 -3
index 86bdced..7acb205 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_unless_positive +3 -3
index c531d5a..bcb4f27 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/fence +1 -1
index 07757d8..067eea5 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/fetch_add_unless +3 -3
index 81d2834..c18b940 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc +2 -2
index 3c2c373..7d838f0 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_and_test +2 -2
index c726a6d..de25aeb 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_not_zero +2 -2
index 9760359..e02206d 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_unless_negative +3 -3
index 95d8ce4..7b85cc5 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/read_acquire +2 -2
index a0ea1d2..26d15ad 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/release +1 -1
index b46feb5..cbbff70 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/set_release +2 -2
index 05cdb7f..104693b 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/sub_and_test +2 -2
index da8a049..8975a49 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/try_cmpxchg +2 -2
index 890f850..4c911a6 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/xchg +2 -2
index 733b898..bdd788a 100644

Lines of this file are not shown. View the raw message.

scripts/atomic/gen-atomic-fallback.sh +153 -83

236 lines not on this page · raw message

scripts/atomic/gen-atomic-raw.sh deleted +0 -80

80 lines not on this page · raw message

scripts/atomic/gen-atomics.sh +0 -1
index 631d351..5b98a83 100755

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

author reviewer
#47 lore raw

PATCH v2 21/27 locking/atomic: scripts: split pfx/name/sfx/order

In reply to Mark Rutland

Currently gen-atomic-long.sh's gen_proto_order_variant() function combines the pfx/name/sfx/order variables immediately, unlike other functions in gen-atomic-*.sh.

This is fine today, but subsequent patches will require the individual individual pfx/name/sfx/order variables within gen-atomic-long.sh's gen_proto_order_variant() function. In preparation for this, split the variables in the style of other gen-atomic-*.sh scripts.

This results in no change to the generated headers, so there should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 scripts/atomic/gen-atomic-long.sh | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
scripts/atomic/gen-atomic-long.sh +8 -3
index 75e91d6da30d3..13832171f7219 100755
@@ -36,10 +36,15 @@ gen_args_cast()
 gen_proto_order_variant()
 {
 	local meta="$1"; shift
-	local name="$1$2$3$4"; shift; shift; shift; shift
+	local pfx="$1"; shift
+	local name="$1"; shift
+	local sfx="$1"; shift
+	local order="$1"; shift
 	local atomic="$1"; shift
 	local int="$1"; shift
 
+	local atomicname="${pfx}${name}${sfx}${order}"
+
 	local ret="$(gen_ret_type "${meta}" "long")"
 	local params="$(gen_params "long" "atomic_long" "$@")"
 	local argscast="$(gen_args_cast "${int}" "${atomic}" "$@")"
@@ -47,9 +52,9 @@ gen_proto_order_variant()
 
 cat <<EOF
 static __always_inline ${ret}
-raw_atomic_long_${name}(${params})
+raw_atomic_long_${atomicname}(${params})
 {
-	${retstmt}raw_${atomic}_${name}(${argscast});
+	${retstmt}raw_${atomic}_${atomicname}(${argscast});
 }
 
 EOF
Signature · 1 line
2.30.2
merge bot Applied
#48 lore raw

[tip: locking/core] locking/atomic: scripts: split pfx/name/sfx/order

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     b916a8c765692444388891f5b9c5b6e941e16d42
Gitweb:        https://git.kernel.org/tip/b916a8c765692444388891f5b9c5b6e941e16d42
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:18 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:21 +02:00

locking/atomic: scripts: split pfx/name/sfx/order

Currently gen-atomic-long.sh's gen_proto_order_variant() function combines the pfx/name/sfx/order variables immediately, unlike other functions in gen-atomic-*.sh.

This is fine today, but subsequent patches will require the individual individual pfx/name/sfx/order variables within gen-atomic-long.sh's gen_proto_order_variant() function. In preparation for this, split the variables in the style of other gen-atomic-*.sh scripts.

This results in no change to the generated headers, so there should be no functional change as a result of this patch.

---
 scripts/atomic/gen-atomic-long.sh | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
scripts/atomic/gen-atomic-long.sh +8 -3
index 75e91d6..1383217 100755
@@ -36,10 +36,15 @@ gen_args_cast()
 gen_proto_order_variant()
 {
 	local meta="$1"; shift
-	local name="$1$2$3$4"; shift; shift; shift; shift
+	local pfx="$1"; shift
+	local name="$1"; shift
+	local sfx="$1"; shift
+	local order="$1"; shift
 	local atomic="$1"; shift
 	local int="$1"; shift
 
+	local atomicname="${pfx}${name}${sfx}${order}"
+
 	local ret="$(gen_ret_type "${meta}" "long")"
 	local params="$(gen_params "long" "atomic_long" "$@")"
 	local argscast="$(gen_args_cast "${int}" "${atomic}" "$@")"
@@ -47,9 +52,9 @@ gen_proto_order_variant()
 
 cat <<EOF
 static __always_inline ${ret}
-raw_atomic_long_${name}(${params})
+raw_atomic_long_${atomicname}(${params})
 {
-	${retstmt}raw_${atomic}_${name}(${argscast});
+	${retstmt}raw_${atomic}_${atomicname}(${argscast});
 }
 
 EOF
author reviewer
#49 lore raw

PATCH v2 22/27 locking/atomic: scripts: simplify raw_atomic_long*() definitions

In reply to Mark Rutland

Currently, atomic-long is split into two sections, one defining the raw_atomic_long_*() ops for CONFIG_64BIT, and one defining the raw atomic_long_*() ops for !CONFIG_64BIT.

With many lines elided, this looks like:

| #ifdef CONFIG_64BIT
| ...
| static __always_inline bool
| raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
| {
|         return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
| }
| ...
| #else /* CONFIG_64BIT */
| ...
| static __always_inline bool
| raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
| {
|         return raw_atomic_try_cmpxchg(v, (int *)old, new);
| }
| ...
| #endif

The two definitions are spread far apart in the file, and duplicate the prototype, making it hard to have a legible set of kerneldoc comments.

Make this simpler by defining the C prototype once, and writing the two definitions inline. For example, the above becomes:

| static __always_inline bool
| raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
| {
| #ifdef CONFIG_64BIT
|         return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
| #else
|         return raw_atomic_try_cmpxchg(v, (int *)old, new);
| #endif
| }

As we now always have a single copy of the C prototype wrapping all the potential definitions, we now have an obvious single location for kerneldoc comments. As a bonus, both the script and the generated file are somewhat shorter.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 include/linux/atomic/atomic-long.h | 857 ++++++++++++-----------------
 scripts/atomic/gen-atomic-long.sh  |  27 +-
 2 files changed, 350 insertions(+), 534 deletions(-)
include/linux/atomic/atomic-long.h +341 -516
index 92dc82ce1ce6d..63e0b4078ebd5 100644
@@ -21,1030 +21,855 @@ typedef atomic_t atomic_long_t;
 #define atomic_long_cond_read_relaxed	atomic_cond_read_relaxed
 #endif
 
-#ifdef CONFIG_64BIT
-
-static __always_inline long
-raw_atomic_long_read(const atomic_long_t *v)
-{
-	return raw_atomic64_read(v);
-}
-
-static __always_inline long
-raw_atomic_long_read_acquire(const atomic_long_t *v)
-{
-	return raw_atomic64_read_acquire(v);
-}
-
-static __always_inline void
-raw_atomic_long_set(atomic_long_t *v, long i)
-{
-	raw_atomic64_set(v, i);
-}
-
-static __always_inline void
-raw_atomic_long_set_release(atomic_long_t *v, long i)
-{
-	raw_atomic64_set_release(v, i);
-}
-
-static __always_inline void
-raw_atomic_long_add(long i, atomic_long_t *v)
-{
-	raw_atomic64_add(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_sub(long i, atomic_long_t *v)
-{
-	raw_atomic64_sub(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_inc(atomic_long_t *v)
-{
-	raw_atomic64_inc(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return_release(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return_relaxed(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc_release(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc_relaxed(v);
-}
-
-static __always_inline void
-raw_atomic_long_dec(atomic_long_t *v)
-{
-	raw_atomic64_dec(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return_release(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return_relaxed(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec_release(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec_relaxed(v);
-}
-
-static __always_inline void
-raw_atomic_long_and(long i, atomic_long_t *v)
-{
-	raw_atomic64_and(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_andnot(long i, atomic_long_t *v)
-{
-	raw_atomic64_andnot(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_or(long i, atomic_long_t *v)
-{
-	raw_atomic64_or(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_xor(long i, atomic_long_t *v)
-{
-	raw_atomic64_xor(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_xchg(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_xchg_acquire(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg_acquire(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_xchg_release(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg_release(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_xchg_relaxed(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg_relaxed(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg(v, old, new);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg_acquire(v, old, new);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg_release(v, old, new);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg_relaxed(v, old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg_acquire(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg_release(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg_relaxed(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_sub_and_test(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_and_test(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_dec_and_test(atomic_long_t *v)
-{
-	return raw_atomic64_dec_and_test(v);
-}
-
-static __always_inline bool
-raw_atomic_long_inc_and_test(atomic_long_t *v)
-{
-	return raw_atomic64_inc_and_test(v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative_acquire(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative_release(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u)
-{
-	return raw_atomic64_fetch_add_unless(v, a, u);
-}
-
-static __always_inline bool
-raw_atomic_long_add_unless(atomic_long_t *v, long a, long u)
-{
-	return raw_atomic64_add_unless(v, a, u);
-}
-
-static __always_inline bool
-raw_atomic_long_inc_not_zero(atomic_long_t *v)
-{
-	return raw_atomic64_inc_not_zero(v);
-}
-
-static __always_inline bool
-raw_atomic_long_inc_unless_negative(atomic_long_t *v)
-{
-	return raw_atomic64_inc_unless_negative(v);
-}
-
-static __always_inline bool
-raw_atomic_long_dec_unless_positive(atomic_long_t *v)
-{
-	return raw_atomic64_dec_unless_positive(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_if_positive(atomic_long_t *v)
-{
-	return raw_atomic64_dec_if_positive(v);
-}
-
-#else /* CONFIG_64BIT */
-
 static __always_inline long
 raw_atomic_long_read(const atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_read(v);
+#else
 	return raw_atomic_read(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_read_acquire(const atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_read_acquire(v);
+#else
 	return raw_atomic_read_acquire(v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_set(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_set(v, i);
+#else
 	raw_atomic_set(v, i);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_set_release(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_set_release(v, i);
+#else
 	raw_atomic_set_release(v, i);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_add(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_add(i, v);
+#else
 	raw_atomic_add(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return(i, v);
+#else
 	return raw_atomic_add_return(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return_acquire(i, v);
+#else
 	return raw_atomic_add_return_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return_release(i, v);
+#else
 	return raw_atomic_add_return_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return_relaxed(i, v);
+#else
 	return raw_atomic_add_return_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add(i, v);
+#else
 	return raw_atomic_fetch_add(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_acquire(i, v);
+#else
 	return raw_atomic_fetch_add_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_release(i, v);
+#else
 	return raw_atomic_fetch_add_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_relaxed(i, v);
+#else
 	return raw_atomic_fetch_add_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_sub(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_sub(i, v);
+#else
 	raw_atomic_sub(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return(i, v);
+#else
 	return raw_atomic_sub_return(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return_acquire(i, v);
+#else
 	return raw_atomic_sub_return_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return_release(i, v);
+#else
 	return raw_atomic_sub_return_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return_relaxed(i, v);
+#else
 	return raw_atomic_sub_return_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub(i, v);
+#else
 	return raw_atomic_fetch_sub(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub_acquire(i, v);
+#else
 	return raw_atomic_fetch_sub_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub_release(i, v);
+#else
 	return raw_atomic_fetch_sub_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub_relaxed(i, v);
+#else
 	return raw_atomic_fetch_sub_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_inc(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_inc(v);
+#else
 	raw_atomic_inc(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return(v);
+#else
 	return raw_atomic_inc_return(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return_acquire(v);
+#else
 	return raw_atomic_inc_return_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return_release(v);
+#else
 	return raw_atomic_inc_return_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return_relaxed(v);
+#else
 	return raw_atomic_inc_return_relaxed(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc(v);
+#else
 	return raw_atomic_fetch_inc(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc_acquire(v);
+#else
 	return raw_atomic_fetch_inc_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc_release(v);
+#else
 	return raw_atomic_fetch_inc_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc_relaxed(v);
+#else
 	return raw_atomic_fetch_inc_relaxed(v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_dec(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_dec(v);
+#else
 	raw_atomic_dec(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return(v);
+#else
 	return raw_atomic_dec_return(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return_acquire(v);
+#else
 	return raw_atomic_dec_return_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return_release(v);
+#else
 	return raw_atomic_dec_return_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return_relaxed(v);
+#else
 	return raw_atomic_dec_return_relaxed(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec(v);
+#else
 	return raw_atomic_fetch_dec(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec_acquire(v);
+#else
 	return raw_atomic_fetch_dec_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec_release(v);
+#else
 	return raw_atomic_fetch_dec_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec_relaxed(v);
+#else
 	return raw_atomic_fetch_dec_relaxed(v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_and(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_and(i, v);
+#else
 	raw_atomic_and(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and(i, v);
+#else
 	return raw_atomic_fetch_and(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and_acquire(i, v);
+#else
 	return raw_atomic_fetch_and_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and_release(i, v);
+#else
 	return raw_atomic_fetch_and_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and_relaxed(i, v);
+#else
 	return raw_atomic_fetch_and_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_andnot(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_andnot(i, v);
+#else
 	raw_atomic_andnot(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot(i, v);
+#else
 	return raw_atomic_fetch_andnot(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot_acquire(i, v);
+#else
 	return raw_atomic_fetch_andnot_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot_release(i, v);
+#else
 	return raw_atomic_fetch_andnot_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot_relaxed(i, v);
+#else
 	return raw_atomic_fetch_andnot_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_or(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_or(i, v);
+#else
 	raw_atomic_or(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or(i, v);
+#else
 	return raw_atomic_fetch_or(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or_acquire(i, v);
+#else
 	return raw_atomic_fetch_or_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or_release(i, v);
+#else
 	return raw_atomic_fetch_or_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or_relaxed(i, v);
+#else
 	return raw_atomic_fetch_or_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_xor(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_xor(i, v);
+#else
 	raw_atomic_xor(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor(i, v);
+#else
 	return raw_atomic_fetch_xor(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor_acquire(i, v);
+#else
 	return raw_atomic_fetch_xor_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor_release(i, v);
+#else
 	return raw_atomic_fetch_xor_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor_relaxed(i, v);
+#else
 	return raw_atomic_fetch_xor_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg(v, i);
+#else
 	return raw_atomic_xchg(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg_acquire(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg_acquire(v, i);
+#else
 	return raw_atomic_xchg_acquire(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg_release(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg_release(v, i);
+#else
 	return raw_atomic_xchg_release(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg_relaxed(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg_relaxed(v, i);
+#else
 	return raw_atomic_xchg_relaxed(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg(v, old, new);
+#else
 	return raw_atomic_cmpxchg(v, old, new);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg_acquire(v, old, new);
+#else
 	return raw_atomic_cmpxchg_acquire(v, old, new);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg_release(v, old, new);
+#else
 	return raw_atomic_cmpxchg_release(v, old, new);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg_relaxed(v, old, new);
+#else
 	return raw_atomic_cmpxchg_relaxed(v, old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg_acquire(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg_acquire(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg_release(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg_release(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg_relaxed(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg_relaxed(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_sub_and_test(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_and_test(i, v);
+#else
 	return raw_atomic_sub_and_test(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_dec_and_test(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_and_test(v);
+#else
 	return raw_atomic_dec_and_test(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_inc_and_test(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_and_test(v);
+#else
 	return raw_atomic_inc_and_test(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative(i, v);
+#else
 	return raw_atomic_add_negative(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative_acquire(i, v);
+#else
 	return raw_atomic_add_negative_acquire(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative_release(i, v);
+#else
 	return raw_atomic_add_negative_release(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative_relaxed(i, v);
+#else
 	return raw_atomic_add_negative_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_unless(v, a, u);
+#else
 	return raw_atomic_fetch_add_unless(v, a, u);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_unless(atomic_long_t *v, long a, long u)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_unless(v, a, u);
+#else
 	return raw_atomic_add_unless(v, a, u);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_inc_not_zero(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_not_zero(v);
+#else
 	return raw_atomic_inc_not_zero(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_inc_unless_negative(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_unless_negative(v);
+#else
 	return raw_atomic_inc_unless_negative(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_dec_unless_positive(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_unless_positive(v);
+#else
 	return raw_atomic_dec_unless_positive(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_if_positive(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_if_positive(v);
+#else
 	return raw_atomic_dec_if_positive(v);
+#endif
 }
 
-#endif /* CONFIG_64BIT */
 #endif /* _LINUX_ATOMIC_LONG_H */
-// 108784846d3bbbb201b8dabe621c5dc30b216206
+// ad09f849db0db5b30c82e497eeb9056a394c5f22
scripts/atomic/gen-atomic-long.sh +9 -18
index 13832171f7219..af27a71b37ef1 100755
@@ -32,7 +32,7 @@ gen_args_cast()
 	done
 }
 
-#gen_proto_order_variant(meta, pfx, name, sfx, order, atomic, int, arg...)
+#gen_proto_order_variant(meta, pfx, name, sfx, order, arg...)
 gen_proto_order_variant()
 {
 	local meta="$1"; shift
@@ -40,21 +40,24 @@ gen_proto_order_variant()
 	local name="$1"; shift
 	local sfx="$1"; shift
 	local order="$1"; shift
-	local atomic="$1"; shift
-	local int="$1"; shift
 
 	local atomicname="${pfx}${name}${sfx}${order}"
 
 	local ret="$(gen_ret_type "${meta}" "long")"
 	local params="$(gen_params "long" "atomic_long" "$@")"
-	local argscast="$(gen_args_cast "${int}" "${atomic}" "$@")"
+	local argscast_32="$(gen_args_cast "int" "atomic" "$@")"
+	local argscast_64="$(gen_args_cast "s64" "atomic64" "$@")"
 	local retstmt="$(gen_ret_stmt "${meta}")"
 
 cat <<EOF
 static __always_inline ${ret}
 raw_atomic_long_${atomicname}(${params})
 {
-	${retstmt}raw_${atomic}_${atomicname}(${argscast});
+#ifdef CONFIG_64BIT
+	${retstmt}raw_atomic64_${atomicname}(${argscast_64});
+#else
+	${retstmt}raw_atomic_${atomicname}(${argscast_32});
+#endif
 }
 
 EOF
@@ -84,24 +87,12 @@ typedef atomic_t atomic_long_t;
 #define atomic_long_cond_read_relaxed	atomic_cond_read_relaxed
 #endif
 
-#ifdef CONFIG_64BIT
-
-EOF
-
-grep '^[a-z]' "$1" | while read name meta args; do
-	gen_proto "${meta}" "${name}" "atomic64" "s64" ${args}
-done
-
-cat <<EOF
-#else /* CONFIG_64BIT */
-
 EOF
 
 grep '^[a-z]' "$1" | while read name meta args; do
-	gen_proto "${meta}" "${name}" "atomic" "int" ${args}
+	gen_proto "${meta}" "${name}" ${args}
 done
 
 cat <<EOF
-#endif /* CONFIG_64BIT */
 #endif /* _LINUX_ATOMIC_LONG_H */
 EOF
Signature · 1 line
2.30.2
merge bot Applied
#50 lore raw

[tip: locking/core] locking/atomic: scripts: simplify raw_atomic_long*() definitions

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     630399469ffcb937936644fbaa5daf61e700a329
Gitweb:        https://git.kernel.org/tip/630399469ffcb937936644fbaa5daf61e700a329
Author:        Mark Rutland <[email protected]>
AuthorDate:    Mon, 05 Jun 2023 08:01:19 +01:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 05 Jun 2023 09:57:22 +02:00

locking/atomic: scripts: simplify raw_atomic_long*() definitions

Currently, atomic-long is split into two sections, one defining the raw_atomic_long_*() ops for CONFIG_64BIT, and one defining the raw atomic_long_*() ops for !CONFIG_64BIT.

With many lines elided, this looks like:

| #ifdef CONFIG_64BIT
| ...
| static __always_inline bool
| raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
| {
|         return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
| }
| ...
| #else /* CONFIG_64BIT */
| ...
| static __always_inline bool
| raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
| {
|         return raw_atomic_try_cmpxchg(v, (int *)old, new);
| }
| ...
| #endif

The two definitions are spread far apart in the file, and duplicate the prototype, making it hard to have a legible set of kerneldoc comments.

Make this simpler by defining the C prototype once, and writing the two definitions inline. For example, the above becomes:

| static __always_inline bool
| raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
| {
| #ifdef CONFIG_64BIT
|         return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
| #else
|         return raw_atomic_try_cmpxchg(v, (int *)old, new);
| #endif
| }

As we now always have a single copy of the C prototype wrapping all the potential definitions, we now have an obvious single location for kerneldoc comments. As a bonus, both the script and the generated file are somewhat shorter.

There should be no functional change as a result of this patch.

---
 include/linux/atomic/atomic-long.h | 857 +++++++++++-----------------
 scripts/atomic/gen-atomic-long.sh  |  27 +-
 2 files changed, 350 insertions(+), 534 deletions(-)
include/linux/atomic/atomic-long.h +341 -516
index 92dc82c..63e0b40 100644
@@ -21,1030 +21,855 @@ typedef atomic_t atomic_long_t;
 #define atomic_long_cond_read_relaxed	atomic_cond_read_relaxed
 #endif
 
-#ifdef CONFIG_64BIT
-
-static __always_inline long
-raw_atomic_long_read(const atomic_long_t *v)
-{
-	return raw_atomic64_read(v);
-}
-
-static __always_inline long
-raw_atomic_long_read_acquire(const atomic_long_t *v)
-{
-	return raw_atomic64_read_acquire(v);
-}
-
-static __always_inline void
-raw_atomic_long_set(atomic_long_t *v, long i)
-{
-	raw_atomic64_set(v, i);
-}
-
-static __always_inline void
-raw_atomic_long_set_release(atomic_long_t *v, long i)
-{
-	raw_atomic64_set_release(v, i);
-}
-
-static __always_inline void
-raw_atomic_long_add(long i, atomic_long_t *v)
-{
-	raw_atomic64_add(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_add_return_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_return_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_add_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_sub(long i, atomic_long_t *v)
-{
-	raw_atomic64_sub(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_sub_return_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_return_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_sub_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_inc(atomic_long_t *v)
-{
-	raw_atomic64_inc(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return_release(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_inc_return_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_inc_return_relaxed(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc_release(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_inc_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_inc_relaxed(v);
-}
-
-static __always_inline void
-raw_atomic_long_dec(atomic_long_t *v)
-{
-	raw_atomic64_dec(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return_release(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_return_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_dec_return_relaxed(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec_acquire(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec_acquire(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec_release(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec_release(v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_dec_relaxed(atomic_long_t *v)
-{
-	return raw_atomic64_fetch_dec_relaxed(v);
-}
-
-static __always_inline void
-raw_atomic_long_and(long i, atomic_long_t *v)
-{
-	raw_atomic64_and(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_and_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_and_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_andnot(long i, atomic_long_t *v)
-{
-	raw_atomic64_andnot(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_andnot_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_or(long i, atomic_long_t *v)
-{
-	raw_atomic64_or(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_or_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_or_relaxed(i, v);
-}
-
-static __always_inline void
-raw_atomic_long_xor(long i, atomic_long_t *v)
-{
-	raw_atomic64_xor(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor_acquire(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor_release(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_fetch_xor_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_xchg(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_xchg_acquire(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg_acquire(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_xchg_release(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg_release(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_xchg_relaxed(atomic_long_t *v, long i)
-{
-	return raw_atomic64_xchg_relaxed(v, i);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg(v, old, new);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg_acquire(v, old, new);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg_release(v, old, new);
-}
-
-static __always_inline long
-raw_atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new)
-{
-	return raw_atomic64_cmpxchg_relaxed(v, old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg_acquire(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg_release(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)
-{
-	return raw_atomic64_try_cmpxchg_relaxed(v, (s64 *)old, new);
-}
-
-static __always_inline bool
-raw_atomic_long_sub_and_test(long i, atomic_long_t *v)
-{
-	return raw_atomic64_sub_and_test(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_dec_and_test(atomic_long_t *v)
-{
-	return raw_atomic64_dec_and_test(v);
-}
-
-static __always_inline bool
-raw_atomic_long_inc_and_test(atomic_long_t *v)
-{
-	return raw_atomic64_inc_and_test(v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative_acquire(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative_acquire(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative_release(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative_release(i, v);
-}
-
-static __always_inline bool
-raw_atomic_long_add_negative_relaxed(long i, atomic_long_t *v)
-{
-	return raw_atomic64_add_negative_relaxed(i, v);
-}
-
-static __always_inline long
-raw_atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u)
-{
-	return raw_atomic64_fetch_add_unless(v, a, u);
-}
-
-static __always_inline bool
-raw_atomic_long_add_unless(atomic_long_t *v, long a, long u)
-{
-	return raw_atomic64_add_unless(v, a, u);
-}
-
-static __always_inline bool
-raw_atomic_long_inc_not_zero(atomic_long_t *v)
-{
-	return raw_atomic64_inc_not_zero(v);
-}
-
-static __always_inline bool
-raw_atomic_long_inc_unless_negative(atomic_long_t *v)
-{
-	return raw_atomic64_inc_unless_negative(v);
-}
-
-static __always_inline bool
-raw_atomic_long_dec_unless_positive(atomic_long_t *v)
-{
-	return raw_atomic64_dec_unless_positive(v);
-}
-
-static __always_inline long
-raw_atomic_long_dec_if_positive(atomic_long_t *v)
-{
-	return raw_atomic64_dec_if_positive(v);
-}
-
-#else /* CONFIG_64BIT */
-
 static __always_inline long
 raw_atomic_long_read(const atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_read(v);
+#else
 	return raw_atomic_read(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_read_acquire(const atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_read_acquire(v);
+#else
 	return raw_atomic_read_acquire(v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_set(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_set(v, i);
+#else
 	raw_atomic_set(v, i);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_set_release(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_set_release(v, i);
+#else
 	raw_atomic_set_release(v, i);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_add(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_add(i, v);
+#else
 	raw_atomic_add(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return(i, v);
+#else
 	return raw_atomic_add_return(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return_acquire(i, v);
+#else
 	return raw_atomic_add_return_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return_release(i, v);
+#else
 	return raw_atomic_add_return_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_add_return_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_return_relaxed(i, v);
+#else
 	return raw_atomic_add_return_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add(i, v);
+#else
 	return raw_atomic_fetch_add(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_acquire(i, v);
+#else
 	return raw_atomic_fetch_add_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_release(i, v);
+#else
 	return raw_atomic_fetch_add_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_relaxed(i, v);
+#else
 	return raw_atomic_fetch_add_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_sub(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_sub(i, v);
+#else
 	raw_atomic_sub(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return(i, v);
+#else
 	return raw_atomic_sub_return(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return_acquire(i, v);
+#else
 	return raw_atomic_sub_return_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return_release(i, v);
+#else
 	return raw_atomic_sub_return_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_sub_return_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_return_relaxed(i, v);
+#else
 	return raw_atomic_sub_return_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub(i, v);
+#else
 	return raw_atomic_fetch_sub(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub_acquire(i, v);
+#else
 	return raw_atomic_fetch_sub_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub_release(i, v);
+#else
 	return raw_atomic_fetch_sub_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_sub_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_sub_relaxed(i, v);
+#else
 	return raw_atomic_fetch_sub_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_inc(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_inc(v);
+#else
 	raw_atomic_inc(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return(v);
+#else
 	return raw_atomic_inc_return(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return_acquire(v);
+#else
 	return raw_atomic_inc_return_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return_release(v);
+#else
 	return raw_atomic_inc_return_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_inc_return_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_return_relaxed(v);
+#else
 	return raw_atomic_inc_return_relaxed(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc(v);
+#else
 	return raw_atomic_fetch_inc(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc_acquire(v);
+#else
 	return raw_atomic_fetch_inc_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc_release(v);
+#else
 	return raw_atomic_fetch_inc_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_inc_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_inc_relaxed(v);
+#else
 	return raw_atomic_fetch_inc_relaxed(v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_dec(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_dec(v);
+#else
 	raw_atomic_dec(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return(v);
+#else
 	return raw_atomic_dec_return(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return_acquire(v);
+#else
 	return raw_atomic_dec_return_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return_release(v);
+#else
 	return raw_atomic_dec_return_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_return_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_return_relaxed(v);
+#else
 	return raw_atomic_dec_return_relaxed(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec(v);
+#else
 	return raw_atomic_fetch_dec(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec_acquire(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec_acquire(v);
+#else
 	return raw_atomic_fetch_dec_acquire(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec_release(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec_release(v);
+#else
 	return raw_atomic_fetch_dec_release(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_dec_relaxed(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_dec_relaxed(v);
+#else
 	return raw_atomic_fetch_dec_relaxed(v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_and(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_and(i, v);
+#else
 	raw_atomic_and(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and(i, v);
+#else
 	return raw_atomic_fetch_and(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and_acquire(i, v);
+#else
 	return raw_atomic_fetch_and_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and_release(i, v);
+#else
 	return raw_atomic_fetch_and_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_and_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_and_relaxed(i, v);
+#else
 	return raw_atomic_fetch_and_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_andnot(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_andnot(i, v);
+#else
 	raw_atomic_andnot(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot(i, v);
+#else
 	return raw_atomic_fetch_andnot(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot_acquire(i, v);
+#else
 	return raw_atomic_fetch_andnot_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot_release(i, v);
+#else
 	return raw_atomic_fetch_andnot_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_andnot_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_andnot_relaxed(i, v);
+#else
 	return raw_atomic_fetch_andnot_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_or(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_or(i, v);
+#else
 	raw_atomic_or(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or(i, v);
+#else
 	return raw_atomic_fetch_or(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or_acquire(i, v);
+#else
 	return raw_atomic_fetch_or_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or_release(i, v);
+#else
 	return raw_atomic_fetch_or_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_or_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_or_relaxed(i, v);
+#else
 	return raw_atomic_fetch_or_relaxed(i, v);
+#endif
 }
 
 static __always_inline void
 raw_atomic_long_xor(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	raw_atomic64_xor(i, v);
+#else
 	raw_atomic_xor(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor(i, v);
+#else
 	return raw_atomic_fetch_xor(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor_acquire(i, v);
+#else
 	return raw_atomic_fetch_xor_acquire(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor_release(i, v);
+#else
 	return raw_atomic_fetch_xor_release(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_xor_relaxed(i, v);
+#else
 	return raw_atomic_fetch_xor_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg(v, i);
+#else
 	return raw_atomic_xchg(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg_acquire(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg_acquire(v, i);
+#else
 	return raw_atomic_xchg_acquire(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg_release(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg_release(v, i);
+#else
 	return raw_atomic_xchg_release(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_xchg_relaxed(atomic_long_t *v, long i)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_xchg_relaxed(v, i);
+#else
 	return raw_atomic_xchg_relaxed(v, i);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg(v, old, new);
+#else
 	return raw_atomic_cmpxchg(v, old, new);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg_acquire(v, old, new);
+#else
 	return raw_atomic_cmpxchg_acquire(v, old, new);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg_release(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg_release(v, old, new);
+#else
 	return raw_atomic_cmpxchg_release(v, old, new);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_cmpxchg_relaxed(v, old, new);
+#else
 	return raw_atomic_cmpxchg_relaxed(v, old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg_acquire(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg_acquire(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg_release(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg_release(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg_release(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_try_cmpxchg_relaxed(v, (s64 *)old, new);
+#else
 	return raw_atomic_try_cmpxchg_relaxed(v, (int *)old, new);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_sub_and_test(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_sub_and_test(i, v);
+#else
 	return raw_atomic_sub_and_test(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_dec_and_test(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_and_test(v);
+#else
 	return raw_atomic_dec_and_test(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_inc_and_test(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_and_test(v);
+#else
 	return raw_atomic_inc_and_test(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative(i, v);
+#else
 	return raw_atomic_add_negative(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative_acquire(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative_acquire(i, v);
+#else
 	return raw_atomic_add_negative_acquire(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative_release(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative_release(i, v);
+#else
 	return raw_atomic_add_negative_release(i, v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_negative_relaxed(long i, atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_negative_relaxed(i, v);
+#else
 	return raw_atomic_add_negative_relaxed(i, v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_fetch_add_unless(v, a, u);
+#else
 	return raw_atomic_fetch_add_unless(v, a, u);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_add_unless(atomic_long_t *v, long a, long u)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_add_unless(v, a, u);
+#else
 	return raw_atomic_add_unless(v, a, u);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_inc_not_zero(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_not_zero(v);
+#else
 	return raw_atomic_inc_not_zero(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_inc_unless_negative(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_inc_unless_negative(v);
+#else
 	return raw_atomic_inc_unless_negative(v);
+#endif
 }
 
 static __always_inline bool
 raw_atomic_long_dec_unless_positive(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_unless_positive(v);
+#else
 	return raw_atomic_dec_unless_positive(v);
+#endif
 }
 
 static __always_inline long
 raw_atomic_long_dec_if_positive(atomic_long_t *v)
 {
+#ifdef CONFIG_64BIT
+	return raw_atomic64_dec_if_positive(v);
+#else
 	return raw_atomic_dec_if_positive(v);
+#endif
 }
 
-#endif /* CONFIG_64BIT */
 #endif /* _LINUX_ATOMIC_LONG_H */
-// 108784846d3bbbb201b8dabe621c5dc30b216206
+// ad09f849db0db5b30c82e497eeb9056a394c5f22
scripts/atomic/gen-atomic-long.sh +9 -18
index 1383217..af27a71 100755
@@ -32,7 +32,7 @@ gen_args_cast()
 	done
 }
 
-#gen_proto_order_variant(meta, pfx, name, sfx, order, atomic, int, arg...)
+#gen_proto_order_variant(meta, pfx, name, sfx, order, arg...)
 gen_proto_order_variant()
 {
 	local meta="$1"; shift
@@ -40,21 +40,24 @@ gen_proto_order_variant()
 	local name="$1"; shift
 	local sfx="$1"; shift
 	local order="$1"; shift
-	local atomic="$1"; shift
-	local int="$1"; shift
 
 	local atomicname="${pfx}${name}${sfx}${order}"
 
 	local ret="$(gen_ret_type "${meta}" "long")"
 	local params="$(gen_params "long" "atomic_long" "$@")"
-	local argscast="$(gen_args_cast "${int}" "${atomic}" "$@")"
+	local argscast_32="$(gen_args_cast "int" "atomic" "$@")"
+	local argscast_64="$(gen_args_cast "s64" "atomic64" "$@")"
 	local retstmt="$(gen_ret_stmt "${meta}")"
 
 cat <<EOF
 static __always_inline ${ret}
 raw_atomic_long_${atomicname}(${params})
 {
-	${retstmt}raw_${atomic}_${atomicname}(${argscast});
+#ifdef CONFIG_64BIT
+	${retstmt}raw_atomic64_${atomicname}(${argscast_64});
+#else
+	${retstmt}raw_atomic_${atomicname}(${argscast_32});
+#endif
 }
 
 EOF
@@ -84,24 +87,12 @@ typedef atomic_t atomic_long_t;
 #define atomic_long_cond_read_relaxed	atomic_cond_read_relaxed
 #endif
 
-#ifdef CONFIG_64BIT
-
-EOF
-
-grep '^[a-z]' "$1" | while read name meta args; do
-	gen_proto "${meta}" "${name}" "atomic64" "s64" ${args}
-done
-
-cat <<EOF
-#else /* CONFIG_64BIT */
-
 EOF
 
 grep '^[a-z]' "$1" | while read name meta args; do
-	gen_proto "${meta}" "${name}" "atomic" "int" ${args}
+	gen_proto "${meta}" "${name}" ${args}
 done
 
 cat <<EOF
-#endif /* CONFIG_64BIT */
 #endif /* _LINUX_ATOMIC_LONG_H */
 EOF
author reviewer
#51 lore raw

PATCH v2 23/27 locking/atomic: scripts: simplify raw_atomic*() definitions

In reply to Mark Rutland

Currently each ordering variant has several potential definitions, with a mixture of preprocessor and C definitions, including several copies of its C prototype, e.g.

| #if defined(arch_atomic_fetch_andnot_acquire)
| #define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot_acquire
| #elif defined(arch_atomic_fetch_andnot_relaxed)
| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
|       int ret = arch_atomic_fetch_andnot_relaxed(i, v);
|       __atomic_acquire_fence();
|       return ret;
| }
| #elif defined(arch_atomic_fetch_andnot)
| #define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot
| #else
| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
|       return raw_atomic_fetch_and_acquire(~i, v);
| }
| #endif

Make this a bit simpler by defining the C prototype once, and writing the various potential definitions as plain C code guarded by ifdeffery. For example, the above becomes:

| static __always_inline int
| raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
| {
| #if defined(arch_atomic_fetch_andnot_acquire)
|         return arch_atomic_fetch_andnot_acquire(i, v);
| #elif defined(arch_atomic_fetch_andnot_relaxed)
|         int ret = arch_atomic_fetch_andnot_relaxed(i, v);
|         __atomic_acquire_fence();
|         return ret;
| #elif defined(arch_atomic_fetch_andnot)
|         return arch_atomic_fetch_andnot(i, v);
| #else
|         return raw_atomic_fetch_and_acquire(~i, v);
| #endif
| }

Which is far easier to read. As we now always have a single copy of the C prototype wrapping all the potential definitions, we now have an obvious single location for kerneldoc comments.

At the same time, the fallbacks for raw_atomic*_xhcg() are made to use 'new' rather than 'i' as the name of the new value. This is what the existing fallback template used, and is more consistent with the raw_atomic{_try,}cmpxchg() fallbacks.

There should be no functional change as a result of this patch.

Cc Boqun Feng <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Will Deacon <[email protected]>
---
 include/linux/atomic/atomic-arch-fallback.h  | 1790 +++++++++---------
 include/linux/atomic/atomic-instrumented.h   |   50 +-
 include/linux/atomic/atomic-long.h           |   26 +-
 scripts/atomic/atomics.tbl                   |    2 +-
 scripts/atomic/fallbacks/acquire             |    4 -
 scripts/atomic/fallbacks/add_negative        |    4 -
 scripts/atomic/fallbacks/add_unless          |    4 -
 scripts/atomic/fallbacks/andnot              |    4 -
 scripts/atomic/fallbacks/cmpxchg             |    4 -
 scripts/atomic/fallbacks/dec                 |    4 -
 scripts/atomic/fallbacks/dec_and_test        |    4 -
 scripts/atomic/fallbacks/dec_if_positive     |    4 -
 scripts/atomic/fallbacks/dec_unless_positive |    4 -
 scripts/atomic/fallbacks/fence               |    4 -
 scripts/atomic/fallbacks/fetch_add_unless    |    4 -
 scripts/atomic/fallbacks/inc                 |    4 -
 scripts/atomic/fallbacks/inc_and_test        |    4 -
 scripts/atomic/fallbacks/inc_not_zero        |    4 -
 scripts/atomic/fallbacks/inc_unless_negative |    4 -
 scripts/atomic/fallbacks/read_acquire        |    4 -
 scripts/atomic/fallbacks/release             |    4 -
 scripts/atomic/fallbacks/set_release         |    4 -
 scripts/atomic/fallbacks/sub_and_test        |    4 -
 scripts/atomic/fallbacks/try_cmpxchg         |    4 -
 scripts/atomic/fallbacks/xchg                |    4 -
 scripts/atomic/gen-atomic-fallback.sh        |   26 +-
 26 files changed, 901 insertions(+), 1077 deletions(-)
include/linux/atomic/atomic-arch-fallback.h +843 -947
index 99bc1a871dc12..470c2890ab8d6 100644
@@ -428,16 +428,20 @@ extern void raw_cmpxchg128_relaxed_not_implemented(void);
 
 #define raw_sync_cmpxchg arch_sync_cmpxchg
 
-#define raw_atomic_read arch_atomic_read
+static __always_inline int
+raw_atomic_read(const atomic_t *v)
+{
+	return arch_atomic_read(v);
+}
 
-#if defined(arch_atomic_read_acquire)
-#define raw_atomic_read_acquire arch_atomic_read_acquire
-#elif defined(arch_atomic_read)
-#define raw_atomic_read_acquire arch_atomic_read
-#else
 static __always_inline int
 raw_atomic_read_acquire(const atomic_t *v)
 {
+#if defined(arch_atomic_read_acquire)
+	return arch_atomic_read_acquire(v);
+#elif defined(arch_atomic_read)
+	return arch_atomic_read(v);
+#else
 	int ret;
 
 	if (__native_word(atomic_t)) {
@@ -448,1144 +452,1088 @@ raw_atomic_read_acquire(const atomic_t *v)
 	}
 
 	return ret;
-}
 #endif
+}
 
-#define raw_atomic_set arch_atomic_set
+static __always_inline void
+raw_atomic_set(atomic_t *v, int i)
+{
+	arch_atomic_set(v, i);
+}
 
-#if defined(arch_atomic_set_release)
-#define raw_atomic_set_release arch_atomic_set_release
-#elif defined(arch_atomic_set)
-#define raw_atomic_set_release arch_atomic_set
-#else
 static __always_inline void
 raw_atomic_set_release(atomic_t *v, int i)
 {
+#if defined(arch_atomic_set_release)
+	arch_atomic_set_release(v, i);
+#elif defined(arch_atomic_set)
+	arch_atomic_set(v, i);
+#else
 	if (__native_word(atomic_t)) {
 		smp_store_release(&(v)->counter, i);
 	} else {
 		__atomic_release_fence();
 		raw_atomic_set(v, i);
 	}
-}
 #endif
+}
 
-#define raw_atomic_add arch_atomic_add
+static __always_inline void
+raw_atomic_add(int i, atomic_t *v)
+{
+	arch_atomic_add(i, v);
+}
 
-#if defined(arch_atomic_add_return)
-#define raw_atomic_add_return arch_atomic_add_return
-#elif defined(arch_atomic_add_return_relaxed)
 static __always_inline int
 raw_atomic_add_return(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_return)
+	return arch_atomic_add_return(i, v);
+#elif defined(arch_atomic_add_return_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_add_return_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_add_return"
 #endif
+}
 
-#if defined(arch_atomic_add_return_acquire)
-#define raw_atomic_add_return_acquire arch_atomic_add_return_acquire
-#elif defined(arch_atomic_add_return_relaxed)
 static __always_inline int
 raw_atomic_add_return_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_return_acquire)
+	return arch_atomic_add_return_acquire(i, v);
+#elif defined(arch_atomic_add_return_relaxed)
 	int ret = arch_atomic_add_return_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_add_return)
-#define raw_atomic_add_return_acquire arch_atomic_add_return
+	return arch_atomic_add_return(i, v);
 #else
 #error "Unable to define raw_atomic_add_return_acquire"
 #endif
+}
 
-#if defined(arch_atomic_add_return_release)
-#define raw_atomic_add_return_release arch_atomic_add_return_release
-#elif defined(arch_atomic_add_return_relaxed)
 static __always_inline int
 raw_atomic_add_return_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_return_release)
+	return arch_atomic_add_return_release(i, v);
+#elif defined(arch_atomic_add_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_add_return_relaxed(i, v);
-}
 #elif defined(arch_atomic_add_return)
-#define raw_atomic_add_return_release arch_atomic_add_return
+	return arch_atomic_add_return(i, v);
 #else
 #error "Unable to define raw_atomic_add_return_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_add_return_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_add_return_relaxed)
-#define raw_atomic_add_return_relaxed arch_atomic_add_return_relaxed
+	return arch_atomic_add_return_relaxed(i, v);
 #elif defined(arch_atomic_add_return)
-#define raw_atomic_add_return_relaxed arch_atomic_add_return
+	return arch_atomic_add_return(i, v);
 #else
 #error "Unable to define raw_atomic_add_return_relaxed"
 #endif
+}
 
-#if defined(arch_atomic_fetch_add)
-#define raw_atomic_fetch_add arch_atomic_fetch_add
-#elif defined(arch_atomic_fetch_add_relaxed)
 static __always_inline int
 raw_atomic_fetch_add(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_add)
+	return arch_atomic_fetch_add(i, v);
+#elif defined(arch_atomic_fetch_add_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_add_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_fetch_add"
 #endif
+}
 
-#if defined(arch_atomic_fetch_add_acquire)
-#define raw_atomic_fetch_add_acquire arch_atomic_fetch_add_acquire
-#elif defined(arch_atomic_fetch_add_relaxed)
 static __always_inline int
 raw_atomic_fetch_add_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_add_acquire)
+	return arch_atomic_fetch_add_acquire(i, v);
+#elif defined(arch_atomic_fetch_add_relaxed)
 	int ret = arch_atomic_fetch_add_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_add)
-#define raw_atomic_fetch_add_acquire arch_atomic_fetch_add
+	return arch_atomic_fetch_add(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_add_acquire"
 #endif
+}
 
-#if defined(arch_atomic_fetch_add_release)
-#define raw_atomic_fetch_add_release arch_atomic_fetch_add_release
-#elif defined(arch_atomic_fetch_add_relaxed)
 static __always_inline int
 raw_atomic_fetch_add_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_add_release)
+	return arch_atomic_fetch_add_release(i, v);
+#elif defined(arch_atomic_fetch_add_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_add_relaxed(i, v);
-}
 #elif defined(arch_atomic_fetch_add)
-#define raw_atomic_fetch_add_release arch_atomic_fetch_add
+	return arch_atomic_fetch_add(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_add_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_fetch_add_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_fetch_add_relaxed)
-#define raw_atomic_fetch_add_relaxed arch_atomic_fetch_add_relaxed
+	return arch_atomic_fetch_add_relaxed(i, v);
 #elif defined(arch_atomic_fetch_add)
-#define raw_atomic_fetch_add_relaxed arch_atomic_fetch_add
+	return arch_atomic_fetch_add(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_add_relaxed"
 #endif
+}
 
-#define raw_atomic_sub arch_atomic_sub
+static __always_inline void
+raw_atomic_sub(int i, atomic_t *v)
+{
+	arch_atomic_sub(i, v);
+}
 
-#if defined(arch_atomic_sub_return)
-#define raw_atomic_sub_return arch_atomic_sub_return
-#elif defined(arch_atomic_sub_return_relaxed)
 static __always_inline int
 raw_atomic_sub_return(int i, atomic_t *v)
 {
+#if defined(arch_atomic_sub_return)
+	return arch_atomic_sub_return(i, v);
+#elif defined(arch_atomic_sub_return_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_sub_return_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_sub_return"
 #endif
+}
 
-#if defined(arch_atomic_sub_return_acquire)
-#define raw_atomic_sub_return_acquire arch_atomic_sub_return_acquire
-#elif defined(arch_atomic_sub_return_relaxed)
 static __always_inline int
 raw_atomic_sub_return_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_sub_return_acquire)
+	return arch_atomic_sub_return_acquire(i, v);
+#elif defined(arch_atomic_sub_return_relaxed)
 	int ret = arch_atomic_sub_return_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_sub_return)
-#define raw_atomic_sub_return_acquire arch_atomic_sub_return
+	return arch_atomic_sub_return(i, v);
 #else
 #error "Unable to define raw_atomic_sub_return_acquire"
 #endif
+}
 
-#if defined(arch_atomic_sub_return_release)
-#define raw_atomic_sub_return_release arch_atomic_sub_return_release
-#elif defined(arch_atomic_sub_return_relaxed)
 static __always_inline int
 raw_atomic_sub_return_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_sub_return_release)
+	return arch_atomic_sub_return_release(i, v);
+#elif defined(arch_atomic_sub_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_sub_return_relaxed(i, v);
-}
 #elif defined(arch_atomic_sub_return)
-#define raw_atomic_sub_return_release arch_atomic_sub_return
+	return arch_atomic_sub_return(i, v);
 #else
 #error "Unable to define raw_atomic_sub_return_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_sub_return_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_sub_return_relaxed)
-#define raw_atomic_sub_return_relaxed arch_atomic_sub_return_relaxed
+	return arch_atomic_sub_return_relaxed(i, v);
 #elif defined(arch_atomic_sub_return)
-#define raw_atomic_sub_return_relaxed arch_atomic_sub_return
+	return arch_atomic_sub_return(i, v);
 #else
 #error "Unable to define raw_atomic_sub_return_relaxed"
 #endif
+}
 
-#if defined(arch_atomic_fetch_sub)
-#define raw_atomic_fetch_sub arch_atomic_fetch_sub
-#elif defined(arch_atomic_fetch_sub_relaxed)
 static __always_inline int
 raw_atomic_fetch_sub(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_sub)
+	return arch_atomic_fetch_sub(i, v);
+#elif defined(arch_atomic_fetch_sub_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_sub_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_fetch_sub"
 #endif
+}
 
-#if defined(arch_atomic_fetch_sub_acquire)
-#define raw_atomic_fetch_sub_acquire arch_atomic_fetch_sub_acquire
-#elif defined(arch_atomic_fetch_sub_relaxed)
 static __always_inline int
 raw_atomic_fetch_sub_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_sub_acquire)
+	return arch_atomic_fetch_sub_acquire(i, v);
+#elif defined(arch_atomic_fetch_sub_relaxed)
 	int ret = arch_atomic_fetch_sub_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_sub)
-#define raw_atomic_fetch_sub_acquire arch_atomic_fetch_sub
+	return arch_atomic_fetch_sub(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_sub_acquire"
 #endif
+}
 
-#if defined(arch_atomic_fetch_sub_release)
-#define raw_atomic_fetch_sub_release arch_atomic_fetch_sub_release
-#elif defined(arch_atomic_fetch_sub_relaxed)
 static __always_inline int
 raw_atomic_fetch_sub_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_sub_release)
+	return arch_atomic_fetch_sub_release(i, v);
+#elif defined(arch_atomic_fetch_sub_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_sub_relaxed(i, v);
-}
 #elif defined(arch_atomic_fetch_sub)
-#define raw_atomic_fetch_sub_release arch_atomic_fetch_sub
+	return arch_atomic_fetch_sub(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_sub_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_fetch_sub_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_fetch_sub_relaxed)
-#define raw_atomic_fetch_sub_relaxed arch_atomic_fetch_sub_relaxed
+	return arch_atomic_fetch_sub_relaxed(i, v);
 #elif defined(arch_atomic_fetch_sub)
-#define raw_atomic_fetch_sub_relaxed arch_atomic_fetch_sub
+	return arch_atomic_fetch_sub(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_sub_relaxed"
 #endif
+}
 
-#if defined(arch_atomic_inc)
-#define raw_atomic_inc arch_atomic_inc
-#else
 static __always_inline void
 raw_atomic_inc(atomic_t *v)
 {
+#if defined(arch_atomic_inc)
+	arch_atomic_inc(v);
+#else
 	raw_atomic_add(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_return)
-#define raw_atomic_inc_return arch_atomic_inc_return
-#elif defined(arch_atomic_inc_return_relaxed)
 static __always_inline int
 raw_atomic_inc_return(atomic_t *v)
 {
+#if defined(arch_atomic_inc_return)
+	return arch_atomic_inc_return(v);
+#elif defined(arch_atomic_inc_return_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_inc_return_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_inc_return(atomic_t *v)
-{
 	return raw_atomic_add_return(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_return_acquire)
-#define raw_atomic_inc_return_acquire arch_atomic_inc_return_acquire
-#elif defined(arch_atomic_inc_return_relaxed)
 static __always_inline int
 raw_atomic_inc_return_acquire(atomic_t *v)
 {
+#if defined(arch_atomic_inc_return_acquire)
+	return arch_atomic_inc_return_acquire(v);
+#elif defined(arch_atomic_inc_return_relaxed)
 	int ret = arch_atomic_inc_return_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_inc_return)
-#define raw_atomic_inc_return_acquire arch_atomic_inc_return
+	return arch_atomic_inc_return(v);
 #else
-static __always_inline int
-raw_atomic_inc_return_acquire(atomic_t *v)
-{
 	return raw_atomic_add_return_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_return_release)
-#define raw_atomic_inc_return_release arch_atomic_inc_return_release
-#elif defined(arch_atomic_inc_return_relaxed)
 static __always_inline int
 raw_atomic_inc_return_release(atomic_t *v)
 {
+#if defined(arch_atomic_inc_return_release)
+	return arch_atomic_inc_return_release(v);
+#elif defined(arch_atomic_inc_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_inc_return_relaxed(v);
-}
 #elif defined(arch_atomic_inc_return)
-#define raw_atomic_inc_return_release arch_atomic_inc_return
+	return arch_atomic_inc_return(v);
 #else
-static __always_inline int
-raw_atomic_inc_return_release(atomic_t *v)
-{
 	return raw_atomic_add_return_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_return_relaxed)
-#define raw_atomic_inc_return_relaxed arch_atomic_inc_return_relaxed
-#elif defined(arch_atomic_inc_return)
-#define raw_atomic_inc_return_relaxed arch_atomic_inc_return
-#else
 static __always_inline int
 raw_atomic_inc_return_relaxed(atomic_t *v)
 {
+#if defined(arch_atomic_inc_return_relaxed)
+	return arch_atomic_inc_return_relaxed(v);
+#elif defined(arch_atomic_inc_return)
+	return arch_atomic_inc_return(v);
+#else
 	return raw_atomic_add_return_relaxed(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_inc)
-#define raw_atomic_fetch_inc arch_atomic_fetch_inc
-#elif defined(arch_atomic_fetch_inc_relaxed)
 static __always_inline int
 raw_atomic_fetch_inc(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_inc)
+	return arch_atomic_fetch_inc(v);
+#elif defined(arch_atomic_fetch_inc_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_inc_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_fetch_inc(atomic_t *v)
-{
 	return raw_atomic_fetch_add(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_inc_acquire)
-#define raw_atomic_fetch_inc_acquire arch_atomic_fetch_inc_acquire
-#elif defined(arch_atomic_fetch_inc_relaxed)
 static __always_inline int
 raw_atomic_fetch_inc_acquire(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_inc_acquire)
+	return arch_atomic_fetch_inc_acquire(v);
+#elif defined(arch_atomic_fetch_inc_relaxed)
 	int ret = arch_atomic_fetch_inc_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_inc)
-#define raw_atomic_fetch_inc_acquire arch_atomic_fetch_inc
+	return arch_atomic_fetch_inc(v);
 #else
-static __always_inline int
-raw_atomic_fetch_inc_acquire(atomic_t *v)
-{
 	return raw_atomic_fetch_add_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_inc_release)
-#define raw_atomic_fetch_inc_release arch_atomic_fetch_inc_release
-#elif defined(arch_atomic_fetch_inc_relaxed)
 static __always_inline int
 raw_atomic_fetch_inc_release(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_inc_release)
+	return arch_atomic_fetch_inc_release(v);
+#elif defined(arch_atomic_fetch_inc_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_inc_relaxed(v);
-}
 #elif defined(arch_atomic_fetch_inc)
-#define raw_atomic_fetch_inc_release arch_atomic_fetch_inc
+	return arch_atomic_fetch_inc(v);
 #else
-static __always_inline int
-raw_atomic_fetch_inc_release(atomic_t *v)
-{
 	return raw_atomic_fetch_add_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_inc_relaxed)
-#define raw_atomic_fetch_inc_relaxed arch_atomic_fetch_inc_relaxed
-#elif defined(arch_atomic_fetch_inc)
-#define raw_atomic_fetch_inc_relaxed arch_atomic_fetch_inc
-#else
 static __always_inline int
 raw_atomic_fetch_inc_relaxed(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_inc_relaxed)
+	return arch_atomic_fetch_inc_relaxed(v);
+#elif defined(arch_atomic_fetch_inc)
+	return arch_atomic_fetch_inc(v);
+#else
 	return raw_atomic_fetch_add_relaxed(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_dec)
-#define raw_atomic_dec arch_atomic_dec
-#else
 static __always_inline void
 raw_atomic_dec(atomic_t *v)
 {
+#if defined(arch_atomic_dec)
+	arch_atomic_dec(v);
+#else
 	raw_atomic_sub(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_return)
-#define raw_atomic_dec_return arch_atomic_dec_return
-#elif defined(arch_atomic_dec_return_relaxed)
 static __always_inline int
 raw_atomic_dec_return(atomic_t *v)
 {
+#if defined(arch_atomic_dec_return)
+	return arch_atomic_dec_return(v);
+#elif defined(arch_atomic_dec_return_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_dec_return_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_dec_return(atomic_t *v)
-{
 	return raw_atomic_sub_return(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_return_acquire)
-#define raw_atomic_dec_return_acquire arch_atomic_dec_return_acquire
-#elif defined(arch_atomic_dec_return_relaxed)
 static __always_inline int
 raw_atomic_dec_return_acquire(atomic_t *v)
 {
+#if defined(arch_atomic_dec_return_acquire)
+	return arch_atomic_dec_return_acquire(v);
+#elif defined(arch_atomic_dec_return_relaxed)
 	int ret = arch_atomic_dec_return_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_dec_return)
-#define raw_atomic_dec_return_acquire arch_atomic_dec_return
+	return arch_atomic_dec_return(v);
 #else
-static __always_inline int
-raw_atomic_dec_return_acquire(atomic_t *v)
-{
 	return raw_atomic_sub_return_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_return_release)
-#define raw_atomic_dec_return_release arch_atomic_dec_return_release
-#elif defined(arch_atomic_dec_return_relaxed)
 static __always_inline int
 raw_atomic_dec_return_release(atomic_t *v)
 {
+#if defined(arch_atomic_dec_return_release)
+	return arch_atomic_dec_return_release(v);
+#elif defined(arch_atomic_dec_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_dec_return_relaxed(v);
-}
 #elif defined(arch_atomic_dec_return)
-#define raw_atomic_dec_return_release arch_atomic_dec_return
+	return arch_atomic_dec_return(v);
 #else
-static __always_inline int
-raw_atomic_dec_return_release(atomic_t *v)
-{
 	return raw_atomic_sub_return_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_return_relaxed)
-#define raw_atomic_dec_return_relaxed arch_atomic_dec_return_relaxed
-#elif defined(arch_atomic_dec_return)
-#define raw_atomic_dec_return_relaxed arch_atomic_dec_return
-#else
 static __always_inline int
 raw_atomic_dec_return_relaxed(atomic_t *v)
 {
+#if defined(arch_atomic_dec_return_relaxed)
+	return arch_atomic_dec_return_relaxed(v);
+#elif defined(arch_atomic_dec_return)
+	return arch_atomic_dec_return(v);
+#else
 	return raw_atomic_sub_return_relaxed(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_dec)
-#define raw_atomic_fetch_dec arch_atomic_fetch_dec
-#elif defined(arch_atomic_fetch_dec_relaxed)
 static __always_inline int
 raw_atomic_fetch_dec(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_dec)
+	return arch_atomic_fetch_dec(v);
+#elif defined(arch_atomic_fetch_dec_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_dec_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_fetch_dec(atomic_t *v)
-{
 	return raw_atomic_fetch_sub(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_dec_acquire)
-#define raw_atomic_fetch_dec_acquire arch_atomic_fetch_dec_acquire
-#elif defined(arch_atomic_fetch_dec_relaxed)
 static __always_inline int
 raw_atomic_fetch_dec_acquire(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_dec_acquire)
+	return arch_atomic_fetch_dec_acquire(v);
+#elif defined(arch_atomic_fetch_dec_relaxed)
 	int ret = arch_atomic_fetch_dec_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_dec)
-#define raw_atomic_fetch_dec_acquire arch_atomic_fetch_dec
+	return arch_atomic_fetch_dec(v);
 #else
-static __always_inline int
-raw_atomic_fetch_dec_acquire(atomic_t *v)
-{
 	return raw_atomic_fetch_sub_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_dec_release)
-#define raw_atomic_fetch_dec_release arch_atomic_fetch_dec_release
-#elif defined(arch_atomic_fetch_dec_relaxed)
 static __always_inline int
 raw_atomic_fetch_dec_release(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_dec_release)
+	return arch_atomic_fetch_dec_release(v);
+#elif defined(arch_atomic_fetch_dec_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_dec_relaxed(v);
-}
 #elif defined(arch_atomic_fetch_dec)
-#define raw_atomic_fetch_dec_release arch_atomic_fetch_dec
+	return arch_atomic_fetch_dec(v);
 #else
-static __always_inline int
-raw_atomic_fetch_dec_release(atomic_t *v)
-{
 	return raw_atomic_fetch_sub_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_dec_relaxed)
-#define raw_atomic_fetch_dec_relaxed arch_atomic_fetch_dec_relaxed
-#elif defined(arch_atomic_fetch_dec)
-#define raw_atomic_fetch_dec_relaxed arch_atomic_fetch_dec
-#else
 static __always_inline int
 raw_atomic_fetch_dec_relaxed(atomic_t *v)
 {
+#if defined(arch_atomic_fetch_dec_relaxed)
+	return arch_atomic_fetch_dec_relaxed(v);
+#elif defined(arch_atomic_fetch_dec)
+	return arch_atomic_fetch_dec(v);
+#else
 	return raw_atomic_fetch_sub_relaxed(1, v);
-}
 #endif
+}
 
-#define raw_atomic_and arch_atomic_and
+static __always_inline void
+raw_atomic_and(int i, atomic_t *v)
+{
+	arch_atomic_and(i, v);
+}
 
-#if defined(arch_atomic_fetch_and)
-#define raw_atomic_fetch_and arch_atomic_fetch_and
-#elif defined(arch_atomic_fetch_and_relaxed)
 static __always_inline int
 raw_atomic_fetch_and(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_and)
+	return arch_atomic_fetch_and(i, v);
+#elif defined(arch_atomic_fetch_and_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_and_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_fetch_and"
 #endif
+}
 
-#if defined(arch_atomic_fetch_and_acquire)
-#define raw_atomic_fetch_and_acquire arch_atomic_fetch_and_acquire
-#elif defined(arch_atomic_fetch_and_relaxed)
 static __always_inline int
 raw_atomic_fetch_and_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_and_acquire)
+	return arch_atomic_fetch_and_acquire(i, v);
+#elif defined(arch_atomic_fetch_and_relaxed)
 	int ret = arch_atomic_fetch_and_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_and)
-#define raw_atomic_fetch_and_acquire arch_atomic_fetch_and
+	return arch_atomic_fetch_and(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_and_acquire"
 #endif
+}
 
-#if defined(arch_atomic_fetch_and_release)
-#define raw_atomic_fetch_and_release arch_atomic_fetch_and_release
-#elif defined(arch_atomic_fetch_and_relaxed)
 static __always_inline int
 raw_atomic_fetch_and_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_and_release)
+	return arch_atomic_fetch_and_release(i, v);
+#elif defined(arch_atomic_fetch_and_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_and_relaxed(i, v);
-}
 #elif defined(arch_atomic_fetch_and)
-#define raw_atomic_fetch_and_release arch_atomic_fetch_and
+	return arch_atomic_fetch_and(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_and_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_fetch_and_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_fetch_and_relaxed)
-#define raw_atomic_fetch_and_relaxed arch_atomic_fetch_and_relaxed
+	return arch_atomic_fetch_and_relaxed(i, v);
 #elif defined(arch_atomic_fetch_and)
-#define raw_atomic_fetch_and_relaxed arch_atomic_fetch_and
+	return arch_atomic_fetch_and(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_and_relaxed"
 #endif
+}
 
-#if defined(arch_atomic_andnot)
-#define raw_atomic_andnot arch_atomic_andnot
-#else
 static __always_inline void
 raw_atomic_andnot(int i, atomic_t *v)
 {
+#if defined(arch_atomic_andnot)
+	arch_atomic_andnot(i, v);
+#else
 	raw_atomic_and(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_andnot)
-#define raw_atomic_fetch_andnot arch_atomic_fetch_andnot
-#elif defined(arch_atomic_fetch_andnot_relaxed)
 static __always_inline int
 raw_atomic_fetch_andnot(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_andnot)
+	return arch_atomic_fetch_andnot(i, v);
+#elif defined(arch_atomic_fetch_andnot_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_andnot_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_fetch_andnot(int i, atomic_t *v)
-{
 	return raw_atomic_fetch_and(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_andnot_acquire)
-#define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot_acquire
-#elif defined(arch_atomic_fetch_andnot_relaxed)
 static __always_inline int
 raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_andnot_acquire)
+	return arch_atomic_fetch_andnot_acquire(i, v);
+#elif defined(arch_atomic_fetch_andnot_relaxed)
 	int ret = arch_atomic_fetch_andnot_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_andnot)
-#define raw_atomic_fetch_andnot_acquire arch_atomic_fetch_andnot
+	return arch_atomic_fetch_andnot(i, v);
 #else
-static __always_inline int
-raw_atomic_fetch_andnot_acquire(int i, atomic_t *v)
-{
 	return raw_atomic_fetch_and_acquire(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_andnot_release)
-#define raw_atomic_fetch_andnot_release arch_atomic_fetch_andnot_release
-#elif defined(arch_atomic_fetch_andnot_relaxed)
 static __always_inline int
 raw_atomic_fetch_andnot_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_andnot_release)
+	return arch_atomic_fetch_andnot_release(i, v);
+#elif defined(arch_atomic_fetch_andnot_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_andnot_relaxed(i, v);
-}
 #elif defined(arch_atomic_fetch_andnot)
-#define raw_atomic_fetch_andnot_release arch_atomic_fetch_andnot
+	return arch_atomic_fetch_andnot(i, v);
 #else
-static __always_inline int
-raw_atomic_fetch_andnot_release(int i, atomic_t *v)
-{
 	return raw_atomic_fetch_and_release(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_andnot_relaxed)
-#define raw_atomic_fetch_andnot_relaxed arch_atomic_fetch_andnot_relaxed
-#elif defined(arch_atomic_fetch_andnot)
-#define raw_atomic_fetch_andnot_relaxed arch_atomic_fetch_andnot
-#else
 static __always_inline int
 raw_atomic_fetch_andnot_relaxed(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_andnot_relaxed)
+	return arch_atomic_fetch_andnot_relaxed(i, v);
+#elif defined(arch_atomic_fetch_andnot)
+	return arch_atomic_fetch_andnot(i, v);
+#else
 	return raw_atomic_fetch_and_relaxed(~i, v);
-}
 #endif
+}
 
-#define raw_atomic_or arch_atomic_or
+static __always_inline void
+raw_atomic_or(int i, atomic_t *v)
+{
+	arch_atomic_or(i, v);
+}
 
-#if defined(arch_atomic_fetch_or)
-#define raw_atomic_fetch_or arch_atomic_fetch_or
-#elif defined(arch_atomic_fetch_or_relaxed)
 static __always_inline int
 raw_atomic_fetch_or(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_or)
+	return arch_atomic_fetch_or(i, v);
+#elif defined(arch_atomic_fetch_or_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_or_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_fetch_or"
 #endif
+}
 
-#if defined(arch_atomic_fetch_or_acquire)
-#define raw_atomic_fetch_or_acquire arch_atomic_fetch_or_acquire
-#elif defined(arch_atomic_fetch_or_relaxed)
 static __always_inline int
 raw_atomic_fetch_or_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_or_acquire)
+	return arch_atomic_fetch_or_acquire(i, v);
+#elif defined(arch_atomic_fetch_or_relaxed)
 	int ret = arch_atomic_fetch_or_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_or)
-#define raw_atomic_fetch_or_acquire arch_atomic_fetch_or
+	return arch_atomic_fetch_or(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_or_acquire"
 #endif
+}
 
-#if defined(arch_atomic_fetch_or_release)
-#define raw_atomic_fetch_or_release arch_atomic_fetch_or_release
-#elif defined(arch_atomic_fetch_or_relaxed)
 static __always_inline int
 raw_atomic_fetch_or_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_or_release)
+	return arch_atomic_fetch_or_release(i, v);
+#elif defined(arch_atomic_fetch_or_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_or_relaxed(i, v);
-}
 #elif defined(arch_atomic_fetch_or)
-#define raw_atomic_fetch_or_release arch_atomic_fetch_or
+	return arch_atomic_fetch_or(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_or_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_fetch_or_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_fetch_or_relaxed)
-#define raw_atomic_fetch_or_relaxed arch_atomic_fetch_or_relaxed
+	return arch_atomic_fetch_or_relaxed(i, v);
 #elif defined(arch_atomic_fetch_or)
-#define raw_atomic_fetch_or_relaxed arch_atomic_fetch_or
+	return arch_atomic_fetch_or(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_or_relaxed"
 #endif
+}
 
-#define raw_atomic_xor arch_atomic_xor
+static __always_inline void
+raw_atomic_xor(int i, atomic_t *v)
+{
+	arch_atomic_xor(i, v);
+}
 
-#if defined(arch_atomic_fetch_xor)
-#define raw_atomic_fetch_xor arch_atomic_fetch_xor
-#elif defined(arch_atomic_fetch_xor_relaxed)
 static __always_inline int
 raw_atomic_fetch_xor(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_xor)
+	return arch_atomic_fetch_xor(i, v);
+#elif defined(arch_atomic_fetch_xor_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_fetch_xor_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic_fetch_xor"
 #endif
+}
 
-#if defined(arch_atomic_fetch_xor_acquire)
-#define raw_atomic_fetch_xor_acquire arch_atomic_fetch_xor_acquire
-#elif defined(arch_atomic_fetch_xor_relaxed)
 static __always_inline int
 raw_atomic_fetch_xor_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_xor_acquire)
+	return arch_atomic_fetch_xor_acquire(i, v);
+#elif defined(arch_atomic_fetch_xor_relaxed)
 	int ret = arch_atomic_fetch_xor_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_fetch_xor)
-#define raw_atomic_fetch_xor_acquire arch_atomic_fetch_xor
+	return arch_atomic_fetch_xor(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_xor_acquire"
 #endif
+}
 
-#if defined(arch_atomic_fetch_xor_release)
-#define raw_atomic_fetch_xor_release arch_atomic_fetch_xor_release
-#elif defined(arch_atomic_fetch_xor_relaxed)
 static __always_inline int
 raw_atomic_fetch_xor_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_fetch_xor_release)
+	return arch_atomic_fetch_xor_release(i, v);
+#elif defined(arch_atomic_fetch_xor_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_fetch_xor_relaxed(i, v);
-}
 #elif defined(arch_atomic_fetch_xor)
-#define raw_atomic_fetch_xor_release arch_atomic_fetch_xor
+	return arch_atomic_fetch_xor(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_xor_release"
 #endif
+}
 
+static __always_inline int
+raw_atomic_fetch_xor_relaxed(int i, atomic_t *v)
+{
 #if defined(arch_atomic_fetch_xor_relaxed)
-#define raw_atomic_fetch_xor_relaxed arch_atomic_fetch_xor_relaxed
+	return arch_atomic_fetch_xor_relaxed(i, v);
 #elif defined(arch_atomic_fetch_xor)
-#define raw_atomic_fetch_xor_relaxed arch_atomic_fetch_xor
+	return arch_atomic_fetch_xor(i, v);
 #else
 #error "Unable to define raw_atomic_fetch_xor_relaxed"
 #endif
+}
 
-#if defined(arch_atomic_xchg)
-#define raw_atomic_xchg arch_atomic_xchg
-#elif defined(arch_atomic_xchg_relaxed)
 static __always_inline int
-raw_atomic_xchg(atomic_t *v, int i)
+raw_atomic_xchg(atomic_t *v, int new)
 {
+#if defined(arch_atomic_xchg)
+	return arch_atomic_xchg(v, new);
+#elif defined(arch_atomic_xchg_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
-	ret = arch_atomic_xchg_relaxed(v, i);
+	ret = arch_atomic_xchg_relaxed(v, new);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_xchg(atomic_t *v, int new)
-{
 	return raw_xchg(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_xchg_acquire)
-#define raw_atomic_xchg_acquire arch_atomic_xchg_acquire
-#elif defined(arch_atomic_xchg_relaxed)
 static __always_inline int
-raw_atomic_xchg_acquire(atomic_t *v, int i)
+raw_atomic_xchg_acquire(atomic_t *v, int new)
 {
-	int ret = arch_atomic_xchg_relaxed(v, i);
+#if defined(arch_atomic_xchg_acquire)
+	return arch_atomic_xchg_acquire(v, new);
+#elif defined(arch_atomic_xchg_relaxed)
+	int ret = arch_atomic_xchg_relaxed(v, new);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_xchg)
-#define raw_atomic_xchg_acquire arch_atomic_xchg
+	return arch_atomic_xchg(v, new);
 #else
-static __always_inline int
-raw_atomic_xchg_acquire(atomic_t *v, int new)
-{
 	return raw_xchg_acquire(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_xchg_release)
-#define raw_atomic_xchg_release arch_atomic_xchg_release
-#elif defined(arch_atomic_xchg_relaxed)
 static __always_inline int
-raw_atomic_xchg_release(atomic_t *v, int i)
+raw_atomic_xchg_release(atomic_t *v, int new)
 {
+#if defined(arch_atomic_xchg_release)
+	return arch_atomic_xchg_release(v, new);
+#elif defined(arch_atomic_xchg_relaxed)
 	__atomic_release_fence();
-	return arch_atomic_xchg_relaxed(v, i);
-}
+	return arch_atomic_xchg_relaxed(v, new);
 #elif defined(arch_atomic_xchg)
-#define raw_atomic_xchg_release arch_atomic_xchg
+	return arch_atomic_xchg(v, new);
 #else
-static __always_inline int
-raw_atomic_xchg_release(atomic_t *v, int new)
-{
 	return raw_xchg_release(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_xchg_relaxed)
-#define raw_atomic_xchg_relaxed arch_atomic_xchg_relaxed
-#elif defined(arch_atomic_xchg)
-#define raw_atomic_xchg_relaxed arch_atomic_xchg
-#else
 static __always_inline int
 raw_atomic_xchg_relaxed(atomic_t *v, int new)
 {
+#if defined(arch_atomic_xchg_relaxed)
+	return arch_atomic_xchg_relaxed(v, new);
+#elif defined(arch_atomic_xchg)
+	return arch_atomic_xchg(v, new);
+#else
 	return raw_xchg_relaxed(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_cmpxchg)
-#define raw_atomic_cmpxchg arch_atomic_cmpxchg
-#elif defined(arch_atomic_cmpxchg_relaxed)
 static __always_inline int
 raw_atomic_cmpxchg(atomic_t *v, int old, int new)
 {
+#if defined(arch_atomic_cmpxchg)
+	return arch_atomic_cmpxchg(v, old, new);
+#elif defined(arch_atomic_cmpxchg_relaxed)
 	int ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_cmpxchg_relaxed(v, old, new);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline int
-raw_atomic_cmpxchg(atomic_t *v, int old, int new)
-{
 	return raw_cmpxchg(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_cmpxchg_acquire)
-#define raw_atomic_cmpxchg_acquire arch_atomic_cmpxchg_acquire
-#elif defined(arch_atomic_cmpxchg_relaxed)
 static __always_inline int
 raw_atomic_cmpxchg_acquire(atomic_t *v, int old, int new)
 {
+#if defined(arch_atomic_cmpxchg_acquire)
+	return arch_atomic_cmpxchg_acquire(v, old, new);
+#elif defined(arch_atomic_cmpxchg_relaxed)
 	int ret = arch_atomic_cmpxchg_relaxed(v, old, new);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_cmpxchg)
-#define raw_atomic_cmpxchg_acquire arch_atomic_cmpxchg
+	return arch_atomic_cmpxchg(v, old, new);
 #else
-static __always_inline int
-raw_atomic_cmpxchg_acquire(atomic_t *v, int old, int new)
-{
 	return raw_cmpxchg_acquire(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_cmpxchg_release)
-#define raw_atomic_cmpxchg_release arch_atomic_cmpxchg_release
-#elif defined(arch_atomic_cmpxchg_relaxed)
 static __always_inline int
 raw_atomic_cmpxchg_release(atomic_t *v, int old, int new)
 {
+#if defined(arch_atomic_cmpxchg_release)
+	return arch_atomic_cmpxchg_release(v, old, new);
+#elif defined(arch_atomic_cmpxchg_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_cmpxchg_relaxed(v, old, new);
-}
 #elif defined(arch_atomic_cmpxchg)
-#define raw_atomic_cmpxchg_release arch_atomic_cmpxchg
+	return arch_atomic_cmpxchg(v, old, new);
 #else
-static __always_inline int
-raw_atomic_cmpxchg_release(atomic_t *v, int old, int new)
-{
 	return raw_cmpxchg_release(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_cmpxchg_relaxed)
-#define raw_atomic_cmpxchg_relaxed arch_atomic_cmpxchg_relaxed
-#elif defined(arch_atomic_cmpxchg)
-#define raw_atomic_cmpxchg_relaxed arch_atomic_cmpxchg
-#else
 static __always_inline int
 raw_atomic_cmpxchg_relaxed(atomic_t *v, int old, int new)
 {
+#if defined(arch_atomic_cmpxchg_relaxed)
+	return arch_atomic_cmpxchg_relaxed(v, old, new);
+#elif defined(arch_atomic_cmpxchg)
+	return arch_atomic_cmpxchg(v, old, new);
+#else
 	return raw_cmpxchg_relaxed(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic_try_cmpxchg)
-#define raw_atomic_try_cmpxchg arch_atomic_try_cmpxchg
-#elif defined(arch_atomic_try_cmpxchg_relaxed)
 static __always_inline bool
 raw_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
 {
+#if defined(arch_atomic_try_cmpxchg)
+	return arch_atomic_try_cmpxchg(v, old, new);
+#elif defined(arch_atomic_try_cmpxchg_relaxed)
 	bool ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_try_cmpxchg_relaxed(v, old, new);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline bool
-raw_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
-{
 	int r, o = *old;
 	r = raw_atomic_cmpxchg(v, o, new);
 	if (unlikely(r != o))
 		*old = r;
 	return likely(r == o);
-}
 #endif
+}
 
-#if defined(arch_atomic_try_cmpxchg_acquire)
-#define raw_atomic_try_cmpxchg_acquire arch_atomic_try_cmpxchg_acquire
-#elif defined(arch_atomic_try_cmpxchg_relaxed)
 static __always_inline bool
 raw_atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
 {
+#if defined(arch_atomic_try_cmpxchg_acquire)
+	return arch_atomic_try_cmpxchg_acquire(v, old, new);
+#elif defined(arch_atomic_try_cmpxchg_relaxed)
 	bool ret = arch_atomic_try_cmpxchg_relaxed(v, old, new);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_try_cmpxchg)
-#define raw_atomic_try_cmpxchg_acquire arch_atomic_try_cmpxchg
+	return arch_atomic_try_cmpxchg(v, old, new);
 #else
-static __always_inline bool
-raw_atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
-{
 	int r, o = *old;
 	r = raw_atomic_cmpxchg_acquire(v, o, new);
 	if (unlikely(r != o))
 		*old = r;
 	return likely(r == o);
-}
 #endif
+}
 
-#if defined(arch_atomic_try_cmpxchg_release)
-#define raw_atomic_try_cmpxchg_release arch_atomic_try_cmpxchg_release
-#elif defined(arch_atomic_try_cmpxchg_relaxed)
 static __always_inline bool
 raw_atomic_try_cmpxchg_release(atomic_t *v, int *old, int new)
 {
+#if defined(arch_atomic_try_cmpxchg_release)
+	return arch_atomic_try_cmpxchg_release(v, old, new);
+#elif defined(arch_atomic_try_cmpxchg_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_try_cmpxchg_relaxed(v, old, new);
-}
 #elif defined(arch_atomic_try_cmpxchg)
-#define raw_atomic_try_cmpxchg_release arch_atomic_try_cmpxchg
+	return arch_atomic_try_cmpxchg(v, old, new);
 #else
-static __always_inline bool
-raw_atomic_try_cmpxchg_release(atomic_t *v, int *old, int new)
-{
 	int r, o = *old;
 	r = raw_atomic_cmpxchg_release(v, o, new);
 	if (unlikely(r != o))
 		*old = r;
 	return likely(r == o);
-}
 #endif
+}
 
-#if defined(arch_atomic_try_cmpxchg_relaxed)
-#define raw_atomic_try_cmpxchg_relaxed arch_atomic_try_cmpxchg_relaxed
-#elif defined(arch_atomic_try_cmpxchg)
-#define raw_atomic_try_cmpxchg_relaxed arch_atomic_try_cmpxchg
-#else
 static __always_inline bool
 raw_atomic_try_cmpxchg_relaxed(atomic_t *v, int *old, int new)
 {
+#if defined(arch_atomic_try_cmpxchg_relaxed)
+	return arch_atomic_try_cmpxchg_relaxed(v, old, new);
+#elif defined(arch_atomic_try_cmpxchg)
+	return arch_atomic_try_cmpxchg(v, old, new);
+#else
 	int r, o = *old;
 	r = raw_atomic_cmpxchg_relaxed(v, o, new);
 	if (unlikely(r != o))
 		*old = r;
 	return likely(r == o);
-}
 #endif
+}
 
-#if defined(arch_atomic_sub_and_test)
-#define raw_atomic_sub_and_test arch_atomic_sub_and_test
-#else
 static __always_inline bool
 raw_atomic_sub_and_test(int i, atomic_t *v)
 {
+#if defined(arch_atomic_sub_and_test)
+	return arch_atomic_sub_and_test(i, v);
+#else
 	return raw_atomic_sub_return(i, v) == 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_and_test)
-#define raw_atomic_dec_and_test arch_atomic_dec_and_test
-#else
 static __always_inline bool
 raw_atomic_dec_and_test(atomic_t *v)
 {
+#if defined(arch_atomic_dec_and_test)
+	return arch_atomic_dec_and_test(v);
+#else
 	return raw_atomic_dec_return(v) == 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_and_test)
-#define raw_atomic_inc_and_test arch_atomic_inc_and_test
-#else
 static __always_inline bool
 raw_atomic_inc_and_test(atomic_t *v)
 {
+#if defined(arch_atomic_inc_and_test)
+	return arch_atomic_inc_and_test(v);
+#else
 	return raw_atomic_inc_return(v) == 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_add_negative)
-#define raw_atomic_add_negative arch_atomic_add_negative
-#elif defined(arch_atomic_add_negative_relaxed)
 static __always_inline bool
 raw_atomic_add_negative(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_negative)
+	return arch_atomic_add_negative(i, v);
+#elif defined(arch_atomic_add_negative_relaxed)
 	bool ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic_add_negative_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline bool
-raw_atomic_add_negative(int i, atomic_t *v)
-{
 	return raw_atomic_add_return(i, v) < 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_add_negative_acquire)
-#define raw_atomic_add_negative_acquire arch_atomic_add_negative_acquire
-#elif defined(arch_atomic_add_negative_relaxed)
 static __always_inline bool
 raw_atomic_add_negative_acquire(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_negative_acquire)
+	return arch_atomic_add_negative_acquire(i, v);
+#elif defined(arch_atomic_add_negative_relaxed)
 	bool ret = arch_atomic_add_negative_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic_add_negative)
-#define raw_atomic_add_negative_acquire arch_atomic_add_negative
+	return arch_atomic_add_negative(i, v);
 #else
-static __always_inline bool
-raw_atomic_add_negative_acquire(int i, atomic_t *v)
-{
 	return raw_atomic_add_return_acquire(i, v) < 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_add_negative_release)
-#define raw_atomic_add_negative_release arch_atomic_add_negative_release
-#elif defined(arch_atomic_add_negative_relaxed)
 static __always_inline bool
 raw_atomic_add_negative_release(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_negative_release)
+	return arch_atomic_add_negative_release(i, v);
+#elif defined(arch_atomic_add_negative_relaxed)
 	__atomic_release_fence();
 	return arch_atomic_add_negative_relaxed(i, v);
-}
 #elif defined(arch_atomic_add_negative)
-#define raw_atomic_add_negative_release arch_atomic_add_negative
+	return arch_atomic_add_negative(i, v);
 #else
-static __always_inline bool
-raw_atomic_add_negative_release(int i, atomic_t *v)
-{
 	return raw_atomic_add_return_release(i, v) < 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_add_negative_relaxed)
-#define raw_atomic_add_negative_relaxed arch_atomic_add_negative_relaxed
-#elif defined(arch_atomic_add_negative)
-#define raw_atomic_add_negative_relaxed arch_atomic_add_negative
-#else
 static __always_inline bool
 raw_atomic_add_negative_relaxed(int i, atomic_t *v)
 {
+#if defined(arch_atomic_add_negative_relaxed)
+	return arch_atomic_add_negative_relaxed(i, v);
+#elif defined(arch_atomic_add_negative)
+	return arch_atomic_add_negative(i, v);
+#else
 	return raw_atomic_add_return_relaxed(i, v) < 0;
-}
 #endif
+}
 
-#if defined(arch_atomic_fetch_add_unless)
-#define raw_atomic_fetch_add_unless arch_atomic_fetch_add_unless
-#else
 static __always_inline int
 raw_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 {
+#if defined(arch_atomic_fetch_add_unless)
+	return arch_atomic_fetch_add_unless(v, a, u);
+#else
 	int c = raw_atomic_read(v);
 
 	do {
@@ -1594,35 +1542,35 @@ raw_atomic_fetch_add_unless(atomic_t *v, int a, int u)
 	} while (!raw_atomic_try_cmpxchg(v, &c, c + a));
 
 	return c;
-}
 #endif
+}
 
-#if defined(arch_atomic_add_unless)
-#define raw_atomic_add_unless arch_atomic_add_unless
-#else
 static __always_inline bool
 raw_atomic_add_unless(atomic_t *v, int a, int u)
 {
+#if defined(arch_atomic_add_unless)
+	return arch_atomic_add_unless(v, a, u);
+#else
 	return raw_atomic_fetch_add_unless(v, a, u) != u;
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_not_zero)
-#define raw_atomic_inc_not_zero arch_atomic_inc_not_zero
-#else
 static __always_inline bool
 raw_atomic_inc_not_zero(atomic_t *v)
 {
+#if defined(arch_atomic_inc_not_zero)
+	return arch_atomic_inc_not_zero(v);
+#else
 	return raw_atomic_add_unless(v, 1, 0);
-}
 #endif
+}
 
-#if defined(arch_atomic_inc_unless_negative)
-#define raw_atomic_inc_unless_negative arch_atomic_inc_unless_negative
-#else
 static __always_inline bool
 raw_atomic_inc_unless_negative(atomic_t *v)
 {
+#if defined(arch_atomic_inc_unless_negative)
+	return arch_atomic_inc_unless_negative(v);
+#else
 	int c = raw_atomic_read(v);
 
 	do {
@@ -1631,15 +1579,15 @@ raw_atomic_inc_unless_negative(atomic_t *v)
 	} while (!raw_atomic_try_cmpxchg(v, &c, c + 1));
 
 	return true;
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_unless_positive)
-#define raw_atomic_dec_unless_positive arch_atomic_dec_unless_positive
-#else
 static __always_inline bool
 raw_atomic_dec_unless_positive(atomic_t *v)
 {
+#if defined(arch_atomic_dec_unless_positive)
+	return arch_atomic_dec_unless_positive(v);
+#else
 	int c = raw_atomic_read(v);
 
 	do {
@@ -1648,15 +1596,15 @@ raw_atomic_dec_unless_positive(atomic_t *v)
 	} while (!raw_atomic_try_cmpxchg(v, &c, c - 1));
 
 	return true;
-}
 #endif
+}
 
-#if defined(arch_atomic_dec_if_positive)
-#define raw_atomic_dec_if_positive arch_atomic_dec_if_positive
-#else
 static __always_inline int
 raw_atomic_dec_if_positive(atomic_t *v)
 {
+#if defined(arch_atomic_dec_if_positive)
+	return arch_atomic_dec_if_positive(v);
+#else
 	int dec, c = raw_atomic_read(v);
 
 	do {
@@ -1666,23 +1614,27 @@ raw_atomic_dec_if_positive(atomic_t *v)
 	} while (!raw_atomic_try_cmpxchg(v, &c, dec));
 
 	return dec;
-}
 #endif
+}
 
 #ifdef CONFIG_GENERIC_ATOMIC64
 #include <asm-generic/atomic64.h>
 #endif
 
-#define raw_atomic64_read arch_atomic64_read
+static __always_inline s64
+raw_atomic64_read(const atomic64_t *v)
+{
+	return arch_atomic64_read(v);
+}
 
-#if defined(arch_atomic64_read_acquire)
-#define raw_atomic64_read_acquire arch_atomic64_read_acquire
-#elif defined(arch_atomic64_read)
-#define raw_atomic64_read_acquire arch_atomic64_read
-#else
 static __always_inline s64
 raw_atomic64_read_acquire(const atomic64_t *v)
 {
+#if defined(arch_atomic64_read_acquire)
+	return arch_atomic64_read_acquire(v);
+#elif defined(arch_atomic64_read)
+	return arch_atomic64_read(v);
+#else
 	s64 ret;
 
 	if (__native_word(atomic64_t)) {
@@ -1693,1144 +1645,1088 @@ raw_atomic64_read_acquire(const atomic64_t *v)
 	}
 
 	return ret;
-}
 #endif
+}
 
-#define raw_atomic64_set arch_atomic64_set
+static __always_inline void
+raw_atomic64_set(atomic64_t *v, s64 i)
+{
+	arch_atomic64_set(v, i);
+}
 
-#if defined(arch_atomic64_set_release)
-#define raw_atomic64_set_release arch_atomic64_set_release
-#elif defined(arch_atomic64_set)
-#define raw_atomic64_set_release arch_atomic64_set
-#else
 static __always_inline void
 raw_atomic64_set_release(atomic64_t *v, s64 i)
 {
+#if defined(arch_atomic64_set_release)
+	arch_atomic64_set_release(v, i);
+#elif defined(arch_atomic64_set)
+	arch_atomic64_set(v, i);
+#else
 	if (__native_word(atomic64_t)) {
 		smp_store_release(&(v)->counter, i);
 	} else {
 		__atomic_release_fence();
 		raw_atomic64_set(v, i);
 	}
-}
 #endif
+}
 
-#define raw_atomic64_add arch_atomic64_add
+static __always_inline void
+raw_atomic64_add(s64 i, atomic64_t *v)
+{
+	arch_atomic64_add(i, v);
+}
 
-#if defined(arch_atomic64_add_return)
-#define raw_atomic64_add_return arch_atomic64_add_return
-#elif defined(arch_atomic64_add_return_relaxed)
 static __always_inline s64
 raw_atomic64_add_return(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_add_return)
+	return arch_atomic64_add_return(i, v);
+#elif defined(arch_atomic64_add_return_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_add_return_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_add_return"
 #endif
+}
 
-#if defined(arch_atomic64_add_return_acquire)
-#define raw_atomic64_add_return_acquire arch_atomic64_add_return_acquire
-#elif defined(arch_atomic64_add_return_relaxed)
 static __always_inline s64
 raw_atomic64_add_return_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_add_return_acquire)
+	return arch_atomic64_add_return_acquire(i, v);
+#elif defined(arch_atomic64_add_return_relaxed)
 	s64 ret = arch_atomic64_add_return_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_add_return)
-#define raw_atomic64_add_return_acquire arch_atomic64_add_return
+	return arch_atomic64_add_return(i, v);
 #else
 #error "Unable to define raw_atomic64_add_return_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_add_return_release)
-#define raw_atomic64_add_return_release arch_atomic64_add_return_release
-#elif defined(arch_atomic64_add_return_relaxed)
 static __always_inline s64
 raw_atomic64_add_return_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_add_return_release)
+	return arch_atomic64_add_return_release(i, v);
+#elif defined(arch_atomic64_add_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_add_return_relaxed(i, v);
-}
 #elif defined(arch_atomic64_add_return)
-#define raw_atomic64_add_return_release arch_atomic64_add_return
+	return arch_atomic64_add_return(i, v);
 #else
 #error "Unable to define raw_atomic64_add_return_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_add_return_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_add_return_relaxed)
-#define raw_atomic64_add_return_relaxed arch_atomic64_add_return_relaxed
+	return arch_atomic64_add_return_relaxed(i, v);
 #elif defined(arch_atomic64_add_return)
-#define raw_atomic64_add_return_relaxed arch_atomic64_add_return
+	return arch_atomic64_add_return(i, v);
 #else
 #error "Unable to define raw_atomic64_add_return_relaxed"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_add)
-#define raw_atomic64_fetch_add arch_atomic64_fetch_add
-#elif defined(arch_atomic64_fetch_add_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_add(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_add)
+	return arch_atomic64_fetch_add(i, v);
+#elif defined(arch_atomic64_fetch_add_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_add_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_fetch_add"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_add_acquire)
-#define raw_atomic64_fetch_add_acquire arch_atomic64_fetch_add_acquire
-#elif defined(arch_atomic64_fetch_add_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_add_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_add_acquire)
+	return arch_atomic64_fetch_add_acquire(i, v);
+#elif defined(arch_atomic64_fetch_add_relaxed)
 	s64 ret = arch_atomic64_fetch_add_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_add)
-#define raw_atomic64_fetch_add_acquire arch_atomic64_fetch_add
+	return arch_atomic64_fetch_add(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_add_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_add_release)
-#define raw_atomic64_fetch_add_release arch_atomic64_fetch_add_release
-#elif defined(arch_atomic64_fetch_add_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_add_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_add_release)
+	return arch_atomic64_fetch_add_release(i, v);
+#elif defined(arch_atomic64_fetch_add_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_add_relaxed(i, v);
-}
 #elif defined(arch_atomic64_fetch_add)
-#define raw_atomic64_fetch_add_release arch_atomic64_fetch_add
+	return arch_atomic64_fetch_add(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_add_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_fetch_add_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_fetch_add_relaxed)
-#define raw_atomic64_fetch_add_relaxed arch_atomic64_fetch_add_relaxed
+	return arch_atomic64_fetch_add_relaxed(i, v);
 #elif defined(arch_atomic64_fetch_add)
-#define raw_atomic64_fetch_add_relaxed arch_atomic64_fetch_add
+	return arch_atomic64_fetch_add(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_add_relaxed"
 #endif
+}
 
-#define raw_atomic64_sub arch_atomic64_sub
+static __always_inline void
+raw_atomic64_sub(s64 i, atomic64_t *v)
+{
+	arch_atomic64_sub(i, v);
+}
 
-#if defined(arch_atomic64_sub_return)
-#define raw_atomic64_sub_return arch_atomic64_sub_return
-#elif defined(arch_atomic64_sub_return_relaxed)
 static __always_inline s64
 raw_atomic64_sub_return(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_sub_return)
+	return arch_atomic64_sub_return(i, v);
+#elif defined(arch_atomic64_sub_return_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_sub_return_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_sub_return"
 #endif
+}
 
-#if defined(arch_atomic64_sub_return_acquire)
-#define raw_atomic64_sub_return_acquire arch_atomic64_sub_return_acquire
-#elif defined(arch_atomic64_sub_return_relaxed)
 static __always_inline s64
 raw_atomic64_sub_return_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_sub_return_acquire)
+	return arch_atomic64_sub_return_acquire(i, v);
+#elif defined(arch_atomic64_sub_return_relaxed)
 	s64 ret = arch_atomic64_sub_return_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_sub_return)
-#define raw_atomic64_sub_return_acquire arch_atomic64_sub_return
+	return arch_atomic64_sub_return(i, v);
 #else
 #error "Unable to define raw_atomic64_sub_return_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_sub_return_release)
-#define raw_atomic64_sub_return_release arch_atomic64_sub_return_release
-#elif defined(arch_atomic64_sub_return_relaxed)
 static __always_inline s64
 raw_atomic64_sub_return_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_sub_return_release)
+	return arch_atomic64_sub_return_release(i, v);
+#elif defined(arch_atomic64_sub_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_sub_return_relaxed(i, v);
-}
 #elif defined(arch_atomic64_sub_return)
-#define raw_atomic64_sub_return_release arch_atomic64_sub_return
+	return arch_atomic64_sub_return(i, v);
 #else
 #error "Unable to define raw_atomic64_sub_return_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_sub_return_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_sub_return_relaxed)
-#define raw_atomic64_sub_return_relaxed arch_atomic64_sub_return_relaxed
+	return arch_atomic64_sub_return_relaxed(i, v);
 #elif defined(arch_atomic64_sub_return)
-#define raw_atomic64_sub_return_relaxed arch_atomic64_sub_return
+	return arch_atomic64_sub_return(i, v);
 #else
 #error "Unable to define raw_atomic64_sub_return_relaxed"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_sub)
-#define raw_atomic64_fetch_sub arch_atomic64_fetch_sub
-#elif defined(arch_atomic64_fetch_sub_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_sub(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_sub)
+	return arch_atomic64_fetch_sub(i, v);
+#elif defined(arch_atomic64_fetch_sub_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_sub_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_fetch_sub"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_sub_acquire)
-#define raw_atomic64_fetch_sub_acquire arch_atomic64_fetch_sub_acquire
-#elif defined(arch_atomic64_fetch_sub_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_sub_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_sub_acquire)
+	return arch_atomic64_fetch_sub_acquire(i, v);
+#elif defined(arch_atomic64_fetch_sub_relaxed)
 	s64 ret = arch_atomic64_fetch_sub_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_sub)
-#define raw_atomic64_fetch_sub_acquire arch_atomic64_fetch_sub
+	return arch_atomic64_fetch_sub(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_sub_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_sub_release)
-#define raw_atomic64_fetch_sub_release arch_atomic64_fetch_sub_release
-#elif defined(arch_atomic64_fetch_sub_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_sub_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_sub_release)
+	return arch_atomic64_fetch_sub_release(i, v);
+#elif defined(arch_atomic64_fetch_sub_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_sub_relaxed(i, v);
-}
 #elif defined(arch_atomic64_fetch_sub)
-#define raw_atomic64_fetch_sub_release arch_atomic64_fetch_sub
+	return arch_atomic64_fetch_sub(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_sub_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_fetch_sub_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_fetch_sub_relaxed)
-#define raw_atomic64_fetch_sub_relaxed arch_atomic64_fetch_sub_relaxed
+	return arch_atomic64_fetch_sub_relaxed(i, v);
 #elif defined(arch_atomic64_fetch_sub)
-#define raw_atomic64_fetch_sub_relaxed arch_atomic64_fetch_sub
+	return arch_atomic64_fetch_sub(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_sub_relaxed"
 #endif
+}
 
-#if defined(arch_atomic64_inc)
-#define raw_atomic64_inc arch_atomic64_inc
-#else
 static __always_inline void
 raw_atomic64_inc(atomic64_t *v)
 {
+#if defined(arch_atomic64_inc)
+	arch_atomic64_inc(v);
+#else
 	raw_atomic64_add(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_inc_return)
-#define raw_atomic64_inc_return arch_atomic64_inc_return
-#elif defined(arch_atomic64_inc_return_relaxed)
 static __always_inline s64
 raw_atomic64_inc_return(atomic64_t *v)
 {
+#if defined(arch_atomic64_inc_return)
+	return arch_atomic64_inc_return(v);
+#elif defined(arch_atomic64_inc_return_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_inc_return_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_inc_return(atomic64_t *v)
-{
 	return raw_atomic64_add_return(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_inc_return_acquire)
-#define raw_atomic64_inc_return_acquire arch_atomic64_inc_return_acquire
-#elif defined(arch_atomic64_inc_return_relaxed)
 static __always_inline s64
 raw_atomic64_inc_return_acquire(atomic64_t *v)
 {
+#if defined(arch_atomic64_inc_return_acquire)
+	return arch_atomic64_inc_return_acquire(v);
+#elif defined(arch_atomic64_inc_return_relaxed)
 	s64 ret = arch_atomic64_inc_return_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_inc_return)
-#define raw_atomic64_inc_return_acquire arch_atomic64_inc_return
+	return arch_atomic64_inc_return(v);
 #else
-static __always_inline s64
-raw_atomic64_inc_return_acquire(atomic64_t *v)
-{
 	return raw_atomic64_add_return_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_inc_return_release)
-#define raw_atomic64_inc_return_release arch_atomic64_inc_return_release
-#elif defined(arch_atomic64_inc_return_relaxed)
 static __always_inline s64
 raw_atomic64_inc_return_release(atomic64_t *v)
 {
+#if defined(arch_atomic64_inc_return_release)
+	return arch_atomic64_inc_return_release(v);
+#elif defined(arch_atomic64_inc_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_inc_return_relaxed(v);
-}
 #elif defined(arch_atomic64_inc_return)
-#define raw_atomic64_inc_return_release arch_atomic64_inc_return
+	return arch_atomic64_inc_return(v);
 #else
-static __always_inline s64
-raw_atomic64_inc_return_release(atomic64_t *v)
-{
 	return raw_atomic64_add_return_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_inc_return_relaxed)
-#define raw_atomic64_inc_return_relaxed arch_atomic64_inc_return_relaxed
-#elif defined(arch_atomic64_inc_return)
-#define raw_atomic64_inc_return_relaxed arch_atomic64_inc_return
-#else
 static __always_inline s64
 raw_atomic64_inc_return_relaxed(atomic64_t *v)
 {
+#if defined(arch_atomic64_inc_return_relaxed)
+	return arch_atomic64_inc_return_relaxed(v);
+#elif defined(arch_atomic64_inc_return)
+	return arch_atomic64_inc_return(v);
+#else
 	return raw_atomic64_add_return_relaxed(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_inc)
-#define raw_atomic64_fetch_inc arch_atomic64_fetch_inc
-#elif defined(arch_atomic64_fetch_inc_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_inc(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_inc)
+	return arch_atomic64_fetch_inc(v);
+#elif defined(arch_atomic64_fetch_inc_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_inc_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_fetch_inc(atomic64_t *v)
-{
 	return raw_atomic64_fetch_add(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_inc_acquire)
-#define raw_atomic64_fetch_inc_acquire arch_atomic64_fetch_inc_acquire
-#elif defined(arch_atomic64_fetch_inc_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_inc_acquire(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_inc_acquire)
+	return arch_atomic64_fetch_inc_acquire(v);
+#elif defined(arch_atomic64_fetch_inc_relaxed)
 	s64 ret = arch_atomic64_fetch_inc_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_inc)
-#define raw_atomic64_fetch_inc_acquire arch_atomic64_fetch_inc
+	return arch_atomic64_fetch_inc(v);
 #else
-static __always_inline s64
-raw_atomic64_fetch_inc_acquire(atomic64_t *v)
-{
 	return raw_atomic64_fetch_add_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_inc_release)
-#define raw_atomic64_fetch_inc_release arch_atomic64_fetch_inc_release
-#elif defined(arch_atomic64_fetch_inc_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_inc_release(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_inc_release)
+	return arch_atomic64_fetch_inc_release(v);
+#elif defined(arch_atomic64_fetch_inc_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_inc_relaxed(v);
-}
 #elif defined(arch_atomic64_fetch_inc)
-#define raw_atomic64_fetch_inc_release arch_atomic64_fetch_inc
+	return arch_atomic64_fetch_inc(v);
 #else
-static __always_inline s64
-raw_atomic64_fetch_inc_release(atomic64_t *v)
-{
 	return raw_atomic64_fetch_add_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_inc_relaxed)
-#define raw_atomic64_fetch_inc_relaxed arch_atomic64_fetch_inc_relaxed
-#elif defined(arch_atomic64_fetch_inc)
-#define raw_atomic64_fetch_inc_relaxed arch_atomic64_fetch_inc
-#else
 static __always_inline s64
 raw_atomic64_fetch_inc_relaxed(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_inc_relaxed)
+	return arch_atomic64_fetch_inc_relaxed(v);
+#elif defined(arch_atomic64_fetch_inc)
+	return arch_atomic64_fetch_inc(v);
+#else
 	return raw_atomic64_fetch_add_relaxed(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_dec)
-#define raw_atomic64_dec arch_atomic64_dec
-#else
 static __always_inline void
 raw_atomic64_dec(atomic64_t *v)
 {
+#if defined(arch_atomic64_dec)
+	arch_atomic64_dec(v);
+#else
 	raw_atomic64_sub(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_dec_return)
-#define raw_atomic64_dec_return arch_atomic64_dec_return
-#elif defined(arch_atomic64_dec_return_relaxed)
 static __always_inline s64
 raw_atomic64_dec_return(atomic64_t *v)
 {
+#if defined(arch_atomic64_dec_return)
+	return arch_atomic64_dec_return(v);
+#elif defined(arch_atomic64_dec_return_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_dec_return_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_dec_return(atomic64_t *v)
-{
 	return raw_atomic64_sub_return(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_dec_return_acquire)
-#define raw_atomic64_dec_return_acquire arch_atomic64_dec_return_acquire
-#elif defined(arch_atomic64_dec_return_relaxed)
 static __always_inline s64
 raw_atomic64_dec_return_acquire(atomic64_t *v)
 {
+#if defined(arch_atomic64_dec_return_acquire)
+	return arch_atomic64_dec_return_acquire(v);
+#elif defined(arch_atomic64_dec_return_relaxed)
 	s64 ret = arch_atomic64_dec_return_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_dec_return)
-#define raw_atomic64_dec_return_acquire arch_atomic64_dec_return
+	return arch_atomic64_dec_return(v);
 #else
-static __always_inline s64
-raw_atomic64_dec_return_acquire(atomic64_t *v)
-{
 	return raw_atomic64_sub_return_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_dec_return_release)
-#define raw_atomic64_dec_return_release arch_atomic64_dec_return_release
-#elif defined(arch_atomic64_dec_return_relaxed)
 static __always_inline s64
 raw_atomic64_dec_return_release(atomic64_t *v)
 {
+#if defined(arch_atomic64_dec_return_release)
+	return arch_atomic64_dec_return_release(v);
+#elif defined(arch_atomic64_dec_return_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_dec_return_relaxed(v);
-}
 #elif defined(arch_atomic64_dec_return)
-#define raw_atomic64_dec_return_release arch_atomic64_dec_return
+	return arch_atomic64_dec_return(v);
 #else
-static __always_inline s64
-raw_atomic64_dec_return_release(atomic64_t *v)
-{
 	return raw_atomic64_sub_return_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_dec_return_relaxed)
-#define raw_atomic64_dec_return_relaxed arch_atomic64_dec_return_relaxed
-#elif defined(arch_atomic64_dec_return)
-#define raw_atomic64_dec_return_relaxed arch_atomic64_dec_return
-#else
 static __always_inline s64
 raw_atomic64_dec_return_relaxed(atomic64_t *v)
 {
+#if defined(arch_atomic64_dec_return_relaxed)
+	return arch_atomic64_dec_return_relaxed(v);
+#elif defined(arch_atomic64_dec_return)
+	return arch_atomic64_dec_return(v);
+#else
 	return raw_atomic64_sub_return_relaxed(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_dec)
-#define raw_atomic64_fetch_dec arch_atomic64_fetch_dec
-#elif defined(arch_atomic64_fetch_dec_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_dec(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_dec)
+	return arch_atomic64_fetch_dec(v);
+#elif defined(arch_atomic64_fetch_dec_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_dec_relaxed(v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_fetch_dec(atomic64_t *v)
-{
 	return raw_atomic64_fetch_sub(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_dec_acquire)
-#define raw_atomic64_fetch_dec_acquire arch_atomic64_fetch_dec_acquire
-#elif defined(arch_atomic64_fetch_dec_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_dec_acquire(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_dec_acquire)
+	return arch_atomic64_fetch_dec_acquire(v);
+#elif defined(arch_atomic64_fetch_dec_relaxed)
 	s64 ret = arch_atomic64_fetch_dec_relaxed(v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_dec)
-#define raw_atomic64_fetch_dec_acquire arch_atomic64_fetch_dec
+	return arch_atomic64_fetch_dec(v);
 #else
-static __always_inline s64
-raw_atomic64_fetch_dec_acquire(atomic64_t *v)
-{
 	return raw_atomic64_fetch_sub_acquire(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_dec_release)
-#define raw_atomic64_fetch_dec_release arch_atomic64_fetch_dec_release
-#elif defined(arch_atomic64_fetch_dec_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_dec_release(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_dec_release)
+	return arch_atomic64_fetch_dec_release(v);
+#elif defined(arch_atomic64_fetch_dec_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_dec_relaxed(v);
-}
 #elif defined(arch_atomic64_fetch_dec)
-#define raw_atomic64_fetch_dec_release arch_atomic64_fetch_dec
+	return arch_atomic64_fetch_dec(v);
 #else
-static __always_inline s64
-raw_atomic64_fetch_dec_release(atomic64_t *v)
-{
 	return raw_atomic64_fetch_sub_release(1, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_dec_relaxed)
-#define raw_atomic64_fetch_dec_relaxed arch_atomic64_fetch_dec_relaxed
-#elif defined(arch_atomic64_fetch_dec)
-#define raw_atomic64_fetch_dec_relaxed arch_atomic64_fetch_dec
-#else
 static __always_inline s64
 raw_atomic64_fetch_dec_relaxed(atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_dec_relaxed)
+	return arch_atomic64_fetch_dec_relaxed(v);
+#elif defined(arch_atomic64_fetch_dec)
+	return arch_atomic64_fetch_dec(v);
+#else
 	return raw_atomic64_fetch_sub_relaxed(1, v);
-}
 #endif
+}
 
-#define raw_atomic64_and arch_atomic64_and
+static __always_inline void
+raw_atomic64_and(s64 i, atomic64_t *v)
+{
+	arch_atomic64_and(i, v);
+}
 
-#if defined(arch_atomic64_fetch_and)
-#define raw_atomic64_fetch_and arch_atomic64_fetch_and
-#elif defined(arch_atomic64_fetch_and_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_and(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_and)
+	return arch_atomic64_fetch_and(i, v);
+#elif defined(arch_atomic64_fetch_and_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_and_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_fetch_and"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_and_acquire)
-#define raw_atomic64_fetch_and_acquire arch_atomic64_fetch_and_acquire
-#elif defined(arch_atomic64_fetch_and_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_and_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_and_acquire)
+	return arch_atomic64_fetch_and_acquire(i, v);
+#elif defined(arch_atomic64_fetch_and_relaxed)
 	s64 ret = arch_atomic64_fetch_and_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_and)
-#define raw_atomic64_fetch_and_acquire arch_atomic64_fetch_and
+	return arch_atomic64_fetch_and(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_and_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_and_release)
-#define raw_atomic64_fetch_and_release arch_atomic64_fetch_and_release
-#elif defined(arch_atomic64_fetch_and_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_and_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_and_release)
+	return arch_atomic64_fetch_and_release(i, v);
+#elif defined(arch_atomic64_fetch_and_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_and_relaxed(i, v);
-}
 #elif defined(arch_atomic64_fetch_and)
-#define raw_atomic64_fetch_and_release arch_atomic64_fetch_and
+	return arch_atomic64_fetch_and(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_and_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_fetch_and_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_fetch_and_relaxed)
-#define raw_atomic64_fetch_and_relaxed arch_atomic64_fetch_and_relaxed
+	return arch_atomic64_fetch_and_relaxed(i, v);
 #elif defined(arch_atomic64_fetch_and)
-#define raw_atomic64_fetch_and_relaxed arch_atomic64_fetch_and
+	return arch_atomic64_fetch_and(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_and_relaxed"
 #endif
+}
 
-#if defined(arch_atomic64_andnot)
-#define raw_atomic64_andnot arch_atomic64_andnot
-#else
 static __always_inline void
 raw_atomic64_andnot(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_andnot)
+	arch_atomic64_andnot(i, v);
+#else
 	raw_atomic64_and(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_andnot)
-#define raw_atomic64_fetch_andnot arch_atomic64_fetch_andnot
-#elif defined(arch_atomic64_fetch_andnot_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_andnot(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_andnot)
+	return arch_atomic64_fetch_andnot(i, v);
+#elif defined(arch_atomic64_fetch_andnot_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_andnot_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_fetch_andnot(s64 i, atomic64_t *v)
-{
 	return raw_atomic64_fetch_and(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_andnot_acquire)
-#define raw_atomic64_fetch_andnot_acquire arch_atomic64_fetch_andnot_acquire
-#elif defined(arch_atomic64_fetch_andnot_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_andnot_acquire)
+	return arch_atomic64_fetch_andnot_acquire(i, v);
+#elif defined(arch_atomic64_fetch_andnot_relaxed)
 	s64 ret = arch_atomic64_fetch_andnot_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_andnot)
-#define raw_atomic64_fetch_andnot_acquire arch_atomic64_fetch_andnot
+	return arch_atomic64_fetch_andnot(i, v);
 #else
-static __always_inline s64
-raw_atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v)
-{
 	return raw_atomic64_fetch_and_acquire(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_andnot_release)
-#define raw_atomic64_fetch_andnot_release arch_atomic64_fetch_andnot_release
-#elif defined(arch_atomic64_fetch_andnot_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_andnot_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_andnot_release)
+	return arch_atomic64_fetch_andnot_release(i, v);
+#elif defined(arch_atomic64_fetch_andnot_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_andnot_relaxed(i, v);
-}
 #elif defined(arch_atomic64_fetch_andnot)
-#define raw_atomic64_fetch_andnot_release arch_atomic64_fetch_andnot
+	return arch_atomic64_fetch_andnot(i, v);
 #else
-static __always_inline s64
-raw_atomic64_fetch_andnot_release(s64 i, atomic64_t *v)
-{
 	return raw_atomic64_fetch_and_release(~i, v);
-}
 #endif
+}
 
-#if defined(arch_atomic64_fetch_andnot_relaxed)
-#define raw_atomic64_fetch_andnot_relaxed arch_atomic64_fetch_andnot_relaxed
-#elif defined(arch_atomic64_fetch_andnot)
-#define raw_atomic64_fetch_andnot_relaxed arch_atomic64_fetch_andnot
-#else
 static __always_inline s64
 raw_atomic64_fetch_andnot_relaxed(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_andnot_relaxed)
+	return arch_atomic64_fetch_andnot_relaxed(i, v);
+#elif defined(arch_atomic64_fetch_andnot)
+	return arch_atomic64_fetch_andnot(i, v);
+#else
 	return raw_atomic64_fetch_and_relaxed(~i, v);
-}
 #endif
+}
 
-#define raw_atomic64_or arch_atomic64_or
+static __always_inline void
+raw_atomic64_or(s64 i, atomic64_t *v)
+{
+	arch_atomic64_or(i, v);
+}
 
-#if defined(arch_atomic64_fetch_or)
-#define raw_atomic64_fetch_or arch_atomic64_fetch_or
-#elif defined(arch_atomic64_fetch_or_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_or(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_or)
+	return arch_atomic64_fetch_or(i, v);
+#elif defined(arch_atomic64_fetch_or_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_or_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_fetch_or"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_or_acquire)
-#define raw_atomic64_fetch_or_acquire arch_atomic64_fetch_or_acquire
-#elif defined(arch_atomic64_fetch_or_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_or_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_or_acquire)
+	return arch_atomic64_fetch_or_acquire(i, v);
+#elif defined(arch_atomic64_fetch_or_relaxed)
 	s64 ret = arch_atomic64_fetch_or_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_or)
-#define raw_atomic64_fetch_or_acquire arch_atomic64_fetch_or
+	return arch_atomic64_fetch_or(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_or_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_or_release)
-#define raw_atomic64_fetch_or_release arch_atomic64_fetch_or_release
-#elif defined(arch_atomic64_fetch_or_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_or_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_or_release)
+	return arch_atomic64_fetch_or_release(i, v);
+#elif defined(arch_atomic64_fetch_or_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_or_relaxed(i, v);
-}
 #elif defined(arch_atomic64_fetch_or)
-#define raw_atomic64_fetch_or_release arch_atomic64_fetch_or
+	return arch_atomic64_fetch_or(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_or_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_fetch_or_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_fetch_or_relaxed)
-#define raw_atomic64_fetch_or_relaxed arch_atomic64_fetch_or_relaxed
+	return arch_atomic64_fetch_or_relaxed(i, v);
 #elif defined(arch_atomic64_fetch_or)
-#define raw_atomic64_fetch_or_relaxed arch_atomic64_fetch_or
+	return arch_atomic64_fetch_or(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_or_relaxed"
 #endif
+}
 
-#define raw_atomic64_xor arch_atomic64_xor
+static __always_inline void
+raw_atomic64_xor(s64 i, atomic64_t *v)
+{
+	arch_atomic64_xor(i, v);
+}
 
-#if defined(arch_atomic64_fetch_xor)
-#define raw_atomic64_fetch_xor arch_atomic64_fetch_xor
-#elif defined(arch_atomic64_fetch_xor_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_xor(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_xor)
+	return arch_atomic64_fetch_xor(i, v);
+#elif defined(arch_atomic64_fetch_xor_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_fetch_xor_relaxed(i, v);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
 #error "Unable to define raw_atomic64_fetch_xor"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_xor_acquire)
-#define raw_atomic64_fetch_xor_acquire arch_atomic64_fetch_xor_acquire
-#elif defined(arch_atomic64_fetch_xor_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_xor_acquire(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_xor_acquire)
+	return arch_atomic64_fetch_xor_acquire(i, v);
+#elif defined(arch_atomic64_fetch_xor_relaxed)
 	s64 ret = arch_atomic64_fetch_xor_relaxed(i, v);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_fetch_xor)
-#define raw_atomic64_fetch_xor_acquire arch_atomic64_fetch_xor
+	return arch_atomic64_fetch_xor(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_xor_acquire"
 #endif
+}
 
-#if defined(arch_atomic64_fetch_xor_release)
-#define raw_atomic64_fetch_xor_release arch_atomic64_fetch_xor_release
-#elif defined(arch_atomic64_fetch_xor_relaxed)
 static __always_inline s64
 raw_atomic64_fetch_xor_release(s64 i, atomic64_t *v)
 {
+#if defined(arch_atomic64_fetch_xor_release)
+	return arch_atomic64_fetch_xor_release(i, v);
+#elif defined(arch_atomic64_fetch_xor_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_fetch_xor_relaxed(i, v);
-}
 #elif defined(arch_atomic64_fetch_xor)
-#define raw_atomic64_fetch_xor_release arch_atomic64_fetch_xor
+	return arch_atomic64_fetch_xor(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_xor_release"
 #endif
+}
 
+static __always_inline s64
+raw_atomic64_fetch_xor_relaxed(s64 i, atomic64_t *v)
+{
 #if defined(arch_atomic64_fetch_xor_relaxed)
-#define raw_atomic64_fetch_xor_relaxed arch_atomic64_fetch_xor_relaxed
+	return arch_atomic64_fetch_xor_relaxed(i, v);
 #elif defined(arch_atomic64_fetch_xor)
-#define raw_atomic64_fetch_xor_relaxed arch_atomic64_fetch_xor
+	return arch_atomic64_fetch_xor(i, v);
 #else
 #error "Unable to define raw_atomic64_fetch_xor_relaxed"
 #endif
+}
 
-#if defined(arch_atomic64_xchg)
-#define raw_atomic64_xchg arch_atomic64_xchg
-#elif defined(arch_atomic64_xchg_relaxed)
 static __always_inline s64
-raw_atomic64_xchg(atomic64_t *v, s64 i)
+raw_atomic64_xchg(atomic64_t *v, s64 new)
 {
+#if defined(arch_atomic64_xchg)
+	return arch_atomic64_xchg(v, new);
+#elif defined(arch_atomic64_xchg_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
-	ret = arch_atomic64_xchg_relaxed(v, i);
+	ret = arch_atomic64_xchg_relaxed(v, new);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_xchg(atomic64_t *v, s64 new)
-{
 	return raw_xchg(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_xchg_acquire)
-#define raw_atomic64_xchg_acquire arch_atomic64_xchg_acquire
-#elif defined(arch_atomic64_xchg_relaxed)
 static __always_inline s64
-raw_atomic64_xchg_acquire(atomic64_t *v, s64 i)
+raw_atomic64_xchg_acquire(atomic64_t *v, s64 new)
 {
-	s64 ret = arch_atomic64_xchg_relaxed(v, i);
+#if defined(arch_atomic64_xchg_acquire)
+	return arch_atomic64_xchg_acquire(v, new);
+#elif defined(arch_atomic64_xchg_relaxed)
+	s64 ret = arch_atomic64_xchg_relaxed(v, new);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_xchg)
-#define raw_atomic64_xchg_acquire arch_atomic64_xchg
+	return arch_atomic64_xchg(v, new);
 #else
-static __always_inline s64
-raw_atomic64_xchg_acquire(atomic64_t *v, s64 new)
-{
 	return raw_xchg_acquire(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_xchg_release)
-#define raw_atomic64_xchg_release arch_atomic64_xchg_release
-#elif defined(arch_atomic64_xchg_relaxed)
 static __always_inline s64
-raw_atomic64_xchg_release(atomic64_t *v, s64 i)
+raw_atomic64_xchg_release(atomic64_t *v, s64 new)
 {
+#if defined(arch_atomic64_xchg_release)
+	return arch_atomic64_xchg_release(v, new);
+#elif defined(arch_atomic64_xchg_relaxed)
 	__atomic_release_fence();
-	return arch_atomic64_xchg_relaxed(v, i);
-}
+	return arch_atomic64_xchg_relaxed(v, new);
 #elif defined(arch_atomic64_xchg)
-#define raw_atomic64_xchg_release arch_atomic64_xchg
+	return arch_atomic64_xchg(v, new);
 #else
-static __always_inline s64
-raw_atomic64_xchg_release(atomic64_t *v, s64 new)
-{
 	return raw_xchg_release(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_xchg_relaxed)
-#define raw_atomic64_xchg_relaxed arch_atomic64_xchg_relaxed
-#elif defined(arch_atomic64_xchg)
-#define raw_atomic64_xchg_relaxed arch_atomic64_xchg
-#else
 static __always_inline s64
 raw_atomic64_xchg_relaxed(atomic64_t *v, s64 new)
 {
+#if defined(arch_atomic64_xchg_relaxed)
+	return arch_atomic64_xchg_relaxed(v, new);
+#elif defined(arch_atomic64_xchg)
+	return arch_atomic64_xchg(v, new);
+#else
 	return raw_xchg_relaxed(&v->counter, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_cmpxchg)
-#define raw_atomic64_cmpxchg arch_atomic64_cmpxchg
-#elif defined(arch_atomic64_cmpxchg_relaxed)
 static __always_inline s64
 raw_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
 {
+#if defined(arch_atomic64_cmpxchg)
+	return arch_atomic64_cmpxchg(v, old, new);
+#elif defined(arch_atomic64_cmpxchg_relaxed)
 	s64 ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_cmpxchg_relaxed(v, old, new);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline s64
-raw_atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
-{
 	return raw_cmpxchg(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_cmpxchg_acquire)
-#define raw_atomic64_cmpxchg_acquire arch_atomic64_cmpxchg_acquire
-#elif defined(arch_atomic64_cmpxchg_relaxed)
 static __always_inline s64
 raw_atomic64_cmpxchg_acquire(atomic64_t *v, s64 old, s64 new)
 {
+#if defined(arch_atomic64_cmpxchg_acquire)
+	return arch_atomic64_cmpxchg_acquire(v, old, new);
+#elif defined(arch_atomic64_cmpxchg_relaxed)
 	s64 ret = arch_atomic64_cmpxchg_relaxed(v, old, new);
 	__atomic_acquire_fence();
 	return ret;
-}
 #elif defined(arch_atomic64_cmpxchg)
-#define raw_atomic64_cmpxchg_acquire arch_atomic64_cmpxchg
+	return arch_atomic64_cmpxchg(v, old, new);
 #else
-static __always_inline s64
-raw_atomic64_cmpxchg_acquire(atomic64_t *v, s64 old, s64 new)
-{
 	return raw_cmpxchg_acquire(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_cmpxchg_release)
-#define raw_atomic64_cmpxchg_release arch_atomic64_cmpxchg_release
-#elif defined(arch_atomic64_cmpxchg_relaxed)
 static __always_inline s64
 raw_atomic64_cmpxchg_release(atomic64_t *v, s64 old, s64 new)
 {
+#if defined(arch_atomic64_cmpxchg_release)
+	return arch_atomic64_cmpxchg_release(v, old, new);
+#elif defined(arch_atomic64_cmpxchg_relaxed)
 	__atomic_release_fence();
 	return arch_atomic64_cmpxchg_relaxed(v, old, new);
-}
 #elif defined(arch_atomic64_cmpxchg)
-#define raw_atomic64_cmpxchg_release arch_atomic64_cmpxchg
+	return arch_atomic64_cmpxchg(v, old, new);
 #else
-static __always_inline s64
-raw_atomic64_cmpxchg_release(atomic64_t *v, s64 old, s64 new)
-{
 	return raw_cmpxchg_release(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_cmpxchg_relaxed)
-#define raw_atomic64_cmpxchg_relaxed arch_atomic64_cmpxchg_relaxed
-#elif defined(arch_atomic64_cmpxchg)
-#define raw_atomic64_cmpxchg_relaxed arch_atomic64_cmpxchg
-#else
 static __always_inline s64
 raw_atomic64_cmpxchg_relaxed(atomic64_t *v, s64 old, s64 new)
 {
+#if defined(arch_atomic64_cmpxchg_relaxed)
+	return arch_atomic64_cmpxchg_relaxed(v, old, new);
+#elif defined(arch_atomic64_cmpxchg)
+	return arch_atomic64_cmpxchg(v, old, new);
+#else
 	return raw_cmpxchg_relaxed(&v->counter, old, new);
-}
 #endif
+}
 
-#if defined(arch_atomic64_try_cmpxchg)
-#define raw_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg
-#elif defined(arch_atomic64_try_cmpxchg_relaxed)
 static __always_inline bool
 raw_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
 {
+#if defined(arch_atomic64_try_cmpxchg)
+	return arch_atomic64_try_cmpxchg(v, old, new);
+#elif defined(arch_atomic64_try_cmpxchg_relaxed)
 	bool ret;
 	__atomic_pre_full_fence();
 	ret = arch_atomic64_try_cmpxchg_relaxed(v, old, new);
 	__atomic_post_full_fence();
 	return ret;
-}
 #else
-static __always_inline bool
-raw_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
-{
 	s64 r, o = *old;
 	r = raw_atomic64_cmpxchg(v, o, new);
 	if (unlikely(r != o))
 		*old = r;
 	return likely(r == o);
-}
 #endif
+}
 
-#if defined(arch_atomic64_try_cmpxchg_acquire)
-#define raw_atomic64_try_cmpxchg_acquire arch_atomic64_try_cmpxchg_acquire
-#elif defined(arch_atomic64_try_cmpxchg_relaxed)
 static __always_inline bool
 raw_atomic64_try_cmpxchg_acquire(atomic64_t *v, s64 *old, s64 new)
 {
+#if defined(arch_atomic64_try_cmpxchg_acquire)
+	return arch_atomic64_try_cmpxchg_acquire(v, old, new);
+#elif defined(arch_atomic64_try_cmpxchg_relaxed)
 	bool ret = arch_atomic64_try_cmpxchg_relaxed(v, old, new);
 	__atomic_acquire_fence();
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

include/linux/atomic/atomic-instrumented.h +25 -25

50 lines not on this page · raw message

include/linux/atomic/atomic-long.h +13 -13
index 63e0b4078ebd5..f564f71ff8afc 100644

Lines of this file are not shown. View the raw message.

scripts/atomic/atomics.tbl +1 -1
index 85ca8d9b5c279..903946cbf1b3e 100644

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/acquire +0 -4
index b0f732a5c46ef..4da0cab3604e2 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/add_negative +0 -4
index 16876118019ec..1d3d4ab3a9d29 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/add_unless +0 -4
index 88593e28b1637..95ecb2b7405be 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/andnot +0 -4
index 5b83bb63f7284..66760457e67a5 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/cmpxchg +0 -4
index 312ee67f1743e..1c8507f62e049 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec +0 -4
index a660ac65994bd..60d286d40300f 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_and_test +0 -4
index 521dfcae03f24..3a0278e0ddd73 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_if_positive +0 -4
index 7acb205e6ce35..f65c11b4b85bd 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/dec_unless_positive +0 -4
index bcb4f27945eaa..d025361d7b85a 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/fence +0 -4
index 067eea553f5e0..40d5b397658f7 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/fetch_add_unless +0 -4
index c18b940153dfd..8db7e9e17facf 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc +0 -4
index 7d838f0b66391..56c770f5919c0 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_and_test +0 -4
index de25aebee715d..7d16a10f2257e 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_not_zero +0 -4
index e02206d017f62..1fcef1e55bc97 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/inc_unless_negative +0 -4
index 7b85cc5b00d2b..7b4b09868842d 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/read_acquire +0 -4
index 26d15ad92d043..e319862d2f1a5 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/release +0 -4
index cbbff708129b8..1e6daf57b4ba5 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/set_release +0 -4
index 104693bc3c660..16a374ae6bb16 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/sub_and_test +0 -4
index 8975a496d495c..d1f746fe0ca4d 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/try_cmpxchg +0 -4
index 4c911a6cced94..d4da82092baf7 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/fallbacks/xchg +0 -4
index bdd788aa575ff..e4def1e0d0926 100755

Lines of this file are not shown. View the raw message.

scripts/atomic/gen-atomic-fallback.sh +19 -7
index 86aca4f9f315a..2b470d31e3539 100755

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.30.2

Part of the diff is not shown here. View the raw message.