locking/atomic: generic arch__atomic_long_*() and arch_ bitops
Subsystems (3)
Mark Rutland
reviewercover locking/atomic: generic arch__atomic_long_*() and arch_ bitops
These patches (based on v5.14-rc1) add arch_*() forms of the atomic_long_*() operations for all architectures, and build atop this to have the generic bitops provide arch_*() forms, allowing for safe usage in noinstr code. For example, I'd like to be able to use these in arm64's entry code.
The first three patches cleanup the current code, simplifiying this by removing some ARCH_ATOMIC remnants and centralizing the generated headers under include/linux/atomic/ for consistency.
Patch four adds arch_atomic_long_*(). This is almost entirely scripted (adding ~600 lines), with the only manual change being to the order of includes in include/linux/atomic.h, as we must now include atomic-long.h before atomic-instrumented.h.
Patch five updates the generic bitop implementaton to provide arch_* forms. Since the non-atomic ops can be implicitly instrumented, some logic is added to omit the redundant checks.
I've pushed the patches to my atomics/arch-atomic-bitops branch [1,2].
I've given this some light build and boot testing so far.
Thanks,
Mark.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=atomics/arch-atomic-bitops
[2] git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git atomics/arch-atomic-bitops
Mark Rutland (5):
locking/atomic: simplify ifdef generation
locking/atomic: remove ARCH_ATOMIC remanants
locking/atomic: centralize generated headers
locking/atomic: add arch_atomic_long*()
locking/atomic: add generic arch_*() bitops
include/asm-generic/atomic-long.h | 1014 --------------------
include/asm-generic/bitops/atomic.h | 32 +-
.../asm-generic/bitops/instrumented-non-atomic.h | 21 +-
include/asm-generic/bitops/lock.h | 39 +-
include/asm-generic/bitops/non-atomic.h | 41 +-
include/linux/atomic.h | 7 +-
include/linux/{ => atomic}/atomic-arch-fallback.h | 0
.../atomic}/atomic-instrumented.h | 586 ++++++++++-
include/linux/atomic/atomic-long.h | 1014 ++++++++++++++++++++
scripts/atomic/check-atomics.sh | 6 +-
scripts/atomic/fallbacks/acquire | 4 +-
scripts/atomic/fallbacks/add_negative | 6 +-
scripts/atomic/fallbacks/add_unless | 6 +-
scripts/atomic/fallbacks/andnot | 4 +-
scripts/atomic/fallbacks/dec | 4 +-
scripts/atomic/fallbacks/dec_and_test | 6 +-
scripts/atomic/fallbacks/dec_if_positive | 6 +-
scripts/atomic/fallbacks/dec_unless_positive | 6 +-
scripts/atomic/fallbacks/fence | 4 +-
scripts/atomic/fallbacks/fetch_add_unless | 8 +-
scripts/atomic/fallbacks/inc | 4 +-
scripts/atomic/fallbacks/inc_and_test | 6 +-
scripts/atomic/fallbacks/inc_not_zero | 6 +-
scripts/atomic/fallbacks/inc_unless_negative | 6 +-
scripts/atomic/fallbacks/read_acquire | 2 +-
scripts/atomic/fallbacks/release | 4 +-
scripts/atomic/fallbacks/set_release | 2 +-
scripts/atomic/fallbacks/sub_and_test | 6 +-
scripts/atomic/fallbacks/try_cmpxchg | 4 +-
scripts/atomic/gen-atomic-fallback.sh | 68 +-
scripts/atomic/gen-atomic-instrumented.sh | 11 +-
scripts/atomic/gen-atomic-long.sh | 10 +-
scripts/atomic/gen-atomics.sh | 6 +-
33 files changed, 1772 insertions(+), 1177 deletions(-)
delete mode 100644 include/asm-generic/atomic-long.h
rename include/linux/{ => atomic}/atomic-arch-fallback.h (100%)
rename include/{asm-generic => linux/atomic}/atomic-instrumented.h (68%)
create mode 100644 include/linux/atomic/atomic-long.hSignature · 1 line
2.11.0
Mark Rutland
reviewerPATCH 1/5 locking/atomic: simplify ifdef generation
In gen-atomic-fallback.sh's gen_proto_order_variants(), we generate some ifdeferry with:
| local basename="${arch}${atomic}_${pfx}${name}${sfx}"
| ...
| printf "#ifdef ${basename}\n"
| ...
| printf "#endif /* ${arch}${atomic}_${pfx}${name}${sfx} */\n\n"For clarity, use ${basename} for both sides, rather than open-coding the string generation.
There is no change to any of the generated headers as a result of this patch.
--- scripts/atomic/gen-atomic-fallback.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/atomic/gen-atomic-fallback.sh +1 -1
@@ -128,7 +128,7 @@ gen_proto_order_variants() gen_basic_fallbacks "${basename}" if [ ! -z "${template}" ]; then - printf "#endif /* ${arch}${atomic}_${pfx}${name}${sfx} */\n\n" + printf "#endif /* ${basename} */\n\n" gen_proto_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "" "$@" gen_proto_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_acquire" "$@" gen_proto_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_release" "$@"
Signature · 1 line
2.11.0
[tip: locking/core] locking/atomic: simplify ifdef generation
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 47401d94947d507ff9f33fccf490baf47638fb69 Gitweb: https://git.kernel.org/tip/47401d94947d507ff9f33fccf490baf47638fb69 Author: Mark Rutland <[email protected]> AuthorDate: Tue, 13 Jul 2021 11:52:49 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 16 Jul 2021 18:46:44 +02:00
locking/atomic: simplify ifdef generation
In gen-atomic-fallback.sh's gen_proto_order_variants(), we generate some ifdeferry with:
| local basename="${arch}${atomic}_${pfx}${name}${sfx}"
| ...
| printf "#ifdef ${basename}\n"
| ...
| printf "#endif /* ${arch}${atomic}_${pfx}${name}${sfx} */\n\n"For clarity, use ${basename} for both sides, rather than open-coding the string generation.
There is no change to any of the generated headers as a result of this patch.
--- scripts/atomic/gen-atomic-fallback.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/atomic/gen-atomic-fallback.sh +1 -1
@@ -128,7 +128,7 @@ gen_proto_order_variants() gen_basic_fallbacks "${basename}" if [ ! -z "${template}" ]; then - printf "#endif /* ${arch}${atomic}_${pfx}${name}${sfx} */\n\n" + printf "#endif /* ${basename} */\n\n" gen_proto_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "" "$@" gen_proto_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_acquire" "$@" gen_proto_fallback "${meta}" "${pfx}" "${name}" "${sfx}" "_release" "$@"
Mark Rutland
reviewerPATCH 2/5 locking/atomic: remove ARCH_ATOMIC remanants
In reply to Mark RutlandNow that gen-atomic-fallback.sh is only used to generate the arch_* fallbacks, we don't need to also generate the non-arch_* forms, and can removethe infrastructure this needed.
There is no change to any of the generated headers as a result of this patch.
--- scripts/atomic/fallbacks/acquire | 4 +- scripts/atomic/fallbacks/add_negative | 6 +-- scripts/atomic/fallbacks/add_unless | 6 +-- scripts/atomic/fallbacks/andnot | 4 +- scripts/atomic/fallbacks/dec | 4 +- scripts/atomic/fallbacks/dec_and_test | 6 +-- scripts/atomic/fallbacks/dec_if_positive | 6 +-- scripts/atomic/fallbacks/dec_unless_positive | 6 +-- scripts/atomic/fallbacks/fence | 4 +- scripts/atomic/fallbacks/fetch_add_unless | 8 ++-- scripts/atomic/fallbacks/inc | 4 +- scripts/atomic/fallbacks/inc_and_test | 6 +-- scripts/atomic/fallbacks/inc_not_zero | 6 +-- scripts/atomic/fallbacks/inc_unless_negative | 6 +-- scripts/atomic/fallbacks/read_acquire | 2 +- scripts/atomic/fallbacks/release | 4 +- scripts/atomic/fallbacks/set_release | 2 +- scripts/atomic/fallbacks/sub_and_test | 6 +-- scripts/atomic/fallbacks/try_cmpxchg | 4 +- scripts/atomic/gen-atomic-fallback.sh | 66 ++++++++++------------------ scripts/atomic/gen-atomics.sh | 2 +- 21 files changed, 71 insertions(+), 91 deletions(-)
scripts/atomic/fallbacks/acquire +2 -2
@@ -1,8 +1,8 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}${name}${sfx}_acquire(${params}) +arch_${atomic}_${pfx}${name}${sfx}_acquire(${params}) { - ${ret} ret = ${arch}${atomic}_${pfx}${name}${sfx}_relaxed(${args}); + ${ret} ret = arch_${atomic}_${pfx}${name}${sfx}_relaxed(${args}); __atomic_acquire_fence(); return ret; }
scripts/atomic/fallbacks/add_negative +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_add_negative - add and test if negative + * arch_${atomic}_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type ${atomic}_t *
@@ -9,8 +9,8 @@ cat <<EOF * result is greater than or equal to zero. */ static __always_inline bool -${arch}${atomic}_add_negative(${int} i, ${atomic}_t *v) +arch_${atomic}_add_negative(${int} i, ${atomic}_t *v) { - return ${arch}${atomic}_add_return(i, v) < 0; + return arch_${atomic}_add_return(i, v) < 0; } EOF
scripts/atomic/fallbacks/add_unless +3 -3
@@ -1,6 +1,6 @@ cat << EOF /** - * ${arch}${atomic}_add_unless - add unless the number is already a given value + * arch_${atomic}_add_unless - add unless the number is already a given value * @v: pointer of type ${atomic}_t * @a: the amount to add to v... * @u: ...unless v is equal to u.
@@ -9,8 +9,8 @@ cat << EOF * Returns true if the addition was done. */ static __always_inline bool -${arch}${atomic}_add_unless(${atomic}_t *v, ${int} a, ${int} u) +arch_${atomic}_add_unless(${atomic}_t *v, ${int} a, ${int} u) { - return ${arch}${atomic}_fetch_add_unless(v, a, u) != u; + return arch_${atomic}_fetch_add_unless(v, a, u) != u; } EOF
scripts/atomic/fallbacks/andnot +2 -2
@@ -1,7 +1,7 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}andnot${sfx}${order}(${int} i, ${atomic}_t *v) +arch_${atomic}_${pfx}andnot${sfx}${order}(${int} i, ${atomic}_t *v) { - ${retstmt}${arch}${atomic}_${pfx}and${sfx}${order}(~i, v); + ${retstmt}arch_${atomic}_${pfx}and${sfx}${order}(~i, v); } EOF
scripts/atomic/fallbacks/dec +2 -2
@@ -1,7 +1,7 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}dec${sfx}${order}(${atomic}_t *v) +arch_${atomic}_${pfx}dec${sfx}${order}(${atomic}_t *v) { - ${retstmt}${arch}${atomic}_${pfx}sub${sfx}${order}(1, v); + ${retstmt}arch_${atomic}_${pfx}sub${sfx}${order}(1, v); } EOF
scripts/atomic/fallbacks/dec_and_test +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_dec_and_test - decrement and test + * arch_${atomic}_dec_and_test - decrement and test * @v: pointer of type ${atomic}_t * * Atomically decrements @v by 1 and
@@ -8,8 +8,8 @@ cat <<EOF * cases. */ static __always_inline bool -${arch}${atomic}_dec_and_test(${atomic}_t *v) +arch_${atomic}_dec_and_test(${atomic}_t *v) { - return ${arch}${atomic}_dec_return(v) == 0; + return arch_${atomic}_dec_return(v) == 0; } EOF
scripts/atomic/fallbacks/dec_if_positive +3 -3
@@ -1,14 +1,14 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_dec_if_positive(${atomic}_t *v) +arch_${atomic}_dec_if_positive(${atomic}_t *v) { - ${int} dec, c = ${arch}${atomic}_read(v); + ${int} dec, c = arch_${atomic}_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, dec)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, dec)); return dec; }
scripts/atomic/fallbacks/dec_unless_positive +3 -3
@@ -1,13 +1,13 @@ cat <<EOF static __always_inline bool -${arch}${atomic}_dec_unless_positive(${atomic}_t *v) +arch_${atomic}_dec_unless_positive(${atomic}_t *v) { - ${int} c = ${arch}${atomic}_read(v); + ${int} c = arch_${atomic}_read(v); do { if (unlikely(c > 0)) return false; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, c - 1)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, c - 1)); return true; }
scripts/atomic/fallbacks/fence +2 -2
@@ -1,10 +1,10 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}${name}${sfx}(${params}) +arch_${atomic}_${pfx}${name}${sfx}(${params}) { ${ret} ret; __atomic_pre_full_fence(); - ret = ${arch}${atomic}_${pfx}${name}${sfx}_relaxed(${args}); + ret = arch_${atomic}_${pfx}${name}${sfx}_relaxed(${args}); __atomic_post_full_fence(); return ret; }
scripts/atomic/fallbacks/fetch_add_unless +4 -4
@@ -1,6 +1,6 @@ cat << EOF /** - * ${arch}${atomic}_fetch_add_unless - add unless the number is already a given value + * arch_${atomic}_fetch_add_unless - add unless the number is already a given value * @v: pointer of type ${atomic}_t * @a: the amount to add to v... * @u: ...unless v is equal to u.
@@ -9,14 +9,14 @@ cat << EOF * Returns original value of @v */ static __always_inline ${int} -${arch}${atomic}_fetch_add_unless(${atomic}_t *v, ${int} a, ${int} u) +arch_${atomic}_fetch_add_unless(${atomic}_t *v, ${int} a, ${int} u) { - ${int} c = ${arch}${atomic}_read(v); + ${int} c = arch_${atomic}_read(v); do { if (unlikely(c == u)) break; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, c + a)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, c + a)); return c; }
scripts/atomic/fallbacks/inc +2 -2
@@ -1,7 +1,7 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}inc${sfx}${order}(${atomic}_t *v) +arch_${atomic}_${pfx}inc${sfx}${order}(${atomic}_t *v) { - ${retstmt}${arch}${atomic}_${pfx}add${sfx}${order}(1, v); + ${retstmt}arch_${atomic}_${pfx}add${sfx}${order}(1, v); } EOF
scripts/atomic/fallbacks/inc_and_test +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_inc_and_test - increment and test + * arch_${atomic}_inc_and_test - increment and test * @v: pointer of type ${atomic}_t * * Atomically increments @v by 1
@@ -8,8 +8,8 @@ cat <<EOF * other cases. */ static __always_inline bool -${arch}${atomic}_inc_and_test(${atomic}_t *v) +arch_${atomic}_inc_and_test(${atomic}_t *v) { - return ${arch}${atomic}_inc_return(v) == 0; + return arch_${atomic}_inc_return(v) == 0; } EOF
scripts/atomic/fallbacks/inc_not_zero +3 -3
@@ -1,14 +1,14 @@ cat <<EOF /** - * ${arch}${atomic}_inc_not_zero - increment unless the number is zero + * arch_${atomic}_inc_not_zero - increment unless the number is zero * @v: pointer of type ${atomic}_t * * Atomically increments @v by 1, if @v is non-zero. * Returns true if the increment was done. */ static __always_inline bool -${arch}${atomic}_inc_not_zero(${atomic}_t *v) +arch_${atomic}_inc_not_zero(${atomic}_t *v) { - return ${arch}${atomic}_add_unless(v, 1, 0); + return arch_${atomic}_add_unless(v, 1, 0); } EOF
scripts/atomic/fallbacks/inc_unless_negative +3 -3
@@ -1,13 +1,13 @@ cat <<EOF static __always_inline bool -${arch}${atomic}_inc_unless_negative(${atomic}_t *v) +arch_${atomic}_inc_unless_negative(${atomic}_t *v) { - ${int} c = ${arch}${atomic}_read(v); + ${int} c = arch_${atomic}_read(v); do { if (unlikely(c < 0)) return false; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, c + 1)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, c + 1)); return true; }
scripts/atomic/fallbacks/read_acquire +1 -1
@@ -1,6 +1,6 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_read_acquire(const ${atomic}_t *v) +arch_${atomic}_read_acquire(const ${atomic}_t *v) { return smp_load_acquire(&(v)->counter); }
scripts/atomic/fallbacks/release +2 -2
@@ -1,8 +1,8 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}${name}${sfx}_release(${params}) +arch_${atomic}_${pfx}${name}${sfx}_release(${params}) { __atomic_release_fence(); - ${retstmt}${arch}${atomic}_${pfx}${name}${sfx}_relaxed(${args}); + ${retstmt}arch_${atomic}_${pfx}${name}${sfx}_relaxed(${args}); } EOF
scripts/atomic/fallbacks/set_release +1 -1
@@ -1,6 +1,6 @@ cat <<EOF static __always_inline void -${arch}${atomic}_set_release(${atomic}_t *v, ${int} i) +arch_${atomic}_set_release(${atomic}_t *v, ${int} i) { smp_store_release(&(v)->counter, i); }
scripts/atomic/fallbacks/sub_and_test +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_sub_and_test - subtract value from variable and test result + * arch_${atomic}_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type ${atomic}_t *
@@ -9,8 +9,8 @@ cat <<EOF * other cases. */ static __always_inline bool -${arch}${atomic}_sub_and_test(${int} i, ${atomic}_t *v) +arch_${atomic}_sub_and_test(${int} i, ${atomic}_t *v) { - return ${arch}${atomic}_sub_return(i, v) == 0; + return arch_${atomic}_sub_return(i, v) == 0; } EOF
scripts/atomic/fallbacks/try_cmpxchg +2 -2
@@ -1,9 +1,9 @@ cat <<EOF static __always_inline bool -${arch}${atomic}_try_cmpxchg${order}(${atomic}_t *v, ${int} *old, ${int} new) +arch_${atomic}_try_cmpxchg${order}(${atomic}_t *v, ${int} *old, ${int} new) { ${int} r, o = *old; - r = ${arch}${atomic}_cmpxchg${order}(v, o, new); + r = arch_${atomic}_cmpxchg${order}(v, o, new); if (unlikely(r != o)) *old = r; return likely(r == o);
scripts/atomic/gen-atomic-fallback.sh +23 -43
155 lines not on this page · raw message
scripts/atomic/gen-atomics.sh +1 -1
@@ -10,7 +10,7 @@ LINUXDIR=${ATOMICDIR}/../.. cat <<EOF | gen-atomic-instrumented.sh asm-generic/atomic-instrumented.h gen-atomic-long.sh asm-generic/atomic-long.h -gen-atomic-fallback.sh linux/atomic-arch-fallback.h arch_ +gen-atomic-fallback.sh linux/atomic-arch-fallback.h EOF while read script header args; do /bin/sh ${ATOMICDIR}/${script} ${ATOMICTBL} ${args} > ${LINUXDIR}/include/${header}
Signature · 1 line
2.11.0
[tip: locking/core] locking/atomic: remove ARCH_ATOMIC remanants
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: f3e615b4db1fb7034f1d76dc307b77cc848f040e Gitweb: https://git.kernel.org/tip/f3e615b4db1fb7034f1d76dc307b77cc848f040e Author: Mark Rutland <[email protected]> AuthorDate: Tue, 13 Jul 2021 11:52:50 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 16 Jul 2021 18:46:44 +02:00
locking/atomic: remove ARCH_ATOMIC remanants
Now that gen-atomic-fallback.sh is only used to generate the arch_* fallbacks, we don't need to also generate the non-arch_* forms, and can removethe infrastructure this needed.
There is no change to any of the generated headers as a result of this patch.
--- scripts/atomic/fallbacks/acquire | 4 +- scripts/atomic/fallbacks/add_negative | 6 +- scripts/atomic/fallbacks/add_unless | 6 +- scripts/atomic/fallbacks/andnot | 4 +- scripts/atomic/fallbacks/dec | 4 +- scripts/atomic/fallbacks/dec_and_test | 6 +- scripts/atomic/fallbacks/dec_if_positive | 6 +- scripts/atomic/fallbacks/dec_unless_positive | 6 +- scripts/atomic/fallbacks/fence | 4 +- scripts/atomic/fallbacks/fetch_add_unless | 8 +- scripts/atomic/fallbacks/inc | 4 +- scripts/atomic/fallbacks/inc_and_test | 6 +- scripts/atomic/fallbacks/inc_not_zero | 6 +- scripts/atomic/fallbacks/inc_unless_negative | 6 +- scripts/atomic/fallbacks/read_acquire | 2 +- scripts/atomic/fallbacks/release | 4 +- scripts/atomic/fallbacks/set_release | 2 +- scripts/atomic/fallbacks/sub_and_test | 6 +- scripts/atomic/fallbacks/try_cmpxchg | 4 +- scripts/atomic/gen-atomic-fallback.sh | 66 ++++++------------- scripts/atomic/gen-atomics.sh | 2 +- 21 files changed, 71 insertions(+), 91 deletions(-)
scripts/atomic/fallbacks/acquire +2 -2
@@ -1,8 +1,8 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}${name}${sfx}_acquire(${params}) +arch_${atomic}_${pfx}${name}${sfx}_acquire(${params}) { - ${ret} ret = ${arch}${atomic}_${pfx}${name}${sfx}_relaxed(${args}); + ${ret} ret = arch_${atomic}_${pfx}${name}${sfx}_relaxed(${args}); __atomic_acquire_fence(); return ret; }
scripts/atomic/fallbacks/add_negative +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_add_negative - add and test if negative + * arch_${atomic}_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type ${atomic}_t *
@@ -9,8 +9,8 @@ cat <<EOF * result is greater than or equal to zero. */ static __always_inline bool -${arch}${atomic}_add_negative(${int} i, ${atomic}_t *v) +arch_${atomic}_add_negative(${int} i, ${atomic}_t *v) { - return ${arch}${atomic}_add_return(i, v) < 0; + return arch_${atomic}_add_return(i, v) < 0; } EOF
scripts/atomic/fallbacks/add_unless +3 -3
@@ -1,6 +1,6 @@ cat << EOF /** - * ${arch}${atomic}_add_unless - add unless the number is already a given value + * arch_${atomic}_add_unless - add unless the number is already a given value * @v: pointer of type ${atomic}_t * @a: the amount to add to v... * @u: ...unless v is equal to u.
@@ -9,8 +9,8 @@ cat << EOF * Returns true if the addition was done. */ static __always_inline bool -${arch}${atomic}_add_unless(${atomic}_t *v, ${int} a, ${int} u) +arch_${atomic}_add_unless(${atomic}_t *v, ${int} a, ${int} u) { - return ${arch}${atomic}_fetch_add_unless(v, a, u) != u; + return arch_${atomic}_fetch_add_unless(v, a, u) != u; } EOF
scripts/atomic/fallbacks/andnot +2 -2
@@ -1,7 +1,7 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}andnot${sfx}${order}(${int} i, ${atomic}_t *v) +arch_${atomic}_${pfx}andnot${sfx}${order}(${int} i, ${atomic}_t *v) { - ${retstmt}${arch}${atomic}_${pfx}and${sfx}${order}(~i, v); + ${retstmt}arch_${atomic}_${pfx}and${sfx}${order}(~i, v); } EOF
scripts/atomic/fallbacks/dec +2 -2
@@ -1,7 +1,7 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}dec${sfx}${order}(${atomic}_t *v) +arch_${atomic}_${pfx}dec${sfx}${order}(${atomic}_t *v) { - ${retstmt}${arch}${atomic}_${pfx}sub${sfx}${order}(1, v); + ${retstmt}arch_${atomic}_${pfx}sub${sfx}${order}(1, v); } EOF
scripts/atomic/fallbacks/dec_and_test +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_dec_and_test - decrement and test + * arch_${atomic}_dec_and_test - decrement and test * @v: pointer of type ${atomic}_t * * Atomically decrements @v by 1 and
@@ -8,8 +8,8 @@ cat <<EOF * cases. */ static __always_inline bool -${arch}${atomic}_dec_and_test(${atomic}_t *v) +arch_${atomic}_dec_and_test(${atomic}_t *v) { - return ${arch}${atomic}_dec_return(v) == 0; + return arch_${atomic}_dec_return(v) == 0; } EOF
scripts/atomic/fallbacks/dec_if_positive +3 -3
@@ -1,14 +1,14 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_dec_if_positive(${atomic}_t *v) +arch_${atomic}_dec_if_positive(${atomic}_t *v) { - ${int} dec, c = ${arch}${atomic}_read(v); + ${int} dec, c = arch_${atomic}_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, dec)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, dec)); return dec; }
scripts/atomic/fallbacks/dec_unless_positive +3 -3
@@ -1,13 +1,13 @@ cat <<EOF static __always_inline bool -${arch}${atomic}_dec_unless_positive(${atomic}_t *v) +arch_${atomic}_dec_unless_positive(${atomic}_t *v) { - ${int} c = ${arch}${atomic}_read(v); + ${int} c = arch_${atomic}_read(v); do { if (unlikely(c > 0)) return false; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, c - 1)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, c - 1)); return true; }
scripts/atomic/fallbacks/fence +2 -2
@@ -1,10 +1,10 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}${name}${sfx}(${params}) +arch_${atomic}_${pfx}${name}${sfx}(${params}) { ${ret} ret; __atomic_pre_full_fence(); - ret = ${arch}${atomic}_${pfx}${name}${sfx}_relaxed(${args}); + ret = arch_${atomic}_${pfx}${name}${sfx}_relaxed(${args}); __atomic_post_full_fence(); return ret; }
scripts/atomic/fallbacks/fetch_add_unless +4 -4
@@ -1,6 +1,6 @@ cat << EOF /** - * ${arch}${atomic}_fetch_add_unless - add unless the number is already a given value + * arch_${atomic}_fetch_add_unless - add unless the number is already a given value * @v: pointer of type ${atomic}_t * @a: the amount to add to v... * @u: ...unless v is equal to u.
@@ -9,14 +9,14 @@ cat << EOF * Returns original value of @v */ static __always_inline ${int} -${arch}${atomic}_fetch_add_unless(${atomic}_t *v, ${int} a, ${int} u) +arch_${atomic}_fetch_add_unless(${atomic}_t *v, ${int} a, ${int} u) { - ${int} c = ${arch}${atomic}_read(v); + ${int} c = arch_${atomic}_read(v); do { if (unlikely(c == u)) break; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, c + a)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, c + a)); return c; }
scripts/atomic/fallbacks/inc +2 -2
@@ -1,7 +1,7 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}inc${sfx}${order}(${atomic}_t *v) +arch_${atomic}_${pfx}inc${sfx}${order}(${atomic}_t *v) { - ${retstmt}${arch}${atomic}_${pfx}add${sfx}${order}(1, v); + ${retstmt}arch_${atomic}_${pfx}add${sfx}${order}(1, v); } EOF
scripts/atomic/fallbacks/inc_and_test +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_inc_and_test - increment and test + * arch_${atomic}_inc_and_test - increment and test * @v: pointer of type ${atomic}_t * * Atomically increments @v by 1
@@ -8,8 +8,8 @@ cat <<EOF * other cases. */ static __always_inline bool -${arch}${atomic}_inc_and_test(${atomic}_t *v) +arch_${atomic}_inc_and_test(${atomic}_t *v) { - return ${arch}${atomic}_inc_return(v) == 0; + return arch_${atomic}_inc_return(v) == 0; } EOF
scripts/atomic/fallbacks/inc_not_zero +3 -3
@@ -1,14 +1,14 @@ cat <<EOF /** - * ${arch}${atomic}_inc_not_zero - increment unless the number is zero + * arch_${atomic}_inc_not_zero - increment unless the number is zero * @v: pointer of type ${atomic}_t * * Atomically increments @v by 1, if @v is non-zero. * Returns true if the increment was done. */ static __always_inline bool -${arch}${atomic}_inc_not_zero(${atomic}_t *v) +arch_${atomic}_inc_not_zero(${atomic}_t *v) { - return ${arch}${atomic}_add_unless(v, 1, 0); + return arch_${atomic}_add_unless(v, 1, 0); } EOF
scripts/atomic/fallbacks/inc_unless_negative +3 -3
@@ -1,13 +1,13 @@ cat <<EOF static __always_inline bool -${arch}${atomic}_inc_unless_negative(${atomic}_t *v) +arch_${atomic}_inc_unless_negative(${atomic}_t *v) { - ${int} c = ${arch}${atomic}_read(v); + ${int} c = arch_${atomic}_read(v); do { if (unlikely(c < 0)) return false; - } while (!${arch}${atomic}_try_cmpxchg(v, &c, c + 1)); + } while (!arch_${atomic}_try_cmpxchg(v, &c, c + 1)); return true; }
scripts/atomic/fallbacks/read_acquire +1 -1
@@ -1,6 +1,6 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_read_acquire(const ${atomic}_t *v) +arch_${atomic}_read_acquire(const ${atomic}_t *v) { return smp_load_acquire(&(v)->counter); }
scripts/atomic/fallbacks/release +2 -2
@@ -1,8 +1,8 @@ cat <<EOF static __always_inline ${ret} -${arch}${atomic}_${pfx}${name}${sfx}_release(${params}) +arch_${atomic}_${pfx}${name}${sfx}_release(${params}) { __atomic_release_fence(); - ${retstmt}${arch}${atomic}_${pfx}${name}${sfx}_relaxed(${args}); + ${retstmt}arch_${atomic}_${pfx}${name}${sfx}_relaxed(${args}); } EOF
scripts/atomic/fallbacks/set_release +1 -1
@@ -1,6 +1,6 @@ cat <<EOF static __always_inline void -${arch}${atomic}_set_release(${atomic}_t *v, ${int} i) +arch_${atomic}_set_release(${atomic}_t *v, ${int} i) { smp_store_release(&(v)->counter, i); }
scripts/atomic/fallbacks/sub_and_test +3 -3
@@ -1,6 +1,6 @@ cat <<EOF /** - * ${arch}${atomic}_sub_and_test - subtract value from variable and test result + * arch_${atomic}_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type ${atomic}_t *
@@ -9,8 +9,8 @@ cat <<EOF * other cases. */ static __always_inline bool -${arch}${atomic}_sub_and_test(${int} i, ${atomic}_t *v) +arch_${atomic}_sub_and_test(${int} i, ${atomic}_t *v) { - return ${arch}${atomic}_sub_return(i, v) == 0; + return arch_${atomic}_sub_return(i, v) == 0; } EOF
scripts/atomic/fallbacks/try_cmpxchg +2 -2
@@ -1,9 +1,9 @@ cat <<EOF static __always_inline bool -${arch}${atomic}_try_cmpxchg${order}(${atomic}_t *v, ${int} *old, ${int} new) +arch_${atomic}_try_cmpxchg${order}(${atomic}_t *v, ${int} *old, ${int} new) { ${int} r, o = *old; - r = ${arch}${atomic}_cmpxchg${order}(v, o, new); + r = arch_${atomic}_cmpxchg${order}(v, o, new); if (unlikely(r != o)) *old = r; return likely(r == o);
scripts/atomic/gen-atomic-fallback.sh +23 -43
155 lines not on this page · raw message
scripts/atomic/gen-atomics.sh +1 -1
@@ -10,7 +10,7 @@ LINUXDIR=${ATOMICDIR}/../.. cat <<EOF | gen-atomic-instrumented.sh asm-generic/atomic-instrumented.h gen-atomic-long.sh asm-generic/atomic-long.h -gen-atomic-fallback.sh linux/atomic-arch-fallback.h arch_ +gen-atomic-fallback.sh linux/atomic-arch-fallback.h EOF while read script header args; do /bin/sh ${ATOMICDIR}/${script} ${ATOMICTBL} ${args} > ${LINUXDIR}/include/${header}
Mark Rutland
reviewerPATCH 3/5 locking/atomic: centralize generated headers
In reply to Mark RutlandThe generated atomic headers are only intended to be included directly by <linux/atomic.h>, but are spread across include/linux/ and include/asm-generic/, where people mnay be encouraged to include them.
This patch centralizes them under include/linux/atomic/.
Other than the header guards and hashes, there is no change to any of the generated headers as a result of this patch.
---
include/linux/atomic.h | 7 +++----
include/linux/{ => atomic}/atomic-arch-fallback.h | 0
include/{asm-generic => linux/atomic}/atomic-instrumented.h | 8 ++++----
include/{asm-generic => linux/atomic}/atomic-long.h | 8 ++++----
scripts/atomic/check-atomics.sh | 6 +++---
scripts/atomic/gen-atomic-instrumented.sh | 6 +++---
scripts/atomic/gen-atomic-long.sh | 6 +++---
scripts/atomic/gen-atomics.sh | 6 +++---
8 files changed, 23 insertions(+), 24 deletions(-)
rename include/linux/{ => atomic}/atomic-arch-fallback.h (100%)
rename include/{asm-generic => linux/atomic}/atomic-instrumented.h (99%)
rename include/{asm-generic => linux/atomic}/atomic-long.h (99%)include/linux/atomic.h +3 -4
@@ -77,9 +77,8 @@ __ret; \ }) -#include <linux/atomic-arch-fallback.h> -#include <asm-generic/atomic-instrumented.h> - -#include <asm-generic/atomic-long.h> +#include <linux/atomic/atomic-arch-fallback.h> +#include <linux/atomic/atomic-instrumented.h> +#include <linux/atomic/atomic-long.h> #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-arch-fallback.h renamed from include/linux/atomic-arch-fallback.h +0 -0
include/linux/atomic/atomic-instrumented.h renamed from include/asm-generic/atomic-instrumented.h +4 -4
@@ -14,8 +14,8 @@* arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid * double instrumentation. */ -#ifndef _ASM_GENERIC_ATOMIC_INSTRUMENTED_H -#define _ASM_GENERIC_ATOMIC_INSTRUMENTED_H +#ifndef _LINUX_ATOMIC_INSTRUMENTED_H +#define _LINUX_ATOMIC_INSTRUMENTED_H#include <linux/build_bug.h> #include <linux/compiler.h>
@@ -1333,5 +1333,5 @@ atomic64_dec_if_positive(atomic64_t *v)arch_cmpxchg_double_local (__ai_ptr, __VA_ARGS__); \ }) -#endif /* _ASM_GENERIC_ATOMIC_INSTRUMENTED_H */ -// 1d7c3a25aca5c7fb031c307be4c3d24c7b48fcd5 +#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ +// 5edd72f105b6f54b7e9492d794abee88e6912d29
include/linux/atomic/atomic-long.h renamed from include/asm-generic/atomic-long.h +4 -4
@@ -3,8 +3,8 @@// Generated by scripts/atomic/gen-atomic-long.sh // DO NOT MODIFY THIS FILE DIRECTLY -#ifndef _ASM_GENERIC_ATOMIC_LONG_H -#define _ASM_GENERIC_ATOMIC_LONG_H +#ifndef _LINUX_ATOMIC_LONG_H +#define _LINUX_ATOMIC_LONG_H#include <linux/compiler.h> #include <asm/types.h>
@@ -1010,5 +1010,5 @@ atomic_long_dec_if_positive(atomic_long_t *v) }#endif /* CONFIG_64BIT */ -#endif /* _ASM_GENERIC_ATOMIC_LONG_H */ -// a624200981f552b2c6be4f32fe44da8289f30d87 +#endif /* _LINUX_ATOMIC_LONG_H */ +// c5552b5d78a0c7584dfd03cba985e78a1a86bbed
scripts/atomic/check-atomics.sh +3 -3
@@ -14,9 +14,9 @@ if [ $? -ne 0 ]; then fi cat <<EOF | -asm-generic/atomic-instrumented.h -asm-generic/atomic-long.h -linux/atomic-arch-fallback.h +linux/atomic/atomic-instrumented.h +linux/atomic/atomic-long.h +linux/atomic/atomic-arch-fallback.h EOF while read header; do OLDSUM="$(tail -n 1 ${LINUXDIR}/include/${header})"
scripts/atomic/gen-atomic-instrumented.sh +3 -3
@@ -121,8 +121,8 @@ cat << EOF * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid * double instrumentation. */ -#ifndef _ASM_GENERIC_ATOMIC_INSTRUMENTED_H -#define _ASM_GENERIC_ATOMIC_INSTRUMENTED_H +#ifndef _LINUX_ATOMIC_INSTRUMENTED_H +#define _LINUX_ATOMIC_INSTRUMENTED_H #include <linux/build_bug.h> #include <linux/compiler.h>
@@ -158,5 +158,5 @@ gen_xchg "cmpxchg_double_local" "2 * " cat <<EOF -#endif /* _ASM_GENERIC_ATOMIC_INSTRUMENTED_H */ +#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ EOF
scripts/atomic/gen-atomic-long.sh +3 -3
@@ -61,8 +61,8 @@ cat << EOF // Generated by $0 // DO NOT MODIFY THIS FILE DIRECTLY -#ifndef _ASM_GENERIC_ATOMIC_LONG_H -#define _ASM_GENERIC_ATOMIC_LONG_H +#ifndef _LINUX_ATOMIC_LONG_H +#define _LINUX_ATOMIC_LONG_H #include <linux/compiler.h> #include <asm/types.h>
@@ -98,5 +98,5 @@ done cat <<EOF #endif /* CONFIG_64BIT */ -#endif /* _ASM_GENERIC_ATOMIC_LONG_H */ +#endif /* _LINUX_ATOMIC_LONG_H */ EOF
scripts/atomic/gen-atomics.sh +3 -3
@@ -8,9 +8,9 @@ ATOMICTBL=${ATOMICDIR}/atomics.tbl LINUXDIR=${ATOMICDIR}/../.. cat <<EOF | -gen-atomic-instrumented.sh asm-generic/atomic-instrumented.h -gen-atomic-long.sh asm-generic/atomic-long.h -gen-atomic-fallback.sh linux/atomic-arch-fallback.h +gen-atomic-instrumented.sh linux/atomic/atomic-instrumented.h +gen-atomic-long.sh linux/atomic/atomic-long.h +gen-atomic-fallback.sh linux/atomic/atomic-arch-fallback.h EOF while read script header args; do /bin/sh ${ATOMICDIR}/${script} ${ATOMICTBL} ${args} > ${LINUXDIR}/include/${header}
Signature · 1 line
2.11.0
[tip: locking/core] locking/atomic: centralize generated headers
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: e3d18cee258b898017b298b5b93f8134dd62aee3 Gitweb: https://git.kernel.org/tip/e3d18cee258b898017b298b5b93f8134dd62aee3 Author: Mark Rutland <[email protected]> AuthorDate: Tue, 13 Jul 2021 11:52:51 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 16 Jul 2021 18:46:45 +02:00
locking/atomic: centralize generated headers
The generated atomic headers are only intended to be included directly by <linux/atomic.h>, but are spread across include/linux/ and include/asm-generic/, where people mnay be encouraged to include them.
This patch centralizes them under include/linux/atomic/.
Other than the header guards and hashes, there is no change to any of the generated headers as a result of this patch.
--- include/asm-generic/atomic-instrumented.h | 1337 +---------- include/asm-generic/atomic-long.h | 1014 +-------- include/linux/atomic-arch-fallback.h | 2361 +------------------ include/linux/atomic.h | 7 +- include/linux/atomic/atomic-arch-fallback.h | 2361 ++++++++++++++++++- include/linux/atomic/atomic-instrumented.h | 1337 ++++++++++- include/linux/atomic/atomic-long.h | 1014 ++++++++- scripts/atomic/check-atomics.sh | 6 +- scripts/atomic/gen-atomic-instrumented.sh | 6 +- scripts/atomic/gen-atomic-long.sh | 6 +- scripts/atomic/gen-atomics.sh | 6 +- 11 files changed, 4727 insertions(+), 4728 deletions(-) delete mode 100644 include/asm-generic/atomic-instrumented.h delete mode 100644 include/asm-generic/atomic-long.h delete mode 100644 include/linux/atomic-arch-fallback.h create mode 100644 include/linux/atomic/atomic-arch-fallback.h create mode 100644 include/linux/atomic/atomic-instrumented.h create mode 100644 include/linux/atomic/atomic-long.h
include/asm-generic/atomic-instrumented.h deleted +0 -1337
@@ -1,1337 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 - -// Generated by scripts/atomic/gen-atomic-instrumented.sh -// DO NOT MODIFY THIS FILE DIRECTLY - -/* -* This file provides wrappers with KASAN instrumentation for atomic operations. -* To use this functionality an arch's atomic.h file needs to define all -* atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include -* this file at the end. This file provides atomic_read() that forwards to -* arch_atomic_read() for actual atomic operation. -* Note: if an arch atomic operation is implemented by means of other atomic -* operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use -* arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid -* double instrumentation. -*/ -#ifndef _ASM_GENERIC_ATOMIC_INSTRUMENTED_H -#define _ASM_GENERIC_ATOMIC_INSTRUMENTED_H - -#include <linux/build_bug.h> -#include <linux/compiler.h> -#include <linux/instrumented.h> - -static __always_inline int -atomic_read (const atomic_t *v) -{ -instrument_atomic_read (v,sizeof (*v)); -return arch_atomic_read (v); -} - -static __always_inline int -atomic_read_acquire (const atomic_t *v) -{ -instrument_atomic_read (v,sizeof (*v)); -return arch_atomic_read_acquire (v); -} - -static __always_inline void -atomic_set (atomic_t *v,int i) -{ -instrument_atomic_write (v,sizeof (*v)); -arch_atomic_set (v, i); -} - -static __always_inline void -atomic_set_release (atomic_t *v,int i) -{ -instrument_atomic_write (v,sizeof (*v)); -arch_atomic_set_release (v, i); -} - -static __always_inline void -atomic_add (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_add (i, v); -} - -static __always_inline int -atomic_add_return (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_add_return (i, v); -} - -static __always_inline int -atomic_add_return_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_add_return_acquire (i, v); -} - -static __always_inline int -atomic_add_return_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_add_return_release (i, v); -} - -static __always_inline int -atomic_add_return_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_add_return_relaxed (i, v); -} - -static __always_inline int -atomic_fetch_add (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_add (i, v); -} - -static __always_inline int -atomic_fetch_add_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_add_acquire (i, v); -} - -static __always_inline int -atomic_fetch_add_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_add_release (i, v); -} - -static __always_inline int -atomic_fetch_add_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_add_relaxed (i, v); -} - -static __always_inline void -atomic_sub (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_sub (i, v); -} - -static __always_inline int -atomic_sub_return (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_sub_return (i, v); -} - -static __always_inline int -atomic_sub_return_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_sub_return_acquire (i, v); -} - -static __always_inline int -atomic_sub_return_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_sub_return_release (i, v); -} - -static __always_inline int -atomic_sub_return_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_sub_return_relaxed (i, v); -} - -static __always_inline int -atomic_fetch_sub (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_sub (i, v); -} - -static __always_inline int -atomic_fetch_sub_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_sub_acquire (i, v); -} - -static __always_inline int -atomic_fetch_sub_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_sub_release (i, v); -} - -static __always_inline int -atomic_fetch_sub_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_sub_relaxed (i, v); -} - -static __always_inline void -atomic_inc (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_inc (v); -} - -static __always_inline int -atomic_inc_return (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_return (v); -} - -static __always_inline int -atomic_inc_return_acquire (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_return_acquire (v); -} - -static __always_inline int -atomic_inc_return_release (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_return_release (v); -} - -static __always_inline int -atomic_inc_return_relaxed (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_return_relaxed (v); -} - -static __always_inline int -atomic_fetch_inc (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_inc (v); -} - -static __always_inline int -atomic_fetch_inc_acquire (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_inc_acquire (v); -} - -static __always_inline int -atomic_fetch_inc_release (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_inc_release (v); -} - -static __always_inline int -atomic_fetch_inc_relaxed (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_inc_relaxed (v); -} - -static __always_inline void -atomic_dec (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_dec (v); -} - -static __always_inline int -atomic_dec_return (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_return (v); -} - -static __always_inline int -atomic_dec_return_acquire (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_return_acquire (v); -} - -static __always_inline int -atomic_dec_return_release (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_return_release (v); -} - -static __always_inline int -atomic_dec_return_relaxed (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_return_relaxed (v); -} - -static __always_inline int -atomic_fetch_dec (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_dec (v); -} - -static __always_inline int -atomic_fetch_dec_acquire (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_dec_acquire (v); -} - -static __always_inline int -atomic_fetch_dec_release (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_dec_release (v); -} - -static __always_inline int -atomic_fetch_dec_relaxed (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_dec_relaxed (v); -} - -static __always_inline void -atomic_and (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_and (i, v); -} - -static __always_inline int -atomic_fetch_and (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_and (i, v); -} - -static __always_inline int -atomic_fetch_and_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_and_acquire (i, v); -} - -static __always_inline int -atomic_fetch_and_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_and_release (i, v); -} - -static __always_inline int -atomic_fetch_and_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_and_relaxed (i, v); -} - -static __always_inline void -atomic_andnot (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_andnot (i, v); -} - -static __always_inline int -atomic_fetch_andnot (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_andnot (i, v); -} - -static __always_inline int -atomic_fetch_andnot_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_andnot_acquire (i, v); -} - -static __always_inline int -atomic_fetch_andnot_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_andnot_release (i, v); -} - -static __always_inline int -atomic_fetch_andnot_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_andnot_relaxed (i, v); -} - -static __always_inline void -atomic_or (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_or (i, v); -} - -static __always_inline int -atomic_fetch_or (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_or (i, v); -} - -static __always_inline int -atomic_fetch_or_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_or_acquire (i, v); -} - -static __always_inline int -atomic_fetch_or_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_or_release (i, v); -} - -static __always_inline int -atomic_fetch_or_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_or_relaxed (i, v); -} - -static __always_inline void -atomic_xor (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic_xor (i, v); -} - -static __always_inline int -atomic_fetch_xor (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_xor (i, v); -} - -static __always_inline int -atomic_fetch_xor_acquire (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_xor_acquire (i, v); -} - -static __always_inline int -atomic_fetch_xor_release (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_xor_release (i, v); -} - -static __always_inline int -atomic_fetch_xor_relaxed (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_xor_relaxed (i, v); -} - -static __always_inline int -atomic_xchg (atomic_t *v,int i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_xchg (v, i); -} - -static __always_inline int -atomic_xchg_acquire (atomic_t *v,int i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_xchg_acquire (v, i); -} - -static __always_inline int -atomic_xchg_release (atomic_t *v,int i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_xchg_release (v, i); -} - -static __always_inline int -atomic_xchg_relaxed (atomic_t *v,int i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_xchg_relaxed (v, i); -} - -static __always_inline int -atomic_cmpxchg (atomic_t *v,int old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_cmpxchg (v, old, new); -} - -static __always_inline int -atomic_cmpxchg_acquire (atomic_t *v,int old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_cmpxchg_acquire (v, old, new); -} - -static __always_inline int -atomic_cmpxchg_release (atomic_t *v,int old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_cmpxchg_release (v, old, new); -} - -static __always_inline int -atomic_cmpxchg_relaxed (atomic_t *v,int old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_cmpxchg_relaxed (v, old, new); -} - -static __always_inline bool -atomic_try_cmpxchg (atomic_t *v,int *old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic_try_cmpxchg (v, old, new); -} - -static __always_inline bool -atomic_try_cmpxchg_acquire (atomic_t *v,int *old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic_try_cmpxchg_acquire (v, old, new); -} - -static __always_inline bool -atomic_try_cmpxchg_release (atomic_t *v,int *old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic_try_cmpxchg_release (v, old, new); -} - -static __always_inline bool -atomic_try_cmpxchg_relaxed (atomic_t *v,int *old,int new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic_try_cmpxchg_relaxed (v, old, new); -} - -static __always_inline bool -atomic_sub_and_test (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_sub_and_test (i, v); -} - -static __always_inline bool -atomic_dec_and_test (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_and_test (v); -} - -static __always_inline bool -atomic_inc_and_test (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_and_test (v); -} - -static __always_inline bool -atomic_add_negative (int i,atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_add_negative (i, v); -} - -static __always_inline int -atomic_fetch_add_unless (atomic_t *v,int a,int u) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_fetch_add_unless (v, a, u); -} - -static __always_inline bool -atomic_add_unless (atomic_t *v,int a,int u) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_add_unless (v, a, u); -} - -static __always_inline bool -atomic_inc_not_zero (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_not_zero (v); -} - -static __always_inline bool -atomic_inc_unless_negative (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_inc_unless_negative (v); -} - -static __always_inline bool -atomic_dec_unless_positive (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_unless_positive (v); -} - -static __always_inline int -atomic_dec_if_positive (atomic_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic_dec_if_positive (v); -} - -static __always_inline s64 -atomic64_read (const atomic64_t *v) -{ -instrument_atomic_read (v,sizeof (*v)); -return arch_atomic64_read (v); -} - -static __always_inline s64 -atomic64_read_acquire (const atomic64_t *v) -{ -instrument_atomic_read (v,sizeof (*v)); -return arch_atomic64_read_acquire (v); -} - -static __always_inline void -atomic64_set (atomic64_t *v,s64 i) -{ -instrument_atomic_write (v,sizeof (*v)); -arch_atomic64_set (v, i); -} - -static __always_inline void -atomic64_set_release (atomic64_t *v,s64 i) -{ -instrument_atomic_write (v,sizeof (*v)); -arch_atomic64_set_release (v, i); -} - -static __always_inline void -atomic64_add (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_add (i, v); -} - -static __always_inline s64 -atomic64_add_return (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_add_return (i, v); -} - -static __always_inline s64 -atomic64_add_return_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_add_return_acquire (i, v); -} - -static __always_inline s64 -atomic64_add_return_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_add_return_release (i, v); -} - -static __always_inline s64 -atomic64_add_return_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_add_return_relaxed (i, v); -} - -static __always_inline s64 -atomic64_fetch_add (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_add (i, v); -} - -static __always_inline s64 -atomic64_fetch_add_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_add_acquire (i, v); -} - -static __always_inline s64 -atomic64_fetch_add_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_add_release (i, v); -} - -static __always_inline s64 -atomic64_fetch_add_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_add_relaxed (i, v); -} - -static __always_inline void -atomic64_sub (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_sub (i, v); -} - -static __always_inline s64 -atomic64_sub_return (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_sub_return (i, v); -} - -static __always_inline s64 -atomic64_sub_return_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_sub_return_acquire (i, v); -} - -static __always_inline s64 -atomic64_sub_return_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_sub_return_release (i, v); -} - -static __always_inline s64 -atomic64_sub_return_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_sub_return_relaxed (i, v); -} - -static __always_inline s64 -atomic64_fetch_sub (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_sub (i, v); -} - -static __always_inline s64 -atomic64_fetch_sub_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_sub_acquire (i, v); -} - -static __always_inline s64 -atomic64_fetch_sub_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_sub_release (i, v); -} - -static __always_inline s64 -atomic64_fetch_sub_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_sub_relaxed (i, v); -} - -static __always_inline void -atomic64_inc (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_inc (v); -} - -static __always_inline s64 -atomic64_inc_return (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_return (v); -} - -static __always_inline s64 -atomic64_inc_return_acquire (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_return_acquire (v); -} - -static __always_inline s64 -atomic64_inc_return_release (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_return_release (v); -} - -static __always_inline s64 -atomic64_inc_return_relaxed (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_return_relaxed (v); -} - -static __always_inline s64 -atomic64_fetch_inc (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_inc (v); -} - -static __always_inline s64 -atomic64_fetch_inc_acquire (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_inc_acquire (v); -} - -static __always_inline s64 -atomic64_fetch_inc_release (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_inc_release (v); -} - -static __always_inline s64 -atomic64_fetch_inc_relaxed (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_inc_relaxed (v); -} - -static __always_inline void -atomic64_dec (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_dec (v); -} - -static __always_inline s64 -atomic64_dec_return (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_return (v); -} - -static __always_inline s64 -atomic64_dec_return_acquire (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_return_acquire (v); -} - -static __always_inline s64 -atomic64_dec_return_release (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_return_release (v); -} - -static __always_inline s64 -atomic64_dec_return_relaxed (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_return_relaxed (v); -} - -static __always_inline s64 -atomic64_fetch_dec (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_dec (v); -} - -static __always_inline s64 -atomic64_fetch_dec_acquire (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_dec_acquire (v); -} - -static __always_inline s64 -atomic64_fetch_dec_release (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_dec_release (v); -} - -static __always_inline s64 -atomic64_fetch_dec_relaxed (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_dec_relaxed (v); -} - -static __always_inline void -atomic64_and (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_and (i, v); -} - -static __always_inline s64 -atomic64_fetch_and (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_and (i, v); -} - -static __always_inline s64 -atomic64_fetch_and_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_and_acquire (i, v); -} - -static __always_inline s64 -atomic64_fetch_and_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_and_release (i, v); -} - -static __always_inline s64 -atomic64_fetch_and_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_and_relaxed (i, v); -} - -static __always_inline void -atomic64_andnot (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_andnot (i, v); -} - -static __always_inline s64 -atomic64_fetch_andnot (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_andnot (i, v); -} - -static __always_inline s64 -atomic64_fetch_andnot_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_andnot_acquire (i, v); -} - -static __always_inline s64 -atomic64_fetch_andnot_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_andnot_release (i, v); -} - -static __always_inline s64 -atomic64_fetch_andnot_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_andnot_relaxed (i, v); -} - -static __always_inline void -atomic64_or (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_or (i, v); -} - -static __always_inline s64 -atomic64_fetch_or (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_or (i, v); -} - -static __always_inline s64 -atomic64_fetch_or_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_or_acquire (i, v); -} - -static __always_inline s64 -atomic64_fetch_or_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_or_release (i, v); -} - -static __always_inline s64 -atomic64_fetch_or_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_or_relaxed (i, v); -} - -static __always_inline void -atomic64_xor (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -arch_atomic64_xor (i, v); -} - -static __always_inline s64 -atomic64_fetch_xor (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_xor (i, v); -} - -static __always_inline s64 -atomic64_fetch_xor_acquire (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_xor_acquire (i, v); -} - -static __always_inline s64 -atomic64_fetch_xor_release (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_xor_release (i, v); -} - -static __always_inline s64 -atomic64_fetch_xor_relaxed (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_xor_relaxed (i, v); -} - -static __always_inline s64 -atomic64_xchg (atomic64_t *v,s64 i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_xchg (v, i); -} - -static __always_inline s64 -atomic64_xchg_acquire (atomic64_t *v,s64 i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_xchg_acquire (v, i); -} - -static __always_inline s64 -atomic64_xchg_release (atomic64_t *v,s64 i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_xchg_release (v, i); -} - -static __always_inline s64 -atomic64_xchg_relaxed (atomic64_t *v,s64 i) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_xchg_relaxed (v, i); -} - -static __always_inline s64 -atomic64_cmpxchg (atomic64_t *v,s64 old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_cmpxchg (v, old, new); -} - -static __always_inline s64 -atomic64_cmpxchg_acquire (atomic64_t *v,s64 old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_cmpxchg_acquire (v, old, new); -} - -static __always_inline s64 -atomic64_cmpxchg_release (atomic64_t *v,s64 old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_cmpxchg_release (v, old, new); -} - -static __always_inline s64 -atomic64_cmpxchg_relaxed (atomic64_t *v,s64 old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_cmpxchg_relaxed (v, old, new); -} - -static __always_inline bool -atomic64_try_cmpxchg (atomic64_t *v,s64 *old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic64_try_cmpxchg (v, old, new); -} - -static __always_inline bool -atomic64_try_cmpxchg_acquire (atomic64_t *v,s64 *old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic64_try_cmpxchg_acquire (v, old, new); -} - -static __always_inline bool -atomic64_try_cmpxchg_release (atomic64_t *v,s64 *old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic64_try_cmpxchg_release (v, old, new); -} - -static __always_inline bool -atomic64_try_cmpxchg_relaxed (atomic64_t *v,s64 *old,s64 new) -{ -instrument_atomic_read_write (v,sizeof (*v)); -instrument_atomic_read_write (old,sizeof (*old)); -return arch_atomic64_try_cmpxchg_relaxed (v, old, new); -} - -static __always_inline bool -atomic64_sub_and_test (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_sub_and_test (i, v); -} - -static __always_inline bool -atomic64_dec_and_test (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_and_test (v); -} - -static __always_inline bool -atomic64_inc_and_test (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_and_test (v); -} - -static __always_inline bool -atomic64_add_negative (s64 i,atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_add_negative (i, v); -} - -static __always_inline s64 -atomic64_fetch_add_unless (atomic64_t *v,s64 a,s64 u) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_fetch_add_unless (v, a, u); -} - -static __always_inline bool -atomic64_add_unless (atomic64_t *v,s64 a,s64 u) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_add_unless (v, a, u); -} - -static __always_inline bool -atomic64_inc_not_zero (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_not_zero (v); -} - -static __always_inline bool -atomic64_inc_unless_negative (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_inc_unless_negative (v); -} - -static __always_inline bool -atomic64_dec_unless_positive (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_unless_positive (v); -} - -static __always_inline s64 -atomic64_dec_if_positive (atomic64_t *v) -{ -instrument_atomic_read_write (v,sizeof (*v)); -return arch_atomic64_dec_if_positive (v); -} - -#define xchg (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_xchg(__ai_ptr, __VA_ARGS__); \ -}) - -#define xchg_acquire (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_xchg_acquire(__ai_ptr, __VA_ARGS__); \ -}) - -#define xchg_release (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_xchg_release(__ai_ptr, __VA_ARGS__); \ -}) - -#define xchg_relaxed (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_xchg_relaxed(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg_acquire (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg_acquire(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg_release (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg_release(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg_relaxed (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg_relaxed(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg64 (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg64(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg64_acquire (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg64_acquire(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg64_release (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg64_release(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg64_relaxed (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \ -}) - -#define try_cmpxchg (ptr, oldp, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - typeof(oldp) __ai_oldp = (oldp); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ - arch_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \ -}) - -#define try_cmpxchg_acquire (ptr, oldp, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - typeof(oldp) __ai_oldp = (oldp); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ - arch_try_cmpxchg_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \ -}) - -#define try_cmpxchg_release (ptr, oldp, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - typeof(oldp) __ai_oldp = (oldp); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ - arch_try_cmpxchg_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \ -}) - -#define try_cmpxchg_relaxed (ptr, oldp, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - typeof(oldp) __ai_oldp = (oldp); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ - arch_try_cmpxchg_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ -}) - -#define cmpxchg_local (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg_local(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg64_local (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \ -}) - -#define sync_cmpxchg (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ - arch_sync_cmpxchg(__ai_ptr, __VA_ARGS__); \ -}) - -#define cmpxchg_double (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ - arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \ -}) - - -#define cmpxchg_double_local (ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ - arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \ -}) - -#endif /* _ASM_GENERIC_ATOMIC_INSTRUMENTED_H */ -// 1d7c3a25aca5c7fb031c307be4c3d24c7b48fcd5
include/asm-generic/atomic-long.h deleted +0 -1014
1,014 lines not on this page · raw message
include/linux/atomic-arch-fallback.h deleted +0 -2361
649 lines not on this page · raw message
include/linux/atomic/atomic-arch-fallback.h new file +2361 -0
2,361 lines not on this page · raw message
include/linux/atomic/atomic-instrumented.h new file +1337 -0
1,337 lines not on this page · raw message
include/linux/atomic/atomic-long.h new file +1014 -0
1,014 lines not on this page · raw message
scripts/atomic/gen-atomic-instrumented.sh +3 -3
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Mark Rutland
reviewerPATCH 4/5 locking/atomic: add arch_atomic_long*()
In reply to Mark RutlandNow that all architectures provide arch_{atomic,atomic64}_*(), we can build arch_atomic_long_*() atop these, which can be safely used in noinstr code. The regular atomic_long_*() wrappers are built atop these, as we do for {atomic,atomic64}_*() atop arch_{atomic,atomic64}_*().
We don't provide arch_* versions of the cond_read*() variants, as we don't have arch_* versions of the underlying atomic/atomic64 functions (nor the smp_cond_load*() helpers these are typically based on).
Note that the headers in this patch under include/linux/atomic/ are generated by the scripts in scripts/atomic/.
--- include/linux/atomic.h | 2 +- include/linux/atomic/atomic-instrumented.h | 580 ++++++++++++++++++++++++- include/linux/atomic/atomic-long.h | 658 ++++++++++++++--------------- scripts/atomic/gen-atomic-instrumented.sh | 5 + scripts/atomic/gen-atomic-long.sh | 4 +- 5 files changed, 916 insertions(+), 333 deletions(-)
include/linux/atomic.h +1 -1
@@ -78,7 +78,7 @@ })#include <linux/atomic/atomic-arch-fallback.h> -#include <linux/atomic/atomic-instrumented.h> #include <linux/atomic/atomic-long.h> +#include <linux/atomic/atomic-instrumented.h> #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-instrumented.h +579 -1
589 lines not on this page · raw message
include/linux/atomic/atomic-long.h +329 -329
1,320 lines not on this page · raw message
scripts/atomic/gen-atomic-instrumented.sh +5 -0
@@ -138,6 +138,11 @@ grep '^[a-z]' "$1" | while read name meta args; do gen_proto "${meta}" "${name}" "atomic64" "s64" ${args} done +grep '^[a-z]' "$1" | while read name meta args; do + gen_proto "${meta}" "${name}" "atomic_long" "long" ${args} +done + + for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg"; do for order in "" "_acquire" "_release" "_relaxed"; do gen_xchg "${xchg}${order}" ""
scripts/atomic/gen-atomic-long.sh +2 -2
@@ -47,9 +47,9 @@ gen_proto_order_variant() cat <<EOF static __always_inline ${ret} -atomic_long_${name}(${params}) +arch_atomic_long_${name}(${params}) { - ${retstmt}${atomic}_${name}(${argscast}); + ${retstmt}arch_${atomic}_${name}(${argscast}); } EOF
Signature · 1 line
2.11.0
[tip: locking/core] locking/atomic: add arch_atomic_long*()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 67d1b0de258ad066e1fc85d0ceaa75e107fb45bb Gitweb: https://git.kernel.org/tip/67d1b0de258ad066e1fc85d0ceaa75e107fb45bb Author: Mark Rutland <[email protected]> AuthorDate: Tue, 13 Jul 2021 11:52:52 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 16 Jul 2021 18:46:45 +02:00
locking/atomic: add arch_atomic_long*()
Now that all architectures provide arch_{atomic,atomic64}_*(), we can build arch_atomic_long_*() atop these, which can be safely used in noinstr code. The regular atomic_long_*() wrappers are built atop these, as we do for {atomic,atomic64}_*() atop arch_{atomic,atomic64}_*().
We don't provide arch_* versions of the cond_read*() variants, as we don't have arch_* versions of the underlying atomic/atomic64 functions (nor the smp_cond_load*() helpers these are typically based on).
Note that the headers in this patch under include/linux/atomic/ are generated by the scripts in scripts/atomic/.
--- include/linux/atomic.h | 2 +- include/linux/atomic/atomic-instrumented.h | 580 +++++++++++++++++- include/linux/atomic/atomic-long.h | 658 ++++++++++---------- scripts/atomic/gen-atomic-instrumented.sh | 5 +- scripts/atomic/gen-atomic-long.sh | 4 +- 5 files changed, 916 insertions(+), 333 deletions(-)
include/linux/atomic.h +1 -1
@@ -78,7 +78,7 @@ })#include <linux/atomic/atomic-arch-fallback.h> -#include <linux/atomic/atomic-instrumented.h> #include <linux/atomic/atomic-long.h> +#include <linux/atomic/atomic-instrumented.h> #endif /* _LINUX_ATOMIC_H */
include/linux/atomic/atomic-instrumented.h +579 -1
589 lines not on this page · raw message
include/linux/atomic/atomic-long.h +329 -329
1,320 lines not on this page · raw message
scripts/atomic/gen-atomic-instrumented.sh +5 -0
@@ -138,6 +138,11 @@ grep '^[a-z]' "$1" | while read name meta args; do gen_proto "${meta}" "${name}" "atomic64" "s64" ${args} done +grep '^[a-z]' "$1" | while read name meta args; do + gen_proto "${meta}" "${name}" "atomic_long" "long" ${args} +done + + for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg"; do for order in "" "_acquire" "_release" "_relaxed"; do gen_xchg "${xchg}${order}" ""
scripts/atomic/gen-atomic-long.sh +2 -2
@@ -47,9 +47,9 @@ gen_proto_order_variant() cat <<EOF static __always_inline ${ret} -atomic_long_${name}(${params}) +arch_atomic_long_${name}(${params}) { - ${retstmt}${atomic}_${name}(${argscast}); + ${retstmt}arch_${atomic}_${name}(${argscast}); } EOF
Mark Rutland
reviewerPATCH 5/5 locking/atomic: add generic arch_*() bitops
In reply to Mark RutlandNow that all architectures provide arch_atomic_long_*(), we can implement the generic bitops atop these rather than atop atomic_long_*(), and provide arch_*() forms of the bitops that are safe to use in noinstr code.
Now that all architectures provide arch_atomic_long_*(), we can build the generic arch_*() bitops atop these, which can be safely used in noinstr code. The regular bitop wrappers are built atop these.
As the generic non-atomic bitops use plain accesses, these will be implicitly instrumented unless they are inlined into noinstr functions (which is similar to arch_atomic*_read() when based on READ_ONCE()). The wrappers are modified so that where the underlying arch_*() function uses a plain access, no explicit instrumentation is added, as this is redundant and could result in confusing reports.
Since function prototypes get excessively long with both an arch_ prefix and __always_inline attribute, the return type and function attributes have been split onto a separate line, matching the style of the generated atomic headers.
--- include/asm-generic/bitops/atomic.h | 32 ++++++++++------- .../asm-generic/bitops/instrumented-non-atomic.h | 21 +++++++---- include/asm-generic/bitops/lock.h | 39 ++++++++++---------- include/asm-generic/bitops/non-atomic.h | 41 +++++++++++++++------- 4 files changed, 83 insertions(+), 50 deletions(-)
include/asm-generic/bitops/atomic.h +20 -12
@@ -11,25 +11,29 @@* See Documentation/atomic_bitops.txt for details. */ -static __always_inline void set_bit(unsigned int nr, volatile unsigned long *p) +static __always_inline void +arch_set_bit (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_or (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_or (BIT_MASK (nr), (atomic_long_t *)p); } -static __always_inline void clear_bit(unsigned int nr, volatile unsigned long *p) +static __always_inline void +arch_clear_bit (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_andnot (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_andnot (BIT_MASK (nr), (atomic_long_t *)p); } -static __always_inline void change_bit(unsigned int nr, volatile unsigned long *p) +static __always_inline void +arch_change_bit (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_xor (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_xor (BIT_MASK (nr), (atomic_long_t *)p); } -static inline int test_and_set_bit (unsigned int nr,volatile unsigned long *p) +static __always_inline int +arch_test_and_set_bit (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr);
@@ -38,11 +42,12 @@ static inline int test_and_set_bit(unsigned int nr, volatile unsigned long *p)if (READ_ONCE (*p) & mask)return 1 ; - old =atomic_long_fetch_or (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_or (mask, (atomic_long_t *)p);return !!(old & mask); } -static inline int test_and_clear_bit (unsigned int nr,volatile unsigned long *p) +static __always_inline int +arch_test_and_clear_bit (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr);
@@ -51,18 +56,21 @@ static inline int test_and_clear_bit(unsigned int nr, volatile unsigned long *p)if (!(READ_ONCE (*p) & mask))return 0 ; - old =atomic_long_fetch_andnot (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_andnot (mask, (atomic_long_t *)p);return !!(old & mask); } -static inline int test_and_change_bit (unsigned int nr,volatile unsigned long *p) +static __always_inline int +arch_test_and_change_bit (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr); p +=BIT_WORD (nr); - old =atomic_long_fetch_xor (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_xor (mask, (atomic_long_t *)p);return !!(old & mask); } +#include <asm-generic/bitops/instrumented-atomic.h> +#endif /* _ASM_GENERIC_BITOPS_ATOMIC_H */
include/asm-generic/bitops/instrumented-non-atomic.h +14 -7
@@ -24,7 +24,8 @@*/ static inline void __set_bit (long nr,volatile unsigned long *addr) { -instrument_write (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch___set_bit_uses_plain_access)) +instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___set_bit (nr, addr); }
@@ -39,7 +40,8 @@ static inline void __set_bit(long nr, volatile unsigned long *addr)*/ static inline void __clear_bit (long nr,volatile unsigned long *addr) { -instrument_write (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch___clear_bit_uses_plain_access)) +instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___clear_bit (nr, addr); }
@@ -54,7 +56,8 @@ static inline void __clear_bit(long nr, volatile unsigned long *addr)*/ static inline void __change_bit (long nr,volatile unsigned long *addr) { -instrument_write (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch___change_bit_uses_plain_access)) +instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___change_bit (nr, addr); }
@@ -92,7 +95,8 @@ static inline void __instrument_read_write_bitop(long nr, volatile unsigned long*/ static inline bool __test_and_set_bit (long nr,volatile unsigned long *addr) { -__instrument_read_write_bitop (nr, addr); +if (!__is_defined (arch___test_and_set_bit_uses_plain_access)) +__instrument_read_write_bitop (nr, addr);return arch___test_and_set_bit (nr, addr); }
@@ -106,7 +110,8 @@ static inline bool __test_and_set_bit(long nr, volatile unsigned long *addr)*/ static inline bool __test_and_clear_bit (long nr,volatile unsigned long *addr) { -__instrument_read_write_bitop (nr, addr); +if (!__is_defined (arch___test_and_clear_bit_uses_plain_access)) +__instrument_read_write_bitop (nr, addr);return arch___test_and_clear_bit (nr, addr); }
@@ -120,7 +125,8 @@ static inline bool __test_and_clear_bit(long nr, volatile unsigned long *addr)*/ static inline bool __test_and_change_bit (long nr,volatile unsigned long *addr) { -__instrument_read_write_bitop (nr, addr); +if (!__is_defined (arch___test_and_change_bit_uses_plain_access)) +__instrument_read_write_bitop (nr, addr);return arch___test_and_change_bit (nr, addr); }
@@ -131,7 +137,8 @@ static inline bool __test_and_change_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_bit (long nr,const volatile unsigned long *addr) { -instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch_test_bit_uses_plain_access)) +instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long ));return arch_test_bit (nr, addr); }
include/asm-generic/bitops/lock.h +21 -18
@@ -7,7 +7,7 @@#include <asm/barrier.h> /** -* test_and_set_bit_lock - Set a bit and return its old value, for lock +* arch_test_and_set_bit_lock - Set a bit and return its old value, for lock * @nr: Bit to set * @addr: Address to count from *
@@ -15,8 +15,8 @@* the returned value is 0. * It can be used to implement bit locks. */ -static inline int test_and_set_bit_lock (unsigned int nr, -volatile unsigned long *p) +static __always_inline int +arch_test_and_set_bit_lock (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr);
@@ -25,26 +25,27 @@ static inline int test_and_set_bit_lock(unsigned int nr,if (READ_ONCE (*p) & mask)return 1 ; - old =atomic_long_fetch_or_acquire (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_or_acquire (mask, (atomic_long_t *)p);return !!(old & mask); }/** -* clear_bit_unlock - Clear a bit in memory, for unlock +* arch_clear_bit_unlock - Clear a bit in memory, for unlock * @nr: the bit to set * @addr: the address to start counting from * * This operation is atomic and provides release barrier semantics. */ -static inline void clear_bit_unlock (unsigned int nr,volatile unsigned long *p) +static __always_inline void +arch_clear_bit_unlock (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_fetch_andnot_release (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_fetch_andnot_release (BIT_MASK (nr), (atomic_long_t *)p); }/** -* __clear_bit_unlock - Clear a bit in memory, for unlock +* arch___clear_bit_unlock - Clear a bit in memory, for unlock * @nr: the bit to set * @addr: the address to start counting from *
@@ -54,38 +55,40 @@ static inline void clear_bit_unlock(unsigned int nr, volatile unsigned long *p)* * See for example x86's implementation. */ -static inline void __clear_bit_unlock (unsigned int nr, -volatile unsigned long *p) +static inline void +arch___clear_bit_unlock (unsigned int nr,volatile unsigned long *p) {unsigned long old; p +=BIT_WORD (nr); old =READ_ONCE (*p); old &= ~BIT_MASK (nr); -atomic_long_set_release ((atomic_long_t *)p, old); +arch_atomic_long_set_release ((atomic_long_t *)p, old); }/** -* clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom -* byte is negative, for unlock. +* arch_clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom +* byte is negative, for unlock. * @nr: the bit to clear * @addr: the address to start counting from * * This is a bit of a one-trick-pony for the filemap code, which clears * PG_locked and tests PG_waiters, */ -#ifndef clear_bit_unlock_is_negative_byte -static inline bool clear_bit_unlock_is_negative_byte (unsigned int nr, -volatile unsigned long *p) +#ifndef arch_clear_bit_unlock_is_negative_byte +static inline bool arch_clear_bit_unlock_is_negative_byte (unsigned int nr, +volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr); p +=BIT_WORD (nr); - old =atomic_long_fetch_andnot_release (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_andnot_release (mask, (atomic_long_t *)p);return !!(old &BIT (7 )); } -#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte +#define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte#endif +#include <asm-generic/bitops/instrumented-lock.h> +#endif /* _ASM_GENERIC_BITOPS_LOCK_H_ */
include/asm-generic/bitops/non-atomic.h +28 -13
123 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [PATCH 5/5] locking/atomic: add generic arch_*() bitops
On Tue, Jul 13, 2021 at 11:52AM +0100, Mark Rutland wrote:
[...]
Quoted 6 lines
As the generic non-atomic bitops use plain accesses, these will be implicitly instrumented unless they are inlined into noinstr functions (which is similar to arch_atomic*_read() when based on
READ_ONCE()). The wrappers are modified so that where the underlying arch_*() function uses a plain access, no explicit instrumentation is added, as this is redundant and could result in confusing reports.
[...]
Quoted 4 lines
include/asm-generic/bitops/instrumented-non-atomic.h +0
-0
[...]
Quoted 9 lines
include/asm-generic/bitops/instrumented-non-atomic.h +2
-1@@ -131,7 +137,8 @@ static inline bool __test_and_change_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_bit (long nr,const volatile unsigned long *addr) { -instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch_test_bit_uses_plain_access)) +instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long ));return arch_test_bit (nr, addr); }
[...]
Quoted 6 lines
include/asm-generic/bitops/non-atomic.h +0
-0@@ -5,7 +5,7 @@#include <asm/types.h>
[...]
Quoted 6 lines
/**
- *test_bit- Determine whether a bit is set
+ *arch_test_bit- Determine whether a bit is set
* @nr: bit number to test
* @addr: Address to start counting from */
include/asm-generic/bitops/non-atomic.h static inline int __test_and_change_bit(int nr, · line 122
Quoted 9 lines
include/asm-generic/bitops/non-atomic.h +5
-1-static inline int test_bit (int nr,const volatile unsigned long *addr) +static __always_inline int +arch_test_bit (int nr,const volatile unsigned long *addr) {return 1UL & (addr[BIT_WORD (nr)] >> (nr & (BITS_PER_LONG -1 ))); } +#define arch_test_bit_uses_plain_access + +#include <asm-generic/bitops/instrumented-non-atomic.h>
Why not just:
#define test_bit arch_test_bit
and similar for the ones that use plain accesses?
And not include instrumented-non-atomic.h here nor do the __is_defined(*_uses_plain_access) change to the instrumented header, which seems to overcomplicate things as it effectively just aliases the non-arch_ name to the arch_ name if *_uses_plain_access is defined.
Thanks,
-- Marco
Mark Rutland
reviewerRe: [PATCH 5/5] locking/atomic: add generic arch_*() bitops
Quoted 49 lines · Marco Elver
On Tue, Jul 13, 2021 at 11:52AM +0100, Mark Rutland wrote:
[...]Quoted 6 lines
As the generic non-atomic bitops use plain accesses, these will be implicitly instrumented unless they are inlined into noinstr functions (which is similar to arch_atomic*_read() when based on
READ_ONCE()). The wrappers are modified so that where the underlying arch_*() function uses a plain access, no explicit instrumentation is added, as this is redundant and could result in confusing reports.[...]
Quoted 4 lines
include/asm-generic/bitops/instrumented-non-atomic.h +0
-0[...]
Quoted 9 lines
include/asm-generic/bitops/instrumented-non-atomic.h +2
-1@@ -131,7 +137,8 @@ static inline bool __test_and_change_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_bit (long nr,const volatile unsigned long *addr) { -instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch_test_bit_uses_plain_access)) +instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long ));return arch_test_bit (nr, addr); }[...]
Quoted 6 lines
include/asm-generic/bitops/non-atomic.h +0
-0@@ -5,7 +5,7 @@#include <asm/types.h> [...]
Quoted 6 lines
/**
- *test_bit- Determine whether a bit is set
+ *arch_test_bit- Determine whether a bit is set
* @nr: bit number to test
* @addr: Address to start counting from */include/asm-generic/bitops/non-atomic.h static inline int __test_and_change_bit(int nr, · line 122
Quoted 9 lines
include/asm-generic/bitops/non-atomic.h +5
-1-static inline int test_bit (int nr,const volatile unsigned long *addr) +static __always_inline int +arch_test_bit (int nr,const volatile unsigned long *addr) {return 1UL & (addr[BIT_WORD (nr)] >> (nr & (BITS_PER_LONG -1 ))); } +#define arch_test_bit_uses_plain_access + +#include <asm-generic/bitops/instrumented-non-atomic.h> Why not just:
Quoted 8 lines
#define test_bit arch_test_bitand similar for the ones that use plain accesses?
And not include
instrumented-non-atomic.hhere nor do the__is_defined(*_uses_plain_access)change to the instrumented header, which seems to overcomplicate things as it effectively just aliases the non-arch_ name to thearch_name if*_uses_plain_accessis defined.
I'd done that to still permit the compiler to out-of-line the non-arch forms if it wanted to. That said, I see that for the atomics we forced those to be __always_inline anyway, so maybe that's not a concern.
I'm happy either way.
Thanks,
Mark.
Re: [PATCH 5/5] locking/atomic: add generic arch_*() bitops
On Fri, 16 Jul 2021 at 14:21, Mark Rutland <[email protected]> wrote:
[...]
Quoted 11 lines
Quoted 10 lines
Why not just:
#define test_bit arch_test_bitand similar for the ones that use plain accesses?
And not include
instrumented-non-atomic.hhere nor do the__is_defined(*_uses_plain_access)change to the instrumented header, which seems to overcomplicate things as it effectively just aliases the non-arch_ name to thearch_name if*_uses_plain_accessis defined.
Quoted 5 lines
I'd done that to still permit the compiler to out-of-line the non-arch forms if it wanted to. That said, I see that for the atomics we forced those to be
__always_inlineanyway, so maybe that's not a concern.I'm happy either way.
I'd prefer simplicity. In an optimized build, I think even if it decided to not inline, the perf and code size difference is a wash. Intuition tells me that inlining would even be preferred either way, but I've been wrong about that in the past.
I think originally we turned the atomics __always_inline because of uaccess regions. Because debugging tools do work in uaccess regions, arch_* aren't necessarily appropriate. So something like a test_bit() in an uaccess region might actually generate a warning. This might be a valid argument for __always_inline irrespective of optimization potential (if there is any).
Thanks,
-- Marco
[tip: locking/core] locking/atomic: add generic arch_*() bitops
In reply to Mark RutlandThe following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: cf3ee3c8c29dc349b2cf52e5e72e8cb805ff5e57 Gitweb: https://git.kernel.org/tip/cf3ee3c8c29dc349b2cf52e5e72e8cb805ff5e57 Author: Mark Rutland <[email protected]> AuthorDate: Tue, 13 Jul 2021 11:52:53 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 16 Jul 2021 18:46:45 +02:00
locking/atomic: add generic arch_*() bitops
Now that all architectures provide arch_atomic_long_*(), we can implement the generic bitops atop these rather than atop atomic_long_*(), and provide arch_*() forms of the bitops that are safe to use in noinstr code.
Now that all architectures provide arch_atomic_long_*(), we can build the generic arch_*() bitops atop these, which can be safely used in noinstr code. The regular bitop wrappers are built atop these.
As the generic non-atomic bitops use plain accesses, these will be implicitly instrumented unless they are inlined into noinstr functions (which is similar to arch_atomic*_read() when based on READ_ONCE()). The wrappers are modified so that where the underlying arch_*() function uses a plain access, no explicit instrumentation is added, as this is redundant and could result in confusing reports.
Since function prototypes get excessively long with both an arch_ prefix and __always_inline attribute, the return type and function attributes have been split onto a separate line, matching the style of the generated atomic headers.
--- include/asm-generic/bitops/atomic.h | 32 +++++---- include/asm-generic/bitops/instrumented-non-atomic.h | 21 ++++-- include/asm-generic/bitops/lock.h | 39 +++++----- include/asm-generic/bitops/non-atomic.h | 41 +++++++---- 4 files changed, 83 insertions(+), 50 deletions(-)
include/asm-generic/bitops/atomic.h +20 -12
@@ -11,25 +11,29 @@* See Documentation/atomic_bitops.txt for details. */ -static __always_inline void set_bit(unsigned int nr, volatile unsigned long *p) +static __always_inline void +arch_set_bit (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_or (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_or (BIT_MASK (nr), (atomic_long_t *)p); } -static __always_inline void clear_bit(unsigned int nr, volatile unsigned long *p) +static __always_inline void +arch_clear_bit (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_andnot (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_andnot (BIT_MASK (nr), (atomic_long_t *)p); } -static __always_inline void change_bit(unsigned int nr, volatile unsigned long *p) +static __always_inline void +arch_change_bit (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_xor (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_xor (BIT_MASK (nr), (atomic_long_t *)p); } -static inline int test_and_set_bit (unsigned int nr,volatile unsigned long *p) +static __always_inline int +arch_test_and_set_bit (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr);
@@ -38,11 +42,12 @@ static inline int test_and_set_bit(unsigned int nr, volatile unsigned long *p)if (READ_ONCE (*p) & mask)return 1 ; - old =atomic_long_fetch_or (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_or (mask, (atomic_long_t *)p);return !!(old & mask); } -static inline int test_and_clear_bit (unsigned int nr,volatile unsigned long *p) +static __always_inline int +arch_test_and_clear_bit (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr);
@@ -51,18 +56,21 @@ static inline int test_and_clear_bit(unsigned int nr, volatile unsigned long *p)if (!(READ_ONCE (*p) & mask))return 0 ; - old =atomic_long_fetch_andnot (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_andnot (mask, (atomic_long_t *)p);return !!(old & mask); } -static inline int test_and_change_bit (unsigned int nr,volatile unsigned long *p) +static __always_inline int +arch_test_and_change_bit (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr); p +=BIT_WORD (nr); - old =atomic_long_fetch_xor (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_xor (mask, (atomic_long_t *)p);return !!(old & mask); } +#include <asm-generic/bitops/instrumented-atomic.h> +#endif /* _ASM_GENERIC_BITOPS_ATOMIC_H */
include/asm-generic/bitops/instrumented-non-atomic.h +14 -7
@@ -24,7 +24,8 @@*/ static inline void __set_bit (long nr,volatile unsigned long *addr) { -instrument_write (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch___set_bit_uses_plain_access)) +instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___set_bit (nr, addr); }
@@ -39,7 +40,8 @@ static inline void __set_bit(long nr, volatile unsigned long *addr)*/ static inline void __clear_bit (long nr,volatile unsigned long *addr) { -instrument_write (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch___clear_bit_uses_plain_access)) +instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___clear_bit (nr, addr); }
@@ -54,7 +56,8 @@ static inline void __clear_bit(long nr, volatile unsigned long *addr)*/ static inline void __change_bit (long nr,volatile unsigned long *addr) { -instrument_write (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch___change_bit_uses_plain_access)) +instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___change_bit (nr, addr); }
@@ -92,7 +95,8 @@ static inline void __instrument_read_write_bitop(long nr, volatile unsigned long*/ static inline bool __test_and_set_bit (long nr,volatile unsigned long *addr) { -__instrument_read_write_bitop (nr, addr); +if (!__is_defined (arch___test_and_set_bit_uses_plain_access)) +__instrument_read_write_bitop (nr, addr);return arch___test_and_set_bit (nr, addr); }
@@ -106,7 +110,8 @@ static inline bool __test_and_set_bit(long nr, volatile unsigned long *addr)*/ static inline bool __test_and_clear_bit (long nr,volatile unsigned long *addr) { -__instrument_read_write_bitop (nr, addr); +if (!__is_defined (arch___test_and_clear_bit_uses_plain_access)) +__instrument_read_write_bitop (nr, addr);return arch___test_and_clear_bit (nr, addr); }
@@ -120,7 +125,8 @@ static inline bool __test_and_clear_bit(long nr, volatile unsigned long *addr)*/ static inline bool __test_and_change_bit (long nr,volatile unsigned long *addr) { -__instrument_read_write_bitop (nr, addr); +if (!__is_defined (arch___test_and_change_bit_uses_plain_access)) +__instrument_read_write_bitop (nr, addr);return arch___test_and_change_bit (nr, addr); }
@@ -131,7 +137,8 @@ static inline bool __test_and_change_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_bit (long nr,const volatile unsigned long *addr) { -instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long )); +if (!__is_defined (arch_test_bit_uses_plain_access)) +instrument_atomic_read (addr +BIT_WORD (nr),sizeof (long ));return arch_test_bit (nr, addr); }
include/asm-generic/bitops/lock.h +21 -18
@@ -7,7 +7,7 @@#include <asm/barrier.h> /** -* test_and_set_bit_lock - Set a bit and return its old value, for lock +* arch_test_and_set_bit_lock - Set a bit and return its old value, for lock * @nr: Bit to set * @addr: Address to count from *
@@ -15,8 +15,8 @@* the returned value is 0. * It can be used to implement bit locks. */ -static inline int test_and_set_bit_lock (unsigned int nr, -volatile unsigned long *p) +static __always_inline int +arch_test_and_set_bit_lock (unsigned int nr,volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr);
@@ -25,26 +25,27 @@ static inline int test_and_set_bit_lock(unsigned int nr,if (READ_ONCE (*p) & mask)return 1 ; - old =atomic_long_fetch_or_acquire (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_or_acquire (mask, (atomic_long_t *)p);return !!(old & mask); }/** -* clear_bit_unlock - Clear a bit in memory, for unlock +* arch_clear_bit_unlock - Clear a bit in memory, for unlock * @nr: the bit to set * @addr: the address to start counting from * * This operation is atomic and provides release barrier semantics. */ -static inline void clear_bit_unlock (unsigned int nr,volatile unsigned long *p) +static __always_inline void +arch_clear_bit_unlock (unsigned int nr,volatile unsigned long *p) { p +=BIT_WORD (nr); -atomic_long_fetch_andnot_release (BIT_MASK (nr), (atomic_long_t *)p); +arch_atomic_long_fetch_andnot_release (BIT_MASK (nr), (atomic_long_t *)p); }/** -* __clear_bit_unlock - Clear a bit in memory, for unlock +* arch___clear_bit_unlock - Clear a bit in memory, for unlock * @nr: the bit to set * @addr: the address to start counting from *
@@ -54,38 +55,40 @@ static inline void clear_bit_unlock(unsigned int nr, volatile unsigned long *p)* * See for example x86's implementation. */ -static inline void __clear_bit_unlock (unsigned int nr, -volatile unsigned long *p) +static inline void +arch___clear_bit_unlock (unsigned int nr,volatile unsigned long *p) {unsigned long old; p +=BIT_WORD (nr); old =READ_ONCE (*p); old &= ~BIT_MASK (nr); -atomic_long_set_release ((atomic_long_t *)p, old); +arch_atomic_long_set_release ((atomic_long_t *)p, old); }/** -* clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom -* byte is negative, for unlock. +* arch_clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom +* byte is negative, for unlock. * @nr: the bit to clear * @addr: the address to start counting from * * This is a bit of a one-trick-pony for the filemap code, which clears * PG_locked and tests PG_waiters, */ -#ifndef clear_bit_unlock_is_negative_byte -static inline bool clear_bit_unlock_is_negative_byte (unsigned int nr, -volatile unsigned long *p) +#ifndef arch_clear_bit_unlock_is_negative_byte +static inline bool arch_clear_bit_unlock_is_negative_byte (unsigned int nr, +volatile unsigned long *p) {long old;unsigned long mask =BIT_MASK (nr); p +=BIT_WORD (nr); - old =atomic_long_fetch_andnot_release (mask, (atomic_long_t *)p); + old =arch_atomic_long_fetch_andnot_release (mask, (atomic_long_t *)p);return !!(old &BIT (7 )); } -#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte +#define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte#endif +#include <asm-generic/bitops/instrumented-lock.h> +#endif /* _ASM_GENERIC_BITOPS_LOCK_H_ */
include/asm-generic/bitops/non-atomic.h +28 -13
123 lines not on this page · raw message