Allow architectures to override __READ_ONCE()
Subsystems (18)
Will Deacon
maintainercover Allow architectures to override __READ_ONCE()
Hi all,
This is version three of the patches I previously posted here:
v1: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/r/[email protected]
Changes since v2 include:
* Actually add the barrier in READ_ONCE() for Alpha!
* Implement Alpha's smp_load_acquire() using __READ_ONCE(), rather than the other way around.
* Further untangling of header files
* Use CONFIG_LTO instead of CONFIG_CLANG_LTO
I have booted this on arm64, and build-tested as follows:
- arm64 allnoconfig, defconfig (also bisected) and allmodconfig
- arm32 allnoconfig, defconfig and allmodconfig
- x86_64 allnoconfig, defconfig and allmodcofig
- alpha defconfig, defconfig+CONFIG_SMP=y
- riscv64 defconfig
- powerpc64 defconfig
- s390 defconfig
- sparc32 defconfig, defconfig+CONFIG_SMP=y
- sparc64 defconfig
Cheers,
Will
--->8
SeongJae Park (1):
Documentation/barriers/kokr: Remove references to
[smp_]read_barrier_depends()
Will Deacon (18):
tools: bpf: Use local copy of headers including uapi/linux/filter.h compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h asm/rwonce: Allow __READ_ONCE to be overridden by the architecture
alpha: Override READ_ONCE() with barriered implementation asm/rwonce: Remove smp_read_barrier_depends() invocation asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h'
vhost: Remove redundant use of read_barrier_depends() barrier
alpha: Replace smp_read_barrier_depends() usage with smp_[r]mb() locking/barriers: Remove definitions for [smp_]read_barrier_depends() Documentation/barriers: Remove references to
[smp_]read_barrier_depends()
tools/memory-model: Remove smp_read_barrier_depends() from informal
doc
include/linux: Remove smp_read_barrier_depends() from comments
checkpatch: Remove checks relating to [smp_]read_barrier_depends()
arm64: Reduce the number of header files pulled into vmlinux.lds.S
arm64: alternatives: Split up alternative.h
arm64: cpufeatures: Add capability for LDAPR instruction
arm64: alternatives: Remove READ_ONCE() usage during patch operation
arm64: lto: Strengthen READ_ONCE() to acquire when CONFIG_LTO=y
.../RCU/Design/Requirements/Requirements.rst | 2 +- Documentation/memory-barriers.txt | 156 +--------- .../translations/ko_KR/memory-barriers.txt | 146 +-------- arch/alpha/include/asm/atomic.h | 16 +- arch/alpha/include/asm/barrier.h | 59 +--- arch/alpha/include/asm/pgtable.h | 10 +- arch/alpha/include/asm/rwonce.h | 35 +++ arch/arm/include/asm/vdso/gettimeofday.h | 1 + arch/arm64/Kconfig | 3 + arch/arm64/include/asm/alternative-macros.h | 276 ++++++++++++++++++ arch/arm64/include/asm/alternative.h | 267 +---------------- arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/include/asm/insn.h | 3 +- arch/arm64/include/asm/kernel-pgtable.h | 2 +- arch/arm64/include/asm/memory.h | 11 +- arch/arm64/include/asm/rwonce.h | 63 ++++ arch/arm64/include/asm/uaccess.h | 1 + .../include/asm/vdso/compat_gettimeofday.h | 1 + arch/arm64/include/asm/vdso/gettimeofday.h | 1 + arch/arm64/kernel/alternative.c | 7 +- arch/arm64/kernel/cpufeature.c | 10 + arch/arm64/kernel/entry.S | 1 + arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vdso32/Makefile | 2 +- arch/arm64/kernel/vmlinux.lds.S | 1 - arch/arm64/kvm/hyp-init.S | 1 + arch/riscv/include/asm/vdso/gettimeofday.h | 1 + drivers/vhost/vhost.c | 5 - include/asm-generic/Kbuild | 1 + include/asm-generic/barrier.h | 19 +- include/asm-generic/rwonce.h | 80 +++++ include/linux/compiler.h | 83 +----- include/linux/nospec.h | 2 + include/linux/percpu-refcount.h | 2 +- include/linux/ptr_ring.h | 2 +- mm/memory.c | 2 +- scripts/checkpatch.pl | 9 +- tools/bpf/Makefile | 3 +- tools/include/uapi/linux/filter.h | 90 ++++++ .../Documentation/explanation.txt | 26 +- 40 files changed, 636 insertions(+), 769 deletions(-) create mode 100644 arch/alpha/include/asm/rwonce.h create mode 100644 arch/arm64/include/asm/alternative-macros.h create mode 100644 arch/arm64/include/asm/rwonce.h create mode 100644 include/asm-generic/rwonce.h create mode 100644 tools/include/uapi/linux/filter.h
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 1/19 tools: bpf: Use local copy of headers including uapi/linux/filter.h
Pulling header files directly out of the kernel sources for inclusion in userspace programs is highly error prone, not least because it bypasses the kbuild infrastructure entirely and so may end up referencing other header files that have not been generated.
Subsequent patches will cause compiler.h to pull in the ungenerated asm/rwonce.h file via filter.h, breaking the build for tools/bpf:
| $ make -C tools/bpf | make: Entering directory '/linux/tools/bpf' | CC bpf_jit_disasm.o | LINK bpf_jit_disasm | CC bpf_dbg.o | In file included from /linux/include/uapi/linux/filter.h:9, | from /linux/tools/bpf/bpf_dbg.c:41: | /linux/include/linux/compiler.h:247:10: fatal error: asm/rwonce.h: No such file or directory | #include <asm/rwonce.h> | ^~~~~~~~~~~~~~ | compilation terminated. | make: *** [Makefile:61: bpf_dbg.o] Error 1 | make: Leaving directory '/linux/tools/bpf'
Take a copy of the installed version of linux/filter.h (i.e. the one created by the 'headers_install' target) into tools/include/uapi/linux/ and adjust the BPF tool Makefile to reference the local include directories instead of those in the main source tree.
--- tools/bpf/Makefile | 3 +- tools/include/uapi/linux/filter.h | 90 +++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+), 1 deletion(-) create mode 100644 tools/include/uapi/linux/filter.h
tools/bpf/Makefile +2 -1
@@ -9,7 +9,8 @@ MAKE = make INSTALL ?= install CFLAGS += -Wall -O2 -CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/include/uapi -I$(srctree)/include +CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \ + -I$(srctree)/tools/include # This will work when bpf is built in tools env. where srctree # isn't set and when invoked from selftests build, where srctree
tools/include/uapi/linux/filter.h new file +90 -0
90 lines not on this page · raw message
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 2/19 compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h
In reply to Will DeaconIn preparation for allowing architectures to define their own implementation of the READ_ONCE() macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'.
--- include/asm-generic/Kbuild | 1 + include/asm-generic/barrier.h | 2 +- include/asm-generic/rwonce.h | 91 +++++++++++++++++++++++++++++++++++ include/linux/compiler.h | 83 +------------------------------- 4 files changed, 95 insertions(+), 82 deletions(-) create mode 100644 include/asm-generic/rwonce.h
include/asm-generic/Kbuild +1 -0
@@ -45,6 +45,7 @@ mandatory-y += pci.h mandatory-y += percpu.h mandatory-y += pgalloc.h mandatory-y += preempt.h +mandatory-y += rwonce.h mandatory-y += sections.h mandatory-y += serial.h mandatory-y += shmparam.h
include/asm-generic/barrier.h +1 -1
@@ -13,7 +13,7 @@#ifndef __ASSEMBLY__ -#include <linux/compiler.h> +#include <asm/rwonce.h> #ifndef nop#define nop () asm volatile ("nop")
include/asm-generic/rwonce.h new file +91 -0
91 lines not on this page · raw message
include/linux/compiler.h +2 -81
99 lines not on this page · raw message
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 3/19 asm/rwonce: Allow __READ_ONCE to be overridden by the architecture
In reply to Will DeaconThe meat and potatoes of READ_ONCE() is defined by the __READ_ONCE() macro, which uses a volatile casts in an attempt to avoid tearing of byte, halfword, word and double-word accesses. Allow this to be overridden by the architecture code in the case that things like memory barriers are also required.
--- include/asm-generic/rwonce.h | 2 ++ 1 file changed, 2 insertions(+)
include/asm-generic/rwonce.h +2 -0
@@ -33,7 +33,9 @@* atomicity or dependency ordering guarantees. Note that this may result * in tears! */ +#ifndef __READ_ONCE#define __READ_ONCE (x) (*(const volatile __unqual_scalar_typeof(x) *)&(x)) +#endif #define __READ_ONCE_SCALAR (x) \ ({ \
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 4/19 alpha: Override READ_ONCE() with barriered implementation
In reply to Will DeaconRather then relying on the core code to use smp_read_barrier_depends() as part of the READ_ONCE() definition, instead override __READ_ONCE() in the Alpha code so that it generates the required mb() and then implement smp_load_acquire() using the new macro to avoid redundant back-to-back barriers from the generic implementation.
--- arch/alpha/include/asm/barrier.h | 59 +++----------------------------- arch/alpha/include/asm/rwonce.h | 35 +++++++++++++++++++ 2 files changed, 40 insertions(+), 54 deletions(-) create mode 100644 arch/alpha/include/asm/rwonce.h
arch/alpha/include/asm/barrier.h +5 -54
@@ -2,64 +2,15 @@#ifndef __BARRIER_H#define __BARRIER_H -#include <asm/compiler.h> -#define mb () __asm__ __volatile__("mb": : :"memory")#define rmb () __asm__ __volatile__("mb": : :"memory")#define wmb () __asm__ __volatile__("wmb": : :"memory") -/** -* read_barrier_depends - Flush all pending reads that subsequents reads -* depend on. -* -* No data-dependent reads from memory-like regions are ever reordered -* over this barrier. All reads preceding this primitive are guaranteed -* to access memory (but not necessarily other CPUs' caches) before any -* reads following this primitive that depend on the data return by -* any of the preceding reads. This primitive is much lighter weight than -* rmb() on most CPUs, and is never heavier weight than is -* rmb(). -* -* These ordering constraints are respected by both the local CPU -* and the compiler. -* -* Ordering is not guaranteed by anything other than these primitives, -* not even by data dependencies. See the documentation for -* memory_barrier() for examples and URLs to more information. -* -* For example, the following code would force ordering (the initial -* value of "a" is zero, "b" is one, and "p" is "&a"): -* -* <programlisting> -* CPU 0 CPU 1 -* -* b = 2; -* memory_barrier(); -* p = &b; q = p; -* read_barrier_depends(); -* d = *q; -* </programlisting> -* -* because the read of "*q" depends on the read of "p" and these -* two reads are separated by a read_barrier_depends(). However, -* the following code, with the same initial values for "a" and "b": -* -* <programlisting> -* CPU 0 CPU 1 -* -* a = 2; -* memory_barrier(); -* b = 3; y = b; -* read_barrier_depends(); -* x = a; -* </programlisting> -* -* does not enforce ordering, since there is no data dependency between -* the read of "a" and the read of "b". Therefore, on some CPUs, such -* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb() -* in cases like this where there are no data dependencies. -*/ -#define read_barrier_depends () __asm__ __volatile__("mb": : :"memory") +#define __smp_load_acquire (p) \ +({ \ + compiletime_assert_atomic_type(*p); \ + __READ_ONCE(*p); \ +})#ifdef CONFIG_SMP #define __ASM_SMP_MB "\tmb\n"
arch/alpha/include/asm/rwonce.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2019 Google LLC. +*/ +#ifndef __ASM_RWONCE_H +#define __ASM_RWONCE_H + +#ifdef CONFIG_SMP + +#include <asm/barrier.h> + +/* +* Alpha is apparently daft enough to reorder address-dependent loads +* on some CPU implementations. Knock some common sense into it with +* a memory barrier in READ_ONCE(). +* +* For the curious, more information about this unusual reordering is +* available in chapter 15 of the "perfbook": +* +* https://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html +* +*/ +#define __READ_ONCE (x) \ +({ \ + __unqual_scalar_typeof(x) __x = \ + (*(volatile typeof(__x) *)(&(x))); \ + mb(); \ + (typeof(x))__x; \ +}) + +#endif /* CONFIG_SMP */ + +#include <asm-generic/rwonce.h> + +#endif /* __ASM_RWONCE_H */
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 5/19 asm/rwonce: Remove smp_read_barrier_depends() invocation
In reply to Will DeaconAlpha overrides __READ_ONCE() directly, so there's no need to use smp_read_barrier_depends() in the core code. This also means that __READ_ONCE() can be relied upon to provide dependency ordering.
--- include/asm-generic/rwonce.h | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-)
include/asm-generic/rwonce.h +4 -15
@@ -30,24 +30,16 @@/* * Use __READ_ONCE() instead of READ_ONCE() if you do not require any -* atomicity or dependency ordering guarantees. Note that this may result -* in tears! +* atomicity. Note that this may result in tears! */ #ifndef __READ_ONCE#define __READ_ONCE (x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))#endif -#define __READ_ONCE_SCALAR (x) \ -({ \ - __unqual_scalar_typeof(x) __x = __READ_ONCE(x); \ - smp_read_barrier_depends(); \ - (typeof(x))__x; \ -}) -#define READ_ONCE (x) \ ({ \ compiletime_assert_rwonce_type(x); \ - __READ_ONCE_SCALAR(x); \ + __READ_ONCE(x); \ })#define __WRITE_ONCE (x, val) \
@@ -74,12 +66,9 @@ unsigned long __read_once_word_nocheck(const void *addr)*/ #define READ_ONCE_NOCHECK (x) \ ({ \ - unsigned long __x; \ - compiletime_assert(sizeof(x) == sizeof(__x), \ + compiletime_assert(sizeof(x) == sizeof(unsigned long), \ "Unsupported access size for READ_ONCE_NOCHECK()."); \ - __x = __read_once_word_nocheck(&(x)); \ - smp_read_barrier_depends(); \ - (typeof(x))__x; \ + (typeof(x))__read_once_word_nocheck(&(x)); \ }) static __no_kasan_or_inline
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 6/19 asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h'
In reply to Will DeaconNow that 'smp_read_barrier_depends()' has gone the way of the Norwegian Blue, drop the inclusion of <asm/barrier.h> in 'asm-generic/rwonce.h'.
This requires fixups to some architecture vdso headers which were previously relying on 'asm/barrier.h' coming in via 'linux/compiler.h'.
--- arch/arm/include/asm/vdso/gettimeofday.h | 1 + arch/arm64/include/asm/vdso/compat_gettimeofday.h | 1 + arch/arm64/include/asm/vdso/gettimeofday.h | 1 + arch/riscv/include/asm/vdso/gettimeofday.h | 1 + include/asm-generic/rwonce.h | 2 -- include/linux/nospec.h | 2 ++ 6 files changed, 6 insertions(+), 2 deletions(-)
arch/arm/include/asm/vdso/gettimeofday.h +1 -0
@@ -7,6 +7,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/errno.h> #include <asm/unistd.h> #include <asm/vdso/cp15.h>
arch/arm64/include/asm/vdso/compat_gettimeofday.h +1 -0
@@ -7,6 +7,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/unistd.h> #include <asm/errno.h>
arch/arm64/include/asm/vdso/gettimeofday.h +1 -0
@@ -7,6 +7,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/unistd.h> #define VDSO_HAS_CLOCK_GETRES 1
arch/riscv/include/asm/vdso/gettimeofday.h +1 -0
@@ -4,6 +4,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/unistd.h> #include <asm/csr.h> #include <uapi/linux/time.h>
include/asm-generic/rwonce.h +0 -2
@@ -26,8 +26,6 @@#include <linux/kasan-checks.h> #include <linux/kcsan-checks.h> -#include <asm/barrier.h> -/* * Use __READ_ONCE() instead of READ_ONCE() if you do not require any * atomicity. Note that this may result in tears!
include/linux/nospec.h +2 -0
@@ -5,6 +5,8 @@#ifndef _LINUX_NOSPEC_H#define _LINUX_NOSPEC_H + +#include <linux/compiler.h> #include <asm/barrier.h> struct task_struct ;
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 7/19 vhost: Remove redundant use of read_barrier_depends() barrier
In reply to Will DeaconSince commit 76ebbe78f739 ("locking/barriers: Add implicit smp_read_barrier_depends() to READ_ONCE()"), there is no need to use smp_read_barrier_depends() outside of the Alpha architecture code.
Unfortunately, there is precisely _one_ user in the vhost code, and there isn't an obvious READ_ONCE() access making the barrier redundant. However, on closer inspection (thanks, Jason), it appears that vring synchronisation between the producer and consumer occurs via the 'avail_idx' field, which is followed up by an rmb() in vhost_get_vq_desc(), making the read_barrier_depends() redundant on Alpha.
Jason says:
| I'm also confused about the barrier here, basically in driver side | we did: | | 1) allocate pages | 2) store pages in indirect->addr | 3) smp_wmb() | 4) increase the avail idx (somehow a tail pointer of vring) | | in vhost we did: | | 1) read avail idx | 2) smp_rmb() | 3) read indirect->addr | 4) read from indirect->addr | | It looks to me even the data dependency barrier is not necessary | since we have rmb() which is sufficient for us to the correct | indirect->addr and driver are not expected to do any writing to | indirect->addr after avail idx is increased
Remove the redundant barrier invocation.
--- drivers/vhost/vhost.c | 5 ----- 1 file changed, 5 deletions(-)
drivers/vhost/vhost.c +0 -5
@@ -2092,11 +2092,6 @@ static int get_indirect(struct vhost_virtqueue *vq,return ret; }iov_iter_init (&from,READ , vq->indirect, ret, len); - -/* We will use the result as an address to read from, so most -* architectures only need a compiler barrier here. */ -read_barrier_depends (); - count = len /sizeof desc;/* Buffers are chained via a 16 bit next field, so * we can have at most 2^16 of these. */
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 8/19 alpha: Replace smp_read_barrier_depends() usage with smp_[r]mb()
In reply to Will DeaconIn preparation for removing smp_read_barrier_depends() altogether, move the Alpha code over to using smp_rmb() and smp_mb() directly.
--- arch/alpha/include/asm/atomic.h | 16 ++++++++-------- arch/alpha/include/asm/pgtable.h | 10 +++++----- mm/memory.c | 2 +- 3 files changed, 14 insertions(+), 14 deletions(-)
arch/alpha/include/asm/atomic.h +8 -8
@@ -16,10 +16,10 @@/* * To ensure dependency ordering is preserved for the _relaxed and -* _release atomics, an smp_read_barrier_depends() is unconditionally -* inserted into the _relaxed variants, which are used to build the -* barriered versions. Avoid redundant back-to-back fences in the -* _acquire and _fence versions. +* _release atomics, an smp_mb() is unconditionally inserted into the +* _relaxed variants, which are used to build the barriered versions. +* Avoid redundant back-to-back fences in the _acquire and _fence +* versions. */ #define __atomic_acquire_fence ()#define __atomic_post_full_fence ()
@@ -70,7 +70,7 @@ static inline int atomic_##op##_return_relaxed(int i, atomic_t *v) \ ".previous" \ :"=&r" (temp), "=m" (v->counter), "=&r" (result) \ :"Ir" (i), "m" (v->counter) : "memory"); \ -smp_read_barrier_depends (); \ +smp_mb (); \return result; \ }
@@ -88,7 +88,7 @@ static inline int atomic_fetch_##op##_relaxed(int i, atomic_t *v) \ ".previous" \ :"=&r" (temp), "=m" (v->counter), "=&r" (result) \ :"Ir" (i), "m" (v->counter) : "memory"); \ -smp_read_barrier_depends (); \ +smp_mb (); \return result; \ }
@@ -123,7 +123,7 @@ static __inline__ s64 atomic64_##op##_return_relaxed(s64 i, atomic64_t * v) \ ".previous" \ :"=&r" (temp), "=m" (v->counter), "=&r" (result) \ :"Ir" (i), "m" (v->counter) : "memory"); \ -smp_read_barrier_depends (); \ +smp_mb (); \return result; \ }
@@ -141,7 +141,7 @@ static __inline__ s64 atomic64_fetch_##op##_relaxed(s64 i, atomic64_t * v) \ ".previous" \ :"=&r" (temp), "=m" (v->counter), "=&r" (result) \ :"Ir" (i), "m" (v->counter) : "memory"); \ -smp_read_barrier_depends (); \ +smp_mb (); \return result; \ }
arch/alpha/include/asm/pgtable.h +5 -5
@@ -277,9 +277,9 @@ extern inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= __DIRTY_BITS; returextern inline pte_t pte_mkyoung (pte_t pte) {pte_val (pte) |= __ACCESS_BITS;return pte; }/* -* The smp_read_barrier_depends() in the following functions are required to -* order the load of *dir (the pointer in the top level page table) with any -* subsequent load of the returned pmd_t *ret (ret is data dependent on *dir). +* The smp_rmb() in the following functions are required to order the load of +* *dir (the pointer in the top level page table) with any subsequent load of +* the returned pmd_t *ret (ret is data dependent on *dir). * * If this ordering is not enforced, the CPU might load an older value of * *ret, which may be uninitialized data. See mm/memory.c:__pte_alloc for
@@ -293,7 +293,7 @@ extern inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= __ACCESS_BITS; retuextern inline pmd_t *pmd_offset (pud_t * dir,unsigned long address) {pmd_t *ret = (pmd_t *)pud_page_vaddr (*dir) + ((address >>PMD_SHIFT ) & (PTRS_PER_PAGE -1 )); -smp_read_barrier_depends ();/* see above */ +smp_rmb ();/* see above */ return ret; }#define pmd_offset pmd_offset
@@ -303,7 +303,7 @@ extern inline pte_t * pte_offset_kernel(pmd_t * dir, unsigned long address) {pte_t *ret = (pte_t *)pmd_page_vaddr (*dir) + ((address >>PAGE_SHIFT ) & (PTRS_PER_PAGE -1 )); -smp_read_barrier_depends ();/* see above */ +smp_rmb ();/* see above */ return ret; }#define pte_offset_kernel pte_offset_kernel
mm/memory.c +1 -1
@@ -437,7 +437,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)* of a chain of data-dependent loads, meaning most CPUs (alpha * being the notable exception) will already guarantee loads are * seen in-order. See the alpha page table accessors for the -* smp_read_barrier_depends() barriers in page table walking code. +* smp_rmb() barriers in page table walking code. */ smp_wmb ();/* Could be smp_wmb__xxx(before|after)_spin_lock */
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 9/19 locking/barriers: Remove definitions for [smp_]read_barrier_depends()
In reply to Will DeaconThere are no remaining users of [smp_]read_barrier_depends(), so remove it from the generic implementation of 'barrier.h'.
--- include/asm-generic/barrier.h | 17 ----------------- 1 file changed, 17 deletions(-)
include/asm-generic/barrier.h +0 -17
@@ -46,10 +46,6 @@#define dma_wmb () wmb()#endif -#ifndef read_barrier_depends -#define read_barrier_depends () do { } while (0) -#endif -#ifndef __smp_mb#define __smp_mb () mb()#endif
@@ -62,10 +58,6 @@#define __smp_wmb () wmb()#endif -#ifndef __smp_read_barrier_depends -#define __smp_read_barrier_depends () read_barrier_depends() -#endif -#ifdef CONFIG_SMP #ifndef smp_mb
@@ -80,10 +72,6 @@#define smp_wmb () __smp_wmb()#endif -#ifndef smp_read_barrier_depends -#define smp_read_barrier_depends () __smp_read_barrier_depends() -#endif -#else /* !CONFIG_SMP */ #ifndef smp_mb
@@ -98,10 +86,6 @@#define smp_wmb () barrier()#endif -#ifndef smp_read_barrier_depends -#define smp_read_barrier_depends () do { } while (0) -#endif -#endif /* CONFIG_SMP */ #ifndef __smp_store_mb
@@ -196,7 +180,6 @@ do { \#define virt_mb () __smp_mb()#define virt_rmb () __smp_rmb()#define virt_wmb () __smp_wmb() -#define virt_read_barrier_depends () __smp_read_barrier_depends()#define virt_store_mb (var, value) __smp_store_mb(var, value)#define virt_mb__before_atomic () __smp_mb__before_atomic()#define virt_mb__after_atomic () __smp_mb__after_atomic()
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 10/19 Documentation/barriers: Remove references to [smp_]read_barrier_depends()
In reply to Will DeaconThe [smp_]read_barrier_depends() barrier macros no longer exist as part of the Linux memory model, so remove all references to them from the Documentation/ directory.
Although this is fairly mechanical on the whole, we drop the "CACHE COHERENCY" section entirely from 'memory-barriers.txt' as it doesn't make any sense now that the dependency barriers have been removed.
--- .../RCU/Design/Requirements/Requirements.rst | 2 +- Documentation/memory-barriers.txt | 156 +----------------- 2 files changed, 9 insertions(+), 149 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.rst +1 -1
@@ -463,7 +463,7 @@ again without disrupting RCU readers. This guarantee was only partially premeditated. DYNIX/ptx used an explicit memory barrier for publication, but had nothing resembling ``rcu_dereference()`` for subscription, nor did it have anything -resembling the ``smp_read_barrier_depends()`` that was later subsumed +resembling the dependency-ordering barrier that was later subsumed into ``rcu_dereference()`` and later still into ``READ_ONCE()``. The need for these operations made itself known quite suddenly at a late-1990s meeting with the DEC Alpha architects, back in the days when
Documentation/memory-barriers.txt +8 -148
174 lines not on this page · raw message
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 11/19 Documentation/barriers/kokr: Remove references to [smp_]read_barrier_depends()
In reply to Will DeaconFrom: SeongJae Park <[email protected]>
This commit translates commit ("Documentation/barriers: Remove references to [smp_]read_barrier_depends()") into Korean.
--- .../translations/ko_KR/memory-barriers.txt | 146 +----------------- 1 file changed, 3 insertions(+), 143 deletions(-)
Documentation/translations/ko_KR/memory-barriers.txt +3 -143
@@ -577,7 +577,7 @@ ACQUIRE 는 해당 오퍼레이션의 로드 부분에만 적용되고 RELEASE 데이터 의존성 배리어 (역사적) ----------------------------- -리눅스 커널 v4.15 기준으로, smp_read_barrier_depends() 가 READ_ONCE() 에 +리눅스 커널 v4.15 기준으로, smp_mb() 가 DEC Alpha 용 READ_ONCE() 코드에 추가되었는데, 이는 이 섹션에 주의를 기울여야 하는 사람들은 DEC Alpha 아키텍쳐 전용 코드를 만드는 사람들과 READ_ONCE() 자체를 만드는 사람들 뿐임을 의미합니다. 그런 분들을 위해, 그리고 역사에 관심 있는 분들을 위해, 여기 데이터 의존성
@@ -2664,144 +2664,6 @@ CPU 코어는 프로그램의 인과성이 유지된다고만 여겨진다면 수도 있습니다. -캐시 일관성 ------------ - -하지만 삶은 앞에서 이야기한 것처럼 단순하지 않습니다: 캐시들은 일관적일 것으로 -기대되지만, 그 일관성이 순서에도 적용될 거라는 보장은 없습니다. 한 CPU 에서 -만들어진 변경 사항은 최종적으로는 시스템의 모든 CPU 에게 보여지게 되지만, 다른 -CPU 들에게도 같은 순서로 보이게 될 거라는 보장은 없다는 뜻입니다. - - -두개의 CPU (1 & 2) 가 달려 있고, 각 CPU 에 두개의 데이터 캐시(CPU 1 은 A/B 를, -CPU 2 는 C/D 를 갖습니다)가 병렬로 연결되어 있는 시스템을 다룬다고 생각해 -봅시다: - - : - : +--------+ - : +---------+ | | - +--------+ : +--->| Cache A |<------->| | - | | : | +---------+ | | - | CPU 1 |<---+ | | - | | : | +---------+ | | - +--------+ : +--->| Cache B |<------->| | - : +---------+ | | - : | Memory | - : +---------+ | System | - +--------+ : +--->| Cache C |<------->| | - | | : | +---------+ | | - | CPU 2 |<---+ | | - | | : | +---------+ | | - +--------+ : +--->| Cache D |<------->| | - : +---------+ | | - : +--------+ - : - -이 시스템이 다음과 같은 특성을 갖는다 생각해 봅시다: - - (*) 홀수번 캐시라인은 캐시 A, 캐시 C 또는 메모리에 위치할 수 있음; - - (*) 짝수번 캐시라인은 캐시 B, 캐시 D 또는 메모리에 위치할 수 있음; - - (*) CPU 코어가 한개의 캐시에 접근하는 동안, 다른 캐시는 - 더티 캐시라인을 - 메모리에 내리거나 추측성 로드를 하거나 하기 위해 - 시스템의 다른 부분에 - 액세스 하기 위해 버스를 사용할 수 있음; - - (*) 각 캐시는 시스템의 나머지 부분들과 일관성을 맞추기 위해 해당 캐시에 - 적용되어야 할 오퍼레이션들의 큐를 가짐; - - (*) 이 일관성 큐는 캐시에 이미 존재하는 라인에 가해지는 평범한 로드에 의해서는 - 비워지지 않는데, 큐의 오퍼레이션들이 이 로드의 결과에 영향을 끼칠 수 있다 - 할지라도 그러함. - -이제, 첫번째 CPU 에서 두개의 쓰기 오퍼레이션을 만드는데, 해당 CPU 의 캐시에 -요청된 순서로 오퍼레이션이 도달됨을 보장하기 위해 두 오퍼레이션 사이에 쓰기 -배리어를 사용하는 상황을 상상해 봅시다: - - CPU 1 CPU 2 COMMENT - =============== =============== ======================================= - u == 0, v == 1 and p == &u, q == &u - v = 2; - smp_wmb(); v 의 변경이 p 의 변경 전에 보일 것을 - 분명히 함 - <A:modify v=2> v 는 이제 캐시 A 에 독점적으로 존재함 - p = &v; - <B:modify p=&v> p 는 이제 캐시 B 에 독점적으로 존재함 - -여기서의 쓰기 메모리 배리어는 CPU 1 의 캐시가 올바른 순서로 업데이트 된 것으로 -시스템의 다른 CPU 들이 인지하게 만듭니다. 하지만, 이제 두번째 CPU 가 그 값들을 -읽으려 하는 상황을 생각해 봅시다: - - CPU 1 CPU 2 COMMENT - =============== =============== ======================================= - ... - q = p; - x = *q; - -위의 두개의 읽기 오퍼레이션은 예상된 순서로 일어나지 못할 수 있는데, 두번째 CPU -의 한 캐시에 다른 캐시 이벤트가 발생해 v 를 담고 있는 캐시라인의 해당 캐시에의 -업데이트가 지연되는 사이, p 를 담고 있는 캐시라인은 두번째 CPU 의 다른 캐시에 -업데이트 되어버렸을 수 있기 때문입니다. - - CPU 1 CPU 2 COMMENT - =============== =============== ======================================= - u == 0, v == 1 and p == &u, q == &u - v = 2; - smp_wmb(); - <A:modify v=2> <C:busy> - <C:queue v=2> - p = &v; q = p; - <D:request p> - <B:modify p=&v> <D:commit p=&v> - <D:read p> - x = *q; - <C:read *q> 캐시에 업데이트 되기 전의 v 를 읽음 - <C:unbusy> - <C:commit v=2> - -기본적으로, 두개의 캐시라인 모두 CPU 2 에 최종적으로는 업데이트 될 것이지만, -별도의 개입 없이는, 업데이트의 순서가 CPU 1 에서 만들어진 순서와 동일할 -것이라는 보장이 없습니다. - - -여기에 개입하기 위해선, 데이터 의존성 배리어나 읽기 배리어를 로드 오퍼레이션들 -사이에 넣어야 합니다 (v4.15 부터는 READ_ONCE() 매크로에 의해 무조건적으로 -그렇게 됩니다). 이렇게 함으로써 캐시가 다음 요청을 처리하기 전에 일관성 큐를 -처리하도록 강제하게 됩니다. - - CPU 1 CPU 2 COMMENT - =============== =============== ======================================= - u == 0, v == 1 and p == &u, q == &u - v = 2; - smp_wmb(); - <A:modify v=2> <C:busy> - <C:queue v=2> - p = &v; q = p; - <D:request p> - <B:modify p=&v> <D:commit p=&v> - <D:read p> - smp_read_barrier_depends() - <C:unbusy> - <C:commit v=2> - x = *q; - <C:read *q> 캐시에 업데이트 된 v 를 읽음 - - -이런 부류의 문제는 DEC Alpha 계열 프로세서들에서 발견될 수 있는데, 이들은 -데이터 버스를 좀 더 잘 사용해 성능을 개선할 수 있는, 분할된 캐시를 가지고 있기 -때문입니다. 대부분의 CPU 는 하나의 읽기 오퍼레이션의 메모리 액세스가 다른 읽기 -오퍼레이션에 의존적이라면 데이터 의존성 배리어를 내포시킵니다만, 모두가 그런건 -아니기 때문에 이점에 의존해선 안됩니다. - -다른 CPU 들도 분할된 캐시를 가지고 있을 수 있지만, 그런 CPU 들은 평범한 메모리 -액세스를 위해서도 이 분할된 캐시들 사이의 조정을 해야만 합니다. Alpha 는 가장 -약한 메모리 순서 시맨틱 (semantic) 을 선택함으로써 메모리 배리어가 명시적으로 -사용되지 않았을 때에는 그런 조정이 필요하지 않게 했으며, 이는 Alpha 가 당시에 -더 높은 CPU 클락 속도를 가질 수 있게 했습니다. 하지만, (다시 말하건대, v4.15 -이후부터는) Alpha 아키텍쳐 전용 코드와 READ_ONCE() 매크로 내부에서를 제외하고는 -smp_read_barrier_depends() 가 사용되지 않아야 함을 알아두시기 바랍니다. - - 캐시 일관성 VS DMA ------------------
@@ -2962,10 +2824,8 @@ Alpha CPU 의 일부 버전은 분할된 데이터 캐시를 가지고 있어서 데이터의 발견을 올바른 순서로 일어나게 하기 때문입니다. 리눅스 커널의 메모리 배리어 모델은 Alpha 에 기초해서 정의되었습니다만, v4.15 -부터는 리눅스 커널이 READ_ONCE() 내에 smp_read_barrier_depends() 를 추가해서 -Alpha 의 메모리 모델로의 영향력이 크게 줄어들긴 했습니다. - -위의 "캐시 일관성" 서브섹션을 참고하세요. +부터는 Alpha 용 READ_ONCE() 코드 내에 smp_mb() 가 추가되어서 메모리 모델로의 +Alpha 의 영향력이 크게 줄어들었습니다. 가상 머신 게스트
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 12/19 tools/memory-model: Remove smp_read_barrier_depends() from informal doc
In reply to Will Deaconsmp_read_barrier_depends() has gone the way of mmiowb() and so many esoteric memory barriers before it. Drop the two mentions of this deceased barrier from the LKMM informal explanation document.
--- .../Documentation/explanation.txt | 26 +++++++++---------- 1 file changed, 12 insertions(+), 14 deletions(-)
tools/memory-model/Documentation/explanation.txt +12 -14
@@ -1122,12 +1122,10 @@ maintain at least the appearance of FIFO order. In practice, this difficulty is solved by inserting a special fence between P1's two loads when the kernel is compiled for the Alpha architecture. In fact, as of version 4.15, the kernel automatically -adds this fence (called smp_read_barrier_depends() and defined as -nothing at all on non-Alpha builds) after every READ_ONCE() and atomic -load. The effect of the fence is to cause the CPU not to execute any -po-later instructions until after the local cache has finished -processing all the stores it has already received. Thus, if the code -was changed to: +adds this fence after every READ_ONCE() and atomic load on Alpha. The +effect of the fence is to cause the CPU not to execute any po-later +instructions until after the local cache has finished processing all +the stores it has already received. Thus, if the code was changed to: P1() {
@@ -1146,14 +1144,14 @@ READ_ONCE() or another synchronization primitive rather than accessed directly. The LKMM requires that smp_rmb(), acquire fences, and strong fences -share this property with smp_read_barrier_depends(): They do not allow -the CPU to execute any po-later instructions (or po-later loads in the -case of smp_rmb()) until all outstanding stores have been processed by -the local cache. In the case of a strong fence, the CPU first has to -wait for all of its po-earlier stores to propagate to every other CPU -in the system; then it has to wait for the local cache to process all -the stores received as of that time -- not just the stores received -when the strong fence began. +share this property: They do not allow the CPU to execute any po-later +instructions (or po-later loads in the case of smp_rmb()) until all +outstanding stores have been processed by the local cache. In the +case of a strong fence, the CPU first has to wait for all of its +po-earlier stores to propagate to every other CPU in the system; then +it has to wait for the local cache to process all the stores received +as of that time -- not just the stores received when the strong fence +began. And of course, none of this matters for any architecture other than Alpha.
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 13/19 include/linux: Remove smp_read_barrier_depends() from comments
In reply to Will Deaconsmp_read_barrier_depends() doesn't exist any more, so reword the two comments that mention it to refer to "dependency ordering" instead.
--- include/linux/percpu-refcount.h | 2 +- include/linux/ptr_ring.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
include/linux/percpu-refcount.h +1 -1
@@ -155,7 +155,7 @@ static inline bool __ref_is_percpu(struct percpu_ref *ref,* between contaminating the pointer value, meaning that * READ_ONCE() is required when fetching it. * -* The smp_read_barrier_depends() implied by READ_ONCE() pairs +* The dependency ordering from the READ_ONCE() pairs * with smp_store_release() in __percpu_ref_switch_to_percpu(). */ percpu_ptr =READ_ONCE (ref->percpu_count_ptr);
include/linux/ptr_ring.h +1 -1
@@ -107,7 +107,7 @@ static inline int __ptr_ring_produce(struct ptr_ring *r, void *ptr)return -ENOSPC ;/* Make sure the pointer we are storing points to a valid data. */ -/* Pairs with smp_read_barrier_depends in __ptr_ring_consume. */ +/* Pairs with the dependency ordering in __ptr_ring_consume. */ smp_wmb ();WRITE_ONCE (r->queue[r->producer++], ptr);
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 14/19 checkpatch: Remove checks relating to [smp_]read_barrier_depends()
In reply to Will DeaconThe [smp_]read_barrier_depends() macros no longer exist, so we don't need to deal with them in the checkpatch script.
--- scripts/checkpatch.pl | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-)
scripts/checkpatch.pl +1 -8
@@ -5903,8 +5903,7 @@ sub process { my $barriers = qr{ mb| rmb| - wmb| - read_barrier_depends + wmb }x; my $barrier_stems = qr{ mb__before_atomic|
@@ -5953,12 +5952,6 @@ sub process { } } -# check for smp_read_barrier_depends and read_barrier_depends - if (!$file && $line =~ /\b(smp_|)read_barrier_depends\s*\(/) { - WARN("READ_BARRIER_DEPENDS", - "$1read_barrier_depends should only be used in READ_ONCE or DEC Alpha code\n" . $herecurr); - } - # check of hardware specific defines if ($line =~ m@^.\s*\#\s*if.*\b(__i386__|__powerpc64__|__sun__|__s390x__)\b@ && $realfile !~ m@include/asm-@) { CHK("ARCH_DEFINES",
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 15/19 arm64: Reduce the number of header files pulled into vmlinux.lds.S
In reply to Will DeaconAlthough vmlinux.lds.S smells like an assembly file and is compiled with __ASSEMBLY__ defined, it's actually just fed to the preprocessor to create our linker script. This means that any assembly macros defined by headers that it includes will result in a helpful link error:
| aarch64-linux-gnu-ld:./arch/arm64/kernel/vmlinux.lds:1: syntax error
In preparation for an arm64-private asm/rwonce.h implementation, which will end up pulling assembly macros into linux/compiler.h, reduce the number of headers we include directly and transitively in vmlinux.lds.S
--- arch/arm64/include/asm/kernel-pgtable.h | 2 +- arch/arm64/include/asm/memory.h | 11 ++++++----- arch/arm64/include/asm/uaccess.h | 1 + arch/arm64/kernel/entry.S | 1 + arch/arm64/kernel/vmlinux.lds.S | 1 - arch/arm64/kvm/hyp-init.S | 1 + 6 files changed, 10 insertions(+), 7 deletions(-)
arch/arm64/include/asm/kernel-pgtable.h +1 -1
@@ -8,7 +8,7 @@#ifndef __ASM_KERNEL_PGTABLE_H#define __ASM_KERNEL_PGTABLE_H -#include <linux/pgtable.h> +#include <asm/pgtable-hwdef.h> #include <asm/sparsemem.h> /*
arch/arm64/include/asm/memory.h +6 -5
@@ -10,11 +10,8 @@#ifndef __ASM_MEMORY_H#define __ASM_MEMORY_H -#include <linux/compiler.h> #include <linux/const.h> #include <linux/sizes.h> -#include <linux/types.h> -#include <asm/bug.h> #include <asm/page-def.h> /*
@@ -157,11 +154,15 @@#endif #ifndef __ASSEMBLY__ -extern u64 vabits_actual; -#define PAGE_END (_PAGE_END(vabits_actual))#include <linux/bitops.h> +#include <linux/compiler.h> #include <linux/mmdebug.h> +#include <linux/types.h> +#include <asm/bug.h> + +extern u64 vabits_actual; +#define PAGE_END (_PAGE_END(vabits_actual))extern s64 physvirt_offset;extern s64 memstart_addr;
arch/arm64/include/asm/uaccess.h +1 -0
@@ -19,6 +19,7 @@#include <linux/string.h> #include <asm/cpufeature.h> +#include <asm/mmu.h> #include <asm/ptrace.h> #include <asm/memory.h> #include <asm/extable.h>
arch/arm64/kernel/entry.S +1 -0
@@ -15,6 +15,7 @@ #include <asm/assembler.h> #include <asm/asm-offsets.h> #include <asm/asm_pointer_auth.h> +#include <asm/bug.h> #include <asm/cpufeature.h> #include <asm/errno.h> #include <asm/esr.h>
arch/arm64/kernel/vmlinux.lds.S +0 -1
@@ -10,7 +10,6 @@ #include <asm-generic/vmlinux.lds.h> #include <asm/cache.h> #include <asm/kernel-pgtable.h> -#include <asm/thread_info.h> #include <asm/memory.h> #include <asm/page.h>
arch/arm64/kvm/hyp-init.S +1 -0
@@ -6,6 +6,7 @@ #include <linux/linkage.h> +#include <asm/alternative.h> #include <asm/assembler.h> #include <asm/kvm_arm.h> #include <asm/kvm_mmu.h>
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 16/19 arm64: alternatives: Split up alternative.h
In reply to Will Deaconasm/alternative.h contains both the macros needed to use alternatives, as well the type definitions and function prototypes for applying them.
Split the header in two, so that alternatives can be used from core header files such as linux/compiler.h without the risk of circular includes
--- arch/arm64/include/asm/alternative-macros.h | 276 ++++++++++++++++++++ arch/arm64/include/asm/alternative.h | 267 +------------------ arch/arm64/include/asm/insn.h | 3 +- 3 files changed, 279 insertions(+), 267 deletions(-) create mode 100644 arch/arm64/include/asm/alternative-macros.h
arch/arm64/include/asm/alternative-macros.h new file +276 -0
@@ -0,0 +1,276 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_ALTERNATIVE_MACROS_H +#define __ASM_ALTERNATIVE_MACROS_H + +#include <asm/cpucaps.h> + +#define ARM64_CB_PATCH ARM64_NCAPS + +/* A64 instructions are always 32 bits. */ +#define AARCH64_INSN_SIZE 4 + +#ifndef __ASSEMBLY__ + +#include <linux/stringify.h> + +#define ALTINSTR_ENTRY (feature) \ + " .word 661b - .\n"/* label */ \ +" .word 663f - .\n" /* new instruction */ \ + " .hword " __stringify(feature) "\n" /* feature bit */ \ +" .byte 662b-661b\n" /* source len */ \ + " .byte 664f-663f\n" /* replacement len */ + +#define ALTINSTR_ENTRY_CB (feature, cb) \ + " .word 661b - .\n"/* label */ \ + " .word " __stringify(cb) "- .\n" /* callback */ \ + " .hword " __stringify(feature) "\n" /* feature bit */ \ +" .byte 662b-661b\n" /* source len */ \ +" .byte 664f-663f\n" /* replacement len */ + +/* +* alternative assembly primitive: +* +* If any of these .org directive fail, it means that insn1 and insn2 +* don't have the same length. This used to be written as +* +* .if ((664b-663b) != (662b-661b)) +* .error "Alternatives instruction length mismatch" +* .endif +* +* but most assemblers die if insn1 or insn2 have a .inst. This should +* be fixed in a binutils release posterior to 2.25.51.0.2 (anything +* containing commit 4e4d08cf7399b606 or c1baaddf8861). +* +* Alternatives with callbacks do not generate replacement instructions. + */ +#define __ALTERNATIVE_CFG (oldinstr, newinstr, feature, cfg_enabled) \ + ".if "__stringify(cfg_enabled)" == 1\n" \ + "661:\n\t" \ + oldinstr "\n" \ + "662:\n" \ + ".pushsection .altinstructions,\"a\"\n" \ + ALTINSTR_ENTRY(feature) \ + ".popsection\n" \ + ".subsection 1\n" \ + "663:\n\t" \ + newinstr "\n" \ + "664:\n\t" \ + ".previous\n\t" \ + ".org . - (664b-663b) + (662b-661b)\n\t" \ + ".org . - (662b-661b) + (664b-663b)\n" \ + ".endif\n" + +#define __ALTERNATIVE_CFG_CB (oldinstr, feature, cfg_enabled, cb) \ + ".if "__stringify(cfg_enabled)" == 1\n" \ + "661:\n\t" \ + oldinstr "\n" \ + "662:\n" \ + ".pushsection .altinstructions,\"a\"\n" \ + ALTINSTR_ENTRY_CB(feature, cb) \ + ".popsection\n" \ + "663:\n\t" \ + "664:\n\t" \ + ".endif\n" + +#define _ALTERNATIVE_CFG (oldinstr, newinstr, feature, cfg, ...) \ + __ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg)) + +#define ALTERNATIVE_CB (oldinstr, cb) \ + __ALTERNATIVE_CFG_CB(oldinstr, ARM64_CB_PATCH, 1, cb) +#else + +#include <asm/assembler.h> + +.macro altinstruction_entry orig_offset alt_offset feature orig_len alt_len + .word \orig_offset - . + .word \alt_offset - . + .hword \feature + .byte \orig_len + .byte \alt_len +.endm + +.macro alternative_insn insn1, insn2, cap, enable = 1 + .if \enable +661: \insn1 +662: .pushsection .altinstructions, "a" + altinstruction_entry 661b, 663f, \cap, 662b-661b, 664f-663f + .popsection + .subsection 1 +663: \insn2 +664: .previous + .org . - (664b-663b) + (662b-661b) + .org . - (662b-661b) + (664b-663b) + .endif +.endm + +/* +* Alternative sequences +* +* The code for the case where the capability is not present will be +* assembled and linked as normal. There are no restrictions on this +* code. +* +* The code for the case where the capability is present will be +* assembled into a special section to be used for dynamic patching. +* Code for that case must: +* +* 1. Be exactly the same length (in bytes) as the default code +* sequence. +* +* 2. Not contain a branch target that is used outside of the +* alternative sequence it is defined in (branches into an +* alternative sequence are not fixed up). +*/ + +/* +* Begin an alternative code sequence. +*/ +.macro alternative_if_not cap + .set .Lasm_alt_mode, 0 + .pushsection .altinstructions, "a" + altinstruction_entry 661f, 663f, \cap, 662f-661f, 664f-663f + .popsection +661: +.endm + +.macro alternative_if cap + .set .Lasm_alt_mode, 1 + .pushsection .altinstructions, "a" + altinstruction_entry 663f, 661f, \cap, 664f-663f, 662f-661f + .popsection + .subsection 1 + .align 2 /* So GAS knows label 661 is suitably aligned */ +661: +.endm + +.macro alternative_cb cb + .set .Lasm_alt_mode, 0 + .pushsection .altinstructions, "a" + altinstruction_entry 661f, \cb, ARM64_CB_PATCH, 662f-661f, 0 + .popsection +661: +.endm + +/* + * Provide the other half of the alternative code sequence. + */ +.macro alternative_else +662: + .if .Lasm_alt_mode==0 + .subsection 1 + .else + .previous + .endif +663: +.endm + +/* + * Complete an alternative code sequence. + */ +.macro alternative_endif +664: + .if .Lasm_alt_mode==0 + .previous + .endif + .org . - (664b-663b) + (662b-661b) + .org . - (662b-661b) + (664b-663b) +.endm + +/* +* Callback-based alternative epilogue +*/ +.macro alternative_cb_end +662: +.endm + +/* +* Provides a trivial alternative or default sequence consisting solely +* of NOPs. The number of NOPs is chosen automatically to match the +* previous case. +*/ +.macro alternative_else_nop_endif +alternative_else + nops (662b-661b) / AARCH64_INSN_SIZE +alternative_endif +.endm + +#define _ALTERNATIVE_CFG (insn1, insn2, cap, cfg, ...) \ + alternative_insn insn1, insn2, cap, IS_ENABLED(cfg) + +.macro user_alt, label, oldinstr, newinstr, cond +9999: alternative_insn "\oldinstr", "\newinstr", \cond + _asm_extable 9999b, \label +.endm + +/* + * Generate the assembly for UAO alternatives with exception table entries. + * This is complicated as there is no post-increment or pair versions of the + * unprivileged instructions, and USER() only works for single instructions. + */ +#ifdef CONFIG_ARM64_UAO + .macro uao_ldp l, reg1, reg2, addr, post_inc + alternative_if_not ARM64_HAS_UAO +8888: ldp \reg1, \reg2, [\addr], \post_inc; +8889: nop; + nop; +alternative_else + ldtr \reg1, [\addr]; + ldtr \reg2, [\addr, #8]; + add \addr, \addr, \post_inc; +alternative_endif + + _asm_extable 8888b,\l; + _asm_extable 8889b,\l; + .endm + + .macro uao_stp l, reg1, reg2, addr, post_inc +alternative_if_not ARM64_HAS_UAO +8888: stp \reg1, \reg2, [\addr], \post_inc; +8889: nop; + nop; +alternative_else + sttr \reg1, [\addr]; + sttr \reg2, [\addr, #8]; + add \addr, \addr, \post_inc; +alternative_endif + + _asm_extable 8888b,\l; + _asm_extable 8889b,\l; + .endm + + .macro uao_user_alternative l, inst, alt_inst, reg, addr, post_inc +alternative_if_not ARM64_HAS_UAO +8888: \inst \reg, [\addr], \post_inc; + nop; +alternative_else + \alt_inst \reg, [\addr]; + add \addr, \addr, \post_inc; +alternative_endif + + _asm_extable 8888b,\l; + .endm +#else + .macro uao_ldp l, reg1, reg2, addr, post_inc + USER(\l, ldp \reg1, \reg2, [\addr], \post_inc) + .endm + .macro uao_stp l, reg1, reg2, addr, post_inc + USER(\l, stp \reg1, \reg2, [\addr], \post_inc) + .endm + .macro uao_user_alternative l, inst, alt_inst, reg, addr, post_inc + USER(\l, \inst \reg, [\addr], \post_inc) + .endm +#endif + +#endif /* __ASSEMBLY__ */ + +/* +* Usage: asm(ALTERNATIVE(oldinstr, newinstr, feature)); +* +* Usage: asm(ALTERNATIVE(oldinstr, newinstr, feature, CONFIG_FOO)); +* N.B. If CONFIG_FOO is specified, but not selected, the whole block +* will be omitted, including oldinstr. +*/ +#define ALTERNATIVE (oldinstr, newinstr, ...) \ + _ALTERNATIVE_CFG(oldinstr, newinstr, __VA_ARGS__, 1) + +#endif /* __ASM_ALTERNATIVE_MACROS_H */
arch/arm64/include/asm/alternative.h +2 -265
283 lines not on this page · raw message
arch/arm64/include/asm/insn.h +1 -2
@@ -10,8 +10,7 @@#include <linux/build_bug.h> #include <linux/types.h> -/* A64 instructions are always 32 bits. */ -#define AARCH64_INSN_SIZE 4 +#include <asm/alternative.h> #ifndef __ASSEMBLY__/*
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 17/19 arm64: cpufeatures: Add capability for LDAPR instruction
In reply to Will DeaconArmv8.3 introduced the LDAPR instruction, which provides weaker memory ordering semantics than LDARi (RCpc vs RCsc). Generally, we provide an RCsc implementation when implementing the Linux memory model, but LDAPR can be used as a useful alternative to dependency ordering, particularly when the compiler is capable of breaking the dependencies.
Since LDAPR is not available on all CPUs, add a cpufeature to detect it at runtime and allow the instruction to be used with alternative code patching.
--- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/kernel/cpufeature.c | 10 ++++++++++ 3 files changed, 15 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +3 -0
@@ -1409,6 +1409,9 @@ config ARM64_PAN The feature is detected at runtime, and will remain as a 'nop' instruction if the cpu does not implement the feature. +config AS_HAS_LDAPR + def_bool $(as-instr,.arch_extension rcpc) + config ARM64_LSE_ATOMICS bool default ARM64_USE_LSE_ATOMICS
arch/arm64/include/asm/cpucaps.h +2 -1
@@ -62,7 +62,8 @@#define ARM64_HAS_GENERIC_AUTH 52#define ARM64_HAS_32BIT_EL1 53#define ARM64_BTI 54 +#define ARM64_HAS_LDAPR 55 -#define ARM64_NCAPS 55 +#define ARM64_NCAPS 56#endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpufeature.c +10 -0
@@ -2058,6 +2058,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .sign = FTR_UNSIGNED, },#endif + { + .desc = "RCpc load-acquire (LDAPR)", + .capability = ARM64_HAS_LDAPR, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .sys_reg = SYS_ID_AA64ISAR1_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64ISAR1_LRCPC_SHIFT, + .matches = has_cpuid_feature, + .min_field_value = 1, + }, {}, };
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 18/19 arm64: alternatives: Remove READ_ONCE() usage during patch operation
In reply to Will DeaconIn preparation for patching the internals of READ_ONCE() itself, replace its usage on the alternatives patching patch with a volatile variable instead.
--- arch/arm64/kernel/alternative.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
arch/arm64/kernel/alternative.c +4 -3
@@ -21,7 +21,8 @@#define ALT_ORIG_PTR (a) __ALT_PTR(a, orig_offset)#define ALT_REPL_PTR (a) __ALT_PTR(a, alt_offset) -static int all_alternatives_applied; +/* Volatile, as we may be patching the guts of READ_ONCE() */ +static volatile int all_alternatives_applied; static DECLARE_BITMAP(applied_alternatives, ARM64_NCAPS);
@@ -217,7 +218,7 @@ static int __apply_alternatives_multi_stop(void *unused)/* We always have a CPU 0 at this point (__init) */ if (smp_processor_id ()) { -while (!READ_ONCE (all_alternatives_applied)) +while (!all_alternatives_applied)cpu_relax ();isb (); }else {
@@ -229,7 +230,7 @@ static int __apply_alternatives_multi_stop(void *unused)BUG_ON (all_alternatives_applied);__apply_alternatives (®ion, false, remaining_capabilities);/* Barriers provided by the cache flushing */ -WRITE_ONCE (all_alternatives_applied,1 ); + all_alternatives_applied =1 ; }return 0 ;
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Will Deacon
maintainerPATCH v3 19/19 arm64: lto: Strengthen READ_ONCE() to acquire when CONFIG_LTO=y
In reply to Will DeaconWhen building with LTO, there is an increased risk of the compiler converting an address dependency headed by a READ_ONCE() invocation into a control dependency and consequently allowing for harmful reordering by the CPU.
Ensure that such transformations are harmless by overriding the generic READ_ONCE() definition with one that provides acquire semantics when building with LTO.
--- arch/arm64/include/asm/rwonce.h | 63 +++++++++++++++++++++++++++++++ arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vdso32/Makefile | 2 +- 3 files changed, 65 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/include/asm/rwonce.h
arch/arm64/include/asm/rwonce.h new file +63 -0
@@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright (C) 2020 Google LLC. +*/ +#ifndef __ASM_RWONCE_H +#define __ASM_RWONCE_H + +#ifdef CONFIG_LTO + +#include <linux/compiler_types.h> +#include <asm/alternative-macros.h> + +#ifndef BUILD_VDSO + +#ifdef CONFIG_AS_HAS_LDAPR +#define __LOAD_RCPC(sfx, regs...) \ + ALTERNATIVE( \ + "ldar" #sfx "\t" #regs, \ + ".arch_extension rcpc\n" \ + "ldapr" #sfx "\t" #regs, \ + ARM64_HAS_LDAPR) +#else +#define __LOAD_RCPC(sfx, regs...) "ldar" #sfx "\t" #regs +#endif /* CONFIG_AS_HAS_LDAPR */ + +#define __READ_ONCE (x) \ +({ \ + typeof(&(x)) __x = &(x); \ + int atomic = 1; \ + union { __unqual_scalar_typeof(*__x) __val; char __c[1]; } __u; \ + switch (sizeof(x)) { \ + case 1: \ + asm volatile(__LOAD_RCPC(b, %w0, %1) \ + : "=r" (*(__u8 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 2: \ + asm volatile(__LOAD_RCPC(h, %w0, %1) \ + : "=r" (*(__u16 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 4: \ + asm volatile(__LOAD_RCPC(, %w0, %1) \ + : "=r" (*(__u32 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 8: \ + asm volatile(__LOAD_RCPC(, %0, %1) \ + : "=r" (*(__u64 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + default: \ + atomic = 0; \ + } \ + atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(__x))__x);\ +}) + +#endif /* !BUILD_VDSO */ +#endif /* CONFIG_LTO */ + +#include <asm-generic/rwonce.h> + +#endif /* __ASM_RWONCE_H */
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \ $(btildflags-y) -T ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18 -ccflags-y += -DDISABLE_BRANCH_PROFILING +ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS) KBUILD_CFLAGS += $(DISABLE_LTO)
arch/arm64/kernel/vdso32/Makefile +1 -1
@@ -43,7 +43,7 @@ cc32-as-instr = $(call try-run,\ # As a result we set our own flags here. # KBUILD_CPPFLAGS and NOSTDINC_FLAGS from top-level Makefile -VDSO_CPPFLAGS := -D__KERNEL__ -nostdinc -isystem $(shell $(CC_COMPAT) -print-file-name=include) +VDSO_CPPFLAGS := -DBUILD_VDSO -D__KERNEL__ -nostdinc -isystem $(shell $(CC_COMPAT) -print-file-name=include) VDSO_CPPFLAGS += $(LINUXINCLUDE) # Common C and assembly flags
Signature · 1 line
2.27.0.383.g050319c2ae-goog
Re: [PATCH v3 06/19] asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h'
In reply to Will DeaconQuoted 83 lines · Will Deacon
Now that '
smp_read_barrier_depends()' has gone the way of the Norwegian Blue, drop the inclusion of <asm/barrier.h> in 'asm-generic/rwonce.h'.This requires fixups to some architecture vdso headers which were previously relying on '
asm/barrier.h' coming in via 'linux/compiler.h'.Signed-off-by Will Deacon <[email protected]>--- arch/arm/include/asm/vdso/gettimeofday.h | 1 + arch/arm64/include/asm/vdso/compat_gettimeofday.h | 1 + arch/arm64/include/asm/vdso/gettimeofday.h | 1 + arch/riscv/include/asm/vdso/gettimeofday.h | 1 + include/asm-generic/rwonce.h | 2 -- include/linux/nospec.h | 2 ++ 6 files changed, 6 insertions(+), 2 deletions(-)arch/arm/include/asm/vdso/gettimeofday.h +1
-0@@ -7,6 +7,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/errno.h> #include <asm/unistd.h> #include <asm/vdso/cp15.h> arch/arm64/include/asm/vdso/compat_gettimeofday.h +1
-0@@ -7,6 +7,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/unistd.h> #include <asm/errno.h> arch/arm64/include/asm/vdso/gettimeofday.h +1
-0@@ -7,6 +7,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/unistd.h> #define VDSO_HAS_CLOCK_GETRES 1arch/riscv/include/asm/vdso/gettimeofday.h +1
-0@@ -4,6 +4,7 @@#ifndef __ASSEMBLY__ +#include <asm/barrier.h> #include <asm/unistd.h> #include <asm/csr.h> #include <uapi/linux/time.h> include/asm-generic/rwonce.h +0
-2@@ -26,8 +26,6 @@#include <linux/kasan-checks.h> #include <linux/kcsan-checks.h> -#include <asm/barrier.h> -/* * Use __READ_ONCE() instead of READ_ONCE() if you do not require any * atomicity. Note that this may result in tears! include/linux/nospec.h +0
-0
include/linux/nospec.h line 9
Quoted 5 lines
include/linux/nospec.h +2
-0@@ -5,6 +5,8 @@#ifndef _LINUX_NOSPEC_H#define _LINUX_NOSPEC_H + +#include <linux/compiler.h>
The other hunks LGTM, but this one is a little more curious to me. Can you walk me through this addition?
Quoted 6 lines
include/linux/nospec.h +0
-1#include <asm/barrier.h> struct task_struct ; --2.27.0.383.g050319c2ae-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Will Deacon
maintainerRe: [PATCH v3 06/19] asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h'
Quoted 7 lines · Nick Desaulniers
Quoted 6 lines · Will Deacon
include/linux/nospec.h +0
-0@@ -5,6 +5,8 @@
Quoted 7 lines
include/linux/nospec.h line 9
Quoted 4 lines
include/linux/nospec.h +2
-0#ifndef _LINUX_NOSPEC_H#define _LINUX_NOSPEC_H + +#include <linux/compiler.h> The other hunks LGTM, but this one is a little more curious to me. Can you walk me through this addition?
Sure. Without it, the build breaks on riscv because it includes this header without first including <linux/compiler.h>, and this header relies on OPTIMIZER_HIDE_VAR() being to defined as it is used in static inline functions.
Perhaps I should squash this hunk into "compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h" instead, as that is where I remove the include of <linux/compiler.h> from 'asm-generic/barrier.h'. I'll check the bisection on riscv...
Will
Quoted 17 lines · Will Deacon
SeongJae Park (1):
Documentation/barriers/kokr: Remove references to
[smp_]read_barrier_depends()Will Deacon (18):
tools: bpf: Use local copy of headers includinguapi/linux/filter.hcompiler.h: Split {READ,WRITE}_ONCE definitions out intorwonce.hasm/rwonce: Allow__READ_ONCEto be overridden by the architecture
alpha: OverrideREAD_ONCE()with barriered implementation asm/rwonce: Removesmp_read_barrier_depends()invocation asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h'
vhost: Remove redundant use ofread_barrier_depends()barrier
alpha: Replacesmp_read_barrier_depends()usage with smp_[r]mb() locking/barriers: Remove definitions for [smp_]read_barrier_depends()Documentation/barriers: Remove references to
[smp_]read_barrier_depends()
tools/memory-model: Removesmp_read_barrier_depends()from informal
Quoted 8 lines
doc
include/linux: Removesmp_read_barrier_depends()from comments
checkpatch: Remove checks relating to [smp_]read_barrier_depends()
arm64: Reduce the number of header files pulled intovmlinux.lds.S
arm64: alternatives: Split upalternative.h
arm64: cpufeatures: Add capability for LDAPR instruction
arm64: alternatives: RemoveREAD_ONCE()usage during patch operation
arm64: lto: StrengthenREAD_ONCE()to acquire whenCONFIG_LTO=y
Re: [PATCH v3 07/19] vhost: Remove redundant use of read_barrier_depends() barrier
In reply to Will DeaconQuoted 34 lines · Will Deacon
Since commit
76ebbe78f739("locking/barriers: Add implicitsmp_read_barrier_depends()toREAD_ONCE()"), there is no need to usesmp_read_barrier_depends()outside of the Alpha architecture code.Unfortunately, there is precisely _one_ user in the vhost code, and there isn't an obvious
READ_ONCE()access making the barrier redundant. However, on closer inspection (thanks, Jason), it appears that vring synchronisation between the producer and consumer occurs via the 'avail_idx' field, which is followed up by anrmb()invhost_get_vq_desc(), making theread_barrier_depends()redundant on Alpha.Jason says:
| I'm also confused about the barrier here, basically in driver side | we did: | | 1) allocate pages | 2) store pages in indirect->addr | 3) smp_wmb() | 4) increase the avail idx (somehow a tail pointer of vring) | | in vhost we did: | | 1) read avail idx | 2) smp_rmb() | 3) read indirect->addr | 4) read from indirect->addr | | It looks to me even the data dependency barrier is not necessary | since we have rmb() which is sufficient for us to the correct | indirect->addr and driver are not expected to do any writing to | indirect->addr after avail idx is increased
Quoted 5 lines
Remove the redundant barrier invocation.
Suggested-by Jason Wang <[email protected]>Acked-by Paul E. McKenney <[email protected]>Signed-off-by Will Deacon <[email protected]>
I agree
Pls merge with the rest of the patchset.
Quoted 22 lines
--- drivers/vhost/vhost.c | 5 ----- 1 file changed, 5 deletions(-)drivers/vhost/vhost.c +0
-5@@ -2092,11 +2092,6 @@ static int get_indirect(struct vhost_virtqueue *vq,return ret; }iov_iter_init (&from,READ , vq->indirect, ret, len); - -/* We will use the result as an address to read from, so most -* architectures only need a compiler barrier here. */ -read_barrier_depends (); - count = len /sizeof desc;/* Buffers are chained via a 16 bit next field, so * we can have at most 2^16 of these. */ Signature · 1 line
2.27.0.383.g050319c2ae-goog
Re: [PATCH v3 02/19] compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h
In reply to Will DeaconQuoted 83 lines · Will Deacon
In preparation for allowing architectures to define their own implementation of the
READ_ONCE()macro, move the generic {READ,WRITE}_ONCE() definitions out of the unwieldy 'linux/compiler.h' file and into a new 'rwonce.h' header under 'asm-generic'.Acked-by Paul E. McKenney <[email protected]>Signed-off-by Will Deacon <[email protected]>--- include/asm-generic/Kbuild | 1 + include/asm-generic/barrier.h | 2 +- include/asm-generic/rwonce.h | 91 +++++++++++++++++++++++++++++++++++ include/linux/compiler.h | 83 +------------------------------- 4 files changed, 95 insertions(+), 82 deletions(-) create mode 100644 include/asm-generic/rwonce.hinclude/asm-generic/Kbuild +1
-0@@ -45,6 +45,7 @@ mandatory-y += pci.h mandatory-y += percpu.h mandatory-y += pgalloc.h mandatory-y += preempt.h +mandatory-y += rwonce.h mandatory-y += sections.h mandatory-y += serial.h mandatory-y += shmparam.hinclude/asm-generic/barrier.h +1
-1@@ -13,7 +13,7 @@#ifndef __ASSEMBLY__ -#include <linux/compiler.h> +#include <asm/rwonce.h> #ifndef nop#define nop () asm volatile ("nop")include/asm-generic/rwonce.h new file +37
-0@@ -0,0 +1,91 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Prevent the compiler from merging or refetching reads or writes. The +* compiler is also forbidden from reordering successive instances of +* READ_ONCE and WRITE_ONCE, but only when the compiler is aware of some +* particular ordering. One way to make the compiler aware of ordering is to +* put the two invocations of READ_ONCE or WRITE_ONCE in different C +* statements. +* +* These two macros will also work on aggregate data types like structs or +* unions. +* +* Their two major use cases are: (1) Mediating communication between +* process-level code and irq/NMI handlers, all running on the same CPU, +* and (2) Ensuring that the compiler does not fold, spindle, or otherwise +* mutilate accesses that either do not require ordering or that interact +* with an explicit memory barrier or atomic instruction that provides the +* required ordering. +*/ +#ifndef __ASM_GENERIC_RWONCE_H +#define __ASM_GENERIC_RWONCE_H + +#ifndef __ASSEMBLY__ + +#include <linux/compiler_types.h> +#include <linux/kasan-checks.h> +#include <linux/kcsan-checks.h> + +#include <asm/barrier.h> + +/* +* Use __READ_ONCE() instead of READ_ONCE() if you do not require any +* atomicity or dependency ordering guarantees. Note that this may result +* in tears! +*/ +#define __READ_ONCE (x) (*(const volatile __unqual_scalar_typeof(x) *)&(x)) +
include/asm-generic/rwonce.h line 47
Quoted 10 lines
include/asm-generic/rwonce.h new file +10
-0+#define __READ_ONCE_SCALAR (x) \ +({ \ + __unqual_scalar_typeof(x) __x = __READ_ONCE(x); \ + smp_read_barrier_depends(); \ + (typeof(x))__x; \ +}) + +#define READ_ONCE (x) \ +({ \ + compiletime_assert_rwonce_type(x); \
Does it make sense if we also move the definition of this compile time assertion into rwonce.h too?
Regards,
Boqun
Quoted 3 lines
include/asm-generic/rwonce.h new file +3
-0+__READ_ONCE_SCALAR (x); \ +}) +
[...]
Will Deacon
maintainerRe: [PATCH v3 02/19] compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h
Quoted 49 lines · On Mon, Jul 13, 2020 at 08:23:22PM +0800, [email protected] wrote:
Quoted 48 lines · Will Deacon
include/asm-generic/rwonce.h new file +42
-0@@ -0,0 +1,91 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Prevent the compiler from merging or refetching reads or writes. The +* compiler is also forbidden from reordering successive instances of +* READ_ONCE and WRITE_ONCE, but only when the compiler is aware of some +* particular ordering. One way to make the compiler aware of ordering is to +* put the two invocations of READ_ONCE or WRITE_ONCE in different C +* statements. +* +* These two macros will also work on aggregate data types like structs or +* unions. +* +* Their two major use cases are: (1) Mediating communication between +* process-level code and irq/NMI handlers, all running on the same CPU, +* and (2) Ensuring that the compiler does not fold, spindle, or otherwise +* mutilate accesses that either do not require ordering or that interact +* with an explicit memory barrier or atomic instruction that provides the +* required ordering. +*/ +#ifndef __ASM_GENERIC_RWONCE_H +#define __ASM_GENERIC_RWONCE_H + +#ifndef __ASSEMBLY__ + +#include <linux/compiler_types.h> +#include <linux/kasan-checks.h> +#include <linux/kcsan-checks.h> + +#include <asm/barrier.h> + +/* +* Use __READ_ONCE() instead of READ_ONCE() if you do not require any +* atomicity or dependency ordering guarantees. Note that this may result +* in tears! +*/ +#define __READ_ONCE (x) (*(const volatile __unqual_scalar_typeof(x) *)&(x)) + +#define __READ_ONCE_SCALAR (x) \ +({ \ + __unqual_scalar_typeof(x) __x = __READ_ONCE(x); \ + smp_read_barrier_depends(); \ + (typeof(x))__x; \
Quoted 8 lines
include/asm-generic/rwonce.h line 47
Quoted 5 lines
include/asm-generic/rwonce.h new file +5
-0+}) + +#define READ_ONCE (x) \ +({ \ + compiletime_assert_rwonce_type(x); \Does it make sense if we also move the definition of this compile time assertion into
rwonce.htoo?
Yes, that looks straightforward enough. Thanks for the suggestion!
I'll also try to get this lot into -next this week.
Will
Re: [PATCH v3 19/19] arm64: lto: Strengthen READ_ONCE() to acquire when CONFIG_LTO=y
In reply to Will DeaconQuoted 8 lines · Will Deacon
When building with LTO, there is an increased risk of the compiler converting an address dependency headed by a
READ_ONCE()invocation into a control dependency and consequently allowing for harmful reordering by the CPU.Ensure that such transformations are harmless by overriding the generic
READ_ONCE()definition with one that provides acquire semantics when building with LTO.
Traditionally, READ_ONCE had only effects on compiler optimalizations, not on special semantics of the load instruction.
Do you have example how LTO optimalizations break the code?
Should some documentation be added? Because I believe users will need to understand what is going on there.
It is not LTO-only problem and it is not arm64-only problem, right?
Best regards,
Pavel
Quoted 77 lines
arch/arm64/include/asm/rwonce.h new file +49
-0+#ifdef CONFIG_AS_HAS_LDAPR +#define __LOAD_RCPC(sfx, regs...) \ + ALTERNATIVE( \ + "ldar" #sfx "\t" #regs, \ + ".arch_extension rcpc\n" \ + "ldapr" #sfx "\t" #regs, \ + ARM64_HAS_LDAPR) +#else +#define __LOAD_RCPC(sfx, regs...) "ldar" #sfx "\t" #regs +#endif /* CONFIG_AS_HAS_LDAPR */ + +#define __READ_ONCE (x) \ +({ \ + typeof(&(x)) __x = &(x); \ + int atomic = 1; \ + union { __unqual_scalar_typeof(*__x) __val; char __c[1]; } __u; \ + switch (sizeof(x)) { \ + case 1: \ + asm volatile(__LOAD_RCPC(b, %w0, %1) \ + : "=r" (*(__u8 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 2: \ + asm volatile(__LOAD_RCPC(h, %w0, %1) \ + : "=r" (*(__u16 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 4: \ + asm volatile(__LOAD_RCPC(, %w0, %1) \ + : "=r" (*(__u32 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 8: \ + asm volatile(__LOAD_RCPC(, %0, %1) \ + : "=r" (*(__u64 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + default: \ + atomic = 0; \ + } \ + atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(__x))__x);\ +}) + +#endif /* !BUILD_VDSO */ +#endif /* CONFIG_LTO */ + +#include <asm-generic/rwonce.h> + +#endif /* __ASM_RWONCE_H */ arch/arm64/kernel/vdso/Makefile +1
-1@@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \ $(btildflags-y) -T ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18 -ccflags-y += -DDISABLE_BRANCH_PROFILING +ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS) KBUILD_CFLAGS += $(DISABLE_LTO)arch/arm64/kernel/vdso32/Makefile +1
-1@@ -43,7 +43,7 @@ cc32-as-instr = $(call try-run,\ # As a result we set our own flags here. # KBUILD_CPPFLAGS and NOSTDINC_FLAGS from top-level Makefile -VDSO_CPPFLAGS := -D__KERNEL__ -nostdinc -isystem $(shell $(CC_COMPAT) -print-file-name=include) +VDSO_CPPFLAGS := -DBUILD_VDSO -D__KERNEL__ -nostdinc -isystem $(shell $(CC_COMPAT) -print-file-name=include) VDSO_CPPFLAGS += $(LINUXINCLUDE) # Common C and assembly flagsSignature · 1 line
2.27.0.383.g050319c2ae-goog
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html