arch: atomic rework
Subsystems (87)
Peter Zijlstra
maintainercover arch: atomic rework
Hi all,
A few too large patches here, mostly as RFC to see if we want to continue with this before I sink more time into it. I hope they make it out to the lists.
This all started with me wanting to implement atomic_sub_release() for all archs, but I got side-tracked a bit and it ended up cleaning up bits and deleting almost 1400 lines of code.
Its been compiled on everything I have a compiler for, however frv and tile are missing because they're special and I was tired.
--- Documentation/atomic_ops.txt | 31 - Documentation/memory-barriers.txt | 44 - a/arch/arc/include/asm/barrier.h | 37 - a/arch/hexagon/include/asm/barrier.h | 37 - arch/alpha/include/asm/atomic.h | 225 +++----- arch/alpha/include/asm/bitops.h | 3 arch/arc/include/asm/atomic.h | 198 ++----- arch/arc/include/asm/bitops.h | 5 arch/arm/include/asm/atomic.h | 301 ++++------- arch/arm/include/asm/barrier.h | 3 arch/arm/include/asm/bitops.h | 4 arch/arm64/include/asm/atomic.h | 212 +++----- arch/arm64/include/asm/barrier.h | 3 arch/arm64/include/asm/bitops.h | 9 arch/avr32/include/asm/atomic.h | 95 +-- arch/avr32/include/asm/bitops.h | 9 arch/blackfin/include/asm/atomic.h | 5 arch/blackfin/include/asm/barrier.h | 3 arch/blackfin/include/asm/bitops.h | 14 arch/blackfin/mach-common/smp.c | 2 arch/c6x/include/asm/bitops.h | 8 arch/cris/include/arch-v10/arch/system.h | 2 arch/cris/include/asm/atomic.h | 66 +- arch/cris/include/asm/bitops.h | 9 arch/frv/include/asm/atomic.h | 7 arch/frv/include/asm/bitops.h | 6 arch/hexagon/include/asm/atomic.h | 74 +- arch/hexagon/include/asm/bitops.h | 4 arch/ia64/include/asm/atomic.h | 212 ++++---- arch/ia64/include/asm/barrier.h | 3 arch/ia64/include/asm/bitops.h | 9 arch/ia64/include/uapi/asm/cmpxchg.h | 9 arch/m32r/include/asm/atomic.h | 191 ++----- arch/m32r/include/asm/bitops.h | 6 arch/m32r/kernel/smp.c | 4 arch/m68k/include/asm/atomic.h | 130 +---- arch/m68k/include/asm/bitops.h | 7 arch/metag/include/asm/atomic.h | 6 arch/metag/include/asm/atomic_lnkget.h | 159 +----- arch/metag/include/asm/atomic_lock1.h | 100 +-- arch/metag/include/asm/barrier.h | 3 arch/metag/include/asm/bitops.h | 6 arch/mips/include/asm/atomic.h | 570 +++++++--------------- arch/mips/include/asm/barrier.h | 3 arch/mips/include/asm/bitops.h | 11 arch/mips/kernel/irq.c | 4 arch/mn10300/include/asm/atomic.h | 199 +------ arch/mn10300/include/asm/bitops.h | 4 arch/mn10300/mm/tlb-smp.c | 6 arch/openrisc/include/asm/bitops.h | 9 arch/parisc/include/asm/atomic.h | 121 ++-- arch/parisc/include/asm/bitops.h | 4 arch/powerpc/include/asm/atomic.h | 214 +++----- arch/powerpc/include/asm/barrier.h | 3 arch/powerpc/include/asm/bitops.h | 6 arch/powerpc/kernel/crash.c | 2 arch/powerpc/kernel/misc_32.S | 19 arch/s390/include/asm/atomic.h | 93 ++- arch/s390/include/asm/barrier.h | 5 arch/s390/include/asm/bitops.h | 1 arch/s390/kernel/time.c | 4 arch/s390/kvm/diag.c | 2 arch/s390/kvm/intercept.c | 2 arch/s390/kvm/interrupt.c | 16 arch/s390/kvm/kvm-s390.c | 14 arch/s390/kvm/sigp.c | 6 arch/score/include/asm/bitops.h | 7 arch/sh/include/asm/atomic-grb.h | 164 +----- arch/sh/include/asm/atomic-irq.h | 88 +-- arch/sh/include/asm/atomic-llsc.h | 135 +---- arch/sh/include/asm/atomic.h | 6 arch/sh/include/asm/bitops.h | 7 arch/sparc/include/asm/atomic_32.h | 30 - arch/sparc/include/asm/atomic_64.h | 53 -- arch/sparc/include/asm/barrier_32.h | 1 arch/sparc/include/asm/barrier_64.h | 3 arch/sparc/include/asm/bitops_32.h | 4 arch/sparc/include/asm/bitops_64.h | 4 arch/sparc/include/asm/processor.h | 2 arch/sparc/kernel/smp_64.c | 2 arch/sparc/lib/atomic32.c | 28 - arch/sparc/lib/atomic_64.S | 167 ++---- arch/sparc/lib/ksyms.c | 20 arch/tile/include/asm/atomic_32.h | 10 arch/tile/include/asm/atomic_64.h | 6 arch/tile/include/asm/barrier.h | 14 arch/tile/include/asm/bitops.h | 1 arch/tile/include/asm/bitops_32.h | 8 arch/tile/include/asm/bitops_64.h | 4 arch/x86/include/asm/atomic.h | 39 - arch/x86/include/asm/atomic64_32.h | 20 arch/x86/include/asm/atomic64_64.h | 22 arch/x86/include/asm/barrier.h | 4 arch/x86/include/asm/bitops.h | 6 arch/x86/include/asm/sync_bitops.h | 2 arch/x86/kernel/apic/hw_nmi.c | 2 arch/xtensa/include/asm/atomic.h | 318 +++--------- arch/xtensa/include/asm/bitops.h | 4 block/blk-iopoll.c | 4 crypto/chainiv.c | 2 drivers/base/power/domain.c | 2 drivers/block/mtip32xx/mtip32xx.c | 4 drivers/cpuidle/coupled.c | 2 drivers/firewire/ohci.c | 2 drivers/gpu/drm/drm_irq.c | 10 drivers/gpu/drm/i915/i915_irq.c | 6 drivers/md/bcache/bcache.h | 2 drivers/md/bcache/closure.h | 2 drivers/md/dm-bufio.c | 8 drivers/md/dm-snap.c | 4 drivers/md/dm.c | 2 drivers/md/raid5.c | 2 drivers/media/usb/dvb-usb-v2/dvb_usb_core.c | 6 drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c | 6 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c | 34 - drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c | 26 - drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c | 12 drivers/net/ethernet/broadcom/cnic.c | 8 drivers/net/ethernet/brocade/bna/bnad.c | 6 drivers/net/ethernet/chelsio/cxgb/cxgb2.c | 2 drivers/net/ethernet/chelsio/cxgb3/sge.c | 6 drivers/net/ethernet/chelsio/cxgb4/sge.c | 2 drivers/net/ethernet/chelsio/cxgb4vf/sge.c | 2 drivers/net/ethernet/intel/i40e/i40e_main.c | 2 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 4 drivers/net/wireless/ti/wlcore/main.c | 2 drivers/pci/xen-pcifront.c | 4 drivers/s390/scsi/zfcp_aux.c | 2 drivers/s390/scsi/zfcp_erp.c | 68 +- drivers/s390/scsi/zfcp_fc.c | 8 drivers/s390/scsi/zfcp_fsf.c | 30 - drivers/s390/scsi/zfcp_qdio.c | 14 drivers/scsi/isci/remote_device.c | 2 drivers/target/loopback/tcm_loop.c | 4 drivers/target/target_core_alua.c | 26 - drivers/target/target_core_device.c | 6 drivers/target/target_core_iblock.c | 2 drivers/target/target_core_pr.c | 56 +- drivers/target/target_core_transport.c | 16 drivers/target/target_core_ua.c | 10 drivers/tty/n_tty.c | 2 drivers/tty/serial/mxs-auart.c | 4 drivers/usb/gadget/tcm_usb_gadget.c | 4 drivers/usb/serial/usb_wwan.c | 2 drivers/vhost/scsi.c | 2 drivers/w1/w1_family.c | 4 drivers/xen/xen-pciback/pciback_ops.c | 4 fs/btrfs/btrfs_inode.h | 2 fs/btrfs/extent_io.c | 2 fs/btrfs/inode.c | 6 fs/buffer.c | 2 fs/ext4/resize.c | 2 fs/gfs2/glock.c | 8 fs/gfs2/glops.c | 2 fs/gfs2/lock_dlm.c | 4 fs/gfs2/recovery.c | 2 fs/gfs2/sys.c | 4 fs/jbd2/commit.c | 6 fs/nfs/dir.c | 12 fs/nfs/inode.c | 2 fs/nfs/nfs4filelayoutdev.c | 4 fs/nfs/nfs4state.c | 4 fs/nfs/pagelist.c | 6 fs/nfs/pnfs.c | 2 fs/nfs/pnfs.h | 2 fs/nfs/write.c | 4 fs/ubifs/lpt_commit.c | 4 fs/ubifs/tnc_commit.c | 4 include/asm-generic/atomic.h | 176 ++---- include/asm-generic/atomic64.h | 17 include/asm-generic/barrier.h | 8 include/asm-generic/bitops.h | 8 include/asm-generic/bitops/atomic.h | 2 include/asm-generic/bitops/lock.h | 2 include/linux/atomic.h | 13 include/linux/buffer_head.h | 2 include/linux/genhd.h | 2 include/linux/interrupt.h | 8 include/linux/netdevice.h | 2 include/linux/sched.h | 6 include/linux/sunrpc/sched.h | 8 include/linux/sunrpc/xprt.h | 8 include/linux/tracehook.h | 2 include/net/ip_vs.h | 4 kernel/debug/debug_core.c | 4 kernel/futex.c | 2 kernel/kmod.c | 2 kernel/rcu/tree.c | 22 kernel/rcu/tree_plugin.h | 8 kernel/sched/cpupri.c | 6 kernel/sched/wait.c | 2 lib/atomic64.c | 77 +- mm/backing-dev.c | 2 mm/filemap.c | 4 net/atm/pppoatm.c | 2 net/bluetooth/hci_event.c | 4 net/core/dev.c | 8 net/core/link_watch.c | 2 net/ipv4/inetpeer.c | 2 net/netfilter/nf_conntrack_core.c | 2 net/rds/ib_recv.c | 4 net/rds/iw_recv.c | 4 net/rds/send.c | 6 net/rds/tcp_send.c | 2 net/sunrpc/auth.c | 2 net/sunrpc/auth_gss/auth_gss.c | 2 net/sunrpc/backchannel_rqst.c | 4 net/sunrpc/xprt.c | 4 net/sunrpc/xprtsock.c | 16 net/unix/af_unix.c | 2 sound/pci/bt87x.c | 4 211 files changed, 2188 insertions(+), 3563 deletions(-)
Peter Zijlstra
maintainerRFC PATCH 1/5 ia64: Fix up smp_mb__{before,after}_clear_bit
IA64 doesn't actually have acquire/release barriers, its a lie!
Add a comment explaining this and fix up the bitop barriers.
--- arch/ia64/include/asm/bitops.h | 7 ++----- arch/ia64/include/uapi/asm/cmpxchg.h | 9 +++++++++ 2 files changed, 11 insertions(+), 5 deletions(-)
arch/ia64/include/asm/bitops.h +2 -5
@@ -65,11 +65,8 @@ __set_bit (int nr, volatile void *addr) *((__u32 *) addr + (nr >>5 )) |= (1 << (nr &31 )); } -/* -* clear_bit() has "acquire" semantics. -*/ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () do {/* skip */ ; }while (0 ) +#define smp_mb__before_clear_bit () barrier(); +#define smp_mb__after_clear_bit () barrier();/** * clear_bit - Clears a bit in memory
arch/ia64/include/uapi/asm/cmpxchg.h +9 -0
@@ -118,6 +118,15 @@ extern long ia64_cmpxchg_called_with_bad#define cmpxchg_rel (ptr, o, n) \ ia64_cmpxchg(rel, (ptr), (o), (n), sizeof(*(ptr))) +/* +* Worse still - early processor implementations actually just ignored +* the acquire/release and did a full fence all the time. Unfortunately +* this meant a lot of badly written code that used .acq when they really +* wanted .rel became legacy out in the wild - so when we made a cpu +* that strictly did the .acq or .rel ... all that code started breaking - so +* we had to back-pedal and keep the "legacy" behavior of a full fence :-( +*/ +/* for compatibility with other platforms: */ #define cmpxchg (ptr, o, n) cmpxchg_acq((ptr), (o), (n))#define cmpxchg64 (ptr, o, n) cmpxchg_acq((ptr), (o), (n))
Peter Zijlstra
maintainerRFC PATCH 2/5 arc,hexagon: Delete asm/barrier.h
In reply to Peter ZijlstraBoth already use asm-generic/barrier.h as per their include/asm/Kbuild. Remove the stale files.
--- arch/arc/include/asm/barrier.h | 37 ------------------------------------- arch/hexagon/include/asm/barrier.h | 37 ------------------------------------- 2 files changed, 74 deletions(-)
arch/arc/include/asm/barrier.h deleted +0 -37
@@ -1,37 +0,0 @@ -/* -* Copyright (C) 2004, 2007-2010, 2011-2012 Synopsys, Inc. (www.synopsys.com) -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 as -* published by the Free Software Foundation. -*/ - -#ifndef __ASM_BARRIER_H -#define __ASM_BARRIER_H - -#ifndef __ASSEMBLY__ - -/* TODO-vineetg: Need to see what this does, don't we need sync anywhere */ -#define mb () __asm__ __volatile__ ("" : : : "memory") -#define rmb () mb() -#define wmb () mb() -#define set_mb (var, value) do { var = value; mb(); } while (0) -#define set_wmb (var, value) do { var = value; wmb(); } while (0) -#define read_barrier_depends () mb() - -/* TODO-vineetg verify the correctness of macros here */ -#ifdef CONFIG_SMP -#define smp_mb () mb() -#define smp_rmb () rmb() -#define smp_wmb () wmb() -#else -#define smp_mb () barrier() -#define smp_rmb () barrier() -#define smp_wmb () barrier() -#endif - -#define smp_read_barrier_depends () do { } while (0) - -#endif - -#endif
arch/hexagon/include/asm/barrier.h deleted +0 -37
@@ -1,37 +0,0 @@ -/* -* Memory barrier definitions for the Hexagon architecture -* -* Copyright (c) 2010-2011, The Linux Foundation. All rights reserved. -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 and -* only version 2 as published by the Free Software Foundation. -* -* This program is distributed in the hope that it will be useful, -* but WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -* GNU General Public License for more details. -* -* You should have received a copy of the GNU General Public License -* along with this program; if not, write to the Free Software -* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA -* 02110-1301, USA. -*/ - -#ifndef _ASM_BARRIER_H -#define _ASM_BARRIER_H - -#define rmb () barrier() -#define read_barrier_depends () barrier() -#define wmb () barrier() -#define mb () barrier() -#define smp_rmb () barrier() -#define smp_read_barrier_depends () barrier() -#define smp_wmb () barrier() -#define smp_mb () barrier() - -/* Set a value and use a memory barrier. Used by the scheduler somewhere. */ -#define set_mb (var, value) \ - do { var = value; mb(); } while (0) - -#endif /* _ASM_BARRIER_H */
Peter Zijlstra
maintainerRFC PATCH 3/5 arch: s/smp_mb__(before|after)_(atomic|clear)_(dec,inc,bit)/smp_mb__\1/g
In reply to Peter ZijlstraBecause atomic ops are implemented the same across an architecture, the current incomplete set of extra barriers:
smp_mb__before_atomic_inc()
smp_mb__after_atomic_inc()
smp_mb__before_atomic_dec()
smp_mb__after_atomic_dec()
smp_mb__before_clear_bit()
smp_mb__after_clear_bit()
is both incomplete and superfluous.
It is incomplete because there are far more atomic operations that do not return values -- such as atomic_add(), set_bit() etc. And it is superfluous because they're all the same anyway.
Simplify things by reducing the triplicate set into a single set of barriers that is valid for all void atomic ops:
smp_mb__before_atomic()
smp_mb__after_atomic()
--- Documentation/atomic_ops.txt | 31 ++++-------- Documentation/memory-barriers.txt | 44 ++++------------- arch/alpha/include/asm/atomic.h | 5 - arch/alpha/include/asm/bitops.h | 3 - arch/arc/include/asm/atomic.h | 5 - arch/arc/include/asm/bitops.h | 5 - arch/arm/include/asm/atomic.h | 5 - arch/arm/include/asm/barrier.h | 3 + arch/arm/include/asm/bitops.h | 4 - arch/arm64/include/asm/atomic.h | 5 - arch/arm64/include/asm/barrier.h | 3 + arch/arm64/include/asm/bitops.h | 9 --- arch/avr32/include/asm/atomic.h | 5 - arch/avr32/include/asm/bitops.h | 9 --- arch/blackfin/include/asm/barrier.h | 3 + arch/blackfin/include/asm/bitops.h | 14 ----- arch/c6x/include/asm/bitops.h | 8 --- arch/cris/include/asm/atomic.h | 7 -- arch/cris/include/asm/bitops.h | 9 --- arch/frv/include/asm/atomic.h | 7 -- arch/frv/include/asm/bitops.h | 6 -- arch/hexagon/include/asm/atomic.h | 6 -- arch/hexagon/include/asm/bitops.h | 4 - arch/ia64/include/asm/atomic.h | 6 -- arch/ia64/include/asm/barrier.h | 3 + arch/ia64/include/asm/bitops.h | 6 -- arch/m32r/include/asm/atomic.h | 7 -- arch/m32r/include/asm/bitops.h | 6 -- arch/m68k/include/asm/atomic.h | 8 --- arch/m68k/include/asm/bitops.h | 7 -- arch/metag/include/asm/atomic.h | 6 -- arch/metag/include/asm/barrier.h | 3 + arch/metag/include/asm/bitops.h | 6 -- arch/mips/include/asm/atomic.h | 9 --- arch/mips/include/asm/barrier.h | 3 + arch/mips/include/asm/bitops.h | 11 ---- arch/mips/kernel/irq.c | 4 - arch/mn10300/include/asm/atomic.h | 7 -- arch/mn10300/include/asm/bitops.h | 4 - arch/mn10300/mm/tlb-smp.c | 4 - arch/openrisc/include/asm/bitops.h | 9 --- arch/parisc/include/asm/atomic.h | 6 -- arch/parisc/include/asm/bitops.h | 4 - arch/powerpc/include/asm/atomic.h | 6 -- arch/powerpc/include/asm/barrier.h | 3 + arch/powerpc/include/asm/bitops.h | 6 -- arch/powerpc/kernel/crash.c | 2 arch/s390/include/asm/atomic.h | 6 -- arch/s390/include/asm/barrier.h | 5 + arch/s390/include/asm/bitops.h | 1 arch/score/include/asm/bitops.h | 7 -- arch/sh/include/asm/atomic.h | 6 -- arch/sh/include/asm/bitops.h | 7 -- arch/sparc/include/asm/atomic_32.h | 7 -- arch/sparc/include/asm/atomic_64.h | 7 -- arch/sparc/include/asm/barrier_64.h | 3 + arch/sparc/include/asm/bitops_32.h | 4 - arch/sparc/include/asm/bitops_64.h | 4 - arch/tile/include/asm/atomic_32.h | 10 --- arch/tile/include/asm/atomic_64.h | 6 -- arch/tile/include/asm/barrier.h | 14 +++++ arch/tile/include/asm/bitops.h | 1 arch/tile/include/asm/bitops_32.h | 8 --- arch/tile/include/asm/bitops_64.h | 4 - arch/x86/include/asm/atomic.h | 7 -- arch/x86/include/asm/barrier.h | 4 + arch/x86/include/asm/bitops.h | 6 -- arch/x86/include/asm/sync_bitops.h | 2 arch/x86/kernel/apic/hw_nmi.c | 2 arch/xtensa/include/asm/atomic.h | 7 -- arch/xtensa/include/asm/bitops.h | 4 - block/blk-iopoll.c | 4 - crypto/chainiv.c | 2 drivers/base/power/domain.c | 2 drivers/block/mtip32xx/mtip32xx.c | 4 - drivers/cpuidle/coupled.c | 2 drivers/firewire/ohci.c | 2 drivers/gpu/drm/drm_irq.c | 10 +-- drivers/gpu/drm/i915/i915_irq.c | 2 drivers/md/bcache/bcache.h | 2 drivers/md/bcache/closure.h | 2 drivers/md/dm-bufio.c | 8 +-- drivers/md/dm-snap.c | 4 - drivers/md/dm.c | 2 drivers/md/raid5.c | 2 drivers/media/usb/dvb-usb-v2/dvb_usb_core.c | 6 +- drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c | 6 +- drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c | 34 ++++++------- drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c | 26 +++++----- drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c | 12 ++-- drivers/net/ethernet/broadcom/cnic.c | 8 +-- drivers/net/ethernet/brocade/bna/bnad.c | 6 +- drivers/net/ethernet/chelsio/cxgb/cxgb2.c | 2 drivers/net/ethernet/chelsio/cxgb3/sge.c | 6 +- drivers/net/ethernet/chelsio/cxgb4/sge.c | 2 drivers/net/ethernet/chelsio/cxgb4vf/sge.c | 2 drivers/net/ethernet/intel/i40e/i40e_main.c | 2 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 4 - drivers/net/wireless/ti/wlcore/main.c | 2 drivers/pci/xen-pcifront.c | 4 - drivers/scsi/isci/remote_device.c | 2 drivers/target/loopback/tcm_loop.c | 4 - drivers/target/target_core_alua.c | 26 +++++----- drivers/target/target_core_device.c | 6 +- drivers/target/target_core_iblock.c | 2 drivers/target/target_core_pr.c | 56 +++++++++++----------- drivers/target/target_core_transport.c | 16 +++--- drivers/target/target_core_ua.c | 10 +-- drivers/tty/n_tty.c | 2 drivers/tty/serial/mxs-auart.c | 4 - drivers/usb/gadget/tcm_usb_gadget.c | 4 - drivers/usb/serial/usb_wwan.c | 2 drivers/vhost/scsi.c | 2 drivers/w1/w1_family.c | 4 - drivers/xen/xen-pciback/pciback_ops.c | 4 - fs/btrfs/btrfs_inode.h | 2 fs/btrfs/extent_io.c | 2 fs/btrfs/inode.c | 6 +- fs/buffer.c | 2 fs/ext4/resize.c | 2 fs/gfs2/glock.c | 8 +-- fs/gfs2/glops.c | 2 fs/gfs2/lock_dlm.c | 4 - fs/gfs2/recovery.c | 2 fs/gfs2/sys.c | 4 - fs/jbd2/commit.c | 6 +- fs/nfs/dir.c | 12 ++-- fs/nfs/inode.c | 2 fs/nfs/nfs4filelayoutdev.c | 4 - fs/nfs/nfs4state.c | 4 - fs/nfs/pagelist.c | 6 +- fs/nfs/pnfs.c | 2 fs/nfs/pnfs.h | 2 fs/nfs/write.c | 4 - fs/ubifs/lpt_commit.c | 4 - fs/ubifs/tnc_commit.c | 4 - include/asm-generic/atomic.h | 7 -- include/asm-generic/barrier.h | 8 +++ include/asm-generic/bitops.h | 8 --- include/asm-generic/bitops/atomic.h | 2 include/asm-generic/bitops/lock.h | 2 include/linux/buffer_head.h | 2 include/linux/genhd.h | 2 include/linux/interrupt.h | 8 +-- include/linux/netdevice.h | 2 include/linux/sched.h | 6 -- include/linux/sunrpc/sched.h | 8 +-- include/linux/sunrpc/xprt.h | 8 +-- include/linux/tracehook.h | 2 include/net/ip_vs.h | 4 - kernel/debug/debug_core.c | 4 - kernel/futex.c | 2 kernel/kmod.c | 2 kernel/rcu/tree.c | 22 ++++---- kernel/rcu/tree_plugin.h | 8 +-- kernel/sched/cpupri.c | 6 +- kernel/sched/wait.c | 2 mm/backing-dev.c | 2 mm/filemap.c | 4 - net/atm/pppoatm.c | 2 net/bluetooth/hci_event.c | 4 - net/core/dev.c | 8 +-- net/core/link_watch.c | 2 net/ipv4/inetpeer.c | 2 net/netfilter/nf_conntrack_core.c | 2 net/rds/ib_recv.c | 4 - net/rds/iw_recv.c | 4 - net/rds/send.c | 6 +- net/rds/tcp_send.c | 2 net/sunrpc/auth.c | 2 net/sunrpc/auth_gss/auth_gss.c | 2 net/sunrpc/backchannel_rqst.c | 4 - net/sunrpc/xprt.c | 4 - net/sunrpc/xprtsock.c | 16 +++--- net/unix/af_unix.c | 2 sound/pci/bt87x.c | 4 - 176 files changed, 415 insertions(+), 642 deletions(-)
Documentation/atomic_ops.txt +12 -19
@@ -285,15 +285,13 @@ If a caller requires memory barrier sema operation which does not return a value, a set of interfaces are defined which accomplish this: - void smp_mb__before_atomic_dec(void); - void smp_mb__after_atomic_dec(void); - void smp_mb__before_atomic_inc(void); - void smp_mb__after_atomic_inc(void); + void smp_mb__before_atomic(void); + void smp_mb__after_atomic(void); -For example, smp_mb__before_atomic_dec() can be used like so: +For example, smp_mb__before_atomic() can be used like so: obj->dead = 1; - smp_mb__before_atomic_dec(); + smp_mb__before_atomic(); atomic_dec(&obj->ref_count); It makes sure that all memory operations preceding the atomic_dec()
@@ -302,15 +300,10 @@ operation. In the above example, it gua "1" to obj->dead will be globally visible to other cpus before the atomic counter decrement. -Without the explicit smp_mb__before_atomic_dec() call, the +Without the explicit smp_mb__before_atomic() call, the implementation could legally allow the atomic counter update visible to other cpus before the "obj->dead = 1;" assignment. -The other three interfaces listed are used to provide explicit -ordering with respect to memory operations after an atomic_dec() call -(smp_mb__after_atomic_dec()) and around atomic_inc() calls -(smp_mb__{before,after}_atomic_inc()). - A missing memory barrier in the cases where they are required by the atomic_t implementation above can have disastrous results. Here is an example, which follows a pattern occurring frequently in the Linux
@@ -487,12 +480,12 @@ memory operation done by test_and_set_bi Which returns a boolean indicating if bit "nr" is set in the bitmask pointed to by "addr". -If explicit memory barriers are required around clear_bit() (which -does not return a value, and thus does not need to provide memory -barrier semantics), two interfaces are provided: +If explicit memory barriers are required around {set,clear}_bit() (which do +not return a value, and thus does not need to provide memory barrier +semantics), two interfaces are provided: - void smp_mb__before_clear_bit(void); - void smp_mb__after_clear_bit(void); + void smp_mb__before_atomic(void); + void smp_mb__after_atomic(void); They are used as follows, and are akin to their atomic_t operation brothers:
@@ -500,13 +493,13 @@ They are used as follows, and are akin t /* All memory operations before this call will * be globally visible before the clear_bit(). */ - smp_mb__before_clear_bit(); + smp_mb__before_atomic(); clear_bit( ... ); /* The clear_bit() will be visible before all * subsequent memory operations. */ - smp_mb__after_clear_bit(); + smp_mb__after_atomic(); There are two special bitops with lock barrier semantics (acquire/release, same as spinlocks). These operate in the same way as their non-_lock/unlock
Documentation/memory-barriers.txt +12 -32
70 lines not on this page · raw message
arch/alpha/include/asm/atomic.h +0 -5
@@ -292,9 +292,4 @@ static inline long atomic64_dec_if_posit#define atomic_dec (v) atomic_sub(1,(v))#define atomic64_dec (v) atomic64_sub(1,(v)) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#endif /* _ALPHA_ATOMIC_H */
arch/alpha/include/asm/bitops.h +0 -3
@@ -53,9 +53,6 @@ __set_bit(unsigned long nr, volatile voi *m |=1 << (nr &31 ); } -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -static inline void clear_bit (unsigned long nr,volatile void * addr) {
arch/arc/include/asm/atomic.h +0 -5
@@ -190,11 +190,6 @@ static inline void atomic_clear_mask(uns#endif /* !CONFIG_ARC_HAS_LLSC */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -/** * __atomic_add_unless - add unless the number is a given value * @v: pointer of type atomic_t
arch/arc/include/asm/bitops.h +1 -4
@@ -19,6 +19,7 @@#include <linux/types.h> #include <linux/compiler.h> +#include <asm/barrier.h> /* * Hardware assisted read-modify-write using ARC700 LLOCK/SCOND insns.
@@ -496,10 +497,6 @@ static inline __attribute__ ((const)) in*/ #define ffz (x) __ffs(~(x)) -/* TODO does this affect uni-processor code */ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#include <asm-generic/bitops/hweight.h> #include <asm-generic/bitops/fls64.h> #include <asm-generic/bitops/sched.h>
arch/arm/include/asm/atomic.h +0 -5
@@ -211,11 +211,6 @@ static inline int __atomic_add_unless(at#define atomic_add_negative (i,v) (atomic_add_return(i, v) < 0) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() - #ifndef CONFIG_GENERIC_ATOMIC64 typedef struct { long long counter;
arch/arm/include/asm/barrier.h +3 -0
@@ -79,5 +79,8 @@ do { \#define set_mb (var, value) do { var = value; smp_mb(); } while (0) +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_BARRIER_H */
arch/arm/include/asm/bitops.h +1 -3
@@ -25,9 +25,7 @@#include <linux/compiler.h> #include <linux/irqflags.h> - -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() +#include <asm/barrier.h> /* * These functions are the basis of our bit ops.
arch/arm64/include/asm/atomic.h +0 -5
@@ -154,11 +154,6 @@ static inline int __atomic_add_unless(at#define atomic_add_negative (i,v) (atomic_add_return(i, v) < 0) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -/* * 64-bit atomic operations. */
arch/arm64/include/asm/barrier.h +3 -0
@@ -97,6 +97,9 @@ do { \#define set_mb (var, value) do { var = value; smp_mb(); } while (0)#define nop () asm volatile("nop"); +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif /* __ASSEMBLY__ */ #endif /* __ASM_BARRIER_H */
arch/arm64/include/asm/bitops.h +0 -9
@@ -17,17 +17,8 @@#define __ASM_BITOPS_H#include <linux/compiler.h> -#include <asm/barrier.h> -/* -* clear_bit may not imply a memory barrier -*/ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif -#ifndef _LINUX_BITOPS_H#error only <linux/bitops.h> can be included directly#endif
arch/avr32/include/asm/atomic.h +0 -5
@@ -183,9 +183,4 @@ static inline int atomic_sub_if_positive#define atomic_dec_if_positive (v) atomic_sub_if_positive(1, v) -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __ASM_AVR32_ATOMIC_H */
arch/avr32/include/asm/bitops.h +2 -7
@@ -13,12 +13,7 @@#endif #include <asm/byteorder.h> - -/* -* clear_bit() doesn't provide any barrier for the compiler -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> /* * set_bit - Atomically set a bit in memory
@@ -67,7 +62,7 @@ static inline void set_bit(int nr, volat* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void clear_bit (int nr,volatile void * addr)
arch/blackfin/include/asm/barrier.h +3 -0
@@ -27,6 +27,9 @@#endif /* !CONFIG_SMP */ +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +#include <asm-generic/barrier.h> #endif /* _BLACKFIN_BARRIER_H */
arch/blackfin/include/asm/bitops.h +2 -12
@@ -27,21 +27,17 @@#include <asm-generic/bitops/ext2-atomic.h> +#include <asm/barrier.h> +#ifndef CONFIG_SMP #include <linux/irqflags.h> -/* * clear_bit may not imply a memory barrier */ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif #include <asm-generic/bitops/atomic.h> #include <asm-generic/bitops/non-atomic.h> #else -#include <asm/barrier.h> #include <asm/byteorder.h> /* swab32 */ #include <linux/linkage.h>
@@ -101,12 +97,6 @@ static inline int test_and_change_bit(inreturn __raw_bit_test_toggle_asm (a, nr &0x1f ); } -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#define test_bit __skip_test_bit#include <asm-generic/bitops/non-atomic.h> #undef test_bit
arch/c6x/include/asm/bitops.h +1 -7
@@ -14,14 +14,8 @@#ifdef __KERNEL__#include <linux/bitops.h> -#include <asm/byteorder.h> - -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> /* * We are lucky, DSP is perfect for bitops: do it in 3 cycles
arch/cris/include/asm/atomic.h +1 -6
@@ -6,6 +6,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #include <arch/atomic.h> /*
@@ -151,10 +152,4 @@ static inline int __atomic_add_unless(at return ret; } -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif
arch/cris/include/asm/bitops.h +2 -7
@@ -21,6 +21,7 @@#include <arch/bitops.h> #include <linux/atomic.h> #include <linux/compiler.h> +#include <asm/barrier.h> /* * set_bit - Atomically set a bit in memory
@@ -42,7 +43,7 @@* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */
@@ -84,12 +85,6 @@ static inline int test_and_set_bit(int nreturn retval; } -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/** * test_and_clear_bit - Clear a bit and return its old value * @nr: Bit to clear
arch/frv/include/asm/atomic.h +1 -6
@@ -17,6 +17,7 @@#include <linux/types.h> #include <asm/spr-regs.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #ifdef CONFIG_SMP #error not SMP safe
@@ -29,12 +30,6 @@* We do not have SMP systems, so we don't have to deal with that. */ -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#define ATOMIC_INIT (i) { (i) }#define atomic_read (v) (*(volatile int *)&(v)->counter)#define atomic_set (v, i) (((v)->counter) = (i))
arch/frv/include/asm/bitops.h +0 -6
@@ -25,12 +25,6 @@#include <asm-generic/bitops/ffz.h> -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#ifndef CONFIG_FRV_OUTOFLINE_ATOMIC_OPS static inline unsigned long atomic_test_and_ANDNOT_mask (unsigned long mask,volatile unsigned long *v)
arch/hexagon/include/asm/atomic.h +1 -5
@@ -24,6 +24,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }#define atomic_set (v, i) ((v)->counter = (i))
@@ -163,9 +164,4 @@ static inline int __atomic_add_unless(at#define atomic_inc_return (v) (atomic_add_return(1, v))#define atomic_dec_return (v) (atomic_sub_return(1, v)) -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif
arch/hexagon/include/asm/bitops.h +1 -3
@@ -25,12 +25,10 @@#include <linux/compiler.h> #include <asm/byteorder.h> #include <asm/atomic.h> +#include <asm/barrier.h> #ifdef __KERNEL__ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/* * The offset calculations for these are based on BITS_PER_LONG == 32 * (i.e. I get to shift by #5-2 (32 bits per long, 4 bytes per access),
arch/ia64/include/asm/atomic.h +0 -6
@@ -208,10 +208,4 @@ atomic64_add_negative (__s64 i, atomic64#define atomic64_inc (v) atomic64_add(1, (v))#define atomic64_dec (v) atomic64_sub(1, (v)) -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* _ASM_IA64_ATOMIC_H */
arch/ia64/include/asm/barrier.h +3 -0
@@ -55,6 +55,9 @@#endif +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +/* * IA64 GCC turns volatile stores into st.rel and volatile loads into ld.acq no * need for asm trickery!
arch/ia64/include/asm/bitops.h +2 -4
@@ -16,6 +16,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/intrinsics.h> +#include <asm/barrier.h> /** * set_bit - Atomically set a bit in memory
@@ -65,9 +66,6 @@ __set_bit (int nr, volatile void *addr) *((__u32 *) addr + (nr >>5 )) |= (1 << (nr &31 )); } -#define smp_mb__before_clear_bit () barrier(); -#define smp_mb__after_clear_bit () barrier(); -/** * clear_bit - Clears a bit in memory * @nr: Bit to clear
@@ -75,7 +73,7 @@ __set_bit (int nr, volatile void *addr)* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static __inline__ void
arch/m32r/include/asm/atomic.h +1 -6
@@ -13,6 +13,7 @@#include <asm/assembler.h> #include <asm/cmpxchg.h> #include <asm/dcache_clear.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for
@@ -308,10 +309,4 @@ static __inline__ void atomic_set_mask(u local_irq_restore(flags); } -/* Atomic operations are already serializing on m32r */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* _ASM_M32R_ATOMIC_H */
arch/m32r/include/asm/bitops.h +2 -4
@@ -21,6 +21,7 @@#include <asm/byteorder.h> #include <asm/dcache_clear.h> #include <asm/types.h> +#include <asm/barrier.h> /* * These have to be done with inline assembly: that way the bit-setting
@@ -73,7 +74,7 @@ static __inline__ void set_bit(int nr, v* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static __inline__void clear_bit (int nr,volatile void * addr)
@@ -103,9 +104,6 @@ static __inline__ void clear_bit(int nr,local_irq_restore (flags); } -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/** * change_bit - Toggle a bit in memory * @nr: Bit to clear
arch/m68k/include/asm/atomic.h +1 -7
@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/irqflags.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for
@@ -209,11 +210,4 @@ static __inline__ int __atomic_add_unles return c; } - -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __ARCH_M68K_ATOMIC __ */
arch/m68k/include/asm/bitops.h +1 -6
@@ -13,6 +13,7 @@#endif #include <linux/compiler.h> +#include <asm/barrier.h> /* * Bit access functions vary across the ColdFire and 68k families.
@@ -67,12 +68,6 @@ static inline void bfset_mem_set_bit(int#define __set_bit (nr, vaddr) set_bit(nr, vaddr) -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -static inline void bclr_reg_clear_bit (int nr,volatile unsigned long *vaddr) {char *p = (char *)vaddr + (nr ^31 ) /8 ;
arch/metag/include/asm/atomic.h +1 -5
@@ -4,6 +4,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #if defined(CONFIG_METAG_ATOMICITY_IRQSOFF )/* The simple UP case. */
@@ -39,11 +40,6 @@#define atomic_inc_not_zero (v) atomic_add_unless((v), 1, 0) -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif #define atomic_dec_if_positive (v) atomic_sub_if_positive(1, v)
arch/metag/include/asm/barrier.h +3 -0
@@ -97,4 +97,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +#endif /* _ASM_METAG_BARRIER_H */
arch/metag/include/asm/bitops.h +0 -6
@@ -5,12 +5,6 @@#include <asm/barrier.h> #include <asm/global_lock.h> -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#ifdef CONFIG_SMP /* * These functions are the basis of our bit ops.
arch/mips/include/asm/atomic.h +0 -9
@@ -761,13 +761,4 @@ static __inline__ int atomic64_add_unles#endif /* CONFIG_64BIT */ -/* -* atomic*_return operations are serializing but not the non-*_return -* versions. -*/ -#define smp_mb__before_atomic_dec () smp_mb__before_llsc() -#define smp_mb__after_atomic_dec () smp_llsc_mb() -#define smp_mb__before_atomic_inc () smp_mb__before_llsc() -#define smp_mb__after_atomic_inc () smp_llsc_mb() -#endif /* _ASM_ATOMIC_H */
arch/mips/include/asm/barrier.h +3 -0
@@ -195,4 +195,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () smp_mb__before_llsc() +#define smp_mb__after_atomic () smp_llsc_mb() +#endif /* __ASM_BARRIER_H */
arch/mips/include/asm/bitops.h +2 -9
@@ -38,13 +38,6 @@#endif /* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () smp_mb__before_llsc() -#define smp_mb__after_clear_bit () smp_llsc_mb() - - -/* * These are the "slower" versions of the functions and are in bitops.c. * These functions call raw_local_irq_{save,restore}(). */
@@ -120,7 +113,7 @@ static inline void set_bit(unsigned long* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void clear_bit (unsigned long nr,volatile unsigned long *addr)
@@ -175,7 +168,7 @@ static inline void clear_bit(unsigned lo*/ static inline void clear_bit_unlock (unsigned long nr,volatile unsigned long *addr) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (nr, addr); }
arch/mips/kernel/irq.c +2 -2
@@ -62,9 +62,9 @@ void __init alloc_legacy_irqno(void)void free_irqno (unsigned int irq) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (irq, irq_map); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }/*
arch/mn10300/include/asm/atomic.h +1 -6
@@ -13,6 +13,7 @@#include <asm/irqflags.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #ifndef CONFIG_SMP #include <asm-generic/atomic.h>
@@ -234,12 +235,6 @@ static inline void atomic_set_mask(unsig#endif } -/* Atomic operations are already serializing on MN10300??? */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __KERNEL__ */ #endif /* CONFIG_SMP */ #endif /* _ASM_ATOMIC_H */
arch/mn10300/include/asm/bitops.h +1 -3
@@ -18,9 +18,7 @@#define __ASM_BITOPS_H#include <asm/cpu-regs.h> - -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> /* * set bit
arch/mn10300/mm/tlb-smp.c +2 -2
@@ -78,9 +78,9 @@ void smp_flush_tlb(void *unused)else local_flush_tlb_page (flush_mm, flush_va); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();cpumask_clear_cpu (cpu_id, &flush_cpumask); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); out:put_cpu (); }
arch/openrisc/include/asm/bitops.h +1 -8
@@ -27,14 +27,7 @@#include <linux/irqflags.h> #include <linux/compiler.h> - -/* -* clear_bit may not imply a memory barrier -*/ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif +#include <asm/barrier.h> #include <asm/bitops/__ffs.h> #include <asm-generic/bitops/ffz.h>
arch/parisc/include/asm/atomic.h +1 -5
@@ -7,6 +7,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for
@@ -143,11 +144,6 @@ static __inline__ int __atomic_add_unles#define ATOMIC_INIT (i) { (i) } -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#ifdef CONFIG_64BIT #define ATOMIC64_INIT (i) { (i) }
arch/parisc/include/asm/bitops.h +1 -3
@@ -8,6 +8,7 @@#include <linux/compiler.h> #include <asm/types.h> /* for BITS_PER_LONG/SHIFT_PER_LONG */ #include <asm/byteorder.h> +#include <asm/barrier.h> #include <linux/atomic.h> /*
@@ -19,9 +20,6 @@#define CHOP_SHIFTCOUNT (x) (((unsigned long) (x)) & (BITS_PER_LONG - 1)) -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -/* See http://marc.theaimsgroup.com/?t=108826637900003 for discussion * on use of volatile and __*_bit() (set/clear/change): * *_bit() want use of volatile.
arch/powerpc/include/asm/atomic.h +1 -5
@@ -8,6 +8,7 @@#ifdef __KERNEL__#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }
@@ -270,11 +271,6 @@ static __inline__ int atomic_dec_if_posi }#define atomic_dec_if_positive atomic_dec_if_positive -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#ifdef __powerpc64__#define ATOMIC64_INIT (i) { (i) }
arch/powerpc/include/asm/barrier.h +3 -0
@@ -84,4 +84,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif /* _ASM_POWERPC_BARRIER_H */
arch/powerpc/include/asm/bitops.h +1 -5
@@ -51,11 +51,7 @@#define PPC_BIT (bit) (1UL << PPC_BITLSHIFT(bit))#define PPC_BITMASK (bs, be) ((PPC_BIT(bs) - PPC_BIT(be)) | PPC_BIT(bs)) -/* -* clear_bit doesn't imply a memory barrier -*/ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() +#include <asm/barrier.h> /* Macro for generating the ***_bits() functions */ #define DEFINE_BITOP (fn, op, prefix) \
arch/powerpc/kernel/crash.c +1 -1
@@ -81,7 +81,7 @@ void crash_ipi_callback(struct pt_regs * }atomic_inc (&cpus_in_crash); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * Starting the kdump boot.
arch/s390/include/asm/atomic.h +1 -5
@@ -16,6 +16,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }
@@ -398,9 +399,4 @@ static inline long long atomic64_dec_if_#define atomic64_dec_and_test (_v) (atomic64_sub_return(1, _v) == 0)#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1, 0) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#endif /* __ARCH_S390_ATOMIC__ */
arch/s390/include/asm/barrier.h +3 -2
@@ -27,8 +27,9 @@#define smp_rmb () rmb()#define smp_wmb () wmb()#define smp_read_barrier_depends () read_barrier_depends() -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() + +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb()#define set_mb (var, value) do { var = value; mb(); } while (0)
arch/s390/include/asm/bitops.h +1 -0
@@ -47,6 +47,7 @@#include <linux/typecheck.h> #include <linux/compiler.h> +#include <asm/barrier.h> #ifndef CONFIG_64BIT
arch/score/include/asm/bitops.h +1 -6
@@ -2,12 +2,7 @@#define _ASM_SCORE_BITOPS_H#include <asm/byteorder.h> /* swab32 */ - -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> #include <asm-generic/bitops.h> #include <asm-generic/bitops/__fls.h>
arch/sh/include/asm/atomic.h +1 -5
@@ -10,6 +10,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }
@@ -62,9 +63,4 @@ static inline int __atomic_add_unless(at return c; } -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#endif /* __ASM_SH_ATOMIC_H */
arch/sh/include/asm/bitops.h +1 -6
@@ -9,6 +9,7 @@/* For __swab32 */ #include <asm/byteorder.h> +#include <asm/barrier.h> #ifdef CONFIG_GUSA_RB #include <asm/bitops-grb.h>
@@ -22,12 +23,6 @@#include <asm-generic/bitops/non-atomic.h> #endif -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#ifdef CONFIG_SUPERH32 static inline unsigned long ffz (unsigned long word) {
arch/sparc/include/asm/atomic_32.h +1 -6
@@ -14,6 +14,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #include <asm-generic/atomic64.h>
@@ -52,10 +53,4 @@ extern void atomic_set(atomic_t *, int);#define atomic_dec_and_test (v) (atomic_dec_return(v) == 0)#define atomic_sub_and_test (i, v) (atomic_sub_return(i, v) == 0) -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* !(__ARCH_SPARC_ATOMIC__) */
arch/sparc/include/asm/atomic_64.h +1 -6
@@ -9,6 +9,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }#define ATOMIC64_INIT (i) { (i) }
@@ -108,10 +109,4 @@ static inline long atomic64_add_unless(aextern long atomic64_dec_if_positive (atomic64_t *v); -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* !(__ARCH_SPARC64_ATOMIC__) */
arch/sparc/include/asm/barrier_64.h +3 -0
@@ -68,4 +68,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +#endif /* !(__SPARC64_BARRIER_H) */
arch/sparc/include/asm/bitops_32.h +1 -3
@@ -11,6 +11,7 @@#include <linux/compiler.h> #include <asm/byteorder.h> +#include <asm/barrier.h> #ifdef __KERNEL__
@@ -90,9 +91,6 @@ static inline void change_bit(unsigned l#include <asm-generic/bitops/non-atomic.h> -#define smp_mb__before_clear_bit () do { } while(0) -#define smp_mb__after_clear_bit () do { } while(0) -#include <asm-generic/bitops/ffz.h> #include <asm-generic/bitops/__ffs.h> #include <asm-generic/bitops/sched.h>
arch/sparc/include/asm/bitops_64.h +1 -3
@@ -13,6 +13,7 @@#include <linux/compiler.h> #include <asm/byteorder.h> +#include <asm/barrier.h> extern int test_and_set_bit (unsigned long nr,volatile unsigned long *addr);extern int test_and_clear_bit (unsigned long nr,volatile unsigned long *addr);
@@ -23,9 +24,6 @@ extern void change_bit(unsigned long nr,#include <asm-generic/bitops/non-atomic.h> -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#include <asm-generic/bitops/fls.h> #include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h>
arch/tile/include/asm/atomic_32.h +0 -10
@@ -169,16 +169,6 @@ static inline void atomic64_set(atomic64#define atomic64_dec_and_test (v) (atomic64_dec_return((v)) == 0)#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1LL, 0LL) -/* -* We need to barrier before modifying the word, since the _atomic_xxx() -* routines just tns the lock and then read/modify/write of the word. -* But after the word is updated, the routine issues an "mf" before returning, -* and since it's a function call, we don't even need a compiler barrier. -*/ -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_dec () do { } while (0) -#define smp_mb__after_atomic_inc () do { } while (0)#endif /* !__ASSEMBLY__ */
arch/tile/include/asm/atomic_64.h +0 -6
@@ -105,12 +105,6 @@ static inline long atomic64_add_unless(a#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1, 0) -/* Atomic dec and inc don't implement barrier, so provide them if needed. */ -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -/* Define this to indicate that cmpxchg is an efficient operation. */ #define __HAVE_ARCH_CMPXCHG
arch/tile/include/asm/barrier.h +14 -0
@@ -72,6 +72,20 @@ mb_incoherent(void)#define mb () fast_mb()#define iob () fast_iob() +#ifndef __tilegx__/* 32 bit */ +/* +* We need to barrier before modifying the word, since the _atomic_xxx() +* routines just tns the lock and then read/modify/write of the word. +* But after the word is updated, the routine issues an "mf" before returning, +* and since it's a function call, we don't even need a compiler barrier. +*/ +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () do { } while (0) +#else /* 64 bit */ +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif +#include <asm-generic/barrier.h> #endif /* !__ASSEMBLY__ */
arch/tile/include/asm/bitops.h +1 -0
@@ -17,6 +17,7 @@#define _ASM_TILE_BITOPS_H#include <linux/types.h> +#include <asm/barrier.h> #ifndef _LINUX_BITOPS_H#error only <linux/bitops.h> can be included directly
arch/tile/include/asm/bitops_32.h +2 -6
@@ -49,8 +49,8 @@ static inline void set_bit(unsigned nr,* restricted to acting on a single-word quantity. * * clear_bit() may not contain a memory barrier, so if it is used for -* locking purposes, you should call smp_mb__before_clear_bit() and/or -* smp_mb__after_clear_bit() to ensure changes are visible on other cpus. +* locking purposes, you should call smp_mb__before_atomic() and/or +* smp_mb__after_atomic() to ensure changes are visible on other cpus. */ static inline void clear_bit (unsigned nr,volatile unsigned long *addr) {
@@ -121,10 +121,6 @@ static inline int test_and_change_bit(un return (_atomic_xor(addr, mask) & mask) != 0; } -/* See discussion at smp_mb__before_atomic_dec() in <asm/atomic_32.h>. */ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () do {} while (0) -#include <asm-generic/bitops/ext2-atomic.h> #endif /* _ASM_TILE_BITOPS_32_H */
arch/tile/include/asm/bitops_64.h +0 -4
@@ -32,10 +32,6 @@ static inline void clear_bit(unsigned nr__insn_fetchand ((void *)(addr + nr /BITS_PER_LONG ), ~mask); } -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() - -static inline void change_bit (unsigned nr,volatile unsigned long *addr) {unsigned long mask = (1UL << (nr %BITS_PER_LONG ));
arch/x86/include/asm/atomic.h +1 -6
@@ -7,6 +7,7 @@#include <asm/alternative.h> #include <asm/cmpxchg.h> #include <asm/rmwcc.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for
@@ -243,12 +244,6 @@ static inline void atomic_or_long(unsign : : "r" ((unsigned)(mask)), "m" (*(addr)) \ : "memory") -/* Atomic operations are already serializing on x86 */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#ifdef CONFIG_X86_32 # include <asm/atomic64_32.h> #else
arch/x86/include/asm/barrier.h +4 -0
@@ -141,6 +141,10 @@ do { \#endif +/* Atomic operations are already serializing on x86 */ +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +/* * Stop RDTSC speculation. This is needed when you need to use RDTSC * (or get_cycles or vread that possibly accesses the TSC) in a defined
arch/x86/include/asm/bitops.h +2 -4
@@ -15,6 +15,7 @@#include <linux/compiler.h> #include <asm/alternative.h> #include <asm/rmwcc.h> +#include <asm/barrier.h> #if BITS_PER_LONG ==32 # define _BITOPS_LONG_SHIFT 5
@@ -102,7 +103,7 @@ static inline void __set_bit(long nr, vo* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static __always_inline void
@@ -156,9 +157,6 @@ static inline void __clear_bit_unlock(lo__clear_bit (nr, addr); } -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/** * __change_bit - Toggle a bit in memory * @nr: the bit to change
arch/x86/include/asm/sync_bitops.h +1 -1
@@ -41,7 +41,7 @@ static inline void sync_set_bit(long nr,* * sync_clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void sync_clear_bit (long nr,volatile unsigned long *addr)
arch/x86/kernel/apic/hw_nmi.c +1 -1
@@ -57,7 +57,7 @@ void arch_trigger_all_cpu_backtrace(void }clear_bit (0 , &backtrace_flag); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); } static int __kprobes
arch/xtensa/include/asm/atomic.h +1 -6
@@ -19,6 +19,7 @@#ifdef __KERNEL__#include <asm/processor.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }
@@ -387,12 +388,6 @@ static inline void atomic_set_mask(unsig#endif } -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __KERNEL__ */ #endif /* _XTENSA_ATOMIC_H */
arch/xtensa/include/asm/bitops.h +1 -3
@@ -21,9 +21,7 @@#include <asm/processor.h> #include <asm/byteorder.h> - -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() +#include <asm/barrier.h> #include <asm-generic/bitops/non-atomic.h>
block/blk-iopoll.c +2 -2
@@ -52,7 +52,7 @@ EXPORT_SYMBOL(blk_iopoll_sched);void __blk_iopoll_complete (struct blk_iopoll *iop) {list_del (&iop->list); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit_unlock (IOPOLL_F_SCHED , &iop->state); }EXPORT_SYMBOL (__blk_iopoll_complete);
@@ -164,7 +164,7 @@ EXPORT_SYMBOL(blk_iopoll_disable);void blk_iopoll_enable (struct blk_iopoll *iop) {BUG_ON (!test_bit (IOPOLL_F_SCHED , &iop->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit_unlock (IOPOLL_F_SCHED , &iop->state); }EXPORT_SYMBOL (blk_iopoll_enable);
crypto/chainiv.c +1 -1
@@ -126,7 +126,7 @@ static int async_chainiv_schedule_work(sint err = ctx->err;if (!ctx->queue.qlen) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (CHAINIV_STATE_INUSE , &ctx->state); if (!ctx->queue.qlen ||
drivers/base/power/domain.c +1 -1
@@ -106,7 +106,7 @@ static bool genpd_sd_counter_dec(structstatic void genpd_sd_counter_inc (struct generic_pm_domain *genpd) {atomic_inc (&genpd->sd_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static void genpd_acquire_lock (struct generic_pm_domain *genpd)
drivers/block/mtip32xx/mtip32xx.c +2 -2
@@ -224,9 +224,9 @@ static int get_slot(struct mtip_port *po*/ static inline void release_slot (struct mtip_port *port,int tag) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (tag, port->allocated); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }/*
drivers/cpuidle/coupled.c +1 -1
@@ -159,7 +159,7 @@ void cpuidle_coupled_parallel_barrier(st {int n = dev->coupled->online_count; -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (a);while (atomic_read (a) < n)
drivers/firewire/ohci.c +1 -1
@@ -3509,7 +3509,7 @@ static int ohci_flush_iso_completions(st }clear_bit_unlock (0 , &ctx->flushing_completions); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }tasklet_enable (&ctx->context.tasklet);
drivers/gpu/drm/drm_irq.c +5 -5
@@ -156,7 +156,7 @@ static void vblank_disable_and_save(stru*/ if ((vblrc >0 ) && (abs64 (diff_ns) >1000000 )) {atomic_inc (&dev->vblank[crtc].count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }/* Invalidate all timestamps while vblank irq's are off. */
@@ -864,9 +864,9 @@ static void drm_update_vblank_count(struvblanktimestamp (dev, crtc, tslot) = t_vblank; } -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_add (diff, &dev->vblank[crtc].count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }/**
@@ -1330,9 +1330,9 @@ bool drm_handle_vblank(struct drm_device/* Increment cooked vblank count. This also atomically commits * the timestamp computed above. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&dev->vblank[crtc].count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }else {DRM_DEBUG ("crtc %d: Redundant vblirq ignored. diff_ns = %d\n" , crtc, (int ) diff_ns);
drivers/gpu/drm/i915/i915_irq.c +1 -1
@@ -1996,7 +1996,7 @@ static void i915_error_work_func(struct* updates before * the counter increment. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&dev_priv->gpu_error.reset_counter); kobject_uevent_env(&dev->primary->kdev->kobj,
drivers/md/bcache/bcache.h +1 -1
@@ -841,7 +841,7 @@ static inline bool cached_dev_get(structreturn false;/* Paired with the mb in cached_dev_attach */ -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return true; }
drivers/md/bcache/closure.h +1 -1
@@ -243,7 +243,7 @@ static inline void set_closure_fn(struct cl->fn = fn; cl->wq = wq;/* between atomic_dec() in closure_put() */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic (); }static inline void closure_queue (struct closure *cl)
drivers/md/dm-bufio.c +4 -4
@@ -607,9 +607,9 @@ static void write_endio(struct bio *bio,BUG_ON (!test_bit (B_WRITING , &b->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (B_WRITING , &b->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&b->state,B_WRITING ); }
@@ -997,9 +997,9 @@ static void read_endio(struct bio *bio,BUG_ON (!test_bit (B_READING , &b->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (B_READING , &b->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&b->state,B_READING ); }
drivers/md/dm-snap.c +2 -2
@@ -642,7 +642,7 @@ static void free_pending_exception(strucstruct dm_snapshot *s = pe->snap;mempool_free (pe, s->pending_pool); -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&s->pending_exceptions_count); }
@@ -783,7 +783,7 @@ static int init_hash_tables(struct dm_snstatic void merge_shutdown (struct dm_snapshot *s) {clear_bit_unlock (RUNNING_MERGE , &s->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&s->state_bits,RUNNING_MERGE ); }
drivers/md/dm.c +1 -1
@@ -2451,7 +2451,7 @@ static void dm_wq_work(struct work_strucstatic void dm_queue_flush (struct mapped_device *md) {clear_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();queue_work (md->wq, &md->work); }
drivers/md/raid5.c +1 -1
@@ -4406,7 +4406,7 @@ static void raid5_unplug(struct blk_plug* STRIPE_ON_UNPLUG_LIST clear but the stripe * is still in our list */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (STRIPE_ON_UNPLUG_LIST , &sh->state);/* * STRIPE_ON_RELEASE_LIST could be set here. In that
drivers/media/usb/dvb-usb-v2/dvb_usb_core.c +3 -3
@@ -399,7 +399,7 @@ static int dvb_usb_stop_feed(struct dvb_/* clear 'streaming' status bit */ clear_bit (ADAP_STREAMING , &adap->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&adap->state_bits,ADAP_STREAMING ); skip_feed_stop:
@@ -550,7 +550,7 @@ static int dvb_usb_fe_init(struct dvb_fr err:if (!adap->suspend_resume_active) {clear_bit (ADAP_INIT , &adap->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&adap->state_bits,ADAP_INIT ); }
@@ -591,7 +591,7 @@ static int dvb_usb_fe_sleep(struct dvb_fif (!adap->suspend_resume_active) { adap->active_fe =-1 ;clear_bit (ADAP_SLEEP , &adap->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&adap->state_bits,ADAP_SLEEP ); }
drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +3 -3
@@ -2779,7 +2779,7 @@ int bnx2x_nic_load(struct bnx2x *bp, intcase LOAD_OPEN :netif_tx_start_all_queues (bp->dev); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();break ;case LOAD_DIAG :
@@ -4773,9 +4773,9 @@ void bnx2x_tx_timeout(struct net_devicebnx2x_panic ();#endif -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_TX_TIMEOUT , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* This allows the netif to be shutdown gracefully before resetting */ schedule_delayed_work (&bp->sp_rtnl_task,0 );
drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +17 -17
@@ -1837,10 +1837,10 @@ void bnx2x_sp_event(struct bnx2x_fastpat/* SRIOV: reschedule any 'in_progress' operations */ bnx2x_iov_sp_event (bp, cid, true); -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&bp->cq_spq_left);/* push the change in bp->spq_left and towards the memory */ -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();DP (BNX2X_MSG_SP ,"bp->cq_spq_left %x\n" ,atomic_read (&bp->cq_spq_left));
@@ -1855,11 +1855,11 @@ void bnx2x_sp_event(struct bnx2x_fastpat* sp_state is cleared, and this order prevents * races */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_AFEX_PENDING_VIFSET_MCP_ACK , &bp->sp_state);wmb ();clear_bit (BNX2X_AFEX_FCOE_Q_UPDATE_PENDING , &bp->sp_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* schedule the sp task as mcp ack is required */ bnx2x_schedule_sp_task (bp);
@@ -3878,9 +3878,9 @@ static void bnx2x_fan_failure(struct bnx* This is due to some boards consuming sufficient power when driver is * up to overheat if fan fails. */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_FAN_FAILURE , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); }
@@ -5137,9 +5137,9 @@ static void bnx2x_after_function_update(__clear_bit (RAMROD_COMP_WAIT , &queue_params.ramrod_flags);/* mark latest Q bit */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_AFEX_FCOE_Q_UPDATE_PENDING , &bp->sp_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* send Q update ramrod for FCoE Q */ rc =bnx2x_queue_state_change (bp, &queue_params);
@@ -5282,10 +5282,10 @@ static void bnx2x_eq_int(struct bnx2x *b* sp_rtnl task as all Queue SP operations * should run under rtnl_lock. */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_AFEX_F_UPDATE , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); }
@@ -5368,7 +5368,7 @@ static void bnx2x_eq_int(struct bnx2x *b spqe_cnt++; }/* for */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_add (spqe_cnt, &bp->eq_spq_left); bp->eq_cons = sw_cons;
@@ -12065,9 +12065,9 @@ static void bnx2x_set_rx_mode(struct net }else {/* Schedule an SP task to handle rest of change */ DP (NETIF_MSG_IFUP ,"Scheduling an Rx mode change\n" ); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_RX_MODE , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); } }
@@ -12101,10 +12101,10 @@ void bnx2x_set_rx_mode_inner(struct bnx2/* configuring mcast to a vf involves sleeping (when we * wait for the pf's response). */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_VFPF_MCAST , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); } }
@@ -13723,9 +13723,9 @@ static int bnx2x_drv_ctl(struct net_devicase DRV_CTL_RET_L2_SPQ_CREDIT_CMD : {int count = ctl->data.credit.credit_count; -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_add (count, &bp->cq_spq_left); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();break ; }case DRV_CTL_ULP_REGISTER_CMD : {
drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c +13 -13
@@ -258,16 +258,16 @@ static bool bnx2x_raw_check_pending(strustatic void bnx2x_raw_clear_pending (struct bnx2x_raw_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (o->state, o->pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static void bnx2x_raw_set_pending (struct bnx2x_raw_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (o->state, o->pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }/**
@@ -2131,7 +2131,7 @@ static int bnx2x_set_rx_mode_e1x(struct/* The operation is completed */ clear_bit (p->state, p->pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return 0 ; }
@@ -3576,16 +3576,16 @@ int bnx2x_config_mcast(struct bnx2x *bp,static void bnx2x_mcast_clear_sched (struct bnx2x_mcast_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (o->sched_state, o->raw.pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static void bnx2x_mcast_set_sched (struct bnx2x_mcast_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (o->sched_state, o->raw.pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static bool bnx2x_mcast_check_sched (struct bnx2x_mcast_obj *o)
@@ -4210,7 +4210,7 @@ int bnx2x_queue_state_change(struct bnx2if (rc) { o->next_state =BNX2X_Q_STATE_MAX ;clear_bit (pending_bit, pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return rc; }
@@ -4298,7 +4298,7 @@ static int bnx2x_queue_comp_cmd(struct bwmb ();clear_bit (cmd, &o->pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return 0 ; }
@@ -5242,7 +5242,7 @@ static inline int bnx2x_func_state_changwmb ();clear_bit (cmd, &o->pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return 0 ; }
@@ -5877,7 +5877,7 @@ int bnx2x_func_state_change(struct bnx2xif (rc) { o->next_state =BNX2X_F_STATE_MAX ;clear_bit (cmd, pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return rc; }
drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c +6 -6
@@ -971,10 +971,10 @@ static void bnx2x_vfop_qsetup(struct bnx op_done:case BNX2X_VFOP_QSETUP_DONE : vf->cfg_flags |=VF_CFG_VLAN ; -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_HYPERVISOR_VLAN , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 );bnx2x_vfop_end (bp, vf, vfop);return ;
@@ -2354,9 +2354,9 @@ staticvoid bnx2x_vf_handle_filters_eqe (struct bnx2x *bp,struct bnx2x_virtf *vf) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNX2X_FILTER_RX_MODE_PENDING , &vf->filter_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }int bnx2x_iov_eq_sp_event (struct bnx2x *bp,union event_ring_elem *elem)
@@ -3737,10 +3737,10 @@ void bnx2x_timer_sriov(struct bnx2x *bp)/* if channel is down we need to self destruct */ if (bp->old_bulletin.valid_bitmap &1 <<CHANNEL_DOWN ) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_VFPF_CHANNEL_DOWN , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); } }
drivers/net/ethernet/broadcom/cnic.c +4 -4
@@ -436,7 +436,7 @@ static int cnic_offld_prep(struct cnic_sstatic int cnic_close_prep (struct cnic_sock *csk) {clear_bit (SK_F_CONNECT_START , &csk->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (test_and_clear_bit (SK_F_OFFLD_COMPLETE , &csk->flags)) {while (test_and_set_bit (SK_F_OFFLD_SCHED , &csk->flags))
@@ -450,7 +450,7 @@ static int cnic_close_prep(struct cnic_sstatic int cnic_abort_prep (struct cnic_sock *csk) {clear_bit (SK_F_CONNECT_START , &csk->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();while (test_and_set_bit (SK_F_OFFLD_SCHED , &csk->flags))msleep (1 );
@@ -3645,7 +3645,7 @@ static int cnic_cm_destroy(struct cnic_scsk_hold (csk);clear_bit (SK_F_INUSE , &csk->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();while (atomic_read (&csk->ref_count) !=1 )msleep (1 );cnic_cm_cleanup (csk);
@@ -4025,7 +4025,7 @@ static void cnic_cm_process_kcqe(struct L4_KCQE_COMPLETION_STATUS_PARITY_ERROR) set_bit(SK_F_HW_ERR, &csk->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (SK_F_OFFLD_SCHED , &csk->flags);cnic_cm_upcall (cp, csk, opcode);break ;
drivers/net/ethernet/brocade/bna/bnad.c +3 -3
@@ -249,7 +249,7 @@ bnad_tx_complete(struct bnad *bnad, struif (likely (test_bit (BNAD_TXQ_TX_STARTED , &tcb->flags)))bna_ib_ack (tcb->i_dbell, sent); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNAD_TXQ_FREE_SENT , &tcb->flags);return sent;
@@ -1125,7 +1125,7 @@ bnad_tx_cleanup(struct delayed_work *worbnad_txq_cleanup (bnad, tcb); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNAD_TXQ_FREE_SENT , &tcb->flags); }
@@ -2999,7 +2999,7 @@ bnad_start_xmit(struct sk_buff *skb, str sent =bnad_txcmpl_process (bnad, tcb);if (likely (test_bit (BNAD_TXQ_TX_STARTED , &tcb->flags)))bna_ib_ack (tcb->i_dbell, sent); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNAD_TXQ_FREE_SENT , &tcb->flags); }else {netif_stop_queue (netdev);
drivers/net/ethernet/chelsio/cxgb/cxgb2.c +1 -1
@@ -281,7 +281,7 @@ static int cxgb_close(struct net_deviceif (adapter->params.stats_update_period && !(adapter->open_device_map &PORT_MASK )) {/* Stop statistics accumulation. */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();spin_lock (&adapter->work_lock);/* sync with update task */ spin_unlock (&adapter->work_lock);cancel_mac_stats_update (adapter);
drivers/net/ethernet/chelsio/cxgb3/sge.c +3 -3
@@ -1379,7 +1379,7 @@ static inline int check_desc_avail(strucstruct sge_qset *qs =txq_to_qset (q, qid);set_bit (qid, &qs->txq_stopped); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (should_restart_tx (q) &&test_and_clear_bit (qid, &qs->txq_stopped))
@@ -1492,7 +1492,7 @@ static void restart_ctrlq(unsigned longif (!skb_queue_empty (&q->sendq)) {set_bit (TXQ_CTRL , &qs->txq_stopped); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (should_restart_tx (q) &&test_and_clear_bit (TXQ_CTRL , &qs->txq_stopped))
@@ -1697,7 +1697,7 @@ again: reclaim_completed_tx(adap, q, TX_if (unlikely (q->size - q->in_use < ndesc)) {set_bit (TXQ_OFLD , &qs->txq_stopped); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (should_restart_tx (q) &&test_and_clear_bit (TXQ_OFLD , &qs->txq_stopped))
drivers/net/ethernet/chelsio/cxgb4/sge.c +1 -1
@@ -2003,7 +2003,7 @@ static void sge_rx_timer_cb(unsigned lonstruct sge_fl *fl = s->egr_map[id];clear_bit (id, s->starving_fl); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (fl_starving (fl)) { rxq = container_of(fl, struct sge_eth_rxq, fl);
drivers/net/ethernet/chelsio/cxgb4vf/sge.c +1 -1
@@ -1951,7 +1951,7 @@ static void sge_rx_timer_cb(unsigned lonstruct sge_fl *fl = s->egr_map[id];clear_bit (id, s->starving_fl); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* * Since we are accessing fl without a lock there's a
drivers/net/ethernet/intel/i40e/i40e_main.c +1 -1
@@ -4576,7 +4576,7 @@ static void i40e_service_event_complete(BUG_ON (!test_bit (__I40E_SERVICE_SCHED, &pf->state));/* flush memory to make sure state is correct before next watchog */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__I40E_SERVICE_SCHED, &pf->state); }
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2 -2
@@ -318,7 +318,7 @@ static void ixgbe_service_event_completeBUG_ON (!test_bit (__IXGBE_SERVICE_SCHED, &adapter->state));/* flush memory to make sure state is correct before next watchdog */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__IXGBE_SERVICE_SCHED, &adapter->state); }
@@ -4607,7 +4607,7 @@ static void ixgbe_up_complete(struct ixgif (hw->mac.ops.enable_tx_laser) hw->mac.ops.enable_tx_laser (hw); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__IXGBE_DOWN, &adapter->state);ixgbe_napi_enable_all (adapter);
drivers/net/wireless/ti/wlcore/main.c +1 -1
@@ -547,7 +547,7 @@ static int wlcore_irq_locked(struct wl12* wl1271_ps_elp_wakeup cannot be called concurrently. */ clear_bit (WL1271_FLAG_IRQ_RUNNING , &wl->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); ret =wlcore_fw_status (wl, wl->fw_status_1, wl->fw_status_2);if (ret <0 )
drivers/pci/xen-pcifront.c +2 -2
@@ -662,9 +662,9 @@ static void pcifront_do_aer(struct work_notify_remote_via_evtchn (pdev->evtchn);/*in case of we lost an aer request in four lines time_window*/ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (_PDEVB_op_active, &pdev->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_pcifront_aer_op (pdev);
drivers/scsi/isci/remote_device.c +1 -1
@@ -1541,7 +1541,7 @@ void isci_remote_device_release(struct kclear_bit (IDEV_STOP_PENDING , &idev->flags);clear_bit (IDEV_IO_READY , &idev->flags);clear_bit (IDEV_GONE , &idev->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (IDEV_ALLOCATED , &idev->flags);wake_up (&ihost->eventq); }
drivers/target/loopback/tcm_loop.c +2 -2
@@ -942,7 +942,7 @@ static int tcm_loop_port_link(struct tcm_loop_hba *tl_hba = tl_tpg->tl_hba;atomic_inc (&tl_tpg->tl_tpg_port_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * Add Linux/SCSI struct scsi_device by HCTL */
@@ -977,7 +977,7 @@ static void tcm_loop_port_unlink(scsi_device_put (sd);atomic_dec (&tl_tpg->tl_tpg_port_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();pr_debug ("TCM_Loop_ConfigFS: Port Unlink Successful\n" ); }
drivers/target/target_core_alua.c +13 -13
@@ -393,7 +393,7 @@ target_emulate_set_target_port_groups(stcontinue ;atomic_inc (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);
@@ -404,7 +404,7 @@ target_emulate_set_target_port_groups(stspin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_dec (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();break ; }spin_unlock (&dev->t10_alua.tg_pt_gps_lock);
@@ -997,7 +997,7 @@ static void core_alua_do_transition_tg_p* TARGET PORT GROUPS command */ atomic_inc (&mem->tg_pt_gp_mem_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&tg_pt_gp->tg_pt_gp_lock);spin_lock_bh (&port->sep_alua_lock);
@@ -1027,7 +1027,7 @@ static void core_alua_do_transition_tg_pspin_lock (&tg_pt_gp->tg_pt_gp_lock);atomic_dec (&mem->tg_pt_gp_mem_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&tg_pt_gp->tg_pt_gp_lock);/*
@@ -1061,7 +1061,7 @@ static void core_alua_do_transition_tg_p core_alua_dump_state(tg_pt_gp->tg_pt_gp_alua_pending_state));spin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_dec (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);if (tg_pt_gp->tg_pt_gp_transition_complete)
@@ -1123,7 +1123,7 @@ static int core_alua_do_transition_tg_pt*/ spin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_inc (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);if (!explicit && tg_pt_gp->tg_pt_gp_implicit_trans_secs) {
@@ -1166,7 +1166,7 @@ int core_alua_do_port_transition(spin_lock (&local_lu_gp_mem->lu_gp_mem_lock); lu_gp = local_lu_gp_mem->lu_gp;atomic_inc (&lu_gp->lu_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&local_lu_gp_mem->lu_gp_mem_lock);/* * For storage objects that are members of the 'default_lu_gp',
@@ -1183,7 +1183,7 @@ int core_alua_do_port_transition( rc =core_alua_do_transition_tg_pt (l_tg_pt_gp, new_state, explicit);atomic_dec (&lu_gp->lu_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();return rc; }/*
@@ -1197,7 +1197,7 @@ int core_alua_do_port_transition( dev = lu_gp_mem->lu_gp_mem_dev;atomic_inc (&lu_gp_mem->lu_gp_mem_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&lu_gp->lu_gp_lock);spin_lock (&dev->t10_alua.tg_pt_gps_lock);
@@ -1226,7 +1226,7 @@ int core_alua_do_port_transition( tg_pt_gp->tg_pt_gp_alua_nacl =NULL ; }atomic_inc (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);/* * core_alua_do_transition_tg_pt() will always return
@@ -1237,7 +1237,7 @@ int core_alua_do_port_transition(spin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_dec (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();if (rc)break ; }
@@ -1245,7 +1245,7 @@ int core_alua_do_port_transition(spin_lock (&lu_gp->lu_gp_lock);atomic_dec (&lu_gp_mem->lu_gp_mem_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&lu_gp->lu_gp_lock);
@@ -1259,7 +1259,7 @@ int core_alua_do_port_transition( }atomic_dec (&lu_gp->lu_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();return rc; }
drivers/target/target_core_device.c +3 -3
@@ -225,7 +225,7 @@ struct se_dev_entry *core_get_se_deve_frcontinue ;atomic_inc (&deve->pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock_irq (&nacl->device_list_lock);return deve;
@@ -1392,7 +1392,7 @@ int core_dev_add_initiator_node_lun_acl(spin_lock (&lun->lun_acl_lock);list_add_tail (&lacl->lacl_list, &lun->lun_acl_list);atomic_inc (&lun->lun_acl_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&lun->lun_acl_lock); pr_debug("%s_TPG[%hu]_LUN[%u->%u] - Added %s ACL for "
@@ -1426,7 +1426,7 @@ int core_dev_del_initiator_node_lun_acl(spin_lock (&lun->lun_acl_lock);list_del (&lacl->lacl_list);atomic_dec (&lun->lun_acl_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();spin_unlock (&lun->lun_acl_lock); core_disable_device_list_for_node(lun, NULL, lacl->mapped_lun,
drivers/target/target_core_iblock.c +1 -1
@@ -324,7 +324,7 @@ static void iblock_bio_done(struct bio ** Bump the ib_bio_err_cnt and release bio. */ atomic_inc (&ibr->ib_bio_err_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }bio_put (bio);
drivers/target/target_core_pr.c +28 -28
214 lines not on this page · raw message
drivers/target/target_core_transport.c +8 -8
@@ -728,7 +728,7 @@ void target_qf_do_work(struct work_struclist_for_each_entry_safe (cmd, cmd_tmp, &qf_cmd_list, se_qf_node) {list_del (&cmd->se_qf_node);atomic_dec (&dev->dev_qf_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); pr_debug("Processing %s cmd: %p QUEUE_FULL in work queue" " context: %s\n", cmd->se_tfo->get_fabric_name(), cmd,
@@ -1140,7 +1140,7 @@ transport_check_alloc_task_attr(struct s* Dormant to Active status. */ cmd->se_ordered_id =atomic_inc_return (&dev->dev_ordered_id); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();pr_debug ("Allocated se_ordered_id: %u for Task Attr: 0x%02x on %s\n" , cmd->se_ordered_id, cmd->sam_task_attr, dev->transport->name);
@@ -1692,7 +1692,7 @@ static bool target_handle_task_attr(strureturn false;case MSG_ORDERED_TAG :atomic_inc (&dev->dev_ordered_sync); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); pr_debug("Added ORDERED for CDB: 0x%02x to ordered list, " " se_ordered_id: %u\n",
@@ -1710,7 +1710,7 @@ static bool target_handle_task_attr(stru* For SIMPLE and UNTAGGED Task Attribute commands */ atomic_inc (&dev->simple_cmds); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();break ; }
@@ -1806,7 +1806,7 @@ static void transport_complete_task_attrif (cmd->sam_task_attr ==MSG_SIMPLE_TAG ) {atomic_dec (&dev->simple_cmds); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); dev->dev_cur_ordered_id++; pr_debug("Incremented dev->dev_cur_ordered_id: %u for" " SIMPLE: %u\n", dev->dev_cur_ordered_id,
@@ -1818,7 +1818,7 @@ static void transport_complete_task_attr cmd->se_ordered_id); }else if (cmd->sam_task_attr ==MSG_ORDERED_TAG ) {atomic_dec (&dev->dev_ordered_sync); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); dev->dev_cur_ordered_id++; pr_debug("Incremented dev_cur_ordered_id: %u for ORDERED:"
@@ -1877,7 +1877,7 @@ static void transport_handle_queue_full(spin_lock_irq (&dev->qf_cmd_lock);list_add_tail (&cmd->se_qf_node, &cmd->se_dev->qf_cmd_list);atomic_inc (&dev->dev_qf_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock_irq (&cmd->se_dev->qf_cmd_lock);schedule_work (&cmd->se_dev->qf_work_queue);
@@ -2805,7 +2805,7 @@ void transport_send_task_abort(struct seif (cmd->data_direction ==DMA_TO_DEVICE ) {if (cmd->se_tfo->write_pending_status (cmd) !=0 ) { cmd->transport_state |=CMD_T_ABORTED ; -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return ; } }
drivers/target/target_core_ua.c +5 -5
@@ -162,7 +162,7 @@ int core_scsi3_ua_allocate(spin_unlock_irq (&nacl->device_list_lock);atomic_inc (&deve->ua_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return 0 ; }list_add_tail (&ua->ua_nacl_list, &deve->ua_list);
@@ -175,7 +175,7 @@ int core_scsi3_ua_allocate( asc, ascq);atomic_inc (&deve->ua_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return 0 ; }
@@ -190,7 +190,7 @@ void core_scsi3_ua_release_all(kmem_cache_free (se_ua_cache, ua);atomic_dec (&deve->ua_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&deve->ua_lock); }
@@ -251,7 +251,7 @@ void core_scsi3_ua_for_check_condition(kmem_cache_free (se_ua_cache, ua);atomic_dec (&deve->ua_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&deve->ua_lock);spin_unlock_irq (&nacl->device_list_lock);
@@ -310,7 +310,7 @@ int core_scsi3_ua_clear_for_request_senskmem_cache_free (se_ua_cache, ua);atomic_dec (&deve->ua_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&deve->ua_lock);spin_unlock_irq (&nacl->device_list_lock);
drivers/tty/n_tty.c +1 -1
@@ -2042,7 +2042,7 @@ static int canon_copy_from_read_buf(struif (found)clear_bit (eol, ldata->read_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); ldata->read_tail += c;if (found) {
drivers/tty/serial/mxs-auart.c +2 -2
@@ -200,7 +200,7 @@ static void dma_tx_callback(void *param)/* clear the bit used to serialize the DMA tx. */ clear_bit (MXS_AUART_DMA_TX_SYNC , &s->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* wake up the possible processes. */ if (uart_circ_chars_pending (xmit) <WAKEUP_CHARS )
@@ -275,7 +275,7 @@ static void mxs_auart_tx_chars(struct mxmxs_auart_dma_tx (s, i); }else {clear_bit (MXS_AUART_DMA_TX_SYNC , &s->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }return ; }
drivers/usb/gadget/tcm_usb_gadget.c +2 -2
@@ -1846,7 +1846,7 @@ static int usbg_port_link(struct se_port struct usbg_tpg *tpg = container_of(se_tpg, struct usbg_tpg, se_tpg);atomic_inc (&tpg->tpg_port_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return 0 ; }
@@ -1856,7 +1856,7 @@ static void usbg_port_unlink(struct se_p struct usbg_tpg *tpg = container_of(se_tpg, struct usbg_tpg, se_tpg);atomic_dec (&tpg->tpg_port_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }static int usbg_check_stop_free (struct se_cmd *se_cmd)
drivers/usb/serial/usb_wwan.c +1 -1
@@ -325,7 +325,7 @@ static void usb_wwan_outdat_callback(strfor (i =0 ; i <N_OUT_URB ; ++i) {if (portdata->out_urbs[i] == urb) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (i, &portdata->out_busy);break ; }
drivers/vhost/scsi.c +1 -1
@@ -1244,7 +1244,7 @@ vhost_scsi_set_endpoint(struct vhost_scs tpg->tv_tpg_vhost_count++; tpg->vhost_scsi = vs; vs_tpg[tpg->tport_tpgt] = tpg; -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); match = true; }mutex_unlock (&tpg->tv_tpg_mutex);
drivers/w1/w1_family.c +2 -2
@@ -131,9 +131,9 @@ void w1_family_get(struct w1_family *f)void __w1_family_get (struct w1_family *f) { -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&f->refcnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }EXPORT_SYMBOL (w1_unregister_family);
drivers/xen/xen-pciback/pciback_ops.c +2 -2
@@ -348,9 +348,9 @@ void xen_pcibk_do_op(struct work_structnotify_remote_via_irq (pdev->evtchn_irq);/* Mark that we're done. */ -smp_mb__before_clear_bit ();/* /after/ clearing PCIF_active */ +smp_mb__before_atomic ();/* /after/ clearing PCIF_active */ clear_bit (_PDEVF_op_active, &pdev->flags); -smp_mb__after_clear_bit ();/* /before/ final check for work */ +smp_mb__after_atomic ();/* /before/ final check for work */ /* Check to see if the driver domain tried to start another request in * between clearing _XEN_PCIF_active and clearing _PDEVF_op_active.
fs/btrfs/btrfs_inode.h +1 -1
@@ -279,7 +279,7 @@ static inline void btrfs_inode_block_unlstatic inline void btrfs_inode_resume_unlocked_dio (struct inode *inode) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BTRFS_INODE_READDIO_NEED_LOCK , &BTRFS_I (inode)->runtime_flags); }
fs/btrfs/extent_io.c +1 -1
@@ -3451,7 +3451,7 @@ static int lock_extent_buffer_for_io(strstatic void end_extent_buffer_writeback (struct extent_buffer *eb) {clear_bit (EXTENT_BUFFER_WRITEBACK , &eb->bflags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&eb->bflags,EXTENT_BUFFER_WRITEBACK ); }
fs/btrfs/inode.c +3 -3
@@ -7078,7 +7078,7 @@ static void btrfs_end_dio_bio(struct bio* before atomic variable goto zero, we must make sure * dip->errors is perceived to be set. */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic (); }/* if there are more bios still pending for this dio, just exit */
@@ -7258,7 +7258,7 @@ static int btrfs_submit_direct_hook(int* before atomic variable goto zero, we must * make sure dip->errors is perceived to be set. */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();if (atomic_dec_and_test (&dip->pending_bios))bio_io_error (dip->orig_bio);
@@ -7401,7 +7401,7 @@ static ssize_t btrfs_direct_IO(int rw, sreturn 0 ;atomic_inc (&inode->i_dio_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * The generic stuff only does filemap_write_and_wait_range, which isn't
fs/buffer.c +1 -1
@@ -77,7 +77,7 @@ EXPORT_SYMBOL(__lock_buffer);void unlock_buffer (struct buffer_head *bh) {clear_bit_unlock (BH_Lock, &bh->b_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&bh->b_state, BH_Lock); }EXPORT_SYMBOL (unlock_buffer);
fs/ext4/resize.c +1 -1
@@ -42,7 +42,7 @@ int ext4_resize_begin(struct super_blockvoid ext4_resize_end (struct super_block *sb) {clear_bit_unlock (EXT4_RESIZING , &EXT4_SB (sb)->s_resize_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); } static ext4_group_t ext4_meta_bg_first_group(struct super_block *sb,
fs/gfs2/glock.c +4 -4
@@ -275,7 +275,7 @@ static inline int may_grant(const structstatic void gfs2_holder_wake (struct gfs2_holder *gh) {clear_bit (HIF_WAIT , &gh->gh_iflags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&gh->gh_iflags,HIF_WAIT ); }
@@ -409,7 +409,7 @@ static void gfs2_demote_wake(struct gfs2 { gl->gl_demote_state =LM_ST_EXCLUSIVE ;clear_bit (GLF_DEMOTE , &gl->gl_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&gl->gl_flags,GLF_DEMOTE ); }
@@ -618,7 +618,7 @@ __acquires(&gl->gl_spin) out_sched:clear_bit (GLF_LOCK , &gl->gl_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); gl->gl_lockref.count++;if (queue_delayed_work (glock_workqueue, &gl->gl_work,0 ) ==0 ) gl->gl_lockref.count--;
@@ -626,7 +626,7 @@ __acquires(&gl->gl_spin) out_unlock:clear_bit (GLF_LOCK , &gl->gl_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return ; }
fs/gfs2/glops.c +1 -1
@@ -219,7 +219,7 @@ static void inode_go_sync(struct gfs2_gl* Writeback of the data mapping may cause the dirty flag to be set * so we have to clear it again here. */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (GLF_DIRTY , &gl->gl_flags); }
fs/gfs2/lock_dlm.c +2 -2
@@ -1132,7 +1132,7 @@ static void gdlm_recover_done(void *arg,queue_delayed_work (gfs2_control_wq, &sdp->sd_control_work,0 );clear_bit (DFL_DLM_RECOVERY , &ls->ls_recover_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&ls->ls_recover_flags,DFL_DLM_RECOVERY );spin_unlock (&ls->ls_recover_spin); }
@@ -1269,7 +1269,7 @@ static int gdlm_mount(struct gfs2_sbd *s ls->ls_first = !!test_bit (DFL_FIRST_MOUNT , &ls->ls_recover_flags);clear_bit (SDF_NOJOURNALID , &sdp->sd_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&sdp->sd_flags,SDF_NOJOURNALID );return 0 ;
fs/gfs2/recovery.c +1 -1
@@ -587,7 +587,7 @@ void gfs2_recover_func(struct work_strucgfs2_recovery_done (sdp, jd->jd_jid,LM_RD_GAVEUP ); done:clear_bit (JDF_RECOVERY , &jd->jd_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&jd->jd_flags,JDF_RECOVERY ); }
fs/gfs2/sys.c +2 -2
@@ -332,7 +332,7 @@ static ssize_t block_store(struct gfs2_sset_bit (DFL_BLOCK_LOCKS , &ls->ls_recover_flags); else if (val == 0) {clear_bit (DFL_BLOCK_LOCKS , &ls->ls_recover_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();gfs2_glock_thaw (sdp); } else { ret = -EINVAL ;
@@ -481,7 +481,7 @@ static ssize_t jid_store(struct gfs2_sbd rv = jid = -EINVAL ; sdp->sd_lockstruct.ls_jid = jid;clear_bit (SDF_NOJOURNALID , &sdp->sd_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&sdp->sd_flags,SDF_NOJOURNALID ); out:spin_unlock (&sdp->sd_jindex_spin);
fs/jbd2/commit.c +3 -3
@@ -43,7 +43,7 @@ static void journal_end_buffer_io_sync(sclear_buffer_uptodate (bh);if (orig_bh) {clear_bit_unlock (BH_Shadow, &orig_bh->b_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&orig_bh->b_state, BH_Shadow); }unlock_buffer (bh);
@@ -239,7 +239,7 @@ static int journal_submit_data_buffers(jspin_lock (&journal->j_list_lock);J_ASSERT (jinode->i_transaction == commit_transaction);clear_bit (__JI_COMMIT_RUNNING, &jinode->i_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&jinode->i_flags, __JI_COMMIT_RUNNING); }spin_unlock (&journal->j_list_lock);
@@ -277,7 +277,7 @@ static int journal_finish_inode_data_buf }spin_lock (&journal->j_list_lock);clear_bit (__JI_COMMIT_RUNNING, &jinode->i_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&jinode->i_flags, __JI_COMMIT_RUNNING); }
fs/nfs/dir.c +6 -6
@@ -1985,9 +1985,9 @@ static void nfs_access_free_entry(struct {put_rpccred (entry->cred);kfree (entry); -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_long_dec (&nfs_access_nr_entries); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }static void nfs_access_free_list (struct list_head *head)
@@ -2035,9 +2035,9 @@ nfs_access_cache_scan(struct shrinker *selse { remove_lru_entry:list_del_init (&nfsi->access_cache_inode_lru); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NFS_INO_ACL_LRU_SET , &nfsi->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }spin_unlock (&inode->i_lock); }
@@ -2185,9 +2185,9 @@ void nfs_access_add_cache(struct inode *nfs_access_add_rbtree (inode, cache);/* Update accounting */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_long_inc (&nfs_access_nr_entries); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* Add inode to global LRU list */ if (!test_bit (NFS_INO_ACL_LRU_SET , &NFS_I (inode)->flags)) {
fs/nfs/inode.c +1 -1
@@ -1058,7 +1058,7 @@ int nfs_revalidate_mapping(struct inodetrace_nfs_invalidate_mapping_exit (inode, ret);clear_bit_unlock (NFS_INO_INVALIDATING , bitlock); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (bitlock,NFS_INO_INVALIDATING ); out:return ret;
fs/nfs/nfs4filelayoutdev.c +2 -2
@@ -789,9 +789,9 @@ static void nfs4_wait_ds_connect(structstatic void nfs4_clear_ds_conn_bit (struct nfs4_pnfs_ds *ds) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NFS4DS_CONNECTING , &ds->ds_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&ds->ds_state,NFS4DS_CONNECTING ); }
fs/nfs/nfs4state.c +2 -2
@@ -1145,9 +1145,9 @@ static int nfs4_run_state_manager(void *static void nfs4_clear_state_manager_bit (struct nfs_client *clp) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NFS4CLNT_MANAGER_RUNNING , &clp->cl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&clp->cl_state,NFS4CLNT_MANAGER_RUNNING );rpc_wake_up (&clp->cl_rpcwaitq); }
fs/nfs/pagelist.c +3 -3
@@ -95,7 +95,7 @@ nfs_iocounter_dec(struct nfs_io_counter {if (atomic_dec_and_test (&c->io_count)) {clear_bit (NFS_IO_INPROGRESS , &c->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&c->flags,NFS_IO_INPROGRESS ); } }
@@ -193,9 +193,9 @@ void nfs_unlock_request(struct nfs_pageprintk (KERN_ERR "NFS: Invalid unlock attempted\n" );BUG (); } -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (PG_BUSY , &req->wb_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&req->wb_flags,PG_BUSY ); }
fs/nfs/pnfs.c +1 -1
@@ -1795,7 +1795,7 @@ static void pnfs_clear_layoutcommitting(unsigned long *bitlock = &NFS_I (inode)->flags;clear_bit_unlock (NFS_INO_LAYOUTCOMMITTING , bitlock); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (bitlock,NFS_INO_LAYOUTCOMMITTING ); }
fs/nfs/pnfs.h +1 -1
@@ -275,7 +275,7 @@ pnfs_get_lseg(struct pnfs_layout_segment {if (lseg) {atomic_inc (&lseg->pls_refcount); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }return lseg; }
fs/nfs/write.c +2 -2
@@ -405,7 +405,7 @@ int nfs_writepages(struct address_spacenfs_pageio_complete (&pgio);clear_bit_unlock (NFS_INO_FLUSHING , bitlock); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (bitlock,NFS_INO_FLUSHING );if (err <0 )
@@ -1458,7 +1458,7 @@ static int nfs_commit_set_lock(struct nfstatic void nfs_commit_clear_lock (struct nfs_inode *nfsi) {clear_bit (NFS_INO_COMMIT , &nfsi->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&nfsi->flags,NFS_INO_COMMIT ); }
fs/ubifs/lpt_commit.c +2 -2
@@ -460,9 +460,9 @@ static int write_cnodes(struct ubifs_inf* important. */ clear_bit (DIRTY_CNODE , &cnode->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (COW_CNODE , &cnode->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); offs += len;dbg_chk_lpt_sz (c,1 , len); cnode = cnode->cnext;
fs/ubifs/tnc_commit.c +2 -2
@@ -895,9 +895,9 @@ static int write_index(struct ubifs_info* the reason for the second barrier. */ clear_bit (DIRTY_ZNODE , &znode->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (COW_ZNODE , &znode->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* * We have marked the znode as clean but have not updated the
include/asm-generic/atomic.h +1 -6
@@ -16,6 +16,7 @@#define __ASM_GENERIC_ATOMIC_H#include <asm/cmpxchg.h> +#include <asm/barrier.h> #ifdef CONFIG_SMP /* Force people to define core atomics */
@@ -182,11 +183,5 @@ static inline void atomic_set_mask(unsig }#endif -/* Assume that atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __KERNEL__ */ #endif /* __ASM_GENERIC_ATOMIC_H */
include/asm-generic/barrier.h +8 -0
@@ -62,6 +62,14 @@#define set_mb (var, value) do { (var) = (value); mb(); } while (0)#endif +#ifndef smp_mb__before_atomic +#define smp_mb__before_atomic () smp_mb() +#endif + +#ifndef smp_mb__after_atomic +#define smp_mb__after_atomic () smp_mb() +#endif +#define smp_store_release (p, v) \ do { \ compiletime_assert_atomic_type(*p); \
include/asm-generic/bitops.h +1 -7
@@ -11,14 +11,8 @@#include <linux/irqflags.h> #include <linux/compiler.h> +#include <asm/barrier.h> -/* -* clear_bit may not imply a memory barrier -*/ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif #include <asm-generic/bitops/__ffs.h> #include <asm-generic/bitops/ffz.h>
include/asm-generic/bitops/atomic.h +1 -1
@@ -80,7 +80,7 @@ static inline void set_bit(int nr, volat* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void clear_bit (int nr,volatile unsigned long *addr)
include/asm-generic/bitops/lock.h +1 -1
@@ -20,7 +20,7 @@*/ #define clear_bit_unlock (nr, addr) \ do { \ - smp_mb__before_clear_bit(); \ + smp_mb__before_atomic(); \ clear_bit(nr, addr); \ } while (0)
include/linux/buffer_head.h +1 -1
@@ -278,7 +278,7 @@ static inline void get_bh(struct buffer_static inline void put_bh (struct buffer_head *bh) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&bh->b_count); }
include/linux/genhd.h +1 -1
@@ -649,7 +649,7 @@ static inline void hd_ref_init(struct hdstatic inline void hd_struct_get (struct hd_struct *part) {atomic_inc (&part->ref); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static inline int hd_struct_try_get (struct hd_struct *part)
include/linux/interrupt.h +4 -4
@@ -447,7 +447,7 @@ static inline int tasklet_trylock(structstatic inline void tasklet_unlock (struct tasklet_struct *t) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (TASKLET_STATE_RUN , &(t)->state); }
@@ -495,7 +495,7 @@ static inline void tasklet_hi_schedule_fstatic inline void tasklet_disable_nosync (struct tasklet_struct *t) {atomic_inc (&t->count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static inline void tasklet_disable (struct tasklet_struct *t)
@@ -507,13 +507,13 @@ static inline void tasklet_disable(strucstatic inline void tasklet_enable (struct tasklet_struct *t) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&t->count); }static inline void tasklet_hi_enable (struct tasklet_struct *t) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&t->count); }
include/linux/netdevice.h +1 -1
@@ -500,7 +500,7 @@ static inline void napi_disable(struct nstatic inline void napi_enable (struct napi_struct *n) {BUG_ON (!test_bit (NAPI_STATE_SCHED , &n->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NAPI_STATE_SCHED , &n->state); }
include/linux/sched.h +2 -4
@@ -2737,10 +2737,8 @@ static inline bool __must_check current_/* * Polling state must be visible before we test NEED_RESCHED, * paired by resched_task() -* -* XXX: assumes set/clear bit are identical barrier wise. */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return unlikely (tif_need_resched ()); }
@@ -2758,7 +2756,7 @@ static inline bool __must_check current_* Polling state must be visible before we test NEED_RESCHED, * paired by resched_task() */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return unlikely (tif_need_resched ()); }
include/linux/sunrpc/sched.h +4 -4
@@ -142,18 +142,18 @@ struct rpc_task_setup { test_and_set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)#define rpc_clear_running (t) \ do { \ - smp_mb__before_clear_bit(); \ + smp_mb__before_atomic(); \ clear_bit(RPC_TASK_RUNNING, &(t)->tk_runstate); \ - smp_mb__after_clear_bit(); \ + smp_mb__after_atomic(); \ } while (0)#define RPC_IS_QUEUED (t) test_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)#define rpc_set_queued (t) set_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)#define rpc_clear_queued (t) \ do { \ - smp_mb__before_clear_bit(); \ + smp_mb__before_atomic(); \ clear_bit(RPC_TASK_QUEUED, &(t)->tk_runstate); \ - smp_mb__after_clear_bit(); \ + smp_mb__after_atomic(); \ } while (0)#define RPC_IS_ACTIVATED (t) test_bit(RPC_TASK_ACTIVE, &(t)->tk_runstate)
include/linux/sunrpc/xprt.h +4 -4
@@ -368,9 +368,9 @@ static inline int xprt_test_and_clear_costatic inline void xprt_clear_connecting (struct rpc_xprt *xprt) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (XPRT_CONNECTING , &xprt->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static inline int xprt_connecting (struct rpc_xprt *xprt)
@@ -400,9 +400,9 @@ static inline void xprt_clear_bound(strustatic inline void xprt_clear_binding (struct rpc_xprt *xprt) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (XPRT_BINDING , &xprt->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static inline int xprt_test_and_set_binding (struct rpc_xprt *xprt)
include/linux/tracehook.h +1 -1
@@ -191,7 +191,7 @@ static inline void tracehook_notify_resu* pairs with task_work_add()->set_notify_resume() after * hlist_add_head(task->task_works); */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (unlikely (current->task_works))task_work_run (); }
include/net/ip_vs.h +2 -2
@@ -1204,7 +1204,7 @@ static inline bool __ip_vs_conn_get(stru/* put back the conn without restarting its timer */ static inline void __ip_vs_conn_put (struct ip_vs_conn *cp) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&cp->refcnt); }void ip_vs_conn_put (struct ip_vs_conn *cp);
@@ -1408,7 +1408,7 @@ static inline void ip_vs_dest_hold(strucstatic inline void ip_vs_dest_put (struct ip_vs_dest *dest) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&dest->refcnt); }
kernel/debug/debug_core.c +2 -2
@@ -526,7 +526,7 @@ static int kgdb_cpu_enter(struct kgdb_st kgdb_info[cpu].exception_state &= ~(DCPU_WANT_MASTER |DCPU_IS_SLAVE ); kgdb_info[cpu].enter_kgdb--; -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&slaves_in_kgdb);dbg_touch_watchdogs ();local_irq_restore (flags);
@@ -654,7 +654,7 @@ static int kgdb_cpu_enter(struct kgdb_st kgdb_info[cpu].exception_state &= ~(DCPU_WANT_MASTER |DCPU_IS_SLAVE ); kgdb_info[cpu].enter_kgdb--; -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&masters_in_kgdb);/* Free kgdb_active */ atomic_set (&kgdb_active,-1 );
kernel/futex.c +1 -1
@@ -250,7 +250,7 @@ static inline void futex_get_mm(union fu* get_futex_key() implies a full barrier. This is relied upon * as full barrier (B), see the ordering comment above. */ -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static inline bool hb_waiters_pending (struct futex_hash_bucket *hb)
kernel/kmod.c +1 -1
@@ -498,7 +498,7 @@ int __usermodehelper_disable(enum umh_distatic void helper_lock (void ) {atomic_inc (&running_helpers); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static void helper_unlock (void )
kernel/rcu/tree.c +11 -11
@@ -389,9 +389,9 @@ static void rcu_eqs_enter_common(struct }rcu_prepare_for_idle (smp_processor_id ());/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */ -smp_mb__before_atomic_inc ();/* See above. */ +smp_mb__before_atomic ();/* See above. */ atomic_inc (&rdtp->dynticks); -smp_mb__after_atomic_inc ();/* Force ordering with next sojourn. */ +smp_mb__after_atomic ();/* Force ordering with next sojourn. */ WARN_ON_ONCE (atomic_read (&rdtp->dynticks) &0x1 );/*
@@ -509,10 +509,10 @@ void rcu_irq_exit(void)static void rcu_eqs_exit_common (struct rcu_dynticks *rdtp,long long oldval,int user) { -smp_mb__before_atomic_inc ();/* Force ordering w/previous sojourn. */ +smp_mb__before_atomic ();/* Force ordering w/previous sojourn. */ atomic_inc (&rdtp->dynticks);/* CPUs seeing atomic_inc() must see later RCU read-side crit sects */ -smp_mb__after_atomic_inc ();/* See above. */ +smp_mb__after_atomic ();/* See above. */ WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks) &0x1 ));rcu_cleanup_after_idle (smp_processor_id ());trace_rcu_dyntick (TPS ("End" ), oldval, rdtp->dynticks_nesting);
@@ -637,10 +637,10 @@ void rcu_nmi_enter(void) (atomic_read(&rdtp->dynticks) & 0x1))return ; rdtp->dynticks_nmi_nesting++; -smp_mb__before_atomic_inc ();/* Force delay from prior write. */ +smp_mb__before_atomic ();/* Force delay from prior write. */ atomic_inc (&rdtp->dynticks);/* CPUs seeing atomic_inc() must see later RCU read-side crit sects */ -smp_mb__after_atomic_inc ();/* See above. */ +smp_mb__after_atomic ();/* See above. */ WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks) &0x1 )); }
@@ -659,9 +659,9 @@ void rcu_nmi_exit(void) --rdtp->dynticks_nmi_nesting != 0)return ;/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */ -smp_mb__before_atomic_inc ();/* See above. */ +smp_mb__before_atomic ();/* See above. */ atomic_inc (&rdtp->dynticks); -smp_mb__after_atomic_inc ();/* Force delay to next write. */ +smp_mb__after_atomic ();/* Force delay to next write. */ WARN_ON_ONCE (atomic_read (&rdtp->dynticks) &0x1 ); }
@@ -2738,7 +2738,7 @@ void synchronize_sched_expedited(void) s =atomic_long_read (&rsp->expedited_done);if (ULONG_CMP_GE ((ulong )s, (ulong )firstsnap)) {/* ensure test happens before caller kfree */ -smp_mb__before_atomic_inc ();/* ^^^ */ +smp_mb__before_atomic ();/* ^^^ */ atomic_long_inc (&rsp->expedited_workdone1);return ; }
@@ -2756,7 +2756,7 @@ void synchronize_sched_expedited(void) s =atomic_long_read (&rsp->expedited_done);if (ULONG_CMP_GE ((ulong )s, (ulong )firstsnap)) {/* ensure test happens before caller kfree */ -smp_mb__before_atomic_inc ();/* ^^^ */ +smp_mb__before_atomic ();/* ^^^ */ atomic_long_inc (&rsp->expedited_workdone2);return ; }
@@ -2785,7 +2785,7 @@ void synchronize_sched_expedited(void) s =atomic_long_read (&rsp->expedited_done);if (ULONG_CMP_GE ((ulong )s, (ulong )snap)) {/* ensure test happens before caller kfree */ -smp_mb__before_atomic_inc ();/* ^^^ */ +smp_mb__before_atomic ();/* ^^^ */ atomic_long_inc (&rsp->expedited_done_lost);break ; }
kernel/rcu/tree_plugin.h +4 -4
@@ -2514,9 +2514,9 @@ static void rcu_sysidle_enter(struct rcu/* Record start of fully idle period. */ j = jiffies;ACCESS_ONCE (rdtp->dynticks_idle_jiffies) = j; -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&rdtp->dynticks_idle); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();WARN_ON_ONCE (atomic_read (&rdtp->dynticks_idle) &0x1 ); }
@@ -2581,9 +2581,9 @@ static void rcu_sysidle_exit(struct rcu_ }/* Record end of idle period. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&rdtp->dynticks_idle); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks_idle) &0x1 ));/*
kernel/sched/cpupri.c +3 -3
@@ -165,7 +165,7 @@ void cpupri_set(struct cpupri *cp, int c* do a write memory barrier, and then update the count, to * make sure the vector is visible when count is set. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&(vec)->count); do_mb =1 ; }
@@ -185,14 +185,14 @@ void cpupri_set(struct cpupri *cp, int c* the new priority vec. */ if (do_mb) -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * When removing from the vector, we decrement the counter first * do a memory barrier and then clear the mask. */ atomic_dec (&(vec)->count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();cpumask_clear_cpu (cpu, vec->mask); }
kernel/sched/wait.c +1 -1
@@ -394,7 +394,7 @@ EXPORT_SYMBOL(__wake_up_bit);* * In order for this to function properly, as it uses waitqueue_active() * internally, some kind of memory barrier must be done prior to calling -* this. Typically, this will be smp_mb__after_clear_bit(), but in some +* this. Typically, this will be smp_mb__after_atomic(), but in some * cases where bitflags are manipulated non-atomically under a lock, one * may need to use a less regular barrier, such fs/inode.c's smp_mb(), * because spin_unlock() does not guarantee a memory barrier.
mm/backing-dev.c +1 -1
@@ -549,7 +549,7 @@ void clear_bdi_congested(struct backing_ bit = sync ? BDI_sync_congested : BDI_async_congested;if (test_and_clear_bit (bit, &bdi->state))atomic_dec (&nr_bdi_congested[sync]); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (waitqueue_active (wqh))wake_up (wqh); }
mm/filemap.c +2 -2
@@ -609,7 +609,7 @@ void unlock_page(struct page *page) {VM_BUG_ON_PAGE (!PageLocked (page), page);clear_bit_unlock (PG_locked, &page->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_page (page, PG_locked); }EXPORT_SYMBOL (unlock_page);
@@ -626,7 +626,7 @@ void end_page_writeback(struct page *pagif (!test_clear_page_writeback (page))BUG (); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_page (page, PG_writeback); }EXPORT_SYMBOL (end_page_writeback);
net/atm/pppoatm.c +1 -1
@@ -252,7 +252,7 @@ static int pppoatm_may_send(struct pppoa* we need to ensure there's a memory barrier after it. The bit * *must* be set before we do the atomic_inc() on pvcc->inflight. * There's no smp_mb__after_set_bit(), so it's this or abuse -* smp_mb__after_clear_bit(). +* smp_mb__after_atomic(). */ test_and_set_bit (BLOCKED , &pvcc->blocked);
net/bluetooth/hci_event.c +2 -2
@@ -45,7 +45,7 @@ static void hci_cc_inquiry_cancel(structreturn ;clear_bit (HCI_INQUIRY , &hdev->flags); -smp_mb__after_clear_bit ();/* wake_up_bit advises about this barrier */ +smp_mb__after_atomic ();/* wake_up_bit advises about this barrier */ wake_up_bit (&hdev->flags,HCI_INQUIRY );hci_conn_check_pending (hdev);
@@ -1531,7 +1531,7 @@ static void hci_inquiry_complete_evt(strif (!test_and_clear_bit (HCI_INQUIRY , &hdev->flags))return ; -smp_mb__after_clear_bit ();/* wake_up_bit advises about this barrier */ +smp_mb__after_atomic ();/* wake_up_bit advises about this barrier */ wake_up_bit (&hdev->flags,HCI_INQUIRY );if (!test_bit (HCI_MGMT , &hdev->dev_flags))
net/core/dev.c +4 -4
@@ -1323,7 +1323,7 @@ static int __dev_close_many(struct list_* dev->stop() will invoke napi_disable() on all of it's * napi_struct instances on this device. */ -smp_mb__after_clear_bit ();/* Commit netif_running(). */ +smp_mb__after_atomic ();/* Commit netif_running(). */ }dev_deactivate_many (head);
@@ -3343,7 +3343,7 @@ static void net_tx_action(struct softirq root_lock =qdisc_lock (q);if (spin_trylock (root_lock)) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__QDISC_STATE_SCHED, &q->state);qdisc_run (q);
@@ -3353,7 +3353,7 @@ static void net_tx_action(struct softirq &q->state)) { __netif_reschedule(q); } else { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__QDISC_STATE_SCHED, &q->state); }
@@ -4216,7 +4216,7 @@ void __napi_complete(struct napi_structBUG_ON (n->gro_list);list_del (&n->poll_list); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NAPI_STATE_SCHED , &n->state); }EXPORT_SYMBOL (__napi_complete);
net/core/link_watch.c +1 -1
@@ -147,7 +147,7 @@ static void linkwatch_do_dev(struct net_* Make sure the above read is complete since it can be * rewritten as soon as we clear the bit below.
Lines of this file are not shown. View the raw message.
net/ipv4/inetpeer.c +1 -1
Lines of this file are not shown. View the raw message.
net/netfilter/nf_conntrack_core.c +1 -1
Lines of this file are not shown. View the raw message.
net/rds/ib_recv.c +2 -2
Lines of this file are not shown. View the raw message.
net/rds/iw_recv.c +2 -2
Lines of this file are not shown. View the raw message.
net/rds/send.c +3 -3
Lines of this file are not shown. View the raw message.
net/rds/tcp_send.c +1 -1
Lines of this file are not shown. View the raw message.
net/sunrpc/auth.c +1 -1
Lines of this file are not shown. View the raw message.
net/sunrpc/auth_gss/auth_gss.c +1 -1
Lines of this file are not shown. View the raw message.
net/sunrpc/backchannel_rqst.c +2 -2
Lines of this file are not shown. View the raw message.
net/sunrpc/xprt.c +2 -2
Lines of this file are not shown. View the raw message.
net/sunrpc/xprtsock.c +8 -8
Lines of this file are not shown. View the raw message.
net/unix/af_unix.c +1 -1
Lines of this file are not shown. View the raw message.
sound/pci/bt87x.c +2 -2
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [RFC][PATCH 3/5] arch: s/smp_mb__(before|after)_(atomic|clear)_(dec,inc,bit)/smp_mb__\1/g
Quoted 16 lines · Peter Zijlstra
Because atomic ops are implemented the same across an architecture, the current incomplete set of extra barriers:
smp_mb__before_atomic_inc()
smp_mb__after_atomic_inc()
smp_mb__before_atomic_dec()
smp_mb__after_atomic_dec()
smp_mb__before_clear_bit()
smp_mb__after_clear_bit()is both incomplete and superfluous.
It is incomplete because there are far more atomic operations that do not return values -- such as
atomic_add(),set_bit()etc. And it is superfluous because they're all the same anyway.
Quoted 7 lines
Simplify things by reducing the triplicate set into a single set of barriers that is valid for all void atomic ops:
smp_mb__before_atomic()
smp_mb__after_atomic()Signed-off-by Peter Zijlstra <[email protected]>
I very much like the API shrinkage. The RCU changes are good, and I believe that the rest is OK too, though my eyes were going a bit buggy towards the end...
Quoted 4,071 lines
--- Documentation/atomic_ops.txt | 31 ++++-------- Documentation/memory-barriers.txt | 44 ++++------------- arch/alpha/include/asm/atomic.h | 5 - arch/alpha/include/asm/bitops.h | 3 - arch/arc/include/asm/atomic.h | 5 - arch/arc/include/asm/bitops.h | 5 - arch/arm/include/asm/atomic.h | 5 - arch/arm/include/asm/barrier.h | 3 + arch/arm/include/asm/bitops.h | 4 - arch/arm64/include/asm/atomic.h | 5 - arch/arm64/include/asm/barrier.h | 3 + arch/arm64/include/asm/bitops.h | 9 --- arch/avr32/include/asm/atomic.h | 5 - arch/avr32/include/asm/bitops.h | 9 --- arch/blackfin/include/asm/barrier.h | 3 + arch/blackfin/include/asm/bitops.h | 14 ----- arch/c6x/include/asm/bitops.h | 8 --- arch/cris/include/asm/atomic.h | 7 -- arch/cris/include/asm/bitops.h | 9 --- arch/frv/include/asm/atomic.h | 7 -- arch/frv/include/asm/bitops.h | 6 -- arch/hexagon/include/asm/atomic.h | 6 -- arch/hexagon/include/asm/bitops.h | 4 - arch/ia64/include/asm/atomic.h | 6 -- arch/ia64/include/asm/barrier.h | 3 + arch/ia64/include/asm/bitops.h | 6 -- arch/m32r/include/asm/atomic.h | 7 -- arch/m32r/include/asm/bitops.h | 6 -- arch/m68k/include/asm/atomic.h | 8 --- arch/m68k/include/asm/bitops.h | 7 -- arch/metag/include/asm/atomic.h | 6 -- arch/metag/include/asm/barrier.h | 3 + arch/metag/include/asm/bitops.h | 6 -- arch/mips/include/asm/atomic.h | 9 --- arch/mips/include/asm/barrier.h | 3 + arch/mips/include/asm/bitops.h | 11 ---- arch/mips/kernel/irq.c | 4 - arch/mn10300/include/asm/atomic.h | 7 -- arch/mn10300/include/asm/bitops.h | 4 - arch/mn10300/mm/tlb-smp.c | 4 - arch/openrisc/include/asm/bitops.h | 9 --- arch/parisc/include/asm/atomic.h | 6 -- arch/parisc/include/asm/bitops.h | 4 - arch/powerpc/include/asm/atomic.h | 6 -- arch/powerpc/include/asm/barrier.h | 3 + arch/powerpc/include/asm/bitops.h | 6 -- arch/powerpc/kernel/crash.c | 2 arch/s390/include/asm/atomic.h | 6 -- arch/s390/include/asm/barrier.h | 5 + arch/s390/include/asm/bitops.h | 1 arch/score/include/asm/bitops.h | 7 -- arch/sh/include/asm/atomic.h | 6 -- arch/sh/include/asm/bitops.h | 7 -- arch/sparc/include/asm/atomic_32.h | 7 -- arch/sparc/include/asm/atomic_64.h | 7 -- arch/sparc/include/asm/barrier_64.h | 3 + arch/sparc/include/asm/bitops_32.h | 4 - arch/sparc/include/asm/bitops_64.h | 4 - arch/tile/include/asm/atomic_32.h | 10 --- arch/tile/include/asm/atomic_64.h | 6 -- arch/tile/include/asm/barrier.h | 14 +++++ arch/tile/include/asm/bitops.h | 1 arch/tile/include/asm/bitops_32.h | 8 --- arch/tile/include/asm/bitops_64.h | 4 - arch/x86/include/asm/atomic.h | 7 -- arch/x86/include/asm/barrier.h | 4 + arch/x86/include/asm/bitops.h | 6 -- arch/x86/include/asm/sync_bitops.h | 2 arch/x86/kernel/apic/hw_nmi.c | 2 arch/xtensa/include/asm/atomic.h | 7 -- arch/xtensa/include/asm/bitops.h | 4 - block/blk-iopoll.c | 4 - crypto/chainiv.c | 2 drivers/base/power/domain.c | 2 drivers/block/mtip32xx/mtip32xx.c | 4 - drivers/cpuidle/coupled.c | 2 drivers/firewire/ohci.c | 2 drivers/gpu/drm/drm_irq.c | 10 +-- drivers/gpu/drm/i915/i915_irq.c | 2 drivers/md/bcache/bcache.h | 2 drivers/md/bcache/closure.h | 2 drivers/md/dm-bufio.c | 8 +-- drivers/md/dm-snap.c | 4 - drivers/md/dm.c | 2 drivers/md/raid5.c | 2 drivers/media/usb/dvb-usb-v2/dvb_usb_core.c | 6 +- drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c | 6 +- drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c | 34 ++++++------- drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c | 26 +++++----- drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c | 12 ++-- drivers/net/ethernet/broadcom/cnic.c | 8 +-- drivers/net/ethernet/brocade/bna/bnad.c | 6 +- drivers/net/ethernet/chelsio/cxgb/cxgb2.c | 2 drivers/net/ethernet/chelsio/cxgb3/sge.c | 6 +- drivers/net/ethernet/chelsio/cxgb4/sge.c | 2 drivers/net/ethernet/chelsio/cxgb4vf/sge.c | 2 drivers/net/ethernet/intel/i40e/i40e_main.c | 2 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 4 - drivers/net/wireless/ti/wlcore/main.c | 2 drivers/pci/xen-pcifront.c | 4 - drivers/scsi/isci/remote_device.c | 2 drivers/target/loopback/tcm_loop.c | 4 - drivers/target/target_core_alua.c | 26 +++++----- drivers/target/target_core_device.c | 6 +- drivers/target/target_core_iblock.c | 2 drivers/target/target_core_pr.c | 56 +++++++++++----------- drivers/target/target_core_transport.c | 16 +++--- drivers/target/target_core_ua.c | 10 +-- drivers/tty/n_tty.c | 2 drivers/tty/serial/mxs-auart.c | 4 - drivers/usb/gadget/tcm_usb_gadget.c | 4 - drivers/usb/serial/usb_wwan.c | 2 drivers/vhost/scsi.c | 2 drivers/w1/w1_family.c | 4 - drivers/xen/xen-pciback/pciback_ops.c | 4 - fs/btrfs/btrfs_inode.h | 2 fs/btrfs/extent_io.c | 2 fs/btrfs/inode.c | 6 +- fs/buffer.c | 2 fs/ext4/resize.c | 2 fs/gfs2/glock.c | 8 +-- fs/gfs2/glops.c | 2 fs/gfs2/lock_dlm.c | 4 - fs/gfs2/recovery.c | 2 fs/gfs2/sys.c | 4 - fs/jbd2/commit.c | 6 +- fs/nfs/dir.c | 12 ++-- fs/nfs/inode.c | 2 fs/nfs/nfs4filelayoutdev.c | 4 - fs/nfs/nfs4state.c | 4 - fs/nfs/pagelist.c | 6 +- fs/nfs/pnfs.c | 2 fs/nfs/pnfs.h | 2 fs/nfs/write.c | 4 - fs/ubifs/lpt_commit.c | 4 - fs/ubifs/tnc_commit.c | 4 - include/asm-generic/atomic.h | 7 -- include/asm-generic/barrier.h | 8 +++ include/asm-generic/bitops.h | 8 --- include/asm-generic/bitops/atomic.h | 2 include/asm-generic/bitops/lock.h | 2 include/linux/buffer_head.h | 2 include/linux/genhd.h | 2 include/linux/interrupt.h | 8 +-- include/linux/netdevice.h | 2 include/linux/sched.h | 6 -- include/linux/sunrpc/sched.h | 8 +-- include/linux/sunrpc/xprt.h | 8 +-- include/linux/tracehook.h | 2 include/net/ip_vs.h | 4 - kernel/debug/debug_core.c | 4 - kernel/futex.c | 2 kernel/kmod.c | 2 kernel/rcu/tree.c | 22 ++++---- kernel/rcu/tree_plugin.h | 8 +-- kernel/sched/cpupri.c | 6 +- kernel/sched/wait.c | 2 mm/backing-dev.c | 2 mm/filemap.c | 4 - net/atm/pppoatm.c | 2 net/bluetooth/hci_event.c | 4 - net/core/dev.c | 8 +-- net/core/link_watch.c | 2 net/ipv4/inetpeer.c | 2 net/netfilter/nf_conntrack_core.c | 2 net/rds/ib_recv.c | 4 - net/rds/iw_recv.c | 4 - net/rds/send.c | 6 +- net/rds/tcp_send.c | 2 net/sunrpc/auth.c | 2 net/sunrpc/auth_gss/auth_gss.c | 2 net/sunrpc/backchannel_rqst.c | 4 - net/sunrpc/xprt.c | 4 - net/sunrpc/xprtsock.c | 16 +++--- net/unix/af_unix.c | 2 sound/pci/bt87x.c | 4 - 176 files changed, 415 insertions(+), 642 deletions(-)Documentation/atomic_ops.txt +12
-19@@ -285,15 +285,13 @@ If a caller requires memory barrier sema operation which does not return a value, a set of interfaces are defined which accomplish this: - void smp_mb__before_atomic_dec(void); - void smp_mb__after_atomic_dec(void); - void smp_mb__before_atomic_inc(void); - void smp_mb__after_atomic_inc(void); + void smp_mb__before_atomic(void); + void smp_mb__after_atomic(void); -For example, smp_mb__before_atomic_dec() can be used like so: +For example, smp_mb__before_atomic() can be used like so: obj->dead = 1; - smp_mb__before_atomic_dec(); + smp_mb__before_atomic(); atomic_dec(&obj->ref_count); It makes sure that all memory operations preceding the atomic_dec()@@ -302,15 +300,10 @@ operation. In the above example, it gua "1" to obj->dead will be globally visible to other cpus before the atomic counter decrement. -Without the explicit smp_mb__before_atomic_dec() call, the +Without the explicit smp_mb__before_atomic() call, the implementation could legally allow the atomic counter update visible to other cpus before the "obj->dead = 1;" assignment. -The other three interfaces listed are used to provide explicit -ordering with respect to memory operations after an atomic_dec() call -(smp_mb__after_atomic_dec()) and around atomic_inc() calls -(smp_mb__{before,after}_atomic_inc()). - A missing memory barrier in the cases where they are required by the atomic_t implementation above can have disastrous results. Here is an example, which follows a pattern occurring frequently in the Linux@@ -487,12 +480,12 @@ memory operation done by test_and_set_bi Which returns a boolean indicating if bit "nr" is set in the bitmask pointed to by "addr". -If explicit memory barriers are required around clear_bit() (which -does not return a value, and thus does not need to provide memory -barrier semantics), two interfaces are provided: +If explicit memory barriers are required around {set,clear}_bit() (which do +not return a value, and thus does not need to provide memory barrier +semantics), two interfaces are provided: - void smp_mb__before_clear_bit(void); - void smp_mb__after_clear_bit(void); + void smp_mb__before_atomic(void); + void smp_mb__after_atomic(void); They are used as follows, and are akin to their atomic_t operation brothers:@@ -500,13 +493,13 @@ They are used as follows, and are akin t /* All memory operations before this call will * be globally visible before the clear_bit(). */ - smp_mb__before_clear_bit(); + smp_mb__before_atomic(); clear_bit( ... ); /* The clear_bit() will be visible before all * subsequent memory operations. */ - smp_mb__after_clear_bit(); + smp_mb__after_atomic(); There are two special bitops with lock barrier semantics (acquire/release, same as spinlocks). These operate in the same way as their non-_lock/unlockDocumentation/memory-barriers.txt +12
-32@@ -1553,20 +1553,21 @@ CPU from reordering them. insert anything more than a compiler barrier in a UP compilation. - (*) smp_mb__before_atomic_dec(); - (*) smp_mb__after_atomic_dec(); - (*) smp_mb__before_atomic_inc(); - (*) smp_mb__after_atomic_inc(); - - These are for use with atomic add, subtract, increment and decrement - functions that don't return a value, especially when used for reference - counting. These functions do not imply memory barriers. + (*) smp_mb__before_atomic(); + (*) smp_mb__after_atomic(); + + These are for use with atomic (such as add, subtract, increment and + decrement) functions that don't return a value, especially when used for + reference counting. These functions do not imply memory barriers. + + These are also used for atomic bitop functions that do not return a + value (such as set_bit and clear_bit). As an example, consider a piece of code that marks an object as being dead and then decrements the object's reference count: obj->dead = 1; - smp_mb__before_atomic_dec(); + smp_mb__before_atomic(); atomic_dec(&obj->ref_count); This makes sure that the death mark on the object is perceived to be set@@ -1576,27 +1577,6 @@ CPU from reordering them. operations" subsection for information on where to use these. - (*) smp_mb__before_clear_bit(void); - (*) smp_mb__after_clear_bit(void); - - These are for use similar to the atomic inc/dec barriers. These are - typically used for bitwise unlocking operations, so care must be taken as - there are no implicit memory barriers here either. - - Consider implementing an unlock operation of some nature by clearing a - locking bit. The clear_bit() would then need to be barriered like this: - - smp_mb__before_clear_bit(); - clear_bit( ... ); - - This prevents memory operations before the clear leaking to after it. See - the subsection on "Locking Functions" with reference to RELEASE operation - implications. - - See Documentation/atomic_ops.txt for more information. See the "Atomic - operations" subsection for information on where to use these. - - MMIO WRITE BARRIER ------------------@@ -2222,11 +2202,11 @@ barriers, but might be used for implemen change_bit(); With these the appropriate explicit memory barrier should be used if necessary -(smp_mb__before_clear_bit() for instance). +(smp_mb__before_atomic() for instance). The following also do _not_ imply memory barriers, and so may require explicit -memory barriers under some circumstances (smp_mb__before_atomic_dec() for +memory barriers under some circumstances (smp_mb__before_atomic() for instance): atomic_add();arch/alpha/include/asm/atomic.h +0
-5@@ -292,9 +292,4 @@ static inline long atomic64_dec_if_posit#define atomic_dec (v) atomic_sub(1,(v))#define atomic64_dec (v) atomic64_sub(1,(v)) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#endif /* _ALPHA_ATOMIC_H */ arch/alpha/include/asm/bitops.h +0
-3@@ -53,9 +53,6 @@ __set_bit(unsigned long nr, volatile voi *m |=1 << (nr &31 ); } -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -static inline void clear_bit (unsigned long nr,volatile void * addr) {arch/arc/include/asm/atomic.h +0
-5@@ -190,11 +190,6 @@ static inline void atomic_clear_mask(uns#endif /* !CONFIG_ARC_HAS_LLSC */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -/** * __atomic_add_unless - add unless the number is a given value * @v: pointer of type atomic_t arch/arc/include/asm/bitops.h +1
-4@@ -19,6 +19,7 @@#include <linux/types.h> #include <linux/compiler.h> +#include <asm/barrier.h> /* * Hardware assisted read-modify-write using ARC700 LLOCK/SCOND insns. @@ -496,10 +497,6 @@ static inline __attribute__ ((const)) in*/ #define ffz (x) __ffs(~(x)) -/* TODO does this affect uni-processor code */ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#include <asm-generic/bitops/hweight.h> #include <asm-generic/bitops/fls64.h> #include <asm-generic/bitops/sched.h> arch/arm/include/asm/atomic.h +0
-5@@ -211,11 +211,6 @@ static inline int __atomic_add_unless(at#define atomic_add_negative (i,v) (atomic_add_return(i, v) < 0) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() - #ifndef CONFIG_GENERIC_ATOMIC64 typedef struct { long long counter;arch/arm/include/asm/barrier.h +3
-0@@ -79,5 +79,8 @@ do { \#define set_mb (var, value) do { var = value; smp_mb(); } while (0) +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_BARRIER_H */ arch/arm/include/asm/bitops.h +1
-3@@ -25,9 +25,7 @@#include <linux/compiler.h> #include <linux/irqflags.h> - -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() +#include <asm/barrier.h> /* * These functions are the basis of our bit ops. arch/arm64/include/asm/atomic.h +0
-5@@ -154,11 +154,6 @@ static inline int __atomic_add_unless(at#define atomic_add_negative (i,v) (atomic_add_return(i, v) < 0) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -/* * 64-bit atomic operations. */ arch/arm64/include/asm/barrier.h +3
-0@@ -97,6 +97,9 @@ do { \#define set_mb (var, value) do { var = value; smp_mb(); } while (0)#define nop () asm volatile("nop"); +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif /* __ASSEMBLY__ */ #endif /* __ASM_BARRIER_H */ arch/arm64/include/asm/bitops.h +0
-9@@ -17,17 +17,8 @@#define __ASM_BITOPS_H#include <linux/compiler.h> -#include <asm/barrier.h> -/* -* clear_bit may not imply a memory barrier -*/ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif -#ifndef _LINUX_BITOPS_H#error only <linux/bitops.h> can be included directly#endif arch/avr32/include/asm/atomic.h +0
-5@@ -183,9 +183,4 @@ static inline int atomic_sub_if_positive#define atomic_dec_if_positive (v) atomic_sub_if_positive(1, v) -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __ASM_AVR32_ATOMIC_H */ arch/avr32/include/asm/bitops.h +2
-7@@ -13,12 +13,7 @@#endif #include <asm/byteorder.h> - -/* -* clear_bit() doesn't provide any barrier for the compiler -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> /* * set_bit - Atomically set a bit in memory @@ -67,7 +62,7 @@ static inline void set_bit(int nr, volat* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void clear_bit (int nr,volatile void * addr)arch/blackfin/include/asm/barrier.h +3
-0@@ -27,6 +27,9 @@#endif /* !CONFIG_SMP */ +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +#include <asm-generic/barrier.h> #endif /* _BLACKFIN_BARRIER_H */ arch/blackfin/include/asm/bitops.h +2
-12@@ -27,21 +27,17 @@#include <asm-generic/bitops/ext2-atomic.h> +#include <asm/barrier.h> +#ifndef CONFIG_SMP #include <linux/irqflags.h> -/* * clear_bit may not imply a memory barrier */ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif #include <asm-generic/bitops/atomic.h> #include <asm-generic/bitops/non-atomic.h> #else -#include <asm/barrier.h> #include <asm/byteorder.h> /* swab32 */ #include <linux/linkage.h> @@ -101,12 +97,6 @@ static inline int test_and_change_bit(inreturn __raw_bit_test_toggle_asm (a, nr &0x1f ); } -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#define test_bit __skip_test_bit#include <asm-generic/bitops/non-atomic.h> #undef test_bitarch/c6x/include/asm/bitops.h +1
-7@@ -14,14 +14,8 @@#ifdef __KERNEL__#include <linux/bitops.h> -#include <asm/byteorder.h> - -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> /* * We are lucky, DSP is perfect for bitops: do it in 3 cycles arch/cris/include/asm/atomic.h +1
-6@@ -6,6 +6,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #include <arch/atomic.h> /* @@ -151,10 +152,4 @@ static inline int __atomic_add_unless(at return ret; } -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif arch/cris/include/asm/bitops.h +2
-7@@ -21,6 +21,7 @@#include <arch/bitops.h> #include <linux/atomic.h> #include <linux/compiler.h> +#include <asm/barrier.h> /* * set_bit - Atomically set a bit in memory @@ -42,7 +43,7 @@* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ @@ -84,12 +85,6 @@ static inline int test_and_set_bit(int nreturn retval; } -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/** * test_and_clear_bit - Clear a bit and return its old value * @nr: Bit to clear arch/frv/include/asm/atomic.h +1
-6@@ -17,6 +17,7 @@#include <linux/types.h> #include <asm/spr-regs.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #ifdef CONFIG_SMP #error not SMP safe@@ -29,12 +30,6 @@* We do not have SMP systems, so we don't have to deal with that. */ -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#define ATOMIC_INIT (i) { (i) }#define atomic_read (v) (*(volatile int *)&(v)->counter)#define atomic_set (v, i) (((v)->counter) = (i))arch/frv/include/asm/bitops.h +0
-6@@ -25,12 +25,6 @@#include <asm-generic/bitops/ffz.h> -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#ifndef CONFIG_FRV_OUTOFLINE_ATOMIC_OPS static inline unsigned long atomic_test_and_ANDNOT_mask (unsigned long mask,volatile unsigned long *v)arch/hexagon/include/asm/atomic.h +1
-5@@ -24,6 +24,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }#define atomic_set (v, i) ((v)->counter = (i))@@ -163,9 +164,4 @@ static inline int __atomic_add_unless(at#define atomic_inc_return (v) (atomic_add_return(1, v))#define atomic_dec_return (v) (atomic_sub_return(1, v)) -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif arch/hexagon/include/asm/bitops.h +1
-3@@ -25,12 +25,10 @@#include <linux/compiler.h> #include <asm/byteorder.h> #include <asm/atomic.h> +#include <asm/barrier.h> #ifdef __KERNEL__ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/* * The offset calculations for these are based on BITS_PER_LONG == 32 * (i.e. I get to shift by #5-2 (32 bits per long, 4 bytes per access), arch/ia64/include/asm/atomic.h +0
-6@@ -208,10 +208,4 @@ atomic64_add_negative (__s64 i, atomic64#define atomic64_inc (v) atomic64_add(1, (v))#define atomic64_dec (v) atomic64_sub(1, (v)) -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* _ASM_IA64_ATOMIC_H */ arch/ia64/include/asm/barrier.h +3
-0@@ -55,6 +55,9 @@#endif +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +/* * IA64 GCC turns volatile stores into st.rel and volatile loads into ld.acq no * need for asm trickery! arch/ia64/include/asm/bitops.h +2
-4@@ -16,6 +16,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/intrinsics.h> +#include <asm/barrier.h> /** * set_bit - Atomically set a bit in memory @@ -65,9 +66,6 @@ __set_bit (int nr, volatile void *addr) *((__u32 *) addr + (nr >>5 )) |= (1 << (nr &31 )); } -#define smp_mb__before_clear_bit () barrier(); -#define smp_mb__after_clear_bit () barrier(); -/** * clear_bit - Clears a bit in memory * @nr: Bit to clear @@ -75,7 +73,7 @@ __set_bit (int nr, volatile void *addr)* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static __inline__ voidarch/m32r/include/asm/atomic.h +1
-6@@ -13,6 +13,7 @@#include <asm/assembler.h> #include <asm/cmpxchg.h> #include <asm/dcache_clear.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for @@ -308,10 +309,4 @@ static __inline__ void atomic_set_mask(u local_irq_restore(flags); } -/* Atomic operations are already serializing on m32r */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* _ASM_M32R_ATOMIC_H */ arch/m32r/include/asm/bitops.h +2
-4@@ -21,6 +21,7 @@#include <asm/byteorder.h> #include <asm/dcache_clear.h> #include <asm/types.h> +#include <asm/barrier.h> /* * These have to be done with inline assembly: that way the bit-setting @@ -73,7 +74,7 @@ static __inline__ void set_bit(int nr, v* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static __inline__void clear_bit (int nr,volatile void * addr)@@ -103,9 +104,6 @@ static __inline__ void clear_bit(int nr,local_irq_restore (flags); } -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/** * change_bit - Toggle a bit in memory * @nr: Bit to clear arch/m68k/include/asm/atomic.h +1
-7@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/irqflags.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for @@ -209,11 +210,4 @@ static __inline__ int __atomic_add_unles return c; } - -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __ARCH_M68K_ATOMIC __ */ arch/m68k/include/asm/bitops.h +1
-6@@ -13,6 +13,7 @@#endif #include <linux/compiler.h> +#include <asm/barrier.h> /* * Bit access functions vary across the ColdFire and 68k families. @@ -67,12 +68,6 @@ static inline void bfset_mem_set_bit(int#define __set_bit (nr, vaddr) set_bit(nr, vaddr) -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -static inline void bclr_reg_clear_bit (int nr,volatile unsigned long *vaddr) {char *p = (char *)vaddr + (nr ^31 ) /8 ;arch/metag/include/asm/atomic.h +1
-5@@ -4,6 +4,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #if defined(CONFIG_METAG_ATOMICITY_IRQSOFF )/* The simple UP case. */ @@ -39,11 +40,6 @@#define atomic_inc_not_zero (v) atomic_add_unless((v), 1, 0) -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif #define atomic_dec_if_positive (v) atomic_sub_if_positive(1, v)arch/metag/include/asm/barrier.h +3
-0@@ -97,4 +97,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +#endif /* _ASM_METAG_BARRIER_H */ arch/metag/include/asm/bitops.h +0
-6@@ -5,12 +5,6 @@#include <asm/barrier.h> #include <asm/global_lock.h> -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#ifdef CONFIG_SMP /* * These functions are the basis of our bit ops. arch/mips/include/asm/atomic.h +0
-9@@ -761,13 +761,4 @@ static __inline__ int atomic64_add_unles#endif /* CONFIG_64BIT */ -/* -* atomic*_return operations are serializing but not the non-*_return -* versions. -*/ -#define smp_mb__before_atomic_dec () smp_mb__before_llsc() -#define smp_mb__after_atomic_dec () smp_llsc_mb() -#define smp_mb__before_atomic_inc () smp_mb__before_llsc() -#define smp_mb__after_atomic_inc () smp_llsc_mb() -#endif /* _ASM_ATOMIC_H */ arch/mips/include/asm/barrier.h +3
-0@@ -195,4 +195,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () smp_mb__before_llsc() +#define smp_mb__after_atomic () smp_llsc_mb() +#endif /* __ASM_BARRIER_H */ arch/mips/include/asm/bitops.h +2
-9@@ -38,13 +38,6 @@#endif /* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () smp_mb__before_llsc() -#define smp_mb__after_clear_bit () smp_llsc_mb() - - -/* * These are the "slower" versions of the functions and are in bitops.c. * These functions call raw_local_irq_{save,restore}(). */ @@ -120,7 +113,7 @@ static inline void set_bit(unsigned long* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void clear_bit (unsigned long nr,volatile unsigned long *addr)@@ -175,7 +168,7 @@ static inline void clear_bit(unsigned lo*/ static inline void clear_bit_unlock (unsigned long nr,volatile unsigned long *addr) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (nr, addr); }arch/mips/kernel/irq.c +2
-2@@ -62,9 +62,9 @@ void __init alloc_legacy_irqno(void)void free_irqno (unsigned int irq) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (irq, irq_map); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }/* arch/mn10300/include/asm/atomic.h +1
-6@@ -13,6 +13,7 @@#include <asm/irqflags.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #ifndef CONFIG_SMP #include <asm-generic/atomic.h> @@ -234,12 +235,6 @@ static inline void atomic_set_mask(unsig#endif } -/* Atomic operations are already serializing on MN10300??? */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __KERNEL__ */ #endif /* CONFIG_SMP */ #endif /* _ASM_ATOMIC_H */ arch/mn10300/include/asm/bitops.h +1
-3@@ -18,9 +18,7 @@#define __ASM_BITOPS_H#include <asm/cpu-regs.h> - -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> /* * set bit arch/mn10300/mm/tlb-smp.c +2
-2@@ -78,9 +78,9 @@ void smp_flush_tlb(void *unused)else local_flush_tlb_page (flush_mm, flush_va); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();cpumask_clear_cpu (cpu_id, &flush_cpumask); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); out:put_cpu (); }arch/openrisc/include/asm/bitops.h +1
-8@@ -27,14 +27,7 @@#include <linux/irqflags.h> #include <linux/compiler.h> - -/* -* clear_bit may not imply a memory barrier -*/ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif +#include <asm/barrier.h> #include <asm/bitops/__ffs.h> #include <asm-generic/bitops/ffz.h> arch/parisc/include/asm/atomic.h +1
-5@@ -7,6 +7,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for @@ -143,11 +144,6 @@ static __inline__ int __atomic_add_unles#define ATOMIC_INIT (i) { (i) } -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#ifdef CONFIG_64BIT #define ATOMIC64_INIT (i) { (i) }arch/parisc/include/asm/bitops.h +1
-3@@ -8,6 +8,7 @@#include <linux/compiler.h> #include <asm/types.h> /* for BITS_PER_LONG/SHIFT_PER_LONG */ #include <asm/byteorder.h> +#include <asm/barrier.h> #include <linux/atomic.h> /* @@ -19,9 +20,6 @@#define CHOP_SHIFTCOUNT (x) (((unsigned long) (x)) & (BITS_PER_LONG - 1)) -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -/* See http://marc.theaimsgroup.com/?t=108826637900003 for discussion * on use of volatile and __*_bit() (set/clear/change): * *_bit() want use of volatile. arch/powerpc/include/asm/atomic.h +1
-5@@ -8,6 +8,7 @@#ifdef __KERNEL__#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }@@ -270,11 +271,6 @@ static __inline__ int atomic_dec_if_posi }#define atomic_dec_if_positive atomic_dec_if_positive -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#ifdef __powerpc64__#define ATOMIC64_INIT (i) { (i) }arch/powerpc/include/asm/barrier.h +3
-0@@ -84,4 +84,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif /* _ASM_POWERPC_BARRIER_H */ arch/powerpc/include/asm/bitops.h +1
-5@@ -51,11 +51,7 @@#define PPC_BIT (bit) (1UL << PPC_BITLSHIFT(bit))#define PPC_BITMASK (bs, be) ((PPC_BIT(bs) - PPC_BIT(be)) | PPC_BIT(bs)) -/* -* clear_bit doesn't imply a memory barrier -*/ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() +#include <asm/barrier.h> /* Macro for generating the ***_bits() functions */ #define DEFINE_BITOP (fn, op, prefix) \arch/powerpc/kernel/crash.c +1
-1@@ -81,7 +81,7 @@ void crash_ipi_callback(struct pt_regs * }atomic_inc (&cpus_in_crash); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * Starting the kdump boot. arch/s390/include/asm/atomic.h +1
-5@@ -16,6 +16,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }@@ -398,9 +399,4 @@ static inline long long atomic64_dec_if_#define atomic64_dec_and_test (_v) (atomic64_sub_return(1, _v) == 0)#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1, 0) -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#endif /* __ARCH_S390_ATOMIC__ */ arch/s390/include/asm/barrier.h +3
-2@@ -27,8 +27,9 @@#define smp_rmb () rmb()#define smp_wmb () wmb()#define smp_read_barrier_depends () read_barrier_depends() -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() + +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb()#define set_mb (var, value) do { var = value; mb(); } while (0)arch/s390/include/asm/bitops.h +1
-0@@ -47,6 +47,7 @@#include <linux/typecheck.h> #include <linux/compiler.h> +#include <asm/barrier.h> #ifndef CONFIG_64BIT arch/score/include/asm/bitops.h +1
-6@@ -2,12 +2,7 @@#define _ASM_SCORE_BITOPS_H#include <asm/byteorder.h> /* swab32 */ - -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() +#include <asm/barrier.h> #include <asm-generic/bitops.h> #include <asm-generic/bitops/__fls.h> arch/sh/include/asm/atomic.h +1
-5@@ -10,6 +10,7 @@#include <linux/compiler.h> #include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }@@ -62,9 +63,4 @@ static inline int __atomic_add_unless(at return c; } -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -#endif /* __ASM_SH_ATOMIC_H */ arch/sh/include/asm/bitops.h +1
-6@@ -9,6 +9,7 @@/* For __swab32 */ #include <asm/byteorder.h> +#include <asm/barrier.h> #ifdef CONFIG_GUSA_RB #include <asm/bitops-grb.h> @@ -22,12 +23,6 @@#include <asm-generic/bitops/non-atomic.h> #endif -/* -* clear_bit() doesn't provide any barrier for the compiler. -*/ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#ifdef CONFIG_SUPERH32 static inline unsigned long ffz (unsigned long word) {arch/sparc/include/asm/atomic_32.h +1
-6@@ -14,6 +14,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #include <asm-generic/atomic64.h> @@ -52,10 +53,4 @@ extern void atomic_set(atomic_t *, int);#define atomic_dec_and_test (v) (atomic_dec_return(v) == 0)#define atomic_sub_and_test (i, v) (atomic_sub_return(i, v) == 0) -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* !(__ARCH_SPARC_ATOMIC__) */ arch/sparc/include/asm/atomic_64.h +1
-6@@ -9,6 +9,7 @@#include <linux/types.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }#define ATOMIC64_INIT (i) { (i) }@@ -108,10 +109,4 @@ static inline long atomic64_add_unless(aextern long atomic64_dec_if_positive (atomic64_t *v); -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* !(__ARCH_SPARC64_ATOMIC__) */ arch/sparc/include/asm/barrier_64.h +3
-0@@ -68,4 +68,7 @@ do { \ ___p1; \ }) +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +#endif /* !(__SPARC64_BARRIER_H) */ arch/sparc/include/asm/bitops_32.h +1
-3@@ -11,6 +11,7 @@#include <linux/compiler.h> #include <asm/byteorder.h> +#include <asm/barrier.h> #ifdef __KERNEL__@@ -90,9 +91,6 @@ static inline void change_bit(unsigned l#include <asm-generic/bitops/non-atomic.h> -#define smp_mb__before_clear_bit () do { } while(0) -#define smp_mb__after_clear_bit () do { } while(0) -#include <asm-generic/bitops/ffz.h> #include <asm-generic/bitops/__ffs.h> #include <asm-generic/bitops/sched.h> arch/sparc/include/asm/bitops_64.h +1
-3@@ -13,6 +13,7 @@#include <linux/compiler.h> #include <asm/byteorder.h> +#include <asm/barrier.h> extern int test_and_set_bit (unsigned long nr,volatile unsigned long *addr);extern int test_and_clear_bit (unsigned long nr,volatile unsigned long *addr);@@ -23,9 +24,6 @@ extern void change_bit(unsigned long nr,#include <asm-generic/bitops/non-atomic.h> -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -#include <asm-generic/bitops/fls.h> #include <asm-generic/bitops/__fls.h> #include <asm-generic/bitops/fls64.h> arch/tile/include/asm/atomic_32.h +0
-10@@ -169,16 +169,6 @@ static inline void atomic64_set(atomic64#define atomic64_dec_and_test (v) (atomic64_dec_return((v)) == 0)#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1LL, 0LL) -/* -* We need to barrier before modifying the word, since the _atomic_xxx() -* routines just tns the lock and then read/modify/write of the word. -* But after the word is updated, the routine issues an "mf" before returning, -* and since it's a function call, we don't even need a compiler barrier. -*/ -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_dec () do { } while (0) -#define smp_mb__after_atomic_inc () do { } while (0)#endif /* !__ASSEMBLY__ */ arch/tile/include/asm/atomic_64.h +0
-6@@ -105,12 +105,6 @@ static inline long atomic64_add_unless(a#define atomic64_inc_not_zero (v) atomic64_add_unless((v), 1, 0) -/* Atomic dec and inc don't implement barrier, so provide them if needed. */ -#define smp_mb__before_atomic_dec () smp_mb() -#define smp_mb__after_atomic_dec () smp_mb() -#define smp_mb__before_atomic_inc () smp_mb() -#define smp_mb__after_atomic_inc () smp_mb() -/* Define this to indicate that cmpxchg is an efficient operation. */ #define __HAVE_ARCH_CMPXCHGarch/tile/include/asm/barrier.h +14
-0@@ -72,6 +72,20 @@ mb_incoherent(void)#define mb () fast_mb()#define iob () fast_iob() +#ifndef __tilegx__/* 32 bit */ +/* +* We need to barrier before modifying the word, since the _atomic_xxx() +* routines just tns the lock and then read/modify/write of the word. +* But after the word is updated, the routine issues an "mf" before returning, +* and since it's a function call, we don't even need a compiler barrier. +*/ +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () do { } while (0) +#else /* 64 bit */ +#define smp_mb__before_atomic () smp_mb() +#define smp_mb__after_atomic () smp_mb() +#endif +#include <asm-generic/barrier.h> #endif /* !__ASSEMBLY__ */ arch/tile/include/asm/bitops.h +1
-0@@ -17,6 +17,7 @@#define _ASM_TILE_BITOPS_H#include <linux/types.h> +#include <asm/barrier.h> #ifndef _LINUX_BITOPS_H#error only <linux/bitops.h> can be included directlyarch/tile/include/asm/bitops_32.h +2
-6@@ -49,8 +49,8 @@ static inline void set_bit(unsigned nr,* restricted to acting on a single-word quantity. * * clear_bit() may not contain a memory barrier, so if it is used for -* locking purposes, you should call smp_mb__before_clear_bit() and/or -* smp_mb__after_clear_bit() to ensure changes are visible on other cpus. +* locking purposes, you should call smp_mb__before_atomic() and/or +* smp_mb__after_atomic() to ensure changes are visible on other cpus. */ static inline void clear_bit (unsigned nr,volatile unsigned long *addr) {@@ -121,10 +121,6 @@ static inline int test_and_change_bit(un return (_atomic_xor(addr, mask) & mask) != 0; } -/* See discussion at smp_mb__before_atomic_dec() in <asm/atomic_32.h>. */ -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () do {} while (0) -#include <asm-generic/bitops/ext2-atomic.h> #endif /* _ASM_TILE_BITOPS_32_H */ arch/tile/include/asm/bitops_64.h +0
-4@@ -32,10 +32,6 @@ static inline void clear_bit(unsigned nr__insn_fetchand ((void *)(addr + nr /BITS_PER_LONG ), ~mask); } -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() - -static inline void change_bit (unsigned nr,volatile unsigned long *addr) {unsigned long mask = (1UL << (nr %BITS_PER_LONG ));arch/x86/include/asm/atomic.h +1
-6@@ -7,6 +7,7 @@#include <asm/alternative.h> #include <asm/cmpxchg.h> #include <asm/rmwcc.h> +#include <asm/barrier.h> /* * Atomic operations that C can't guarantee us. Useful for @@ -243,12 +244,6 @@ static inline void atomic_or_long(unsign : : "r" ((unsigned)(mask)), "m" (*(addr)) \ : "memory") -/* Atomic operations are already serializing on x86 */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#ifdef CONFIG_X86_32 # include <asm/atomic64_32.h> #else arch/x86/include/asm/barrier.h +4
-0@@ -141,6 +141,10 @@ do { \#endif +/* Atomic operations are already serializing on x86 */ +#define smp_mb__before_atomic () barrier() +#define smp_mb__after_atomic () barrier() +/* * Stop RDTSC speculation. This is needed when you need to use RDTSC * (or get_cycles or vread that possibly accesses the TSC) in a defined arch/x86/include/asm/bitops.h +2
-4@@ -15,6 +15,7 @@#include <linux/compiler.h> #include <asm/alternative.h> #include <asm/rmwcc.h> +#include <asm/barrier.h> #if BITS_PER_LONG ==32 # define _BITOPS_LONG_SHIFT 5@@ -102,7 +103,7 @@ static inline void __set_bit(long nr, vo* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static __always_inline void@@ -156,9 +157,6 @@ static inline void __clear_bit_unlock(lo__clear_bit (nr, addr); } -#define smp_mb__before_clear_bit () barrier() -#define smp_mb__after_clear_bit () barrier() -/** * __change_bit - Toggle a bit in memory * @nr: the bit to change arch/x86/include/asm/sync_bitops.h +1
-1@@ -41,7 +41,7 @@ static inline void sync_set_bit(long nr,* * sync_clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void sync_clear_bit (long nr,volatile unsigned long *addr)arch/x86/kernel/apic/hw_nmi.c +1
-1@@ -57,7 +57,7 @@ void arch_trigger_all_cpu_backtrace(void }clear_bit (0 , &backtrace_flag); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); } static int __kprobesarch/xtensa/include/asm/atomic.h +1
-6@@ -19,6 +19,7 @@#ifdef __KERNEL__#include <asm/processor.h> #include <asm/cmpxchg.h> +#include <asm/barrier.h> #define ATOMIC_INIT (i) { (i) }@@ -387,12 +388,6 @@ static inline void atomic_set_mask(unsig#endif } -/* Atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __KERNEL__ */ #endif /* _XTENSA_ATOMIC_H */ arch/xtensa/include/asm/bitops.h +1
-3@@ -21,9 +21,7 @@#include <asm/processor.h> #include <asm/byteorder.h> - -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() +#include <asm/barrier.h> #include <asm-generic/bitops/non-atomic.h> block/blk-iopoll.c +2
-2@@ -52,7 +52,7 @@ EXPORT_SYMBOL(blk_iopoll_sched);void __blk_iopoll_complete (struct blk_iopoll *iop) {list_del (&iop->list); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit_unlock (IOPOLL_F_SCHED , &iop->state); }EXPORT_SYMBOL (__blk_iopoll_complete);@@ -164,7 +164,7 @@ EXPORT_SYMBOL(blk_iopoll_disable);void blk_iopoll_enable (struct blk_iopoll *iop) {BUG_ON (!test_bit (IOPOLL_F_SCHED , &iop->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit_unlock (IOPOLL_F_SCHED , &iop->state); }EXPORT_SYMBOL (blk_iopoll_enable);crypto/chainiv.c +1
-1@@ -126,7 +126,7 @@ static int async_chainiv_schedule_work(sint err = ctx->err;if (!ctx->queue.qlen) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (CHAINIV_STATE_INUSE , &ctx->state); if (!ctx->queue.qlen ||drivers/base/power/domain.c +1
-1@@ -106,7 +106,7 @@ static bool genpd_sd_counter_dec(structstatic void genpd_sd_counter_inc (struct generic_pm_domain *genpd) {atomic_inc (&genpd->sd_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static void genpd_acquire_lock (struct generic_pm_domain *genpd)drivers/block/mtip32xx/mtip32xx.c +2
-2@@ -224,9 +224,9 @@ static int get_slot(struct mtip_port *po*/ static inline void release_slot (struct mtip_port *port,int tag) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (tag, port->allocated); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }/* drivers/cpuidle/coupled.c +1
-1@@ -159,7 +159,7 @@ void cpuidle_coupled_parallel_barrier(st {int n = dev->coupled->online_count; -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (a);while (atomic_read (a) < n)drivers/firewire/ohci.c +1
-1@@ -3509,7 +3509,7 @@ static int ohci_flush_iso_completions(st }clear_bit_unlock (0 , &ctx->flushing_completions); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }tasklet_enable (&ctx->context.tasklet);drivers/gpu/drm/drm_irq.c +5
-5@@ -156,7 +156,7 @@ static void vblank_disable_and_save(stru*/ if ((vblrc >0 ) && (abs64 (diff_ns) >1000000 )) {atomic_inc (&dev->vblank[crtc].count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }/* Invalidate all timestamps while vblank irq's are off. */ @@ -864,9 +864,9 @@ static void drm_update_vblank_count(struvblanktimestamp (dev, crtc, tslot) = t_vblank; } -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_add (diff, &dev->vblank[crtc].count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }/** @@ -1330,9 +1330,9 @@ bool drm_handle_vblank(struct drm_device/* Increment cooked vblank count. This also atomically commits * the timestamp computed above. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&dev->vblank[crtc].count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }else {DRM_DEBUG ("crtc %d: Redundant vblirq ignored. diff_ns = %d\n" , crtc, (int ) diff_ns);drivers/gpu/drm/i915/i915_irq.c +1
-1@@ -1996,7 +1996,7 @@ static void i915_error_work_func(struct* updates before * the counter increment. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&dev_priv->gpu_error.reset_counter); kobject_uevent_env(&dev->primary->kdev->kobj,drivers/md/bcache/bcache.h +1
-1@@ -841,7 +841,7 @@ static inline bool cached_dev_get(structreturn false;/* Paired with the mb in cached_dev_attach */ -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return true; }drivers/md/bcache/closure.h +1
-1@@ -243,7 +243,7 @@ static inline void set_closure_fn(struct cl->fn = fn; cl->wq = wq;/* between atomic_dec() in closure_put() */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic (); }static inline void closure_queue (struct closure *cl)drivers/md/dm-bufio.c +4
-4@@ -607,9 +607,9 @@ static void write_endio(struct bio *bio,BUG_ON (!test_bit (B_WRITING , &b->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (B_WRITING , &b->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&b->state,B_WRITING ); }@@ -997,9 +997,9 @@ static void read_endio(struct bio *bio,BUG_ON (!test_bit (B_READING , &b->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (B_READING , &b->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&b->state,B_READING ); }drivers/md/dm-snap.c +2
-2@@ -642,7 +642,7 @@ static void free_pending_exception(strucstruct dm_snapshot *s = pe->snap;mempool_free (pe, s->pending_pool); -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&s->pending_exceptions_count); }@@ -783,7 +783,7 @@ static int init_hash_tables(struct dm_snstatic void merge_shutdown (struct dm_snapshot *s) {clear_bit_unlock (RUNNING_MERGE , &s->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&s->state_bits,RUNNING_MERGE ); }drivers/md/dm.c +1
-1@@ -2451,7 +2451,7 @@ static void dm_wq_work(struct work_strucstatic void dm_queue_flush (struct mapped_device *md) {clear_bit (DMF_BLOCK_IO_FOR_SUSPEND , &md->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();queue_work (md->wq, &md->work); }drivers/md/raid5.c +1
-1@@ -4406,7 +4406,7 @@ static void raid5_unplug(struct blk_plug* STRIPE_ON_UNPLUG_LIST clear but the stripe * is still in our list */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (STRIPE_ON_UNPLUG_LIST , &sh->state);/* * STRIPE_ON_RELEASE_LIST could be set here. In that drivers/media/usb/dvb-usb-v2/dvb_usb_core.c +3
-3@@ -399,7 +399,7 @@ static int dvb_usb_stop_feed(struct dvb_/* clear 'streaming' status bit */ clear_bit (ADAP_STREAMING , &adap->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&adap->state_bits,ADAP_STREAMING ); skip_feed_stop:@@ -550,7 +550,7 @@ static int dvb_usb_fe_init(struct dvb_fr err:if (!adap->suspend_resume_active) {clear_bit (ADAP_INIT , &adap->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&adap->state_bits,ADAP_INIT ); }@@ -591,7 +591,7 @@ static int dvb_usb_fe_sleep(struct dvb_fif (!adap->suspend_resume_active) { adap->active_fe =-1 ;clear_bit (ADAP_SLEEP , &adap->state_bits); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&adap->state_bits,ADAP_SLEEP ); }drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +3
-3@@ -2779,7 +2779,7 @@ int bnx2x_nic_load(struct bnx2x *bp, intcase LOAD_OPEN :netif_tx_start_all_queues (bp->dev); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();break ;case LOAD_DIAG :@@ -4773,9 +4773,9 @@ void bnx2x_tx_timeout(struct net_devicebnx2x_panic ();#endif -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_TX_TIMEOUT , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* This allows the netif to be shutdown gracefully before resetting */ schedule_delayed_work (&bp->sp_rtnl_task,0 );drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +17
-17@@ -1837,10 +1837,10 @@ void bnx2x_sp_event(struct bnx2x_fastpat/* SRIOV: reschedule any 'in_progress' operations */ bnx2x_iov_sp_event (bp, cid, true); -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&bp->cq_spq_left);/* push the change in bp->spq_left and towards the memory */ -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();DP (BNX2X_MSG_SP ,"bp->cq_spq_left %x\n" ,atomic_read (&bp->cq_spq_left));@@ -1855,11 +1855,11 @@ void bnx2x_sp_event(struct bnx2x_fastpat* sp_state is cleared, and this order prevents * races */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_AFEX_PENDING_VIFSET_MCP_ACK , &bp->sp_state);wmb ();clear_bit (BNX2X_AFEX_FCOE_Q_UPDATE_PENDING , &bp->sp_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* schedule the sp task as mcp ack is required */ bnx2x_schedule_sp_task (bp);@@ -3878,9 +3878,9 @@ static void bnx2x_fan_failure(struct bnx* This is due to some boards consuming sufficient power when driver is * up to overheat if fan fails. */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_FAN_FAILURE , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); }@@ -5137,9 +5137,9 @@ static void bnx2x_after_function_update(__clear_bit (RAMROD_COMP_WAIT , &queue_params.ramrod_flags);/* mark latest Q bit */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_AFEX_FCOE_Q_UPDATE_PENDING , &bp->sp_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* send Q update ramrod for FCoE Q */ rc =bnx2x_queue_state_change (bp, &queue_params);@@ -5282,10 +5282,10 @@ static void bnx2x_eq_int(struct bnx2x *b* sp_rtnl task as all Queue SP operations * should run under rtnl_lock. */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_AFEX_F_UPDATE , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); }@@ -5368,7 +5368,7 @@ static void bnx2x_eq_int(struct bnx2x *b spqe_cnt++; }/* for */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_add (spqe_cnt, &bp->eq_spq_left); bp->eq_cons = sw_cons;@@ -12065,9 +12065,9 @@ static void bnx2x_set_rx_mode(struct net }else {/* Schedule an SP task to handle rest of change */ DP (NETIF_MSG_IFUP ,"Scheduling an Rx mode change\n" ); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_RX_MODE , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); } }@@ -12101,10 +12101,10 @@ void bnx2x_set_rx_mode_inner(struct bnx2/* configuring mcast to a vf involves sleeping (when we * wait for the pf's response). */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_VFPF_MCAST , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); } }@@ -13723,9 +13723,9 @@ static int bnx2x_drv_ctl(struct net_devicase DRV_CTL_RET_L2_SPQ_CREDIT_CMD : {int count = ctl->data.credit.credit_count; -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_add (count, &bp->cq_spq_left); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();break ; }case DRV_CTL_ULP_REGISTER_CMD : {drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c +13
-13@@ -258,16 +258,16 @@ static bool bnx2x_raw_check_pending(strustatic void bnx2x_raw_clear_pending (struct bnx2x_raw_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (o->state, o->pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static void bnx2x_raw_set_pending (struct bnx2x_raw_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (o->state, o->pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }/** @@ -2131,7 +2131,7 @@ static int bnx2x_set_rx_mode_e1x(struct/* The operation is completed */ clear_bit (p->state, p->pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return 0 ; }@@ -3576,16 +3576,16 @@ int bnx2x_config_mcast(struct bnx2x *bp,static void bnx2x_mcast_clear_sched (struct bnx2x_mcast_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (o->sched_state, o->raw.pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static void bnx2x_mcast_set_sched (struct bnx2x_mcast_obj *o) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (o->sched_state, o->raw.pstate); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static bool bnx2x_mcast_check_sched (struct bnx2x_mcast_obj *o)@@ -4210,7 +4210,7 @@ int bnx2x_queue_state_change(struct bnx2if (rc) { o->next_state =BNX2X_Q_STATE_MAX ;clear_bit (pending_bit, pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return rc; }@@ -4298,7 +4298,7 @@ static int bnx2x_queue_comp_cmd(struct bwmb ();clear_bit (cmd, &o->pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return 0 ; }@@ -5242,7 +5242,7 @@ static inline int bnx2x_func_state_changwmb ();clear_bit (cmd, &o->pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return 0 ; }@@ -5877,7 +5877,7 @@ int bnx2x_func_state_change(struct bnx2xif (rc) { o->next_state =BNX2X_F_STATE_MAX ;clear_bit (cmd, pending); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return rc; }drivers/net/ethernet/broadcom/bnx2x/bnx2x_sriov.c +6
-6@@ -971,10 +971,10 @@ static void bnx2x_vfop_qsetup(struct bnx op_done:case BNX2X_VFOP_QSETUP_DONE : vf->cfg_flags |=VF_CFG_VLAN ; -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_HYPERVISOR_VLAN , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 );bnx2x_vfop_end (bp, vf, vfop);return ;@@ -2354,9 +2354,9 @@ staticvoid bnx2x_vf_handle_filters_eqe (struct bnx2x *bp,struct bnx2x_virtf *vf) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNX2X_FILTER_RX_MODE_PENDING , &vf->filter_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }int bnx2x_iov_eq_sp_event (struct bnx2x *bp,union event_ring_elem *elem)@@ -3737,10 +3737,10 @@ void bnx2x_timer_sriov(struct bnx2x *bp)/* if channel is down we need to self destruct */ if (bp->old_bulletin.valid_bitmap &1 <<CHANNEL_DOWN ) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();set_bit (BNX2X_SP_RTNL_VFPF_CHANNEL_DOWN , &bp->sp_rtnl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_delayed_work (&bp->sp_rtnl_task,0 ); } }drivers/net/ethernet/broadcom/cnic.c +4
-4@@ -436,7 +436,7 @@ static int cnic_offld_prep(struct cnic_sstatic int cnic_close_prep (struct cnic_sock *csk) {clear_bit (SK_F_CONNECT_START , &csk->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (test_and_clear_bit (SK_F_OFFLD_COMPLETE , &csk->flags)) {while (test_and_set_bit (SK_F_OFFLD_SCHED , &csk->flags))@@ -450,7 +450,7 @@ static int cnic_close_prep(struct cnic_sstatic int cnic_abort_prep (struct cnic_sock *csk) {clear_bit (SK_F_CONNECT_START , &csk->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();while (test_and_set_bit (SK_F_OFFLD_SCHED , &csk->flags))msleep (1 );@@ -3645,7 +3645,7 @@ static int cnic_cm_destroy(struct cnic_scsk_hold (csk);clear_bit (SK_F_INUSE , &csk->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();while (atomic_read (&csk->ref_count) !=1 )msleep (1 );cnic_cm_cleanup (csk);@@ -4025,7 +4025,7 @@ static void cnic_cm_process_kcqe(struct L4_KCQE_COMPLETION_STATUS_PARITY_ERROR) set_bit(SK_F_HW_ERR, &csk->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (SK_F_OFFLD_SCHED , &csk->flags);cnic_cm_upcall (cp, csk, opcode);break ;drivers/net/ethernet/brocade/bna/bnad.c +3
-3@@ -249,7 +249,7 @@ bnad_tx_complete(struct bnad *bnad, struif (likely (test_bit (BNAD_TXQ_TX_STARTED , &tcb->flags)))bna_ib_ack (tcb->i_dbell, sent); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNAD_TXQ_FREE_SENT , &tcb->flags);return sent;@@ -1125,7 +1125,7 @@ bnad_tx_cleanup(struct delayed_work *worbnad_txq_cleanup (bnad, tcb); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNAD_TXQ_FREE_SENT , &tcb->flags); }@@ -2999,7 +2999,7 @@ bnad_start_xmit(struct sk_buff *skb, str sent =bnad_txcmpl_process (bnad, tcb);if (likely (test_bit (BNAD_TXQ_TX_STARTED , &tcb->flags)))bna_ib_ack (tcb->i_dbell, sent); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BNAD_TXQ_FREE_SENT , &tcb->flags); }else {netif_stop_queue (netdev);drivers/net/ethernet/chelsio/cxgb/cxgb2.c +1
-1@@ -281,7 +281,7 @@ static int cxgb_close(struct net_deviceif (adapter->params.stats_update_period && !(adapter->open_device_map &PORT_MASK )) {/* Stop statistics accumulation. */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();spin_lock (&adapter->work_lock);/* sync with update task */ spin_unlock (&adapter->work_lock);cancel_mac_stats_update (adapter);drivers/net/ethernet/chelsio/cxgb3/sge.c +3
-3@@ -1379,7 +1379,7 @@ static inline int check_desc_avail(strucstruct sge_qset *qs =txq_to_qset (q, qid);set_bit (qid, &qs->txq_stopped); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (should_restart_tx (q) &&test_and_clear_bit (qid, &qs->txq_stopped))@@ -1492,7 +1492,7 @@ static void restart_ctrlq(unsigned longif (!skb_queue_empty (&q->sendq)) {set_bit (TXQ_CTRL , &qs->txq_stopped); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (should_restart_tx (q) &&test_and_clear_bit (TXQ_CTRL , &qs->txq_stopped))@@ -1697,7 +1697,7 @@ again: reclaim_completed_tx(adap, q, TX_if (unlikely (q->size - q->in_use < ndesc)) {set_bit (TXQ_OFLD , &qs->txq_stopped); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (should_restart_tx (q) &&test_and_clear_bit (TXQ_OFLD , &qs->txq_stopped))drivers/net/ethernet/chelsio/cxgb4/sge.c +1
-1@@ -2003,7 +2003,7 @@ static void sge_rx_timer_cb(unsigned lonstruct sge_fl *fl = s->egr_map[id];clear_bit (id, s->starving_fl); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (fl_starving (fl)) { rxq = container_of(fl, struct sge_eth_rxq, fl);drivers/net/ethernet/chelsio/cxgb4vf/sge.c +1
-1@@ -1951,7 +1951,7 @@ static void sge_rx_timer_cb(unsigned lonstruct sge_fl *fl = s->egr_map[id];clear_bit (id, s->starving_fl); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* * Since we are accessing fl without a lock there's a drivers/net/ethernet/intel/i40e/i40e_main.c +1
-1@@ -4576,7 +4576,7 @@ static void i40e_service_event_complete(BUG_ON (!test_bit (__I40E_SERVICE_SCHED, &pf->state));/* flush memory to make sure state is correct before next watchog */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__I40E_SERVICE_SCHED, &pf->state); }drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2
-2@@ -318,7 +318,7 @@ static void ixgbe_service_event_completeBUG_ON (!test_bit (__IXGBE_SERVICE_SCHED, &adapter->state));/* flush memory to make sure state is correct before next watchdog */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__IXGBE_SERVICE_SCHED, &adapter->state); }@@ -4607,7 +4607,7 @@ static void ixgbe_up_complete(struct ixgif (hw->mac.ops.enable_tx_laser) hw->mac.ops.enable_tx_laser (hw); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__IXGBE_DOWN, &adapter->state);ixgbe_napi_enable_all (adapter);drivers/net/wireless/ti/wlcore/main.c +1
-1@@ -547,7 +547,7 @@ static int wlcore_irq_locked(struct wl12* wl1271_ps_elp_wakeup cannot be called concurrently. */ clear_bit (WL1271_FLAG_IRQ_RUNNING , &wl->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); ret =wlcore_fw_status (wl, wl->fw_status_1, wl->fw_status_2);if (ret <0 )drivers/pci/xen-pcifront.c +2
-2@@ -662,9 +662,9 @@ static void pcifront_do_aer(struct work_notify_remote_via_evtchn (pdev->evtchn);/*in case of we lost an aer request in four lines time_window*/ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (_PDEVB_op_active, &pdev->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();schedule_pcifront_aer_op (pdev);drivers/scsi/isci/remote_device.c +1
-1@@ -1541,7 +1541,7 @@ void isci_remote_device_release(struct kclear_bit (IDEV_STOP_PENDING , &idev->flags);clear_bit (IDEV_IO_READY , &idev->flags);clear_bit (IDEV_GONE , &idev->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (IDEV_ALLOCATED , &idev->flags);wake_up (&ihost->eventq); }drivers/target/loopback/tcm_loop.c +2
-2@@ -942,7 +942,7 @@ static int tcm_loop_port_link(struct tcm_loop_hba *tl_hba = tl_tpg->tl_hba;atomic_inc (&tl_tpg->tl_tpg_port_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * Add Linux/SCSI struct scsi_device by HCTL */ @@ -977,7 +977,7 @@ static void tcm_loop_port_unlink(scsi_device_put (sd);atomic_dec (&tl_tpg->tl_tpg_port_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();pr_debug ("TCM_Loop_ConfigFS: Port Unlink Successful\n" ); }drivers/target/target_core_alua.c +13
-13@@ -393,7 +393,7 @@ target_emulate_set_target_port_groups(stcontinue ;atomic_inc (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);@@ -404,7 +404,7 @@ target_emulate_set_target_port_groups(stspin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_dec (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();break ; }spin_unlock (&dev->t10_alua.tg_pt_gps_lock);@@ -997,7 +997,7 @@ static void core_alua_do_transition_tg_p* TARGET PORT GROUPS command */ atomic_inc (&mem->tg_pt_gp_mem_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&tg_pt_gp->tg_pt_gp_lock);spin_lock_bh (&port->sep_alua_lock);@@ -1027,7 +1027,7 @@ static void core_alua_do_transition_tg_pspin_lock (&tg_pt_gp->tg_pt_gp_lock);atomic_dec (&mem->tg_pt_gp_mem_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&tg_pt_gp->tg_pt_gp_lock);/* @@ -1061,7 +1061,7 @@ static void core_alua_do_transition_tg_p core_alua_dump_state(tg_pt_gp->tg_pt_gp_alua_pending_state));spin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_dec (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);if (tg_pt_gp->tg_pt_gp_transition_complete)@@ -1123,7 +1123,7 @@ static int core_alua_do_transition_tg_pt*/ spin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_inc (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);if (!explicit && tg_pt_gp->tg_pt_gp_implicit_trans_secs) {@@ -1166,7 +1166,7 @@ int core_alua_do_port_transition(spin_lock (&local_lu_gp_mem->lu_gp_mem_lock); lu_gp = local_lu_gp_mem->lu_gp;atomic_inc (&lu_gp->lu_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&local_lu_gp_mem->lu_gp_mem_lock);/* * For storage objects that are members of the 'default_lu_gp', @@ -1183,7 +1183,7 @@ int core_alua_do_port_transition( rc =core_alua_do_transition_tg_pt (l_tg_pt_gp, new_state, explicit);atomic_dec (&lu_gp->lu_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();return rc; }/* @@ -1197,7 +1197,7 @@ int core_alua_do_port_transition( dev = lu_gp_mem->lu_gp_mem_dev;atomic_inc (&lu_gp_mem->lu_gp_mem_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&lu_gp->lu_gp_lock);spin_lock (&dev->t10_alua.tg_pt_gps_lock);@@ -1226,7 +1226,7 @@ int core_alua_do_port_transition( tg_pt_gp->tg_pt_gp_alua_nacl =NULL ; }atomic_inc (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->t10_alua.tg_pt_gps_lock);/* * core_alua_do_transition_tg_pt() will always return @@ -1237,7 +1237,7 @@ int core_alua_do_port_transition(spin_lock (&dev->t10_alua.tg_pt_gps_lock);atomic_dec (&tg_pt_gp->tg_pt_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();if (rc)break ; }@@ -1245,7 +1245,7 @@ int core_alua_do_port_transition(spin_lock (&lu_gp->lu_gp_lock);atomic_dec (&lu_gp_mem->lu_gp_mem_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&lu_gp->lu_gp_lock);@@ -1259,7 +1259,7 @@ int core_alua_do_port_transition( }atomic_dec (&lu_gp->lu_gp_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();return rc; }drivers/target/target_core_device.c +3
-3@@ -225,7 +225,7 @@ struct se_dev_entry *core_get_se_deve_frcontinue ;atomic_inc (&deve->pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock_irq (&nacl->device_list_lock);return deve;@@ -1392,7 +1392,7 @@ int core_dev_add_initiator_node_lun_acl(spin_lock (&lun->lun_acl_lock);list_add_tail (&lacl->lacl_list, &lun->lun_acl_list);atomic_inc (&lun->lun_acl_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&lun->lun_acl_lock); pr_debug("%s_TPG[%hu]_LUN[%u->%u] - Added %s ACL for "@@ -1426,7 +1426,7 @@ int core_dev_del_initiator_node_lun_acl(spin_lock (&lun->lun_acl_lock);list_del (&lacl->lacl_list);atomic_dec (&lun->lun_acl_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();spin_unlock (&lun->lun_acl_lock); core_disable_device_list_for_node(lun, NULL, lacl->mapped_lun,drivers/target/target_core_iblock.c +1
-1@@ -324,7 +324,7 @@ static void iblock_bio_done(struct bio ** Bump the ib_bio_err_cnt and release bio. */ atomic_inc (&ibr->ib_bio_err_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }bio_put (bio);drivers/target/target_core_pr.c +28
-28@@ -675,7 +675,7 @@ static struct t10_pr_registration *__corspin_lock (&dev->se_port_lock);list_for_each_entry_safe (port, port_tmp, &dev->dev_sep_list, sep_list) {atomic_inc (&port->sep_tg_pt_ref_cnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->se_port_lock);spin_lock_bh (&port->sep_alua_lock);@@ -710,7 +710,7 @@ static struct t10_pr_registration *__corcontinue ;atomic_inc (&deve_tmp->pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock_bh (&port->sep_alua_lock);/* * Grab a configfs group dependency that is released @@ -723,9 +723,9 @@ static struct t10_pr_registration *__corpr_err ("core_scsi3_lunacl_depend" "_item() failed\n" );atomic_dec (&port->sep_tg_pt_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();atomic_dec (&deve_tmp->pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); goto out; }/* @@ -740,9 +740,9 @@ static struct t10_pr_registration *__cor sa_res_key, all_tg_pt, aptpl);if (!pr_reg_atp) {atomic_dec (&port->sep_tg_pt_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();atomic_dec (&deve_tmp->pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();core_scsi3_lunacl_undepend_item (deve_tmp); goto out; }@@ -755,7 +755,7 @@ static struct t10_pr_registration *__corspin_lock (&dev->se_port_lock);atomic_dec (&port->sep_tg_pt_ref_cnt); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&dev->se_port_lock);@@ -1110,7 +1110,7 @@ static struct t10_pr_registration *__corcontinue ; }atomic_inc (&pr_reg->pr_res_holders); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&pr_tmpl->registration_lock);return pr_reg; }@@ -1125,7 +1125,7 @@ static struct t10_pr_registration *__corcontinue ;atomic_inc (&pr_reg->pr_res_holders); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&pr_tmpl->registration_lock);return pr_reg; }@@ -1155,7 +1155,7 @@ static struct t10_pr_registration *core_static void core_scsi3_put_pr_reg (struct t10_pr_registration *pr_reg) {atomic_dec (&pr_reg->pr_res_holders); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); } static int core_scsi3_check_implicit_release(@@ -1349,7 +1349,7 @@ static void core_scsi3_tpg_undepend_item &tpg->tpg_group.cg_item);atomic_dec (&tpg->tpg_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }static int core_scsi3_nodeacl_depend_item (struct se_node_acl *nacl)@@ -1369,7 +1369,7 @@ static void core_scsi3_nodeacl_undepend_if (nacl->dynamic_node_acl) {atomic_dec (&nacl->acl_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();return ; }@@ -1377,7 +1377,7 @@ static void core_scsi3_nodeacl_undepend_ &nacl->acl_group.cg_item);atomic_dec (&nacl->acl_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }static int core_scsi3_lunacl_depend_item (struct se_dev_entry *se_deve)@@ -1408,7 +1408,7 @@ static void core_scsi3_lunacl_undepend_i*/ if (!lun_acl) {atomic_dec (&se_deve->pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();return ; } nacl = lun_acl->se_lun_nacl;@@ -1418,7 +1418,7 @@ static void core_scsi3_lunacl_undepend_i &lun_acl->se_lun_group.cg_item);atomic_dec (&se_deve->pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); } static sense_reason_t@@ -1552,14 +1552,14 @@ core_scsi3_decode_spec_i_port(continue ;atomic_inc (&tmp_tpg->tpg_pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->se_port_lock);if (core_scsi3_tpg_depend_item (tmp_tpg)) {pr_err (" core_scsi3_tpg_depend_item()" " for tmp_tpg\n" );atomic_dec (&tmp_tpg->tpg_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); ret =TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; goto out_unmap; }@@ -1573,7 +1573,7 @@ core_scsi3_decode_spec_i_port( tmp_tpg, i_str);if (dest_node_acl) {atomic_inc (&dest_node_acl->acl_pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }spin_unlock_irq (&tmp_tpg->acl_node_lock);@@ -1587,7 +1587,7 @@ core_scsi3_decode_spec_i_port(pr_err ("configfs_depend_item() failed" " for dest_node_acl->acl_group\n" );atomic_dec (&dest_node_acl->acl_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();core_scsi3_tpg_undepend_item (tmp_tpg); ret =TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; goto out_unmap;@@ -1647,7 +1647,7 @@ core_scsi3_decode_spec_i_port(pr_err ("core_scsi3_lunacl_depend_item()" " failed\n" );atomic_dec (&dest_se_deve->pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();core_scsi3_nodeacl_undepend_item (dest_node_acl);core_scsi3_tpg_undepend_item (dest_tpg); ret =TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ;@@ -3165,14 +3165,14 @@ core_scsi3_emulate_pro_register_and_movecontinue ;atomic_inc (&dest_se_tpg->tpg_pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&dev->se_port_lock);if (core_scsi3_tpg_depend_item (dest_se_tpg)) {pr_err ("core_scsi3_tpg_depend_item() failed" " for dest_se_tpg\n" );atomic_dec (&dest_se_tpg->tpg_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); ret =TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; goto out_put_pr_reg; }@@ -3270,7 +3270,7 @@ core_scsi3_emulate_pro_register_and_move initiator_str);if (dest_node_acl) {atomic_inc (&dest_node_acl->acl_pr_ref_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }spin_unlock_irq (&dest_se_tpg->acl_node_lock);@@ -3286,7 +3286,7 @@ core_scsi3_emulate_pro_register_and_movepr_err ("core_scsi3_nodeacl_depend_item() for" " dest_node_acl\n" );atomic_dec (&dest_node_acl->acl_pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); dest_node_acl =NULL ; ret =TCM_INVALID_PARAMETER_LIST ; goto out;@@ -3311,7 +3311,7 @@ core_scsi3_emulate_pro_register_and_moveif (core_scsi3_lunacl_depend_item (dest_se_deve)) {pr_err ("core_scsi3_lunacl_depend_item() failed\n" );atomic_dec (&dest_se_deve->pr_ref_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); dest_se_deve =NULL ; ret =TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE ; goto out;@@ -3877,7 +3877,7 @@ core_scsi3_pri_read_full_status(struct s add_desc_len =0 ;atomic_inc (&pr_reg->pr_res_holders); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock (&pr_tmpl->registration_lock);/* * Determine expected length of $FABRIC_MOD specific @@ -3891,7 +3891,7 @@ core_scsi3_pri_read_full_status(struct s " out of buffer: %d\n", cmd->data_length);spin_lock (&pr_tmpl->registration_lock);atomic_dec (&pr_reg->pr_res_holders); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();break ; }/* @@ -3953,7 +3953,7 @@ core_scsi3_pri_read_full_status(struct sspin_lock (&pr_tmpl->registration_lock);atomic_dec (&pr_reg->pr_res_holders); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic ();/* * Set the ADDITIONAL DESCRIPTOR LENGTH */ drivers/target/target_core_transport.c +8
-8@@ -728,7 +728,7 @@ void target_qf_do_work(struct work_struclist_for_each_entry_safe (cmd, cmd_tmp, &qf_cmd_list, se_qf_node) {list_del (&cmd->se_qf_node);atomic_dec (&dev->dev_qf_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); pr_debug("Processing %s cmd: %p QUEUE_FULL in work queue" " context: %s\n", cmd->se_tfo->get_fabric_name(), cmd,@@ -1140,7 +1140,7 @@ transport_check_alloc_task_attr(struct s* Dormant to Active status. */ cmd->se_ordered_id =atomic_inc_return (&dev->dev_ordered_id); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();pr_debug ("Allocated se_ordered_id: %u for Task Attr: 0x%02x on %s\n" , cmd->se_ordered_id, cmd->sam_task_attr, dev->transport->name);@@ -1692,7 +1692,7 @@ static bool target_handle_task_attr(strureturn false;case MSG_ORDERED_TAG :atomic_inc (&dev->dev_ordered_sync); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); pr_debug("Added ORDERED for CDB: 0x%02x to ordered list, " " se_ordered_id: %u\n",@@ -1710,7 +1710,7 @@ static bool target_handle_task_attr(stru* For SIMPLE and UNTAGGED Task Attribute commands */ atomic_inc (&dev->simple_cmds); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();break ; }@@ -1806,7 +1806,7 @@ static void transport_complete_task_attrif (cmd->sam_task_attr ==MSG_SIMPLE_TAG ) {atomic_dec (&dev->simple_cmds); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); dev->dev_cur_ordered_id++; pr_debug("Incremented dev->dev_cur_ordered_id: %u for" " SIMPLE: %u\n", dev->dev_cur_ordered_id,@@ -1818,7 +1818,7 @@ static void transport_complete_task_attr cmd->se_ordered_id); }else if (cmd->sam_task_attr ==MSG_ORDERED_TAG ) {atomic_dec (&dev->dev_ordered_sync); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); dev->dev_cur_ordered_id++; pr_debug("Incremented dev_cur_ordered_id: %u for ORDERED:"@@ -1877,7 +1877,7 @@ static void transport_handle_queue_full(spin_lock_irq (&dev->qf_cmd_lock);list_add_tail (&cmd->se_qf_node, &cmd->se_dev->qf_cmd_list);atomic_inc (&dev->dev_qf_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();spin_unlock_irq (&cmd->se_dev->qf_cmd_lock);schedule_work (&cmd->se_dev->qf_work_queue);@@ -2805,7 +2805,7 @@ void transport_send_task_abort(struct seif (cmd->data_direction ==DMA_TO_DEVICE ) {if (cmd->se_tfo->write_pending_status (cmd) !=0 ) { cmd->transport_state |=CMD_T_ABORTED ; -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return ; } }drivers/target/target_core_ua.c +5
-5@@ -162,7 +162,7 @@ int core_scsi3_ua_allocate(spin_unlock_irq (&nacl->device_list_lock);atomic_inc (&deve->ua_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return 0 ; }list_add_tail (&ua->ua_nacl_list, &deve->ua_list);@@ -175,7 +175,7 @@ int core_scsi3_ua_allocate( asc, ascq);atomic_inc (&deve->ua_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return 0 ; }@@ -190,7 +190,7 @@ void core_scsi3_ua_release_all(kmem_cache_free (se_ua_cache, ua);atomic_dec (&deve->ua_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&deve->ua_lock); }@@ -251,7 +251,7 @@ void core_scsi3_ua_for_check_condition(kmem_cache_free (se_ua_cache, ua);atomic_dec (&deve->ua_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&deve->ua_lock);spin_unlock_irq (&nacl->device_list_lock);@@ -310,7 +310,7 @@ int core_scsi3_ua_clear_for_request_senskmem_cache_free (se_ua_cache, ua);atomic_dec (&deve->ua_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }spin_unlock (&deve->ua_lock);spin_unlock_irq (&nacl->device_list_lock);drivers/tty/n_tty.c +1
-1@@ -2042,7 +2042,7 @@ static int canon_copy_from_read_buf(struif (found)clear_bit (eol, ldata->read_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); ldata->read_tail += c;if (found) {drivers/tty/serial/mxs-auart.c +2
-2@@ -200,7 +200,7 @@ static void dma_tx_callback(void *param)/* clear the bit used to serialize the DMA tx. */ clear_bit (MXS_AUART_DMA_TX_SYNC , &s->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* wake up the possible processes. */ if (uart_circ_chars_pending (xmit) <WAKEUP_CHARS )@@ -275,7 +275,7 @@ static void mxs_auart_tx_chars(struct mxmxs_auart_dma_tx (s, i); }else {clear_bit (MXS_AUART_DMA_TX_SYNC , &s->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }return ; }drivers/usb/gadget/tcm_usb_gadget.c +2
-2@@ -1846,7 +1846,7 @@ static int usbg_port_link(struct se_port struct usbg_tpg *tpg = container_of(se_tpg, struct usbg_tpg, se_tpg);atomic_inc (&tpg->tpg_port_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();return 0 ; }@@ -1856,7 +1856,7 @@ static void usbg_port_unlink(struct se_p struct usbg_tpg *tpg = container_of(se_tpg, struct usbg_tpg, se_tpg);atomic_dec (&tpg->tpg_port_count); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }static int usbg_check_stop_free (struct se_cmd *se_cmd)drivers/usb/serial/usb_wwan.c +1
-1@@ -325,7 +325,7 @@ static void usb_wwan_outdat_callback(strfor (i =0 ; i <N_OUT_URB ; ++i) {if (portdata->out_urbs[i] == urb) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (i, &portdata->out_busy);break ; }drivers/vhost/scsi.c +1
-1@@ -1244,7 +1244,7 @@ vhost_scsi_set_endpoint(struct vhost_scs tpg->tv_tpg_vhost_count++; tpg->vhost_scsi = vs; vs_tpg[tpg->tport_tpgt] = tpg; -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); match = true; }mutex_unlock (&tpg->tv_tpg_mutex);drivers/w1/w1_family.c +2
-2@@ -131,9 +131,9 @@ void w1_family_get(struct w1_family *f)void __w1_family_get (struct w1_family *f) { -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&f->refcnt); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }EXPORT_SYMBOL (w1_unregister_family);drivers/xen/xen-pciback/pciback_ops.c +2
-2@@ -348,9 +348,9 @@ void xen_pcibk_do_op(struct work_structnotify_remote_via_irq (pdev->evtchn_irq);/* Mark that we're done. */ -smp_mb__before_clear_bit ();/* /after/ clearing PCIF_active */ +smp_mb__before_atomic ();/* /after/ clearing PCIF_active */ clear_bit (_PDEVF_op_active, &pdev->flags); -smp_mb__after_clear_bit ();/* /before/ final check for work */ +smp_mb__after_atomic ();/* /before/ final check for work */ /* Check to see if the driver domain tried to start another request in * between clearing _XEN_PCIF_active and clearing _PDEVF_op_active. fs/btrfs/btrfs_inode.h +1
-1@@ -279,7 +279,7 @@ static inline void btrfs_inode_block_unlstatic inline void btrfs_inode_resume_unlocked_dio (struct inode *inode) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (BTRFS_INODE_READDIO_NEED_LOCK , &BTRFS_I (inode)->runtime_flags); }fs/btrfs/extent_io.c +1
-1@@ -3451,7 +3451,7 @@ static int lock_extent_buffer_for_io(strstatic void end_extent_buffer_writeback (struct extent_buffer *eb) {clear_bit (EXTENT_BUFFER_WRITEBACK , &eb->bflags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&eb->bflags,EXTENT_BUFFER_WRITEBACK ); }fs/btrfs/inode.c +3
-3@@ -7078,7 +7078,7 @@ static void btrfs_end_dio_bio(struct bio* before atomic variable goto zero, we must make sure * dip->errors is perceived to be set. */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic (); }/* if there are more bios still pending for this dio, just exit */ @@ -7258,7 +7258,7 @@ static int btrfs_submit_direct_hook(int* before atomic variable goto zero, we must * make sure dip->errors is perceived to be set. */ -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();if (atomic_dec_and_test (&dip->pending_bios))bio_io_error (dip->orig_bio);@@ -7401,7 +7401,7 @@ static ssize_t btrfs_direct_IO(int rw, sreturn 0 ;atomic_inc (&inode->i_dio_count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * The generic stuff only does filemap_write_and_wait_range, which isn't fs/buffer.c +1
-1@@ -77,7 +77,7 @@ EXPORT_SYMBOL(__lock_buffer);void unlock_buffer (struct buffer_head *bh) {clear_bit_unlock (BH_Lock, &bh->b_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&bh->b_state, BH_Lock); }EXPORT_SYMBOL (unlock_buffer);fs/ext4/resize.c +1
-1@@ -42,7 +42,7 @@ int ext4_resize_begin(struct super_blockvoid ext4_resize_end (struct super_block *sb) {clear_bit_unlock (EXT4_RESIZING , &EXT4_SB (sb)->s_resize_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); } static ext4_group_t ext4_meta_bg_first_group(struct super_block *sb,fs/gfs2/glock.c +4
-4@@ -275,7 +275,7 @@ static inline int may_grant(const structstatic void gfs2_holder_wake (struct gfs2_holder *gh) {clear_bit (HIF_WAIT , &gh->gh_iflags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&gh->gh_iflags,HIF_WAIT ); }@@ -409,7 +409,7 @@ static void gfs2_demote_wake(struct gfs2 { gl->gl_demote_state =LM_ST_EXCLUSIVE ;clear_bit (GLF_DEMOTE , &gl->gl_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&gl->gl_flags,GLF_DEMOTE ); }@@ -618,7 +618,7 @@ __acquires(&gl->gl_spin) out_sched:clear_bit (GLF_LOCK , &gl->gl_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); gl->gl_lockref.count++;if (queue_delayed_work (glock_workqueue, &gl->gl_work,0 ) ==0 ) gl->gl_lockref.count--;@@ -626,7 +626,7 @@ __acquires(&gl->gl_spin) out_unlock:clear_bit (GLF_LOCK , &gl->gl_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return ; }fs/gfs2/glops.c +1
-1@@ -219,7 +219,7 @@ static void inode_go_sync(struct gfs2_gl* Writeback of the data mapping may cause the dirty flag to be set * so we have to clear it again here. */ -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (GLF_DIRTY , &gl->gl_flags); }fs/gfs2/lock_dlm.c +2
-2@@ -1132,7 +1132,7 @@ static void gdlm_recover_done(void *arg,queue_delayed_work (gfs2_control_wq, &sdp->sd_control_work,0 );clear_bit (DFL_DLM_RECOVERY , &ls->ls_recover_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&ls->ls_recover_flags,DFL_DLM_RECOVERY );spin_unlock (&ls->ls_recover_spin); }@@ -1269,7 +1269,7 @@ static int gdlm_mount(struct gfs2_sbd *s ls->ls_first = !!test_bit (DFL_FIRST_MOUNT , &ls->ls_recover_flags);clear_bit (SDF_NOJOURNALID , &sdp->sd_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&sdp->sd_flags,SDF_NOJOURNALID );return 0 ;fs/gfs2/recovery.c +1
-1@@ -587,7 +587,7 @@ void gfs2_recover_func(struct work_strucgfs2_recovery_done (sdp, jd->jd_jid,LM_RD_GAVEUP ); done:clear_bit (JDF_RECOVERY , &jd->jd_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&jd->jd_flags,JDF_RECOVERY ); }fs/gfs2/sys.c +2
-2@@ -332,7 +332,7 @@ static ssize_t block_store(struct gfs2_sset_bit (DFL_BLOCK_LOCKS , &ls->ls_recover_flags); else if (val == 0) {clear_bit (DFL_BLOCK_LOCKS , &ls->ls_recover_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();gfs2_glock_thaw (sdp); } else { ret = -EINVAL ;@@ -481,7 +481,7 @@ static ssize_t jid_store(struct gfs2_sbd rv = jid = -EINVAL ; sdp->sd_lockstruct.ls_jid = jid;clear_bit (SDF_NOJOURNALID , &sdp->sd_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&sdp->sd_flags,SDF_NOJOURNALID ); out:spin_unlock (&sdp->sd_jindex_spin);fs/jbd2/commit.c +3
-3@@ -43,7 +43,7 @@ static void journal_end_buffer_io_sync(sclear_buffer_uptodate (bh);if (orig_bh) {clear_bit_unlock (BH_Shadow, &orig_bh->b_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&orig_bh->b_state, BH_Shadow); }unlock_buffer (bh);@@ -239,7 +239,7 @@ static int journal_submit_data_buffers(jspin_lock (&journal->j_list_lock);J_ASSERT (jinode->i_transaction == commit_transaction);clear_bit (__JI_COMMIT_RUNNING, &jinode->i_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&jinode->i_flags, __JI_COMMIT_RUNNING); }spin_unlock (&journal->j_list_lock);@@ -277,7 +277,7 @@ static int journal_finish_inode_data_buf }spin_lock (&journal->j_list_lock);clear_bit (__JI_COMMIT_RUNNING, &jinode->i_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&jinode->i_flags, __JI_COMMIT_RUNNING); }fs/nfs/dir.c +6
-6@@ -1985,9 +1985,9 @@ static void nfs_access_free_entry(struct {put_rpccred (entry->cred);kfree (entry); -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_long_dec (&nfs_access_nr_entries); -smp_mb__after_atomic_dec (); +smp_mb__after_atomic (); }static void nfs_access_free_list (struct list_head *head)@@ -2035,9 +2035,9 @@ nfs_access_cache_scan(struct shrinker *selse { remove_lru_entry:list_del_init (&nfsi->access_cache_inode_lru); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NFS_INO_ACL_LRU_SET , &nfsi->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }spin_unlock (&inode->i_lock); }@@ -2185,9 +2185,9 @@ void nfs_access_add_cache(struct inode *nfs_access_add_rbtree (inode, cache);/* Update accounting */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_long_inc (&nfs_access_nr_entries); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* Add inode to global LRU list */ if (!test_bit (NFS_INO_ACL_LRU_SET , &NFS_I (inode)->flags)) {fs/nfs/inode.c +1
-1@@ -1058,7 +1058,7 @@ int nfs_revalidate_mapping(struct inodetrace_nfs_invalidate_mapping_exit (inode, ret);clear_bit_unlock (NFS_INO_INVALIDATING , bitlock); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (bitlock,NFS_INO_INVALIDATING ); out:return ret;fs/nfs/nfs4filelayoutdev.c +2
-2@@ -789,9 +789,9 @@ static void nfs4_wait_ds_connect(structstatic void nfs4_clear_ds_conn_bit (struct nfs4_pnfs_ds *ds) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NFS4DS_CONNECTING , &ds->ds_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&ds->ds_state,NFS4DS_CONNECTING ); }fs/nfs/nfs4state.c +2
-2@@ -1145,9 +1145,9 @@ static int nfs4_run_state_manager(void *static void nfs4_clear_state_manager_bit (struct nfs_client *clp) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NFS4CLNT_MANAGER_RUNNING , &clp->cl_state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&clp->cl_state,NFS4CLNT_MANAGER_RUNNING );rpc_wake_up (&clp->cl_rpcwaitq); }fs/nfs/pagelist.c +3
-3@@ -95,7 +95,7 @@ nfs_iocounter_dec(struct nfs_io_counter {if (atomic_dec_and_test (&c->io_count)) {clear_bit (NFS_IO_INPROGRESS , &c->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&c->flags,NFS_IO_INPROGRESS ); } }@@ -193,9 +193,9 @@ void nfs_unlock_request(struct nfs_pageprintk (KERN_ERR "NFS: Invalid unlock attempted\n" );BUG (); } -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (PG_BUSY , &req->wb_flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&req->wb_flags,PG_BUSY ); }fs/nfs/pnfs.c +1
-1@@ -1795,7 +1795,7 @@ static void pnfs_clear_layoutcommitting(unsigned long *bitlock = &NFS_I (inode)->flags;clear_bit_unlock (NFS_INO_LAYOUTCOMMITTING , bitlock); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (bitlock,NFS_INO_LAYOUTCOMMITTING ); }fs/nfs/pnfs.h +1
-1@@ -275,7 +275,7 @@ pnfs_get_lseg(struct pnfs_layout_segment {if (lseg) {atomic_inc (&lseg->pls_refcount); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }return lseg; }fs/nfs/write.c +2
-2@@ -405,7 +405,7 @@ int nfs_writepages(struct address_spacenfs_pageio_complete (&pgio);clear_bit_unlock (NFS_INO_FLUSHING , bitlock); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (bitlock,NFS_INO_FLUSHING );if (err <0 )@@ -1458,7 +1458,7 @@ static int nfs_commit_set_lock(struct nfstatic void nfs_commit_clear_lock (struct nfs_inode *nfsi) {clear_bit (NFS_INO_COMMIT , &nfsi->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_bit (&nfsi->flags,NFS_INO_COMMIT ); }fs/ubifs/lpt_commit.c +2
-2@@ -460,9 +460,9 @@ static int write_cnodes(struct ubifs_inf* important. */ clear_bit (DIRTY_CNODE , &cnode->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (COW_CNODE , &cnode->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); offs += len;dbg_chk_lpt_sz (c,1 , len); cnode = cnode->cnext;fs/ubifs/tnc_commit.c +2
-2@@ -895,9 +895,9 @@ static int write_index(struct ubifs_info* the reason for the second barrier. */ clear_bit (DIRTY_ZNODE , &znode->flags); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (COW_ZNODE , &znode->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();/* * We have marked the znode as clean but have not updated the include/asm-generic/atomic.h +1
-6@@ -16,6 +16,7 @@#define __ASM_GENERIC_ATOMIC_H#include <asm/cmpxchg.h> +#include <asm/barrier.h> #ifdef CONFIG_SMP /* Force people to define core atomics */ @@ -182,11 +183,5 @@ static inline void atomic_set_mask(unsig }#endif -/* Assume that atomic operations are already serializing */ -#define smp_mb__before_atomic_dec () barrier() -#define smp_mb__after_atomic_dec () barrier() -#define smp_mb__before_atomic_inc () barrier() -#define smp_mb__after_atomic_inc () barrier() -#endif /* __KERNEL__ */ #endif /* __ASM_GENERIC_ATOMIC_H */ include/asm-generic/barrier.h +8
-0@@ -62,6 +62,14 @@#define set_mb (var, value) do { (var) = (value); mb(); } while (0)#endif +#ifndef smp_mb__before_atomic +#define smp_mb__before_atomic () smp_mb() +#endif + +#ifndef smp_mb__after_atomic +#define smp_mb__after_atomic () smp_mb() +#endif +#define smp_store_release (p, v) \ do { \ compiletime_assert_atomic_type(*p); \include/asm-generic/bitops.h +1
-7@@ -11,14 +11,8 @@#include <linux/irqflags.h> #include <linux/compiler.h> +#include <asm/barrier.h> -/* -* clear_bit may not imply a memory barrier -*/ -#ifndef smp_mb__before_clear_bit -#define smp_mb__before_clear_bit () smp_mb() -#define smp_mb__after_clear_bit () smp_mb() -#endif #include <asm-generic/bitops/__ffs.h> #include <asm-generic/bitops/ffz.h> include/asm-generic/bitops/atomic.h +1
-1@@ -80,7 +80,7 @@ static inline void set_bit(int nr, volat* * clear_bit() is atomic and may not be reordered. However, it does * not contain a memory barrier, so if it is used for locking purposes, -* you should call smp_mb__before_clear_bit() and/or smp_mb__after_clear_bit() +* you should call smp_mb__before_atomic() and/or smp_mb__after_atomic() * in order to ensure changes are visible on other processors. */ static inline void clear_bit (int nr,volatile unsigned long *addr)include/asm-generic/bitops/lock.h +1
-1@@ -20,7 +20,7 @@*/ #define clear_bit_unlock (nr, addr) \ do { \ - smp_mb__before_clear_bit(); \ + smp_mb__before_atomic(); \ clear_bit(nr, addr); \ } while (0)include/linux/buffer_head.h +1
-1@@ -278,7 +278,7 @@ static inline void get_bh(struct buffer_static inline void put_bh (struct buffer_head *bh) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&bh->b_count); }include/linux/genhd.h +1
-1@@ -649,7 +649,7 @@ static inline void hd_ref_init(struct hdstatic inline void hd_struct_get (struct hd_struct *part) {atomic_inc (&part->ref); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static inline int hd_struct_try_get (struct hd_struct *part)include/linux/interrupt.h +4
-4@@ -447,7 +447,7 @@ static inline int tasklet_trylock(structstatic inline void tasklet_unlock (struct tasklet_struct *t) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (TASKLET_STATE_RUN , &(t)->state); }@@ -495,7 +495,7 @@ static inline void tasklet_hi_schedule_fstatic inline void tasklet_disable_nosync (struct tasklet_struct *t) {atomic_inc (&t->count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static inline void tasklet_disable (struct tasklet_struct *t)@@ -507,13 +507,13 @@ static inline void tasklet_disable(strucstatic inline void tasklet_enable (struct tasklet_struct *t) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&t->count); }static inline void tasklet_hi_enable (struct tasklet_struct *t) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&t->count); }include/linux/netdevice.h +1
-1@@ -500,7 +500,7 @@ static inline void napi_disable(struct nstatic inline void napi_enable (struct napi_struct *n) {BUG_ON (!test_bit (NAPI_STATE_SCHED , &n->state)); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NAPI_STATE_SCHED , &n->state); }include/linux/sched.h +2
-4@@ -2737,10 +2737,8 @@ static inline bool __must_check current_/* * Polling state must be visible before we test NEED_RESCHED, * paired by resched_task() -* -* XXX: assumes set/clear bit are identical barrier wise. */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return unlikely (tif_need_resched ()); }@@ -2758,7 +2756,7 @@ static inline bool __must_check current_* Polling state must be visible before we test NEED_RESCHED, * paired by resched_task() */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();return unlikely (tif_need_resched ()); }include/linux/sunrpc/sched.h +4
-4@@ -142,18 +142,18 @@ struct rpc_task_setup { test_and_set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)#define rpc_clear_running (t) \ do { \ - smp_mb__before_clear_bit(); \ + smp_mb__before_atomic(); \ clear_bit(RPC_TASK_RUNNING, &(t)->tk_runstate); \ - smp_mb__after_clear_bit(); \ + smp_mb__after_atomic(); \ } while (0)#define RPC_IS_QUEUED (t) test_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)#define rpc_set_queued (t) set_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)#define rpc_clear_queued (t) \ do { \ - smp_mb__before_clear_bit(); \ + smp_mb__before_atomic(); \ clear_bit(RPC_TASK_QUEUED, &(t)->tk_runstate); \ - smp_mb__after_clear_bit(); \ + smp_mb__after_atomic(); \ } while (0)#define RPC_IS_ACTIVATED (t) test_bit(RPC_TASK_ACTIVE, &(t)->tk_runstate)include/linux/sunrpc/xprt.h +4
-4@@ -368,9 +368,9 @@ static inline int xprt_test_and_clear_costatic inline void xprt_clear_connecting (struct rpc_xprt *xprt) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (XPRT_CONNECTING , &xprt->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static inline int xprt_connecting (struct rpc_xprt *xprt)@@ -400,9 +400,9 @@ static inline void xprt_clear_bound(strustatic inline void xprt_clear_binding (struct rpc_xprt *xprt) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (XPRT_BINDING , &xprt->state); -smp_mb__after_clear_bit (); +smp_mb__after_atomic (); }static inline int xprt_test_and_set_binding (struct rpc_xprt *xprt)include/linux/tracehook.h +1
-1@@ -191,7 +191,7 @@ static inline void tracehook_notify_resu* pairs with task_work_add()->set_notify_resume() after * hlist_add_head(task->task_works); */ -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (unlikely (current->task_works))task_work_run (); }include/net/ip_vs.h +2
-2@@ -1204,7 +1204,7 @@ static inline bool __ip_vs_conn_get(stru/* put back the conn without restarting its timer */ static inline void __ip_vs_conn_put (struct ip_vs_conn *cp) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&cp->refcnt); }void ip_vs_conn_put (struct ip_vs_conn *cp);@@ -1408,7 +1408,7 @@ static inline void ip_vs_dest_hold(strucstatic inline void ip_vs_dest_put (struct ip_vs_dest *dest) { -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&dest->refcnt); }kernel/debug/debug_core.c +2
-2@@ -526,7 +526,7 @@ static int kgdb_cpu_enter(struct kgdb_st kgdb_info[cpu].exception_state &= ~(DCPU_WANT_MASTER |DCPU_IS_SLAVE ); kgdb_info[cpu].enter_kgdb--; -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&slaves_in_kgdb);dbg_touch_watchdogs ();local_irq_restore (flags);@@ -654,7 +654,7 @@ static int kgdb_cpu_enter(struct kgdb_st kgdb_info[cpu].exception_state &= ~(DCPU_WANT_MASTER |DCPU_IS_SLAVE ); kgdb_info[cpu].enter_kgdb--; -smp_mb__before_atomic_dec (); +smp_mb__before_atomic ();atomic_dec (&masters_in_kgdb);/* Free kgdb_active */ atomic_set (&kgdb_active,-1 );kernel/futex.c +1
-1@@ -250,7 +250,7 @@ static inline void futex_get_mm(union fu* get_futex_key() implies a full barrier. This is relied upon * as full barrier (B), see the ordering comment above. */ -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static inline bool hb_waiters_pending (struct futex_hash_bucket *hb)kernel/kmod.c +1
-1@@ -498,7 +498,7 @@ int __usermodehelper_disable(enum umh_distatic void helper_lock (void ) {atomic_inc (&running_helpers); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic (); }static void helper_unlock (void )kernel/rcu/tree.c +11
-11@@ -389,9 +389,9 @@ static void rcu_eqs_enter_common(struct }rcu_prepare_for_idle (smp_processor_id ());/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */ -smp_mb__before_atomic_inc ();/* See above. */ +smp_mb__before_atomic ();/* See above. */ atomic_inc (&rdtp->dynticks); -smp_mb__after_atomic_inc ();/* Force ordering with next sojourn. */ +smp_mb__after_atomic ();/* Force ordering with next sojourn. */ WARN_ON_ONCE (atomic_read (&rdtp->dynticks) &0x1 );/* @@ -509,10 +509,10 @@ void rcu_irq_exit(void)static void rcu_eqs_exit_common (struct rcu_dynticks *rdtp,long long oldval,int user) { -smp_mb__before_atomic_inc ();/* Force ordering w/previous sojourn. */ +smp_mb__before_atomic ();/* Force ordering w/previous sojourn. */ atomic_inc (&rdtp->dynticks);/* CPUs seeing atomic_inc() must see later RCU read-side crit sects */ -smp_mb__after_atomic_inc ();/* See above. */ +smp_mb__after_atomic ();/* See above. */ WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks) &0x1 ));rcu_cleanup_after_idle (smp_processor_id ());trace_rcu_dyntick (TPS ("End" ), oldval, rdtp->dynticks_nesting);@@ -637,10 +637,10 @@ void rcu_nmi_enter(void) (atomic_read(&rdtp->dynticks) & 0x1))return ; rdtp->dynticks_nmi_nesting++; -smp_mb__before_atomic_inc ();/* Force delay from prior write. */ +smp_mb__before_atomic ();/* Force delay from prior write. */ atomic_inc (&rdtp->dynticks);/* CPUs seeing atomic_inc() must see later RCU read-side crit sects */ -smp_mb__after_atomic_inc ();/* See above. */ +smp_mb__after_atomic ();/* See above. */ WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks) &0x1 )); }@@ -659,9 +659,9 @@ void rcu_nmi_exit(void) --rdtp->dynticks_nmi_nesting != 0)return ;/* CPUs seeing atomic_inc() must see prior RCU read-side crit sects */ -smp_mb__before_atomic_inc ();/* See above. */ +smp_mb__before_atomic ();/* See above. */ atomic_inc (&rdtp->dynticks); -smp_mb__after_atomic_inc ();/* Force delay to next write. */ +smp_mb__after_atomic ();/* Force delay to next write. */ WARN_ON_ONCE (atomic_read (&rdtp->dynticks) &0x1 ); }@@ -2738,7 +2738,7 @@ void synchronize_sched_expedited(void) s =atomic_long_read (&rsp->expedited_done);if (ULONG_CMP_GE ((ulong )s, (ulong )firstsnap)) {/* ensure test happens before caller kfree */ -smp_mb__before_atomic_inc ();/* ^^^ */ +smp_mb__before_atomic ();/* ^^^ */ atomic_long_inc (&rsp->expedited_workdone1);return ; }@@ -2756,7 +2756,7 @@ void synchronize_sched_expedited(void) s =atomic_long_read (&rsp->expedited_done);if (ULONG_CMP_GE ((ulong )s, (ulong )firstsnap)) {/* ensure test happens before caller kfree */ -smp_mb__before_atomic_inc ();/* ^^^ */ +smp_mb__before_atomic ();/* ^^^ */ atomic_long_inc (&rsp->expedited_workdone2);return ; }@@ -2785,7 +2785,7 @@ void synchronize_sched_expedited(void) s =atomic_long_read (&rsp->expedited_done);if (ULONG_CMP_GE ((ulong )s, (ulong )snap)) {/* ensure test happens before caller kfree */ -smp_mb__before_atomic_inc ();/* ^^^ */ +smp_mb__before_atomic ();/* ^^^ */ atomic_long_inc (&rsp->expedited_done_lost);break ; }kernel/rcu/tree_plugin.h +4
-4@@ -2514,9 +2514,9 @@ static void rcu_sysidle_enter(struct rcu/* Record start of fully idle period. */ j = jiffies;ACCESS_ONCE (rdtp->dynticks_idle_jiffies) = j; -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&rdtp->dynticks_idle); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();WARN_ON_ONCE (atomic_read (&rdtp->dynticks_idle) &0x1 ); }@@ -2581,9 +2581,9 @@ static void rcu_sysidle_exit(struct rcu_ }/* Record end of idle period. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&rdtp->dynticks_idle); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks_idle) &0x1 ));/* kernel/sched/cpupri.c +3
-3@@ -165,7 +165,7 @@ void cpupri_set(struct cpupri *cp, int c* do a write memory barrier, and then update the count, to * make sure the vector is visible when count is set. */ -smp_mb__before_atomic_inc (); +smp_mb__before_atomic ();atomic_inc (&(vec)->count); do_mb =1 ; }@@ -185,14 +185,14 @@ void cpupri_set(struct cpupri *cp, int c* the new priority vec. */ if (do_mb) -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();/* * When removing from the vector, we decrement the counter first * do a memory barrier and then clear the mask. */ atomic_dec (&(vec)->count); -smp_mb__after_atomic_inc (); +smp_mb__after_atomic ();cpumask_clear_cpu (cpu, vec->mask); }kernel/sched/wait.c +1
-1@@ -394,7 +394,7 @@ EXPORT_SYMBOL(__wake_up_bit);* * In order for this to function properly, as it uses waitqueue_active() * internally, some kind of memory barrier must be done prior to calling -* this. Typically, this will be smp_mb__after_clear_bit(), but in some +* this. Typically, this will be smp_mb__after_atomic(), but in some * cases where bitflags are manipulated non-atomically under a lock, one * may need to use a less regular barrier, such fs/inode.c's smp_mb(), * because spin_unlock() does not guarantee a memory barrier. mm/backing-dev.c +1
-1@@ -549,7 +549,7 @@ void clear_bdi_congested(struct backing_ bit = sync ? BDI_sync_congested : BDI_async_congested;if (test_and_clear_bit (bit, &bdi->state))atomic_dec (&nr_bdi_congested[sync]); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();if (waitqueue_active (wqh))wake_up (wqh); }mm/filemap.c +2
-2@@ -609,7 +609,7 @@ void unlock_page(struct page *page) {VM_BUG_ON_PAGE (!PageLocked (page), page);clear_bit_unlock (PG_locked, &page->flags); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_page (page, PG_locked); }EXPORT_SYMBOL (unlock_page);@@ -626,7 +626,7 @@ void end_page_writeback(struct page *pagif (!test_clear_page_writeback (page))BUG (); -smp_mb__after_clear_bit (); +smp_mb__after_atomic ();wake_up_page (page, PG_writeback); }EXPORT_SYMBOL (end_page_writeback);net/atm/pppoatm.c +1
-1@@ -252,7 +252,7 @@ static int pppoatm_may_send(struct pppoa* we need to ensure there's a memory barrier after it. The bit * *must* be set before we do the atomic_inc() on pvcc->inflight. * There's no smp_mb__after_set_bit(), so it's this or abuse -* smp_mb__after_clear_bit(). +* smp_mb__after_atomic(). */ test_and_set_bit (BLOCKED , &pvcc->blocked);net/bluetooth/hci_event.c +2
-2@@ -45,7 +45,7 @@ static void hci_cc_inquiry_cancel(structreturn ;clear_bit (HCI_INQUIRY , &hdev->flags); -smp_mb__after_clear_bit ();/* wake_up_bit advises about this barrier */ +smp_mb__after_atomic ();/* wake_up_bit advises about this barrier */ wake_up_bit (&hdev->flags,HCI_INQUIRY );hci_conn_check_pending (hdev);@@ -1531,7 +1531,7 @@ static void hci_inquiry_complete_evt(strif (!test_and_clear_bit (HCI_INQUIRY , &hdev->flags))return ; -smp_mb__after_clear_bit ();/* wake_up_bit advises about this barrier */ +smp_mb__after_atomic ();/* wake_up_bit advises about this barrier */ wake_up_bit (&hdev->flags,HCI_INQUIRY );if (!test_bit (HCI_MGMT , &hdev->dev_flags))net/core/dev.c +4
-4@@ -1323,7 +1323,7 @@ static int __dev_close_many(struct list_* dev->stop() will invoke napi_disable() on all of it's * napi_struct instances on this device. */ -smp_mb__after_clear_bit ();/* Commit netif_running(). */ +smp_mb__after_atomic ();/* Commit netif_running(). */ }dev_deactivate_many (head);@@ -3343,7 +3343,7 @@ static void net_tx_action(struct softirq root_lock =qdisc_lock (q);if (spin_trylock (root_lock)) { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__QDISC_STATE_SCHED, &q->state);qdisc_run (q);@@ -3353,7 +3353,7 @@ static void net_tx_action(struct softirq &q->state)) { __netif_reschedule(q); } else { -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (__QDISC_STATE_SCHED, &q->state); }@@ -4216,7 +4216,7 @@ void __napi_complete(struct napi_structBUG_ON (n->gro_list);list_del (&n->poll_list); -smp_mb__before_clear_bit (); +smp_mb__before_atomic ();clear_bit (NAPI_STATE_SCHED , &n->state); }EXPORT_SYMBOL (__napi_complete);net/core/link_watch.c +1
-1@@ -147,7 +147,7 @@ static void linkwatch_do_dev(struct net_* Make sure the above read is complete since it can be * rewritten as soon as we clear the bit below. Lines of this file are not shown. View the raw message.
net/ipv4/inetpeer.c +1
-1Lines of this file are not shown. View the raw message.
net/netfilter/nf_conntrack_core.c +1
-1Lines of this file are not shown. View the raw message.
net/rds/ib_recv.c +2
-2Lines of this file are not shown. View the raw message.
net/rds/iw_recv.c +2
-2Lines of this file are not shown. View the raw message.
net/rds/send.c +3
-3Lines of this file are not shown. View the raw message.
net/rds/tcp_send.c +1
-1Lines of this file are not shown. View the raw message.
net/sunrpc/auth.c +1
-1Lines of this file are not shown. View the raw message.
net/sunrpc/auth_gss/auth_gss.c +1
-1Lines of this file are not shown. View the raw message.
net/sunrpc/backchannel_rqst.c +2
-2Lines of this file are not shown. View the raw message.
net/sunrpc/xprt.c +2
-2Lines of this file are not shown. View the raw message.
net/sunrpc/xprtsock.c +8
-8Lines of this file are not shown. View the raw message.
net/unix/af_unix.c +1
-1Lines of this file are not shown. View the raw message.
sound/pci/bt87x.c +2
-2Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Will Deacon
Acked-byRe: [RFC][PATCH 3/5] arch: s/smp_mb__(before|after)_(atomic|clear)_(dec,inc,bit)/smp_mb__\1/g
Quoted 23 lines · Paul E. McKenney
Quoted 22 lines · Peter Zijlstra
Because atomic ops are implemented the same across an architecture, the current incomplete set of extra barriers:
smp_mb__before_atomic_inc()
smp_mb__after_atomic_inc()
smp_mb__before_atomic_dec()
smp_mb__after_atomic_dec()
smp_mb__before_clear_bit()
smp_mb__after_clear_bit()is both incomplete and superfluous.
It is incomplete because there are far more atomic operations that do not return values -- such as
atomic_add(),set_bit()etc. And it is superfluous because they're all the same anyway.Simplify things by reducing the triplicate set into a single set of barriers that is valid for all void atomic ops:
smp_mb__before_atomic()
smp_mb__after_atomic()
Quoted 7 lines
Quoted 1 line
Signed-off-by Peter Zijlstra <[email protected]>I very much like the API shrinkage. The RCU changes are good, and I believe that the rest is OK too, though my eyes were going a bit buggy towards the end...
Acked-by Paul E. McKenney <[email protected]>
... and the arm[64] parts look fine to me.
Usual comment about upcoming conflicts :)
Will
Peter Zijlstra
maintainerRFC PATCH 4/5 arch: Generic atomic.h cleanup
In reply to Peter ZijlstraThis patch should mostly be a NO-OP and only removes some duplication. With a bit of effort these files could be further reduced.
Compile tested on everything I have a compiler for; missing are frv and tile -- because they're special and I got tired.
XXX: split up in per arch patches?
--- arch/alpha/include/asm/atomic.h | 217 ++++-------- arch/arc/include/asm/atomic.h | 156 +++----- arch/arm/include/asm/atomic.h | 283 ++++++---------- arch/arm64/include/asm/atomic.h | 193 +++------- arch/avr32/include/asm/atomic.h | 87 ++-- arch/cris/include/arch-v10/arch/system.h | 2 arch/cris/include/asm/atomic.h | 52 +- arch/hexagon/include/asm/atomic.h | 65 +-- arch/ia64/include/asm/atomic.h | 182 ++++------ arch/m32r/include/asm/atomic.h | 140 +++---- arch/m68k/include/asm/atomic.h | 109 ++---- arch/metag/include/asm/atomic_lnkget.h | 118 ++---- arch/metag/include/asm/atomic_lock1.h | 73 +--- arch/mips/include/asm/atomic.h | 547 +++++++++---------------------- arch/mn10300/include/asm/atomic.h | 124 ++----- arch/parisc/include/asm/atomic.h | 107 +++--- arch/powerpc/include/asm/atomic.h | 194 +++------- arch/sh/include/asm/atomic-grb.h | 118 ++---- arch/sh/include/asm/atomic-irq.h | 63 +-- arch/sh/include/asm/atomic-llsc.h | 100 ++--- arch/sparc/include/asm/atomic_32.h | 19 - arch/sparc/include/asm/atomic_64.h | 43 +- arch/sparc/include/asm/barrier_32.h | 1 arch/sparc/include/asm/processor.h | 2 arch/sparc/kernel/smp_64.c | 2 arch/sparc/lib/atomic32.c | 25 - arch/sparc/lib/atomic_64.S | 159 +++------ arch/sparc/lib/ksyms.c | 17 arch/xtensa/include/asm/atomic.h | 227 ++++-------- include/asm-generic/atomic.h | 143 ++------ 30 files changed, 1341 insertions(+), 2227 deletions(-)
arch/alpha/include/asm/atomic.h +78 -139
@@ -29,145 +29,84 @@* branch back to restart the operation. */ -static __inline__void atomic_add (int i,atomic_t * v) -{ -unsigned long temp; - __asm__ __volatile__( -"1: ldl_l %0,%1\n" -" addl %0,%2,%0\n" -" stl_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter) - :"Ir" (i),"m" (v->counter)); -} - -static __inline__void atomic64_add (long i,atomic64_t * v) -{ -unsigned long temp; - __asm__ __volatile__( -"1: ldq_l %0,%1\n" -" addq %0,%2,%0\n" -" stq_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter) - :"Ir" (i),"m" (v->counter)); -} - -static __inline__void atomic_sub (int i,atomic_t * v) -{ -unsigned long temp; - __asm__ __volatile__( -"1: ldl_l %0,%1\n" -" subl %0,%2,%0\n" -" stl_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter) - :"Ir" (i),"m" (v->counter)); -} - -static __inline__void atomic64_sub (long i,atomic64_t * v) -{ -unsigned long temp; - __asm__ __volatile__( -"1: ldq_l %0,%1\n" -" subq %0,%2,%0\n" -" stq_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter) - :"Ir" (i),"m" (v->counter)); -} - - -/* -* Same as above, but return the result value -*/ -static inline int atomic_add_return (int i,atomic_t *v) -{ -long temp, result; -smp_mb (); - __asm__ __volatile__( -"1: ldl_l %0,%1\n" -" addl %0,%3,%2\n" -" addl %0,%3,%0\n" -" stl_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter),"=&r" (result) - :"Ir" (i),"m" (v->counter) :"memory" ); -smp_mb (); -return result; -} - -static __inline__long atomic64_add_return (long i,atomic64_t * v) -{ -long temp, result; -smp_mb (); - __asm__ __volatile__( -"1: ldq_l %0,%1\n" -" addq %0,%3,%2\n" -" addq %0,%3,%0\n" -" stq_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter),"=&r" (result) - :"Ir" (i),"m" (v->counter) :"memory" ); -smp_mb (); -return result; -} - -static __inline__long atomic_sub_return (int i,atomic_t * v) -{ -long temp, result; -smp_mb (); - __asm__ __volatile__( -"1: ldl_l %0,%1\n" -" subl %0,%3,%2\n" -" subl %0,%3,%0\n" -" stl_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter),"=&r" (result) - :"Ir" (i),"m" (v->counter) :"memory" ); -smp_mb (); -return result; -} - -static __inline__long atomic64_sub_return (long i,atomic64_t * v) -{ -long temp, result; -smp_mb (); - __asm__ __volatile__( -"1: ldq_l %0,%1\n" -" subq %0,%3,%2\n" -" subq %0,%3,%0\n" -" stq_c %0,%1\n" -" beq %0,2f\n" -".subsection 2\n" -"2: br 1b\n" -".previous" - :"=&r" (temp),"=m" (v->counter),"=&r" (result) - :"Ir" (i),"m" (v->counter) :"memory" ); -smp_mb (); -return result; -} +#define ATOMIC_OP (op) \ +static __inline__ void atomic_##op(int i, atomic_t * v) \ +{ \ + unsigned long temp; \ + __asm__ __volatile__( \ + "1: ldl_l %0,%1\n" \ + " " #op "l %0,%2,%0\n" \ + " stl_c %0,%1\n" \ + " beq %0,2f\n" \ + ".subsection 2\n" \ + "2: br 1b\n" \ + ".previous" \ + :"=&r" (temp), "=m" (v->counter) \ + :"Ir" (i), "m" (v->counter)); \ +} \ + \ +static inline int atomic_##op##_return(int i, atomic_t *v) \ +{ \ + long temp, result; \ + smp_mb(); \ + __asm__ __volatile__( \ + "1: ldl_l %0,%1\n" \ + " " #op "l %0,%3,%2\n" \ + " " #op "l %0,%3,%0\n" \ + " stl_c %0,%1\n" \ + " beq %0,2f\n" \ + ".subsection 2\n" \ + "2: br 1b\n" \ + ".previous" \ + :"=&r" (temp), "=m" (v->counter), "=&r" (result) \ + :"Ir" (i), "m" (v->counter) : "memory"); \ + smp_mb(); \ + return result; \ +} + +#define ATOMIC64_OP (op) \ +static __inline__ void atomic64_##op(long i, atomic64_t * v) \ +{ \ + unsigned long temp; \ + __asm__ __volatile__( \ + "1: ldq_l %0,%1\n" \ + " " #op "q %0,%2,%0\n" \ + " stq_c %0,%1\n" \ + " beq %0,2f\n" \ + ".subsection 2\n" \ + "2: br 1b\n" \ + ".previous" \ + :"=&r" (temp), "=m" (v->counter) \ + :"Ir" (i), "m" (v->counter)); \ +} \ + \ +static __inline__ long atomic64_##op##_return(long i, atomic64_t * v) \ +{ \ + long temp, result; \ + smp_mb(); \ + __asm__ __volatile__( \ + "1: ldq_l %0,%1\n" \ + " " #op "q %0,%3,%2\n" \ + " " #op "q %0,%3,%0\n" \ + " stq_c %0,%1\n" \ + " beq %0,2f\n" \ + ".subsection 2\n" \ + "2: br 1b\n" \ + ".previous" \ + :"=&r" (temp), "=m" (v->counter), "=&r" (result) \ + :"Ir" (i), "m" (v->counter) : "memory"); \ + smp_mb(); \ + return result; \ +} + +#define ATOMIC_OPS (op) ATOMIC_OP(op) ATOMIC64_OP(op) + +ATOMIC_OPS(add) +ATOMIC_OPS(sub) + +#undef ATOMIC_OPS +#undef ATOMIC64_OP +#undef ATOMIC_OP#define atomic64_cmpxchg (v, old, new) (cmpxchg(&((v)->counter), old, new))#define atomic64_xchg (v, new) (xchg(&((v)->counter), new))
arch/arc/include/asm/atomic.h +57 -99
174 lines not on this page · raw message
arch/arm/include/asm/atomic.h +108 -175
311 lines not on this page · raw message
arch/arm64/include/asm/atomic.h +70 -123
212 lines not on this page · raw message
arch/avr32/include/asm/atomic.h +36 -51
102 lines not on this page · raw message
arch/cris/include/arch-v10/arch/system.h +0 -2
@@ -36,8 +36,6 @@ static inline unsigned long _get_base(chreturn 0 ; } -#define nop () __asm__ __volatile__ ("nop"); -#define xchg (ptr,x) ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))#define tas (ptr) (xchg((ptr),1))
arch/cris/include/asm/atomic.h +20 -32
64 lines not on this page · raw message
arch/hexagon/include/asm/atomic.h +33 -32
74 lines not on this page · raw message
arch/ia64/include/asm/atomic.h +83 -99
217 lines not on this page · raw message
arch/m32r/include/asm/atomic.h +53 -87
174 lines not on this page · raw message
arch/m68k/include/asm/atomic.h +46 -63
129 lines not on this page · raw message
arch/metag/include/asm/atomic_lnkget.h +42 -76
127 lines not on this page · raw message
arch/metag/include/asm/atomic_lock1.h +27 -46
82 lines not on this page · raw message
arch/mips/include/asm/atomic.h +175 -372
565 lines not on this page · raw message
arch/mn10300/include/asm/atomic.h +39 -85
148 lines not on this page · raw message
arch/parisc/include/asm/atomic.h +61 -46
134 lines not on this page · raw message
arch/powerpc/include/asm/atomic.h +70 -124
218 lines not on this page · raw message
arch/sh/include/asm/atomic-grb.h +42 -76
38 lines not on this page · raw message
arch/sh/include/asm/atomic-irq.h +24 -39
63 lines not on this page · raw message
arch/sh/include/asm/atomic-llsc.h +38 -62
100 lines not on this page · raw message
arch/sparc/include/asm/atomic_32.h +9 -10
Lines of this file are not shown. View the raw message.
arch/sparc/include/asm/atomic_64.h +19 -24
43 lines not on this page · raw message
arch/sparc/include/asm/barrier_32.h +0 -1
Lines of this file are not shown. View the raw message.
arch/sparc/include/asm/processor.h +0 -2
Lines of this file are not shown. View the raw message.
arch/sparc/kernel/smp_64.c +1 -1
Lines of this file are not shown. View the raw message.
arch/sparc/lib/atomic32.c +15 -10
Lines of this file are not shown. View the raw message.
arch/sparc/lib/atomic_64.S +58 -101
159 lines not on this page · raw message
arch/sparc/lib/ksyms.c +9 -8
Lines of this file are not shown. View the raw message.
arch/xtensa/include/asm/atomic.h +76 -151
227 lines not on this page · raw message
include/asm-generic/atomic.h +52 -91
143 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [RFC][PATCH 4/5] arch: Generic atomic.h cleanup
Quoted 1 line · Peter Zijlstra
XXX: split up in per arch patches?
Yes please -- I have a bunch of changes in this area pending for both arm and arm64.
Cheers,
Will
Peter Zijlstra
maintainerRFC PATCH 5/5 arch: Sanitize atomic_t bitwise ops
In reply to Peter ZijlstraMany archs have atomic_{set,clear}_mask() but not all. Remove these and provide a comprehensive set of bitops:
atomic{,64}_{and,or,xor}{,_return}()
for everybody.
--- arch/alpha/include/asm/atomic.h | 3 + arch/arc/include/asm/atomic.h | 29 ++--------- arch/arm/include/asm/atomic.h | 23 ++++---- arch/arm64/include/asm/atomic.h | 8 ++- arch/avr32/include/asm/atomic.h | 3 + arch/blackfin/include/asm/atomic.h | 5 + arch/blackfin/mach-common/smp.c | 2 arch/cris/include/asm/atomic.h | 3 + arch/hexagon/include/asm/atomic.h | 3 + arch/ia64/include/asm/atomic.h | 32 ++++++++++-- arch/m32r/include/asm/atomic.h | 44 +--------------- arch/m32r/kernel/smp.c | 4 - arch/m68k/include/asm/atomic.h | 13 +--- arch/metag/include/asm/atomic_lnkget.h | 37 +------------- arch/metag/include/asm/atomic_lock1.h | 23 +------- arch/mips/include/asm/atomic.h | 6 ++ arch/mn10300/include/asm/atomic.h | 70 +------------------------- arch/mn10300/mm/tlb-smp.c | 2 arch/parisc/include/asm/atomic.h | 6 ++ arch/powerpc/include/asm/atomic.h | 8 ++- arch/powerpc/kernel/misc_32.S | 19 ------- arch/s390/include/asm/atomic.h | 87 ++++++++++++++++++++------------- arch/s390/kernel/time.c | 4 - arch/s390/kvm/diag.c | 2 arch/s390/kvm/intercept.c | 2 arch/s390/kvm/interrupt.c | 16 +++--- arch/s390/kvm/kvm-s390.c | 14 ++--- arch/s390/kvm/sigp.c | 6 +- arch/sh/include/asm/atomic-grb.h | 42 +-------------- arch/sh/include/asm/atomic-irq.h | 21 +------ arch/sh/include/asm/atomic-llsc.h | 31 +---------- arch/sparc/include/asm/atomic_32.h | 4 + arch/sparc/include/asm/atomic_64.h | 3 + arch/sparc/lib/atomic32.c | 5 + arch/sparc/lib/atomic_64.S | 6 ++ arch/sparc/lib/ksyms.c | 3 + arch/x86/include/asm/atomic.h | 32 ++++++++---- arch/x86/include/asm/atomic64_32.h | 20 +++++++ arch/x86/include/asm/atomic64_64.h | 22 ++++++++ arch/xtensa/include/asm/atomic.h | 72 +-------------------------- drivers/gpu/drm/i915/i915_irq.c | 4 - drivers/s390/scsi/zfcp_aux.c | 2 drivers/s390/scsi/zfcp_erp.c | 68 ++++++++++++------------- drivers/s390/scsi/zfcp_fc.c | 8 +-- drivers/s390/scsi/zfcp_fsf.c | 30 +++++------ drivers/s390/scsi/zfcp_qdio.c | 14 ++--- include/asm-generic/atomic.h | 34 +++++++----- include/asm-generic/atomic64.h | 17 ++++-- include/linux/atomic.h | 13 ---- lib/atomic64.c | 77 +++++++++++------------------ 50 files changed, 404 insertions(+), 598 deletions(-)
arch/alpha/include/asm/atomic.h +3 -0
@@ -103,6 +103,9 @@ static __inline__ long atomic64_##op##_r ATOMIC_OPS(add) ATOMIC_OPS(sub) +ATOMIC_OPS(and) +ATOMIC_OPS(or) +ATOMIC_OPS(xor) #undef ATOMIC_OPS#undef ATOMIC64_OP
arch/arc/include/asm/atomic.h +6 -23
@@ -58,23 +58,12 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #undef ATOMIC_OP -static inline void atomic_clear_mask (unsigned long mask,unsigned long *addr) -{ -unsigned int temp; - - __asm__ __volatile__( -"1: llock %0, [%1] \n" -" bic %0, %0, %2 \n" -" scond %0, [%1] \n" -" bnz 1b \n" - :"=&r" (temp) - :"r" (addr),"ir" (mask) - :"cc" ); -} -#else /* !CONFIG_ARC_HAS_LLSC */ #ifndef CONFIG_SMP
@@ -134,18 +123,12 @@ static inline int atomic_##opn##_return( ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) #undef ATOMIC_OP -static inline void atomic_clear_mask(unsigned long mask, unsigned long *addr) -{ - unsigned long flags; - - atomic_ops_lock(flags); - *addr &= ~mask; - atomic_ops_unlock(flags); -} - #endif /* !CONFIG_ARC_HAS_LLSC */ /**
arch/arm/include/asm/atomic.h +12 -11
@@ -38,7 +38,7 @@* to ensure that the update happens. */ -#define ATOMIC_OP (op) \ +#define ATOMIC_OP (op, cop) \ static inline void atomic_##op(int i, atomic_t *v) \ { \ unsigned long tmp; \
@@ -78,11 +78,6 @@ static inline int atomic_##op##_return(ireturn result; \ } -ATOMIC_OP(add) -ATOMIC_OP(sub) - -#undef ATOMIC_OP -static inline int atomic_cmpxchg (atomic_t *ptr,int old,int new) {int oldval;
@@ -135,11 +130,6 @@ static inline int atomic_##opn##_return(return val; \ } -ATOMIC_OP(add, +=) -ATOMIC_OP(sub, -=) - -#undef ATOMIC_OP -static inline int atomic_cmpxchg (atomic_t *v,int old,int new) {int ret;
@@ -156,6 +146,14 @@ static inline int atomic_cmpxchg(atomic_ #endif /* __LINUX_ARM_ARCH__ */ +ATOMIC_OP(add, +=) +ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) + +#undef ATOMIC_OP + #define atomic_xchg(v, new) (xchg(&((v)->counter), new)) static inline int __atomic_add_unless(atomic_t *v, int a, int u)
@@ -282,6 +280,9 @@ static inline long long atomic64_##opn## ATOMIC64_OP(add, adds, adc) ATOMIC64_OP(sub, subs, sbc) +ATOMIC64_OP(and, and, and) +ATOMIC64_OP(or, or, or) +ATOMIC64_OP(xor, xor, xor)#undef ATOMIC64_OP
arch/arm64/include/asm/atomic.h +7 -1
@@ -80,6 +80,9 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #undef ATOMIC_OP
@@ -136,7 +139,7 @@ static inline int __atomic_add_unless(at#define atomic64_read (v) (*(volatile long long *)&(v)->counter)#define atomic64_set (v,i) (((v)->counter) = (i)) -#define ATOMIC64_OP (op) \ +#define ATOMIC_OP (op) \ static inline void atomic64_##op(long i, atomic64_t *v) \ { \ long result; \
@@ -172,6 +175,9 @@ static inline long atomic64_##op##_retur ATOMIC64_OP(add) ATOMIC64_OP(sub) +ATOMIC64_OP(and) +ATOMIC64_OP(or) +ATOMIC64_OP(xor) #undef ATOMIC64_OP
arch/avr32/include/asm/atomic.h +3 -0
@@ -59,6 +59,9 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #undef ATOMIC_OP
arch/blackfin/include/asm/atomic.h +3 -2
@@ -27,8 +27,9 @@ asmlinkage int __raw_atomic_test_asm(con#define atomic_add_return (i, v) __raw_atomic_update_asm(&(v)->counter, i)#define atomic_sub_return (i, v) __raw_atomic_update_asm(&(v)->counter, -(i)) -#define atomic_clear_mask (m, v) __raw_atomic_clear_asm(&(v)->counter, m) -#define atomic_set_mask (m, v) __raw_atomic_set_asm(&(v)->counter, m) +#define atomic_and_return (m, v) __raw_atomic_clear_asm(&(v)->counter, ~m) +#define atomic_or_return (m, v) __raw_atomic_set_asm(&(v)->counter, m) +#define atomic_xor_return (m, v) __raw_atomic_xor_asm(&(v)->counter, m)#endif
arch/blackfin/mach-common/smp.c +1 -1
@@ -195,7 +195,7 @@ void send_ipi(const struct cpumask *cpumlocal_irq_save (flags);for_each_cpu (cpu, cpumask) { bfin_ipi_data = &per_cpu (bfin_ipi, cpu); -atomic_set_mask ((1 << msg), &bfin_ipi_data->bits); +atomic_or ((1 << msg), &bfin_ipi_data->bits);atomic_inc (&bfin_ipi_data->count); }local_irq_restore (flags);
arch/cris/include/asm/atomic.h +3 -0
@@ -42,6 +42,9 @@ static inline int atomic_##opn##_return( ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) #undef ATOMIC_OP
arch/hexagon/include/asm/atomic.h +3 -0
@@ -115,6 +115,9 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #undef ATOMIC_OP
arch/ia64/include/asm/atomic.h +28 -4
@@ -44,8 +44,6 @@ ia64_atomic_##opn (int i, atomic_t *v) ATOMIC_OP(add, +) ATOMIC_OP(sub, -) -#undef ATOMIC_OP - #define atomic_add_return(i,v) \ ({ \ int __ia64_aar_i = (i); \
@@ -70,6 +68,20 @@ ATOMIC_OP(sub, -) : ia64_atomic_sub(__ia64_asr_i, v); \ }) +ATOMIC_OP(and, &) +ATOMIC_OP(or, |) +ATOMIC_OP(xor, ^) + +#define atomic_and_return(i,v) ia64_atomic_and(i,v) +#define atomic_or_return(i,v) ia64_atomic_or(i,v) +#define atomic_xor_return(i,v) ia64_atomic_xor(i,v) + +#define atomic_and(i,v) (void)ia64_atomic_and(i,v) +#define atomic_or(i,v) (void)ia64_atomic_or(i,v) +#define atomic_xor(i,v) (void)ia64_atomic_xor(i,v) + +#undef ATOMIC_OP + #define ATOMIC64_OP(opn, op) \ static __inline__ long \ ia64_atomic64_##opn (__s64 i, atomic64_t *v) \
@@ -88,8 +100,6 @@ ia64_atomic64_##opn (__s64 i, atomic64_t ATOMIC64_OP(add, +) ATOMIC64_OP(sub, -) -#undef ATOMIC64_OP - #define atomic64_add_return(i,v) \ ({ \ long __ia64_aar_i = (i); \
@@ -114,6 +124,20 @@ ATOMIC64_OP(sub, -) : ia64_atomic64_sub(__ia64_asr_i, v); \ }) +ATOMIC64_OP(and, &) +ATOMIC64_OP(or, |) +ATOMIC64_OP(xor, ^) + +#define atomic64_and_return(i,v) ia64_atomic64_and(i,v) +#define atomic64_or_return(i,v) ia64_atomic64_or(i,v) +#define atomic64_xor_return(i,v) ia64_atomic64_xor(i,v) + +#define atomic64_and(i,v) (void)ia64_atomic64_and(i,v) +#define atomic64_or(i,v) (void)ia64_atomic64_or(i,v) +#define atomic64_xor(i,v) (void)ia64_atomic64_xor(i,v) + +#undef ATOMIC64_OP + #define atomic_cmpxchg(v, old, new) (cmpxchg(&((v)->counter), old, new)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
arch/m32r/include/asm/atomic.h +3 -41
54 lines not on this page · raw message
arch/m32r/kernel/smp.c +2 -2
@@ -156,7 +156,7 @@ void smp_flush_cache_all(void)cpumask_clear_cpu (smp_processor_id (), &cpumask);spin_lock (&flushcache_lock); mask=cpumask_bits (&cpumask); -atomic_set_mask (*mask, (atomic_t *)&flushcache_cpumask); +atomic_or (*mask, (atomic_t *)&flushcache_cpumask);send_IPI_mask (&cpumask,INVALIDATE_CACHE_IPI ,0 );_flush_cache_copyback_all ();while (flushcache_cpumask)
@@ -407,7 +407,7 @@ static void flush_tlb_others(cpumask_t c flush_vma = vma; flush_va = va; mask=cpumask_bits (&cpumask); -atomic_set_mask (*mask, (atomic_t *)&flush_cpumask); +atomic_or (*mask, (atomic_t *)&flush_cpumask);/* * We have to send the IPI only to
arch/m68k/include/asm/atomic.h +3 -10
@@ -76,6 +76,9 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or , |=) +ATOMIC_OP(xor, ^=) #undef ATOMIC_OP
@@ -168,16 +171,6 @@ static inline int atomic_add_negative(inreturn c !=0 ; } -static inline void atomic_clear_mask (unsigned long mask,unsigned long *v) -{ - __asm__ __volatile__("andl %1,%0" : "+m" (*v) : ASM_DI (~(mask))); -} - -static inline void atomic_set_mask (unsigned long mask,unsigned long *v) -{ - __asm__ __volatile__("orl %1,%0" : "+m" (*v) : ASM_DI (mask)); -} -static __inline__int __atomic_add_unless (atomic_t *v,int a,int u) {int c, old;
arch/metag/include/asm/atomic_lnkget.h +3 -34
@@ -70,43 +70,12 @@ static inline int atomic_##op##_return(iATOMIC_OP (add)ATOMIC_OP (sub ) +ATOMIC_OP (and) +ATOMIC_OP (or) +ATOMIC_OP (xor) #undef ATOMIC_OP -static inline void atomic_clear_mask (unsigned int mask,atomic_t *v) -{ -int temp; - - asmvolatile ( -"1: LNKGETD %0, [%1]\n" -" AND %0, %0, %2\n" -" LNKSETD [%1] %0\n" -" DEFR %0, TXSTAT\n" -" ANDT %0, %0, #HI(0x3f000000)\n" -" CMPT %0, #HI(0x02000000)\n" -" BNZ 1b\n" - :"=&d" (temp) - :"da" (&v->counter),"bd" (~mask) - :"cc" ); -} - -static inline void atomic_set_mask (unsigned int mask,atomic_t *v) -{ -int temp; - - asmvolatile ( -"1: LNKGETD %0, [%1]\n" -" OR %0, %0, %2\n" -" LNKSETD [%1], %0\n" -" DEFR %0, TXSTAT\n" -" ANDT %0, %0, #HI(0x3f000000)\n" -" CMPT %0, #HI(0x02000000)\n" -" BNZ 1b\n" - :"=&d" (temp) - :"da" (&v->counter),"bd" (mask) - :"cc" ); -} - static inline int atomic_cmpxchg(atomic_t *v, int old, int new) {int result, temp;
arch/metag/include/asm/atomic_lock1.h +3 -20
@@ -65,29 +65,12 @@ static inline int atomic_##opn##_return( ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) #undef ATOMIC_OP -static inline void atomic_clear_mask(unsigned int mask, atomic_t *v) -{ -unsigned long flags; - -__global_lock1 (flags); -fence (); - v->counter &= ~mask; -__global_unlock1 (flags); -} - -static inline void atomic_set_mask(unsigned int mask, atomic_t *v) -{ -unsigned long flags; - -__global_lock1 (flags); -fence (); - v->counter |= mask; -__global_unlock1 (flags); -} - static inline int atomic_cmpxchg(atomic_t *v, int old, int new) { int ret;
arch/mips/include/asm/atomic.h +6 -0
@@ -128,6 +128,9 @@ static __inline__ int atomic_##op##_retu ATOMIC_OP(add, addu, +=) ATOMIC_OP(sub, subu, -=) +ATOMIC_OP(and, and, &=) +ATOMIC_OP(or, or, |=) +ATOMIC_OP(xor, xor, ^=) #undef ATOMIC_OP
@@ -397,6 +400,9 @@ static __inline__ long atomic64_##op##_r ATOMIC64_OP(add, daddu, +=) ATOMIC64_OP(sub, dsubu, -=) +ATOMIC64_OP(and, and, &=) +ATOMIC64_OP(or, or, |=) +ATOMIC64_OP(xor, xor, ^=) #undef ATOMIC64_OP
arch/mn10300/include/asm/atomic.h +3 -67
82 lines not on this page · raw message
arch/mn10300/mm/tlb-smp.c +1 -1
@@ -119,7 +119,7 @@ static void flush_tlb_others(cpumask_t c flush_mm = mm; flush_va = va;#if NR_CPUS <=BITS_PER_LONG -atomic_set_mask (cpumask.bits[0 ], &flush_cpumask.bits[0 ]); +atomic_or (cpumask.bits[0 ], &flush_cpumask.bits[0 ]);#else #error Not supported.#endif
arch/parisc/include/asm/atomic.h +6 -0
@@ -122,6 +122,9 @@ static __inline__ int atomic_##op##_retu ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) #undef ATOMIC_OP
@@ -177,6 +180,9 @@ static __inline__ s64 atomic64_##op##_re ATOMIC64_OP(add, +=) ATOMIC64_OP(sub, -=) +ATOMIC64_OP(and, &=) +ATOMIC64_OP(or, |=) +ATOMIC64_OP(xor, ^=) #undef ATOMIC64_OP
arch/powerpc/include/asm/atomic.h +7 -1
@@ -63,6 +63,9 @@ static __inline__ int atomic_##op##_retu ATOMIC_OP(add, add) ATOMIC_OP(sub, subf) +ATOMIC_OP(and, and) +ATOMIC_OP(or, or) +ATOMIC_OP(xor, xor)#undef ATOMIC_OP
@@ -295,7 +298,10 @@ static __inline__ long atomic64_##op##_r } ATOMIC64_OP(add, add) -ATOMIC64_OP(add, subf) +ATOMIC64_OP(sub, subf) +ATOMIC64_OP(and, and) +ATOMIC64_OP(or, or) +ATOMIC64_OP(xor, xor)#undef ATOMIC64_OP
arch/powerpc/kernel/misc_32.S +0 -19
@@ -593,25 +593,6 @@ _GLOBAL(copy_page) b 2b /* - * void atomic_clear_mask(atomic_t mask, atomic_t *addr) - * void atomic_set_mask(atomic_t mask, atomic_t *addr); - */ -_GLOBAL(atomic_clear_mask) -10: lwarx r5,0,r4 - andc r5,r5,r3 - PPC405_ERR77(0,r4) - stwcx. r5,0,r4 - bne- 10b - blr -_GLOBAL(atomic_set_mask) -10: lwarx r5,0,r4 - or r5,r5,r3 - PPC405_ERR77(0,r4) - stwcx. r5,0,r4 - bne- 10b - blr - -/* * Extended precision shifts. * * Updated to be valid for shift counts from 0 to 63 inclusive.
arch/s390/include/asm/atomic.h +54 -33
140 lines not on this page · raw message
arch/s390/kernel/time.c +2 -2
@@ -367,7 +367,7 @@ static void disable_sync_clock(void *dum* increase the "sequence" counter to avoid the race of an * etr event and the complete recovery against get_sync_clock. */ -atomic_clear_mask (0x80000000 , sw_ptr); +atomic_and (~0x80000000 , sw_ptr);atomic_inc (sw_ptr); }
@@ -378,7 +378,7 @@ static void disable_sync_clock(void *dumstatic void enable_sync_clock (void ) {atomic_t *sw_ptr = &__get_cpu_var (clock_sync_word); -atomic_set_mask (0x80000000 , sw_ptr); +atomic_or (0x80000000 , sw_ptr); }/*
arch/s390/kvm/diag.c +1 -1
@@ -94,7 +94,7 @@ static int __diag_ipl_functions(struct kreturn -EOPNOTSUPP ; } -atomic_set_mask (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); +atomic_or (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); vcpu->run->s390_reset_flags |=KVM_S390_RESET_SUBSYSTEM ; vcpu->run->s390_reset_flags |=KVM_S390_RESET_IPL ; vcpu->run->s390_reset_flags |=KVM_S390_RESET_CPU_INIT ;
arch/s390/kvm/intercept.c +1 -1
@@ -63,7 +63,7 @@ static int handle_stop(struct kvm_vcpu *trace_kvm_s390_stop_request (vcpu->arch.local_int.action_bits);if (vcpu->arch.local_int.action_bits &ACTION_STOP_ON_STOP ) { -atomic_set_mask (CPUSTAT_STOPPED , +atomic_or (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); vcpu->arch.local_int.action_bits &= ~ACTION_STOP_ON_STOP ;VCPU_EVENT (vcpu,3 ,"%s" ,"cpu stopped" );
arch/s390/kvm/interrupt.c +8 -8
@@ -118,21 +118,21 @@ static int __interrupt_is_deliverable(ststatic void __set_cpu_idle (struct kvm_vcpu *vcpu) {BUG_ON (vcpu->vcpu_id >KVM_MAX_VCPUS -1 ); -atomic_set_mask (CPUSTAT_WAIT , &vcpu->arch.sie_block->cpuflags); +atomic_or (CPUSTAT_WAIT , &vcpu->arch.sie_block->cpuflags);set_bit (vcpu->vcpu_id, vcpu->arch.local_int.float_int->idle_mask); }static void __unset_cpu_idle (struct kvm_vcpu *vcpu) {BUG_ON (vcpu->vcpu_id >KVM_MAX_VCPUS -1 ); -atomic_clear_mask (CPUSTAT_WAIT , &vcpu->arch.sie_block->cpuflags); +atomic_and (~CPUSTAT_WAIT , &vcpu->arch.sie_block->cpuflags);clear_bit (vcpu->vcpu_id, vcpu->arch.local_int.float_int->idle_mask); }static void __reset_intercept_indicators (struct kvm_vcpu *vcpu) { -atomic_clear_mask (CPUSTAT_ECALL_PEND | -CPUSTAT_IO_INT |CPUSTAT_EXT_INT |CPUSTAT_STOP_INT , +atomic_and (~(CPUSTAT_ECALL_PEND | +CPUSTAT_IO_INT |CPUSTAT_EXT_INT |CPUSTAT_STOP_INT ), &vcpu->arch.sie_block->cpuflags); vcpu->arch.sie_block->lctl =0x0000 ; vcpu->arch.sie_block->ictl &= ~ICTL_LPSW ;
@@ -140,7 +140,7 @@ static void __reset_intercept_indicatorsstatic void __set_cpuflag (struct kvm_vcpu *vcpu,u32 flag) { -atomic_set_mask (flag, &vcpu->arch.sie_block->cpuflags); +atomic_or (flag, &vcpu->arch.sie_block->cpuflags); } static void __set_intercept_indicator(struct kvm_vcpu *vcpu,
@@ -269,7 +269,7 @@ static void __do_deliver_interrupt(struc rc |=copy_from_guest (vcpu, &vcpu->arch.sie_block->gpsw, offsetof(struct _lowcore , restart_psw),sizeof (psw_t)); -atomic_clear_mask (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); +atomic_and (~CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags);break ;case KVM_S390_PROGRAM_INT : VCPU_EVENT(vcpu, 4, "interrupt: pgm check code:%x, ilc:%x",
@@ -748,7 +748,7 @@ int kvm_s390_inject_vm(struct kvm *kvm, } li = fi->local_int[sigcpu];spin_lock_bh (&li->lock); -atomic_set_mask (CPUSTAT_EXT_INT , li->cpuflags); +atomic_or (CPUSTAT_EXT_INT , li->cpuflags);if (waitqueue_active (li->wq))wake_up_interruptible (li->wq);spin_unlock_bh (&li->lock);
@@ -834,7 +834,7 @@ int kvm_s390_inject_vcpu(struct kvm_vcpuatomic_set (&li->active,1 );if (inti->type ==KVM_S390_SIGP_STOP ) li->action_bits |=ACTION_STOP_ON_STOP ; -atomic_set_mask (CPUSTAT_EXT_INT , li->cpuflags); +atomic_or (CPUSTAT_EXT_INT , li->cpuflags);if (waitqueue_active (&vcpu->wq))wake_up_interruptible (&vcpu->wq);spin_unlock_bh (&li->lock);
arch/s390/kvm/kvm-s390.c +7 -7
@@ -350,12 +350,12 @@ void kvm_arch_vcpu_load(struct kvm_vcpurestore_fp_regs (vcpu->arch.guest_fpregs.fprs);restore_access_regs (vcpu->run->s.regs.acrs);gmap_enable (vcpu->arch.gmap); -atomic_set_mask (CPUSTAT_RUNNING , &vcpu->arch.sie_block->cpuflags); +atomic_or (CPUSTAT_RUNNING , &vcpu->arch.sie_block->cpuflags); }void kvm_arch_vcpu_put (struct kvm_vcpu *vcpu) { -atomic_clear_mask (CPUSTAT_RUNNING , &vcpu->arch.sie_block->cpuflags); +atomic_and (~CPUSTAT_RUNNING , &vcpu->arch.sie_block->cpuflags);gmap_disable (vcpu->arch.gmap);save_fp_ctl (&vcpu->arch.guest_fpregs.fpc);save_fp_regs (vcpu->arch.guest_fpregs.fprs);
@@ -380,7 +380,7 @@ static void kvm_s390_vcpu_initial_reset( vcpu->arch.guest_fpregs.fpc =0 ; asmvolatile ("lfpc %0" : :"Q" (vcpu->arch.guest_fpregs.fpc)); vcpu->arch.sie_block->gbea =1 ; -atomic_set_mask (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); +atomic_or (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); }int kvm_arch_vcpu_postcreate (struct kvm_vcpu *vcpu)
@@ -482,12 +482,12 @@ int kvm_arch_vcpu_runnable(struct kvm_vcvoid s390_vcpu_block (struct kvm_vcpu *vcpu) { -atomic_set_mask (PROG_BLOCK_SIE , &vcpu->arch.sie_block->prog20); +atomic_or (PROG_BLOCK_SIE , &vcpu->arch.sie_block->prog20); }void s390_vcpu_unblock (struct kvm_vcpu *vcpu) { -atomic_clear_mask (PROG_BLOCK_SIE , &vcpu->arch.sie_block->prog20); +atomic_and (~PROG_BLOCK_SIE , &vcpu->arch.sie_block->prog20); }/*
@@ -496,7 +496,7 @@ void s390_vcpu_unblock(struct kvm_vcpu ** return immediately. */ void exit_sie (struct kvm_vcpu *vcpu) { -atomic_set_mask (CPUSTAT_STOP_INT , &vcpu->arch.sie_block->cpuflags); +atomic_or (CPUSTAT_STOP_INT , &vcpu->arch.sie_block->cpuflags);while (vcpu->arch.sie_block->prog0c &PROG_IN_SIE )cpu_relax (); }
@@ -804,7 +804,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vif (vcpu->sigset_active)sigprocmask (SIG_SETMASK , &vcpu->sigset, &sigsaved); -atomic_clear_mask (CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags); +atomic_and (~CPUSTAT_STOPPED , &vcpu->arch.sie_block->cpuflags);BUG_ON (vcpu->kvm->arch.float_int.local_int[vcpu->vcpu_id] ==NULL );
arch/s390/kvm/sigp.c +3 -3
@@ -78,7 +78,7 @@ static int __sigp_emergency(struct kvm_vspin_lock_bh (&li->lock);list_add_tail (&inti->list, &li->list);atomic_set (&li->active,1 ); -atomic_set_mask (CPUSTAT_EXT_INT , li->cpuflags); +atomic_or (CPUSTAT_EXT_INT , li->cpuflags);if (waitqueue_active (li->wq))wake_up_interruptible (li->wq);spin_unlock_bh (&li->lock);
@@ -147,7 +147,7 @@ static int __sigp_external_call(struct kspin_lock_bh (&li->lock);list_add_tail (&inti->list, &li->list);atomic_set (&li->active,1 ); -atomic_set_mask (CPUSTAT_EXT_INT , li->cpuflags); +atomic_or (CPUSTAT_EXT_INT , li->cpuflags);if (waitqueue_active (li->wq))wake_up_interruptible (li->wq);spin_unlock_bh (&li->lock);
@@ -177,7 +177,7 @@ static int __inject_sigp_stop(struct kvm }list_add_tail (&inti->list, &li->list);atomic_set (&li->active,1 ); -atomic_set_mask (CPUSTAT_STOP_INT , li->cpuflags); +atomic_or (CPUSTAT_STOP_INT , li->cpuflags); li->action_bits |= action;if (waitqueue_active (li->wq))wake_up_interruptible (li->wq);
arch/sh/include/asm/atomic-grb.h +3 -39
49 lines not on this page · raw message
arch/sh/include/asm/atomic-irq.h +3 -18
@@ -34,25 +34,10 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) #undef ATOMIC_OP -static inline void atomic_clear_mask(unsigned int mask, atomic_t *v) -{ -unsigned long flags; - -raw_local_irq_save (flags); - v->counter &= ~mask; -raw_local_irq_restore (flags); -} - -static inline void atomic_set_mask(unsigned int mask, atomic_t *v) -{ -unsigned long flags; - -raw_local_irq_save (flags); - v->counter |= mask; -raw_local_irq_restore (flags); -} - #endif /* __ASM_SH_ATOMIC_IRQ_H */
arch/sh/include/asm/atomic-llsc.h +3 -28
@@ -49,35 +49,10 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #undef ATOMIC_OP -static inline void atomic_clear_mask (unsigned int mask,atomic_t *v) -{ -unsigned long tmp; - - __asm__ __volatile__ ( -"1: movli.l @%2, %0 ! atomic_clear_mask \n" -" and %1, %0 \n" -" movco.l %0, @%2 \n" -" bf 1b \n" - :"=&z" (tmp) - :"r" (~mask),"r" (&v->counter) - :"t" ); -} - -static inline void atomic_set_mask (unsigned int mask,atomic_t *v) -{ -unsigned long tmp; - - __asm__ __volatile__ ( -"1: movli.l @%2, %0 ! atomic_set_mask \n" -" or %1, %0 \n" -" movco.l %0, @%2 \n" -" bf 1b \n" - :"=&z" (tmp) - :"r" (mask),"r" (&v->counter) - :"t" ); -} -#endif /* __ASM_SH_ATOMIC_LLSC_H */
arch/sparc/include/asm/atomic_32.h +4 -0
@@ -39,6 +39,10 @@ extern void atomic_set(atomic_t *, int);#define atomic_add_negative (a, v) (atomic_add_return((a), (v)) < 0) +#define atomic_and (i, v) ((void)atomic_and_return(i, v)) +#define atomic_or (i, v) ((void)atomic_or_return(i, v)) +#define atomic_xor (i, v) ((void)atomic_xor_return(i, v)) +/* * atomic_inc_and_test - increment and test * @v: pointer of type atomic_t
arch/sparc/include/asm/atomic_64.h +3 -0
@@ -28,6 +28,9 @@ extern long atomic64_##op##_return(long, ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #undef ATOMIC_OP
arch/sparc/lib/atomic32.c +3 -2
@@ -42,8 +42,9 @@ int atomic_##op##_return(int i, atomic_t EXPORT_SYMBOL(atomic_##op##_return); ATOMIC_OP(add, +=) - -#undef ATOMIC_OP +ATOMIC_OP(and, &=) +ATOMIC_OP(or, |=) +ATOMIC_OP(xor, ^=) int atomic_cmpxchg(atomic_t *v, int old, int new) {
arch/sparc/lib/atomic_64.S +6 -0
@@ -44,6 +44,9 @@ ENDPROC(atomic_##op##_return); ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) #define ATOMIC64_OP(op) \ ENTRY(atomic64_##op) /* %o0 = increment, %o1 = atomic_ptr */ \
@@ -74,6 +77,9 @@ ENDPROC(atomic64_##op##_return); ATOMIC64_OP(add) ATOMIC64_OP(sub) +ATOMIC64_OP(and) +ATOMIC64_OP(or) +ATOMIC64_OP(xor) ENTRY(atomic64_dec_if_positive) /* %o0 = atomic_ptr */ BACKOFF_SETUP(%o2)
arch/sparc/lib/ksyms.c +3 -0
@@ -107,6 +107,9 @@ EXPORT_SYMBOL(atomic64_##op##_return); ATOMIC_OP(add) ATOMIC_OP(sub) +ATOMIC_OP(and) +ATOMIC_OP(or) +ATOMIC_OP(xor) EXPORT_SYMBOL(atomic64_dec_if_positive);
arch/x86/include/asm/atomic.h +22 -10
@@ -182,6 +182,28 @@ static inline int atomic_xchg(atomic_t *return xchg (&v->counter, new); } +#define ATOMIC_OP (op, cop) \ +static inline void atomic_##op(int i, atomic_t *v) \ +{ \ + asm volatile(LOCK_PREFIX #op"l %1,%0" \ + : "+m" (v->counter) \ + : "ir" (i)); \ +} \ + \ +static inline int atomic_##op##_return(int i, atomic_t *v) \ +{ \ + int old, c = atomic_read(v); \ + while ((old = atomic_cmpxchg(v, c, c cop i)) != c) \ + c = old; \ + return c cop i; \ +} + +ATOMIC_OP(and, &) +ATOMIC_OP(or, |) +ATOMIC_OP(xor, ^) + +#undef ATOMIC_OP +/** * __atomic_add_unless - add unless the number is already a given value * @v: pointer of type atomic_t
@@ -234,16 +256,6 @@ static inline void atomic_or_long(unsign }#endif -/* These are x86-specific, used by some header files */ -#define atomic_clear_mask (mask, addr) \ - asm volatile(LOCK_PREFIX "andl %0,%1" \ - : : "r" (~(mask)), "m" (*(addr)) : "memory") - -#define atomic_set_mask (mask, addr) \ - asm volatile(LOCK_PREFIX "orl %0,%1" \ - : : "r" ((unsigned)(mask)), "m" (*(addr)) \ - : "memory") -#ifdef CONFIG_X86_32 # include <asm/atomic64_32.h> #else
arch/x86/include/asm/atomic64_32.h +20 -0
@@ -313,4 +313,24 @@ static inline long long atomic64_dec_if_ #undef alternative_atomic64 #undef __alternative_atomic64 +#define ATOMIC_OP(op, cop) \ +static inline long long atomic64_##op##_return(long long i, atomic64_t *v)\ +{ \ + long long old, c = 0; \ + while ((old = atomic64_cmpxchg(v, c, c cop i)) != c) \ + c = old; \ + return c cop i; \ +} \ + \ +static inline void atomic64_##op(long long i, atomic64_t *v) \ +{ \ + atomic64_##op##_return(i, v); \ +} + +ATOMIC_OP(and, &) +ATOMIC_OP(or, |) +ATOMIC_OP(xor, ^) + +#undef ATOMIC_OP + #endif /* _ASM_X86_ATOMIC64_32_H */
arch/x86/include/asm/atomic64_64.h +22 -0
@@ -220,4 +220,26 @@ static inline long atomic64_dec_if_posit return dec; } +#define ATOMIC64_OP(op, cop) \ +static inline void atomic64_##op(long i, atomic64_t *v) \ +{ \ + asm volatile(LOCK_PREFIX #op"q %1,%0" \ + : "+m" (v->counter) \ + : "ir" (i)); \ +} \ + \ +static inline long atomic64_##op##_return(long i, atomic64_t *v) \ +{ \ + long old, c = atomic64_read(v); \ + while ((old = atomic64_cmpxchg(v, c, c cop i)) != c) \ + c = old; \ + return c cop i; \ +} + +ATOMIC64_OP(and, &) +ATOMIC64_OP(or, |) +ATOMIC64_OP(xor, ^) + +#undef ATOMIC64_OP + #endif /* _ASM_X86_ATOMIC64_64_H */
arch/xtensa/include/asm/atomic.h +3 -69
84 lines not on this page · raw message
drivers/gpu/drm/i915/i915_irq.c +2 -2
@@ -2002,7 +2002,7 @@ static void i915_error_work_func(structkobject_uevent_env (&dev->primary->kdev->kobj,KOBJ_CHANGE , reset_done_event); }else { -atomic_set_mask (I915_WEDGED , &error->reset_counter); +atomic_or (I915_WEDGED , &error->reset_counter); }/*
@@ -2123,7 +2123,7 @@ void i915_handle_error(struct drm_devicei915_report_and_clear_eir (dev);if (wedged) { -atomic_set_mask (I915_RESET_IN_PROGRESS_FLAG , +atomic_or (I915_RESET_IN_PROGRESS_FLAG , &dev_priv->gpu_error.reset_counter);/*
drivers/s390/scsi/zfcp_aux.c +1 -1
@@ -527,7 +527,7 @@ struct zfcp_port *zfcp_port_enqueue(strulist_add_tail (&port->list, &adapter->port_list);write_unlock_irq (&adapter->port_list_lock); -atomic_set_mask (status |ZFCP_STATUS_COMMON_RUNNING , &port->status); +atomic_or (status |ZFCP_STATUS_COMMON_RUNNING , &port->status);return port;
drivers/s390/scsi/zfcp_erp.c +34 -34
250 lines not on this page · raw message
drivers/s390/scsi/zfcp_fc.c +4 -4
@@ -465,7 +465,7 @@ static void zfcp_fc_adisc_handler(void */* port is good, unblock rport without going through erp */ zfcp_scsi_schedule_rport_register (port); out: -atomic_clear_mask (ZFCP_STATUS_PORT_LINK_TEST , &port->status); +atomic_and (~ZFCP_STATUS_PORT_LINK_TEST , &port->status);put_device (&port->dev);kmem_cache_free (zfcp_fc_req_cache, fc_req); }
@@ -521,14 +521,14 @@ void zfcp_fc_link_test_work(struct work_if (atomic_read (&port->status) &ZFCP_STATUS_PORT_LINK_TEST ) goto out; -atomic_set_mask (ZFCP_STATUS_PORT_LINK_TEST , &port->status); +atomic_or (ZFCP_STATUS_PORT_LINK_TEST , &port->status); retval =zfcp_fc_adisc (port);if (retval ==0 )return ;/* send of ADISC was not possible */ -atomic_clear_mask (ZFCP_STATUS_PORT_LINK_TEST , &port->status); +atomic_and (~ZFCP_STATUS_PORT_LINK_TEST , &port->status);zfcp_erp_port_forced_reopen (port,0 ,"fcltwk1" ); out:
@@ -597,7 +597,7 @@ static void zfcp_fc_validate_port(structif (!(atomic_read (&port->status) &ZFCP_STATUS_COMMON_NOESC ))return ; -atomic_clear_mask (ZFCP_STATUS_COMMON_NOESC , &port->status); +atomic_and (~ZFCP_STATUS_COMMON_NOESC , &port->status);if ((port->supported_classes !=0 ) || !list_empty (&port->unit_list))
drivers/s390/scsi/zfcp_fsf.c +15 -15
@@ -114,7 +114,7 @@ static void zfcp_fsf_link_down_info_evalif (atomic_read (&adapter->status) &ZFCP_STATUS_ADAPTER_LINK_UNPLUGGED )return ; -atomic_set_mask (ZFCP_STATUS_ADAPTER_LINK_UNPLUGGED , &adapter->status); +atomic_or (ZFCP_STATUS_ADAPTER_LINK_UNPLUGGED , &adapter->status);zfcp_scsi_schedule_rports_block (adapter);
@@ -345,7 +345,7 @@ static void zfcp_fsf_protstatus_eval(strzfcp_erp_adapter_shutdown (adapter,0 ,"fspse_3" );break ;case FSF_PROT_HOST_CONNECTION_INITIALIZING : -atomic_set_mask (ZFCP_STATUS_ADAPTER_HOST_CON_INIT , +atomic_or (ZFCP_STATUS_ADAPTER_HOST_CON_INIT , &adapter->status);break ;case FSF_PROT_DUPLICATE_REQUEST_ID :
@@ -554,7 +554,7 @@ static void zfcp_fsf_exchange_config_datzfcp_erp_adapter_shutdown (adapter,0 ,"fsecdh1" );return ; } -atomic_set_mask (ZFCP_STATUS_ADAPTER_XCONFIG_OK , +atomic_or (ZFCP_STATUS_ADAPTER_XCONFIG_OK , &adapter->status);break ;case FSF_EXCHANGE_CONFIG_DATA_INCOMPLETE :
@@ -567,7 +567,7 @@ static void zfcp_fsf_exchange_config_dat/* avoids adapter shutdown to be able to recognize * events such as LINK UP */ -atomic_set_mask (ZFCP_STATUS_ADAPTER_XCONFIG_OK , +atomic_or (ZFCP_STATUS_ADAPTER_XCONFIG_OK , &adapter->status);zfcp_fsf_link_down_info_eval (req, &qtcb->header.fsf_status_qual.link_down_info);
@@ -1394,10 +1394,10 @@ static void zfcp_fsf_open_port_handler(sbreak ;case FSF_GOOD : port->handle = header->port_handle; -atomic_set_mask (ZFCP_STATUS_COMMON_OPEN | +atomic_or (ZFCP_STATUS_COMMON_OPEN |ZFCP_STATUS_PORT_PHYS_OPEN , &port->status); -atomic_clear_mask (ZFCP_STATUS_COMMON_ACCESS_DENIED | -ZFCP_STATUS_COMMON_ACCESS_BOXED , +atomic_and (~(ZFCP_STATUS_COMMON_ACCESS_DENIED | +ZFCP_STATUS_COMMON_ACCESS_BOXED ), &port->status);/* check whether D_ID has changed during open */ /*
@@ -1678,10 +1678,10 @@ static void zfcp_fsf_close_physical_portcase FSF_PORT_BOXED :/* can't use generic zfcp_erp_modify_port_status because * ZFCP_STATUS_COMMON_OPEN must not be reset for the port */ -atomic_clear_mask (ZFCP_STATUS_PORT_PHYS_OPEN , &port->status); +atomic_and (~ZFCP_STATUS_PORT_PHYS_OPEN , &port->status);shost_for_each_device (sdev, port->adapter->scsi_host)if (sdev_to_zfcp (sdev)->port == port) -atomic_clear_mask (ZFCP_STATUS_COMMON_OPEN , +atomic_and (~ZFCP_STATUS_COMMON_OPEN , &sdev_to_zfcp (sdev)->status);zfcp_erp_set_port_status (port,ZFCP_STATUS_COMMON_ACCESS_BOXED ); zfcp_erp_port_reopen(port, ZFCP_STATUS_COMMON_ERP_FAILED,
@@ -1701,10 +1701,10 @@ static void zfcp_fsf_close_physical_port/* can't use generic zfcp_erp_modify_port_status because * ZFCP_STATUS_COMMON_OPEN must not be reset for the port */ -atomic_clear_mask (ZFCP_STATUS_PORT_PHYS_OPEN , &port->status); +atomic_and (~ZFCP_STATUS_PORT_PHYS_OPEN , &port->status);shost_for_each_device (sdev, port->adapter->scsi_host)if (sdev_to_zfcp (sdev)->port == port) -atomic_clear_mask (ZFCP_STATUS_COMMON_OPEN , +atomic_and (~ZFCP_STATUS_COMMON_OPEN , &sdev_to_zfcp (sdev)->status);break ; }
@@ -1767,8 +1767,8 @@ static void zfcp_fsf_open_lun_handler(st zfcp_sdev =sdev_to_zfcp (sdev); -atomic_clear_mask (ZFCP_STATUS_COMMON_ACCESS_DENIED | -ZFCP_STATUS_COMMON_ACCESS_BOXED , +atomic_and (~(ZFCP_STATUS_COMMON_ACCESS_DENIED | +ZFCP_STATUS_COMMON_ACCESS_BOXED ), &zfcp_sdev->status);switch (header->fsf_status) {
@@ -1823,7 +1823,7 @@ static void zfcp_fsf_open_lun_handler(stcase FSF_GOOD : zfcp_sdev->lun_handle = header->lun_handle; -atomic_set_mask (ZFCP_STATUS_COMMON_OPEN , &zfcp_sdev->status); +atomic_or (ZFCP_STATUS_COMMON_OPEN , &zfcp_sdev->status);break ; } }
@@ -1914,7 +1914,7 @@ static void zfcp_fsf_close_lun_handler(s }break ;case FSF_GOOD : -atomic_clear_mask (ZFCP_STATUS_COMMON_OPEN , &zfcp_sdev->status); +atomic_and (~ZFCP_STATUS_COMMON_OPEN , &zfcp_sdev->status);break ; } }
drivers/s390/scsi/zfcp_qdio.c +7 -7
@@ -351,7 +351,7 @@ void zfcp_qdio_close(struct zfcp_qdio *q/* clear QDIOUP flag, thus do_QDIO is not called during qdio_shutdown */ spin_lock_irq (&qdio->req_q_lock); -atomic_clear_mask (ZFCP_STATUS_ADAPTER_QDIOUP , &adapter->status); +atomic_and (~ZFCP_STATUS_ADAPTER_QDIOUP , &adapter->status);spin_unlock_irq (&qdio->req_q_lock);wake_up (&qdio->req_q_wq);
@@ -386,7 +386,7 @@ int zfcp_qdio_open(struct zfcp_qdio *qdiif (atomic_read (&adapter->status) &ZFCP_STATUS_ADAPTER_QDIOUP )return -EIO ; -atomic_clear_mask (ZFCP_STATUS_ADAPTER_SIOSL_ISSUED , +atomic_and (~ZFCP_STATUS_ADAPTER_SIOSL_ISSUED , &qdio->adapter->status);zfcp_qdio_setup_init_data (&init_data, qdio);
@@ -398,14 +398,14 @@ int zfcp_qdio_open(struct zfcp_qdio *qdi goto failed_qdio;if (ssqd.qdioac2 &CHSC_AC2_DATA_DIV_ENABLED ) -atomic_set_mask (ZFCP_STATUS_ADAPTER_DATA_DIV_ENABLED , +atomic_or (ZFCP_STATUS_ADAPTER_DATA_DIV_ENABLED , &qdio->adapter->status);if (ssqd.qdioac2 &CHSC_AC2_MULTI_BUFFER_ENABLED ) { -atomic_set_mask (ZFCP_STATUS_ADAPTER_MB_ACT , &adapter->status); +atomic_or (ZFCP_STATUS_ADAPTER_MB_ACT , &adapter->status); qdio->max_sbale_per_sbal =QDIO_MAX_ELEMENTS_PER_BUFFER ; }else { -atomic_clear_mask (ZFCP_STATUS_ADAPTER_MB_ACT , &adapter->status); +atomic_and (~ZFCP_STATUS_ADAPTER_MB_ACT , &adapter->status); qdio->max_sbale_per_sbal =QDIO_MAX_ELEMENTS_PER_BUFFER -1 ; }
@@ -429,7 +429,7 @@ int zfcp_qdio_open(struct zfcp_qdio *qdi/* set index of first available SBALS / number of available SBALS */ qdio->req_q_idx =0 ;atomic_set (&qdio->req_q_free,QDIO_MAX_BUFFERS_PER_Q ); -atomic_set_mask (ZFCP_STATUS_ADAPTER_QDIOUP , &qdio->adapter->status); +atomic_or (ZFCP_STATUS_ADAPTER_QDIOUP , &qdio->adapter->status);if (adapter->scsi_host) { adapter->scsi_host->sg_tablesize = qdio->max_sbale_per_req;
@@ -506,6 +506,6 @@ void zfcp_qdio_siosl(struct zfcp_adapter rc =ccw_device_siosl (adapter->ccw_device);if (!rc) -atomic_set_mask (ZFCP_STATUS_ADAPTER_SIOSL_ISSUED , +atomic_or (ZFCP_STATUS_ADAPTER_SIOSL_ISSUED , &adapter->status); }
include/asm-generic/atomic.h +20 -14
@@ -61,16 +61,32 @@ GEN_ATOMIC_OP(add, +) GEN_ATOMIC_OP(sub, -)#endif -#ifndef atomic_clear_mask +#ifndef atomic_and_return GEN_ATOMIC_OP(and, &) -#define atomic_clear_mask(i, v) (void)atomic_and_return(~(i), (v))#endif -#ifndef atomic_set_mask +#ifndef atomic_or_return GEN_ATOMIC_OP(or, |) -#define atomic_set_mask(i, v) (void)atomic_or_return((i), (v))#endif +#ifndef atomic_xor_return +GEN_ATOMIC_OP(xor, ^) +#endif + +#undef GEN_ATOMIC_OP + +#define GEN_ATOMIC_OP (op) \ +static inline void atomic_##op(int i, atomic_t *v) \ +{ \ + atomic_##op##_return(i, v); \ +} + +GEN_ATOMIC_OP(add) +GEN_ATOMIC_OP(sub) +GEN_ATOMIC_OP(and) +GEN_ATOMIC_OP(or) +GEN_ATOMIC_OP(xor) + #undef GEN_ATOMIC_OP/*
@@ -106,16 +122,6 @@ static inline int atomic_add_negative(inreturn atomic_add_return (i, v) <0 ; } -static inline void atomic_add (int i,atomic_t *v) -{ -atomic_add_return (i, v); -} - -static inline void atomic_sub (int i,atomic_t *v) -{ -atomic_sub_return (i, v); -} -static inline void atomic_inc (atomic_t *v) {atomic_add_return (1 , v);
include/asm-generic/atomic64.h +13 -4
@@ -20,10 +20,19 @@ typedef struct {extern long long atomic64_read (const atomic64_t *v);extern void atomic64_set (atomic64_t *v,long long i); -extern void atomic64_add (long long a,atomic64_t *v); -extern long long atomic64_add_return (long long a,atomic64_t *v); -extern void atomic64_sub (long long a,atomic64_t *v); -extern long long atomic64_sub_return (long long a,atomic64_t *v); + +#define ATOMIC64_OP (op) \ +extern void atomic64_##op(long long a, atomic64_t *v); \ +extern long long atomic64_##op##_return(long long a, atomic64_t *v); + +ATOMIC64_OP(add) +ATOMIC64_OP(sub) +ATOMIC64_OP(and) +ATOMIC64_OP(or ) +ATOMIC64_OP(xor) + +#undef ATOMIC64_OP + extern long long atomic64_dec_if_positive(atomic64_t *v);extern long long atomic64_cmpxchg (atomic64_t *v,long long o,long long n);extern long long atomic64_xchg (atomic64_t *v,long long new);
include/linux/atomic.h +0 -13
@@ -111,19 +111,6 @@ static inline int atomic_dec_if_positive }#endif -#ifndef CONFIG_ARCH_HAS_ATOMIC_OR -static inline void atomic_or (int i,atomic_t *v) -{ -int old; -int new; - -do { - old =atomic_read (v); - new = old | i; - }while (atomic_cmpxchg (v, old, new) != old); -} -#endif /* #ifndef CONFIG_ARCH_HAS_ATOMIC_OR */ -#include <asm-generic/atomic-long.h> #ifdef CONFIG_GENERIC_ATOMIC64 #include <asm-generic/atomic64.h>
lib/atomic64.c +31 -46
84 lines not on this page · raw message
Geert Uytterhoeven
maintainerRe: [RFC][PATCH 5/5] arch: Sanitize atomic_t bitwise ops
arch/m68k/include/asm/atomic.h static inline int atomic_##op##_return(i · line 81
Quoted 9 lines · Peter Zijlstra
arch/m68k/include/asm/atomic.h +3
-0@@ -76,6 +76,9 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or , |=) +ATOMIC_OP(xor, ^=)
This last one ain't gonna fly. Given
static inline void atomic_##op(int i, atomic_t *v) \
{ \
__asm__ __volatile__(#op "l %1,%0" : "+m" (*v) : ASM_DI (i)); \}
from the previous patch, it'll generate the "xorl" mnemonic, which IIRC does not exist. It's called "eorl".
Verified with m68k-linux-gnu-as:
Error: Unknown operator -- statement `xorl %d0,%d1' ignored
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Peter Zijlstra
maintainerRe: [RFC][PATCH 5/5] arch: Sanitize atomic_t bitwise ops
Quoted 13 lines · Geert Uytterhoeven
arch/m68k/include/asm/atomic.h static inline int atomic_##op##_return(i · line 81
Quoted 9 lines · Peter Zijlstra
arch/m68k/include/asm/atomic.h +3
-0@@ -76,6 +76,9 @@ static inline int atomic_##op##_return(i ATOMIC_OP(add, +=) ATOMIC_OP(sub, -=) +ATOMIC_OP(and, &=) +ATOMIC_OP(or , |=) +ATOMIC_OP(xor, ^=)This last one ain't gonna fly. Given
Quoted 7 lines
static inline void atomic_##op(int i, atomic_t *v) \ { \ __asm__ __volatile__(#op "l %1,%0" : "+m" (*v) : ASM_DI (i)); \}
from the previous patch, it'll generate the "xorl" mnemonic, which IIRC does not exist. It's called "eorl".
Some instruction sets I googled, others I didn't; guess what I did (or rather didn't do) for m68k :-)
ok I'll make that:
#define ATOMIC_OP(op, aop, cop)
ATOMIC_OP(xor, eor, ^=)
I had to so similar things on a few other archs as well, asm instruction names are odd sometimes.
Thanks!
Re: [RFC][PATCH 5/5] arch: Sanitize atomic_t bitwise ops
In reply to Peter ZijlstraQuoted 4 lines · Peter Zijlstra
Many archs have atomic_{set,clear}_mask() but not all. Remove these and provide a comprehensive set of bitops:
atomic{,64}_{and,or,xor}{,_return}()
Who uses these, and why?
The "_return()" versions of atomic ops are noticeably slower and more complex on common architectures (ie x86), and apparently there is no use of them since they didn't exist.
So why add them? Just to encourage people to do bad things?
Linus
Peter Zijlstra
maintainerRe: [RFC][PATCH 5/5] arch: Sanitize atomic_t bitwise ops
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Peter Zijlstra
Many archs have atomic_{set,clear}_mask() but not all. Remove these and provide a comprehensive set of bitops:
atomic{,64}_{and,or,xor}{,_return}()
Who uses these, and why?
s390 is stuffed with atomic_{set,clear}_mask usage, no clue whatfor.
There's 2 atomic_set_mask() users in drm/i915, again haven't looked if it makes sense.
Various archs use atomic_set_mask() for their tlb flush mask, which arguably should be done using the bitmap functions we have.
Quoted 5 lines
The "
_return()" versions of atomic ops are noticeably slower and more complex on common architectures (ie x86), and apparently there is no use of them since they didn't exist.So why add them? Just to encourage people to do bad things?
Fair enough, they're mostly an accident of how I implemented the macro generation magic. I suppose I can change that and avoid generating the _return thingies.
I don't particularly care for this patch too much; but I do dislike the atomic_{set,clear}_mask() things -- mostly because they don't really fit the normal atomic_t functions and because some archs have then and others do no.
And atomic_{set,clear}_mask() can be used for setting/clearing multiple bits as opposed to {set,clear}_bit, which only does a single bit.
Re: [RFC][PATCH 5/5] arch: Sanitize atomic_t bitwise ops
Quoted 2 lines · Peter Zijlstra
s390 is stuffed with atomic_{set,clear}_mask usage, no clue whatfor.
It's not the set/clear_mask ones I would worry about.
It's the "_return" variants. As far as I can tell, there are exactly ZERO users of that stuff, and they are BAD BAD BAD.
On x86, those things would cause a cmpxchg loop, and the bad read-for-shared-before-acquire cacheline pattern.
So why indirectly encourage people to add users for a bad operation?
Linus
Peter Zijlstra
maintainerRe: [RFC][PATCH 5/5] arch: Sanitize atomic_t bitwise ops
Quoted 8 lines · Linus Torvalds
On Thu, Feb 6, 2014 at 9:52 AM, Peter Zijlstra <[email protected]> wrote:
It's the "_return" variants. As far as I can tell, there are exactly ZERO users of that stuff, and they are BAD BAD BAD.On x86, those things would cause a cmpxchg loop, and the bad read-for-shared-before-acquire cacheline pattern.
So why indirectly encourage people to add users for a bad operation?
OK, I'll not generate the _return() variants.
David Howells
maintainerIs it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
One thing I'm not sure of, though, is how well gcc's atomics will cope with interrupt handlers touching atomics on CPUs without suitable atomic instructions - that said, userspace does have to deal with signals getting underfoot. but then userspace can't normally disable interrupts.
David
Peter Zijlstra
maintainerQuoted 9 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
One thing I'm not sure of, though, is how well gcc's atomics will cope with interrupt handlers touching atomics on CPUs without suitable atomic instructions - that said, userspace does have to deal with signals getting underfoot. but then userspace can't normally disable interrupts.
I can do an asm-generic/atomic_c11.h if people want.
Quoted 4 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
Makes sense to me!
Quoted 4 lines
One thing I'm not sure of, though, is how well gcc's atomics will cope with interrupt handlers touching atomics on CPUs without suitable atomic instructions - that said, userspace does have to deal with signals getting underfoot. but then userspace can't normally disable interrupts.
Perhaps make the C11 definitions so that any arch can override any specific definition?
Thanx, Paul
Quoted 4 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
It sounds interesting to me, if we can make it work properly and reliably. + [email protected] for others in the GCC community to chip in.
Quoted 8 lines
One thing I'm not sure of, though, is how well gcc's atomics will cope with interrupt handlers touching atomics on CPUs without suitable atomic instructions - that said, userspace does have to deal with signals getting underfoot. but then userspace can't normally disable interrupts.
David
Ramana
Quoted 9 lines · Ramana Radhakrishnan
Quoted 4 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
It sounds interesting to me, if we can make it work properly and reliably. + [email protected] for others in the GCC community to chip in.
Given my (albeit limited) experience playing with the C11 spec and GCC, I really think this is a bad idea for the kernel. It seems that nobody really agrees on exactly how the C11 atomics map to real architectural instructions on anything but the trivial architectures. For example, should the following code fire the assert?
extern atomic<int> foo, bar, baz;
void thread1(void)
{
foo.store(42, memory_order_relaxed);
bar.fetch_add(1, memory_order_seq_cst);
baz.store(42, memory_order_relaxed);
}void thread2(void)
{
while (baz.load(memory_order_seq_cst) != 42) {
/* do nothing */
}
assert(foo.load(memory_order_seq_cst) == 42);
}To answer that question, you need to go and look at the definitions of synchronises-with, happens-before, dependency_ordered_before and a whole pile of vaguely written waffle to realise that you don't know. Certainly, the code that arm64 GCC currently spits out would allow the assertion to fire on some microarchitectures.
There are also so many ways to blow your head off it's untrue. For example, cmpxchg takes a separate memory model parameter for failure and success, but then there are restrictions on the sets you can use for each. It's not hard to find well-known memory-ordering experts shouting "Just use memory_model_seq_cst for everything, it's too hard otherwise". Then there's the fun of load-consume vs load-acquire (arm64 GCC completely ignores consume atm and optimises all of the data dependencies away) as well as the definition of "data races", which seem to be used as an excuse to miscompile a program at the earliest opportunity.
Trying to introduce system concepts (writes to devices, interrupts, non-coherent agents) into this mess is going to be an uphill battle IMHO. I'd just rather stick to the semantics we have and the asm volatile barriers.
That's not to say I don't there's no room for improvement in what we have in the kernel. Certainly, I'd welcome allowing more relaxed operations on architectures that support them, but it needs to be something that at least the different architecture maintainers can understand how to implement efficiently behind an uncomplicated interface. I don't think that interface is C11.
Just my thoughts on the matter...
Will
Quoted 37 lines · Will Deacon
Quoted 9 lines · Ramana Radhakrishnan
Quoted 4 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
It sounds interesting to me, if we can make it work properly and reliably. + [email protected] for others in the GCC community to chip in.
Given my (albeit limited) experience playing with the C11 spec and GCC, I really think this is a bad idea for the kernel. It seems that nobody really agrees on exactly how the C11 atomics map to real architectural instructions on anything but the trivial architectures. For example, should the following code fire the assert?
extern atomic<int> foo, bar, baz;
void thread1(void)
{ foo.store(42, memory_order_relaxed); bar.fetch_add(1, memory_order_seq_cst); baz.store(42, memory_order_relaxed); }void thread2(void)
{ while (baz.load(memory_order_seq_cst) != 42) { /* do nothing */ } assert(foo.load(memory_order_seq_cst) == 42); }
Quoted 5 lines
To answer that question, you need to go and look at the definitions of synchronises-with, happens-before,
dependency_ordered_beforeand a whole pile of vaguely written waffle to realise that you don't know. Certainly, the code that arm64 GCC currently spits out would allow the assertion to fire on some microarchitectures.
Yep! I believe that a memory_order_seq_cst fence in combination with the fetch_add() would do the trick on many architectures, however. All of this is one reason that any C11 definitions need to be individually overridable by individual architectures.
Quoted 9 lines
There are also so many ways to blow your head off it's untrue. For example, cmpxchg takes a separate memory model parameter for failure and success, but then there are restrictions on the sets you can use for each. It's not hard to find well-known memory-ordering experts shouting "Just use
memory_model_seq_cstfor everything, it's too hard otherwise". Then there's the fun of load-consume vs load-acquire (arm64 GCC completely ignores consume atm and optimises all of the data dependencies away) as well as the definition of "data races", which seem to be used as an excuse to miscompile a program at the earliest opportunity.
Trust me, rcu_dereference() is not going to be defined in terms of memory_order_consume until the compilers implement it both correctly and efficiently. They are not there yet, and there is currently no shortage of compiler writers who would prefer to ignore memory_order_consume. And rcu_dereference() will need per-arch overrides for some time during any transition to memory_order_consume.
Quoted 3 lines
Trying to introduce system concepts (writes to devices, interrupts, non-coherent agents) into this mess is going to be an uphill battle IMHO. I'd just rather stick to the semantics we have and the asm volatile barriers.
And barrier() isn't going to go away any time soon, either. And ACCESS_ONCE() needs to keep volatile semantics until there is some memory_order_whatever that prevents loads and stores from being coalesced.
Quoted 8 lines
That's not to say I don't there's no room for improvement in what we have in the kernel. Certainly, I'd welcome allowing more relaxed operations on architectures that support them, but it needs to be something that at least the different architecture maintainers can understand how to implement efficiently behind an uncomplicated interface. I don't think that interface is C11.
Just my thoughts on the matter...
C11 does not provide a good interface for the Linux kernel, nor was it intended to do so. It might provide good implementations for some of the atomic ops for some architectures. This could reduce the amount of assembly written for new architectures, and could potentially allow the compiler to do a better job of optimizing (scary thought!). But for this to work, that architecture's Linux-kernel maintainer and gcc maintainer would need to be working together.
Thanx, Paul
Quoted 44 lines · Paul E. McKenney
Quoted 37 lines · Will Deacon
Quoted 9 lines · Ramana Radhakrishnan
Quoted 4 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
It sounds interesting to me, if we can make it work properly and reliably. + [email protected] for others in the GCC community to chip in.
Given my (albeit limited) experience playing with the C11 spec and GCC, I really think this is a bad idea for the kernel. It seems that nobody really agrees on exactly how the C11 atomics map to real architectural instructions on anything but the trivial architectures. For example, should the following code fire the assert?
extern atomic<int> foo, bar, baz;
void thread1(void)
{ foo.store(42, memory_order_relaxed); bar.fetch_add(1, memory_order_seq_cst); baz.store(42, memory_order_relaxed); }void thread2(void)
{ while (baz.load(memory_order_seq_cst) != 42) { /* do nothing */ } assert(foo.load(memory_order_seq_cst) == 42); }Quoted 5 lines
To answer that question, you need to go and look at the definitions of synchronises-with, happens-before,
dependency_ordered_beforeand a whole pile of vaguely written waffle to realise that you don't know. Certainly, the code that arm64 GCC currently spits out would allow the assertion to fire on some microarchitectures.
Quoted 4 lines
Yep! I believe that a
memory_order_seq_cstfence in combination with thefetch_add()would do the trick on many architectures, however. All of this is one reason that any C11 definitions need to be individually overridable by individual architectures.
"Overridable" in which sense? Do you want to change the semantics on the language level in the sense of altering the memory model, or rather use a different implementation under the hood to, for example, fix deficiencies in the compilers?
Quoted 10 lines
Quoted 9 lines
There are also so many ways to blow your head off it's untrue. For example, cmpxchg takes a separate memory model parameter for failure and success, but then there are restrictions on the sets you can use for each. It's not hard to find well-known memory-ordering experts shouting "Just use
memory_model_seq_cstfor everything, it's too hard otherwise". Then there's the fun of load-consume vs load-acquire (arm64 GCC completely ignores consume atm and optimises all of the data dependencies away) as well as the definition of "data races", which seem to be used as an excuse to miscompile a program at the earliest opportunity.
Quoted 4 lines
Trust me,
rcu_dereference()is not going to be defined in terms ofmemory_order_consumeuntil the compilers implement it both correctly and efficiently. They are not there yet, and there is currently no shortage of compiler writers who would prefer to ignorememory_order_consume.
Do you have any input on
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59448? In particular, the language standard's definition of dependencies?
Quoted 10 lines
And
rcu_dereference()will need per-arch overrides for some time during any transition tomemory_order_consume.Quoted 3 lines
Trying to introduce system concepts (writes to devices, interrupts, non-coherent agents) into this mess is going to be an uphill battle IMHO. I'd just rather stick to the semantics we have and the asm volatile barriers.
And
barrier()isn't going to go away any time soon, either. AndACCESS_ONCE()needs to keep volatile semantics until there is somememory_order_whateverthat prevents loads and stores from being coalesced.
I'd be happy to discuss something like this in ISO C++ SG1 (or has this been discussed in the past already?). But it needs to have a paper I suppose.
Will you be in Issaquah for the C++ meeting next week?
Quoted 50 lines · Torvald Riegel
Quoted 44 lines · Paul E. McKenney
Quoted 37 lines · Will Deacon
Quoted 9 lines · Ramana Radhakrishnan
Quoted 4 lines · David Howells
Is it worth considering a move towards using C11 atomics and barriers and compiler intrinsics inside the kernel? The compiler _ought_ to be able to do these.
It sounds interesting to me, if we can make it work properly and reliably. + [email protected] for others in the GCC community to chip in.
Given my (albeit limited) experience playing with the C11 spec and GCC, I really think this is a bad idea for the kernel. It seems that nobody really agrees on exactly how the C11 atomics map to real architectural instructions on anything but the trivial architectures. For example, should the following code fire the assert?
extern atomic<int> foo, bar, baz;
void thread1(void)
{ foo.store(42, memory_order_relaxed); bar.fetch_add(1, memory_order_seq_cst); baz.store(42, memory_order_relaxed); }void thread2(void)
{ while (baz.load(memory_order_seq_cst) != 42) { /* do nothing */ } assert(foo.load(memory_order_seq_cst) == 42); }Quoted 5 lines
To answer that question, you need to go and look at the definitions of synchronises-with, happens-before,
dependency_ordered_beforeand a whole pile of vaguely written waffle to realise that you don't know. Certainly, the code that arm64 GCC currently spits out would allow the assertion to fire on some microarchitectures.Quoted 4 lines
Yep! I believe that a
memory_order_seq_cstfence in combination with thefetch_add()would do the trick on many architectures, however. All of this is one reason that any C11 definitions need to be individually overridable by individual architectures.
Quoted 4 lines
"Overridable" in which sense? Do you want to change the semantics on the language level in the sense of altering the memory model, or rather use a different implementation under the hood to, for example, fix deficiencies in the compilers?
We need the architecture maintainer to be able to select either an assembly-language implementation or a C11-atomics implementation for any given Linux-kernel operation. For example, a given architecture might be able to use fetch_add(1, memory_order_relaxed) for atomic_inc() but assembly for atomic_add_return(). This is because atomic_inc() is not required to have any particular ordering properties, while as discussed previously, atomic_add_return() requires tighter ordering than the C11 standard provides.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
There are also so many ways to blow your head off it's untrue. For example, cmpxchg takes a separate memory model parameter for failure and success, but then there are restrictions on the sets you can use for each. It's not hard to find well-known memory-ordering experts shouting "Just use
memory_model_seq_cstfor everything, it's too hard otherwise". Then there's the fun of load-consume vs load-acquire (arm64 GCC completely ignores consume atm and optimises all of the data dependencies away) as well as the definition of "data races", which seem to be used as an excuse to miscompile a program at the earliest opportunity.
Quoted 8 lines
Quoted 4 lines
Trust me,
rcu_dereference()is not going to be defined in terms ofmemory_order_consumeuntil the compilers implement it both correctly and efficiently. They are not there yet, and there is currently no shortage of compiler writers who would prefer to ignorememory_order_consume.Do you have any input on
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59448? In particular, the language standard's definition of dependencies?
Let's see... 1.10p9 says that a dependency must be carried unless:
— B is an invocation of any specialization of std::kill_dependency (29.3), or
— A is the left operand of a built-in logical AND (&&, see 5.14) or logical OR (||, see 5.15) operator, or
— A is the left operand of a conditional (?:, see 5.16) operator, or
— A is the left operand of the built-in comma (,) operator (5.18);
So the use of "flag" before the "?" is ignored. But the "flag - flag" after the "?" will carry a dependency, so the code fragment in 59448 needs to do the ordering rather than just optimizing "flag - flag" out of existence. One way to do that on both ARM and Power is to actually emit code for "flag - flag", but there are a number of other ways to make that work.
BTW, there is some discussion on 1.10p9's handling of && and ||, and that clause is likely to change. And yes, I am behind on analyzing usage in the Linux kernel to find out if Linux cares...
Quoted 7 lines
Quoted 7 lines
And
rcu_dereference()will need per-arch overrides for some time during any transition tomemory_order_consume.Quoted 3 lines
Trying to introduce system concepts (writes to devices, interrupts, non-coherent agents) into this mess is going to be an uphill battle IMHO. I'd just rather stick to the semantics we have and the asm volatile barriers.
Quoted 7 lines
Quoted 3 lines
And
barrier()isn't going to go away any time soon, either. AndACCESS_ONCE()needs to keep volatile semantics until there is somememory_order_whateverthat prevents loads and stores from being coalesced.I'd be happy to discuss something like this in ISO C++ SG1 (or has this been discussed in the past already?). But it needs to have a paper I suppose.
The current position of the usual suspects other than me is that this falls into the category of forward-progress guarantees, which are considers (again, by the usual suspects other than me) to be out of scope.
Quoted 1 line
Will you be in Issaquah for the C++ meeting next week?
Weather permitting, I will be there!
Thanx, Paul
Quoted 30 lines · Paul E. McKenney
Quoted 12 lines · Torvald Riegel
Quoted 11 lines · Paul E. McKenney
Quoted 9 lines · Will Deacon
There are also so many ways to blow your head off it's untrue. For example, cmpxchg takes a separate memory model parameter for failure and success, but then there are restrictions on the sets you can use for each. It's not hard to find well-known memory-ordering experts shouting "Just use
memory_model_seq_cstfor everything, it's too hard otherwise". Then there's the fun of load-consume vs load-acquire (arm64 GCC completely ignores consume atm and optimises all of the data dependencies away) as well as the definition of "data races", which seem to be used as an excuse to miscompile a program at the earliest opportunity.Quoted 8 lines
Quoted 4 lines
Trust me,
rcu_dereference()is not going to be defined in terms ofmemory_order_consumeuntil the compilers implement it both correctly and efficiently. They are not there yet, and there is currently no shortage of compiler writers who would prefer to ignorememory_order_consume.Do you have any input on
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59448? In particular, the language standard's definition of dependencies?Let's see... 1.10p9 says that a dependency must be carried unless:
— B is an invocation of any specialization of
std::kill_dependency(29.3), or
— A is the left operand of a built-in logical AND (&&, see 5.14) or logical OR (||, see 5.15) operator, or
— A is the left operand of a conditional (?:, see 5.16) operator, or
— A is the left operand of the built-in comma (,) operator (5.18);
Quoted 6 lines
So the use of "flag" before the "?" is ignored. But the "flag - flag" after the "?" will carry a dependency, so the code fragment in 59448 needs to do the ordering rather than just optimizing "flag - flag" out of existence. One way to do that on both ARM and Power is to actually emit code for "flag - flag", but there are a number of other ways to make that work.
And that's what would concern me, considering that these requirements seem to be able to creep out easily. Also, whereas the other atomics just constrain compilers wrt. reordering across atomic accesses or changes to the atomic accesses themselves, the dependencies are new requirements on pieces of otherwise non-synchronizing code. The latter seems far more involved to me.
Quoted 3 lines
BTW, there is some discussion on 1.10p9's handling of && and ||, and that clause is likely to change. And yes, I am behind on analyzing usage in the Linux kernel to find out if Linux cares...
Do you have any pointers to these discussions (e.g., LWG issues)?
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
And
rcu_dereference()will need per-arch overrides for some time during any transition tomemory_order_consume.Quoted 3 lines
Trying to introduce system concepts (writes to devices, interrupts, non-coherent agents) into this mess is going to be an uphill battle IMHO. I'd just rather stick to the semantics we have and the asm volatile barriers.
And
barrier()isn't going to go away any time soon, either. AndACCESS_ONCE()needs to keep volatile semantics until there is somememory_order_whateverthat prevents loads and stores from being coalesced.
Quoted 8 lines
Quoted 3 lines
I'd be happy to discuss something like this in ISO C++ SG1 (or has this been discussed in the past already?). But it needs to have a paper I suppose.
The current position of the usual suspects other than me is that this falls into the category of forward-progress guarantees, which are considers (again, by the usual suspects other than me) to be out of scope.
But I think we need to better describe forward progress, even though that might be tricky. We made at least some progress on
http://cplusplus.github.io/LWG/lwg-active.html#2159 in Chicago, even though we can't constrain the OS schedulers too much, and for lock-free we're in this weird position that on most general-purpose schedulers and machines, obstruction-free algorithms are likely to work just fine like lock-free, most of the time, in practice...
We also need to discuss forward progress guarantees for any parallelism/concurrency abstractions, I believe:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3874.pdf
Hopefully we'll get some more acceptance of this being in scope...
Quoted 3 lines
Quoted 1 line
Will you be in Issaquah for the C++ meeting next week?
Weather permitting, I will be there!
Great, maybe we can find some time in SG1 to discuss this then. Even if the standard doesn't want to include it, SG1 should be a good forum to understand everyone's concerns around that, with the hope that this would help potential non-standard extensions to be still checked by the same folks that did the rest of the memory model.
Quoted 38 lines · Torvald Riegel
Quoted 30 lines · Paul E. McKenney
Quoted 12 lines · Torvald Riegel
Quoted 11 lines · Paul E. McKenney
Quoted 9 lines · Will Deacon
There are also so many ways to blow your head off it's untrue. For example, cmpxchg takes a separate memory model parameter for failure and success, but then there are restrictions on the sets you can use for each. It's not hard to find well-known memory-ordering experts shouting "Just use
memory_model_seq_cstfor everything, it's too hard otherwise". Then there's the fun of load-consume vs load-acquire (arm64 GCC completely ignores consume atm and optimises all of the data dependencies away) as well as the definition of "data races", which seem to be used as an excuse to miscompile a program at the earliest opportunity.Quoted 8 lines
Quoted 4 lines
Trust me,
rcu_dereference()is not going to be defined in terms ofmemory_order_consumeuntil the compilers implement it both correctly and efficiently. They are not there yet, and there is currently no shortage of compiler writers who would prefer to ignorememory_order_consume.Do you have any input on
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59448? In particular, the language standard's definition of dependencies?Let's see... 1.10p9 says that a dependency must be carried unless:
— B is an invocation of any specialization of
std::kill_dependency(29.3), or
— A is the left operand of a built-in logical AND (&&, see 5.14) or logical OR (||, see 5.15) operator, or
— A is the left operand of a conditional (?:, see 5.16) operator, or
— A is the left operand of the built-in comma (,) operator (5.18);Quoted 6 lines
So the use of "flag" before the "?" is ignored. But the "flag - flag" after the "?" will carry a dependency, so the code fragment in 59448 needs to do the ordering rather than just optimizing "flag - flag" out of existence. One way to do that on both ARM and Power is to actually emit code for "flag - flag", but there are a number of other ways to make that work.
Quoted 6 lines
And that's what would concern me, considering that these requirements seem to be able to creep out easily. Also, whereas the other atomics just constrain compilers wrt. reordering across atomic accesses or changes to the atomic accesses themselves, the dependencies are new requirements on pieces of otherwise non-synchronizing code. The latter seems far more involved to me.
Well, the wording of 1.10p9 is pretty explicit on this point. There are only a few exceptions to the rule that dependencies from memory_order_consume loads must be tracked. And to your point about requirements being placed on pieces of otherwise non-synchronizing code, we already have that with plain old load acquire and store release -- both of these put ordering constraints that affect the surrounding non-synchronizing code.
This issue got a lot of discussion, and the compromise is that dependencies cannot leak into or out of functions unless the relevant parameters or return values are annotated with [[carries_dependency]]. This means that the compiler can see all the places where dependencies must be tracked. This is described in 7.6.4. If a dependency chain headed by a memory_order_consume load goes into or out of a function without the aid of the [[carries_dependency]] attribute, the compiler needs to do something else to enforce ordering, e.g., emit a memory barrier.
From a Linux-kernel viewpoint, this is a bit ugly, as it requires annotations and use of kill_dependency, but it was the best I could do at the time. If things go as they usually do, there will be some other reason why those are needed...
Quoted 5 lines
Quoted 3 lines
BTW, there is some discussion on 1.10p9's handling of && and ||, and that clause is likely to change. And yes, I am behind on analyzing usage in the Linux kernel to find out if Linux cares...
Do you have any pointers to these discussions (e.g., LWG issues)?
Nope, just a bare email thread. I would guess that it will come up next week.
The question is whether dependencies should be carried through && or || at all, and if so how. My current guess is that && and || should not carry dependencies.
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
And
rcu_dereference()will need per-arch overrides for some time during any transition tomemory_order_consume.Quoted 3 lines
Trying to introduce system concepts (writes to devices, interrupts, non-coherent agents) into this mess is going to be an uphill battle IMHO. I'd just rather stick to the semantics we have and the asm volatile barriers.
And
barrier()isn't going to go away any time soon, either. AndACCESS_ONCE()needs to keep volatile semantics until there is somememory_order_whateverthat prevents loads and stores from being coalesced.Quoted 8 lines
Quoted 3 lines
I'd be happy to discuss something like this in ISO C++ SG1 (or has this been discussed in the past already?). But it needs to have a paper I suppose.
The current position of the usual suspects other than me is that this falls into the category of forward-progress guarantees, which are considers (again, by the usual suspects other than me) to be out of scope.
Quoted 7 lines
But I think we need to better describe forward progress, even though that might be tricky. We made at least some progress on
http://cplusplus.github.io/LWG/lwg-active.html#2159 in Chicago, even though we can't constrain the OS schedulers too much, and for lock-free we're in this weird position that on most general-purpose schedulers and machines, obstruction-free algorithms are likely to work just fine like lock-free, most of the time, in practice...
Yep, there is a draft paper by Alistarh et al. making this point. They could go quite a bit further. With a reasonably short set of additional constraints, you can get bounded execution times out of locking as well. They were not amused when I suggested this, Bjoern Brandenberg's dissertation notwithstanding. ;-)
Quoted 5 lines
We also need to discuss forward progress guarantees for any parallelism/concurrency abstractions, I believe:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3874.pdfHopefully we'll get some more acceptance of this being in scope...
That would be good. Just in case C11 is to be applicable to real-time software.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
Will you be in Issaquah for the C++ meeting next week?
Weather permitting, I will be there!
Great, maybe we can find some time in SG1 to discuss this then. Even if the standard doesn't want to include it, SG1 should be a good forum to understand everyone's concerns around that, with the hope that this would help potential non-standard extensions to be still checked by the same folks that did the rest of the memory model.
Sounds good! Hopefully some discussion of out-of-thin-air values as well.
Thanx, Paul
Peter Zijlstra
maintainerQuoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Thanx, Paul
Hi Paul,
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
Quoted 7 lines
Quoted 2 lines
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
Will
Peter Zijlstra
maintainerQuoted 18 lines · Will Deacon
Hi Paul,
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
Quoted 7 lines
Quoted 2 lines
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Quoted 6 lines
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
This is more about the compiler speculating on stores; imagine:
if (x) y = 1; else y = 2;
The compiler is allowed to change that into:
y = 2; if (x) y = 1;
Which is of course a big problem when you want to rely on the ordering.
There's further problems where things like memset() can write outside the specified address range. Examples are memset() using single instructions to wipe entire cachelines and then 'restoring' the tail bit.
While valid for single threaded, its a complete disaster for concurrent code.
There's more, but it all boils down to doing stores you don't expect in a 'sane' concurrent environment and/or don't respect the control flow.
Quoted 33 lines · Peter Zijlstra
Quoted 18 lines · Will Deacon
Hi Paul,
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
Quoted 7 lines
Quoted 2 lines
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Quoted 6 lines
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
This is more about the compiler speculating on stores; imagine:
if (x) y = 1; else y = 2;
Quoted 7 lines
The compiler is allowed to change that into:
y = 2; if (x) y = 1;Which is of course a big problem when you want to rely on the ordering.
Understood, but that doesn't explain why Paul wants to add ISB/isync instructions which affect the *CPU* rather than the compiler!
Will
Peter Zijlstra
maintainerQuoted 2 lines · Will Deacon
Understood, but that doesn't explain why Paul wants to add ISB/isync instructions which affect the *CPU* rather than the compiler!
I doubt Paul wants it, but yeah, I'm curious about that proposal as well, sounds like someone took a big toke from the bong again; it seems a favourite past time amongst committees.
Quoted 36 lines · Will Deacon
Quoted 35 lines · Peter Zijlstra
Quoted 18 lines · Will Deacon
Hi Paul,
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
Quoted 7 lines
Quoted 2 lines
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Quoted 6 lines
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
This is more about the compiler speculating on stores; imagine:
if (x) y = 1; else y = 2;The compiler is allowed to change that into:
Quoted 8 lines
Quoted 5 lines
y = 2; if (x) y = 1;Which is of course a big problem when you want to rely on the ordering.
Understood, but that doesn't explain why Paul wants to add ISB/isync instructions which affect the *CPU* rather than the compiler!
Hey!!! -I- don't want to add those instructions! Others do. Unfortunately, lots of others.
Thanx, Paul
Quoted 4 lines · Peter Zijlstra
There's further problems where things like
memset()can write outside the specified address range. Examples arememset()using single instructions to wipe entire cachelines and then 'restoring' the tail bit.
If memset (or any C library function) modifies bytes it's not permitted to modify in the abstract machine, that's a simple bug and should be reported as usual. We've made GCC follow that part of the memory model by default (so a store to a non-bit-field structure field doesn't do a read-modify-write to a word containing another field, for example) and I think it's pretty obvious that glibc should do so as well.
(Of course, memset is not an atomic operation, and you need to allow for that if you use it on an _Atomic object - which is I think valid, unless the object is also volatile, but perhaps ill-advised.)
Signature · 2 lines
Joseph S. Myers [email protected]
Quoted 33 lines · Peter Zijlstra
Quoted 18 lines · Will Deacon
Hi Paul,
Quoted 7 lines · Paul E. McKenney
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
Quoted 7 lines
Quoted 2 lines
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Quoted 6 lines
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
This is more about the compiler speculating on stores; imagine:
if (x) y = 1; else y = 2;
Quoted 5 lines
The compiler is allowed to change that into:
y = 2; if (x) y = 1;
If you write the example like that, this is indeed allowed because it's all sequential code (and there's no volatiles in there, at least you didn't show them :). A store to y would happen in either case. You cannot observe the difference between both examples in a data-race-free program.
Are there supposed to be atomic/non-sequential accesses in there? If so, please update the example.
Quoted 6 lines
Which is of course a big problem when you want to rely on the ordering.
There's further problems where things like
memset()can write outside the specified address range. Examples arememset()using single instructions to wipe entire cachelines and then 'restoring' the tail bit.
As Joseph said, this would be a bug IMO.
Quoted 5 lines
While valid for single threaded, its a complete disaster for concurrent code.
There's more, but it all boils down to doing stores you don't expect in a 'sane' concurrent environment and/or don't respect the control flow.
A few of those got fixed already, because they violated the memory model's requirements. If you have further examples that are valid code in the C11/C++11 model, please report them.
Quoted 13 lines · Will Deacon
Hi Paul,
Quoted 10 lines · Paul E. McKenney
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
Quoted 8 lines
Quoted 4 lines
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Requiring a compare/branch/ISB after each relaxed load enables a simple(r) proof that out-of-thin-air values cannot be observed in the face of any compiler optimization that refrains from reordering a prior relaxed load with a subsequent relaxed store.
Quoted 2 lines
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
So let me try an example. I am sure that Torvald Riegel will jump in with any needed corrections or amplifications:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(r1, y, memory_order_relaxed);
T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
One would intuitively expect r1 == r2 == 0 as the only possible outcome. But suppose that the compiler used specialization optimizations, as it would if there was a function that has a very lightweight implementation for some values and a very heavyweight one for other. In particular, suppose that the lightweight implementation was for the value 42. Then the compiler might do something like the following:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); if (r1 == 42) atomic_store_explicit(42, y, memory_order_relaxed); else atomic_store_explicit(r1, y, memory_order_relaxed);
T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
Suddenly we have an explicit constant 42 showing up. Of course, if the compiler carefully avoided speculative stores (as both Peter and I believe that it should if its code generation is to be regarded as anything other than an act of vandalism, the words in the standard notwithstanding), there would be no problem. But currently, a number of compiler writers see absolutely nothing wrong with transforming the optimized-for-42 version above with something like this:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(42, y, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);
T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
And then it is a short and uncontroversial step to the following:
Initial state: x == y == 0
T1: atomic_store_explicit(42, y, memory_order_relaxed); r1 = atomic_load_explicit(x, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);
T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
This can of course result in r1 == r2 == 42, even though the constant 42 never appeared in the original code. This is one way to generate an out-of-thin-air value.
As near as I can tell, compiler writers hate the idea of prohibiting speculative-store optimizations because it requires them to introduce both control and data dependency tracking into their compilers. Many of them seem to hate dependency tracking with a purple passion. At least, such a hatred would go a long way towards explaining the incomplete and high-overhead implementations of memory_order_consume, the long and successful use of idioms based on the memory_order_consume pattern notwithstanding [*]. ;-)
That said, the Java guys are talking about introducing something vaguely resembling memory_order_consume (and thus resembling the rcu_assign_pointer() and rcu_dereference() portions of RCU) to solve Java out-of-thin-air issues involving initialization, so perhaps there is hope.
Thanx, Paul
[*] http://queue.acm.org/detail.cfm?id=2488549
http://www.rdrop.com/users/paulmck/RCU/rclockpdcsproof.pdf
Quoted 70 lines · Paul E. McKenney
Quoted 13 lines · Will Deacon
Hi Paul,
Quoted 10 lines · Paul E. McKenney
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
Quoted 8 lines
Quoted 4 lines
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Requiring a compare/branch/ISB after each relaxed load enables a simple(r) proof that out-of-thin-air values cannot be observed in the face of any compiler optimization that refrains from reordering a prior relaxed load with a subsequent relaxed store.
Quoted 2 lines
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
So let me try an example. I am sure that Torvald Riegel will jump in with any needed corrections or amplifications:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);One would intuitively expect r1 == r2 == 0 as the only possible outcome. But suppose that the compiler used specialization optimizations, as it would if there was a function that has a very lightweight implementation for some values and a very heavyweight one for other. In particular, suppose that the lightweight implementation was for the value 42. Then the compiler might do something like the following:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); if (r1 == 42) atomic_store_explicit(42, y, memory_order_relaxed); else atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);Suddenly we have an explicit constant 42 showing up. Of course, if the compiler carefully avoided speculative stores (as both Peter and I believe that it should if its code generation is to be regarded as anything other than an act of vandalism, the words in the standard notwithstanding), there would be no problem. But currently, a number of compiler writers see absolutely nothing wrong with transforming the optimized-for-42 version above with something like this:
Initial state: x == y == 0
Quoted 7 lines
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(42, y, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
For this to be correct, the compiler would actually have to prove that the speculative store is "as-if correct", which in turn would mean that it needs to be aware of all potential observers, and check whether those observers aren't actually affected by the speculative store.
I would guess that the compilers you have in mind don't really do that. If they do, then I don't see why this should be okay, unless you think out-of-thin-air values are something good (which I wouldn't agree with).
Quoted 12 lines
And then it is a short and uncontroversial step to the following:
Initial state: x == y == 0
T1: atomic_store_explicit(42, y, memory_order_relaxed); r1 = atomic_load_explicit(x, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
Quoted 7 lines
This can of course result in r1 == r2 == 42, even though the constant 42 never appeared in the original code. This is one way to generate an out-of-thin-air value.
As near as I can tell, compiler writers hate the idea of prohibiting speculative-store optimizations because it requires them to introduce both control and data dependency tracking into their compilers.
I wouldn't characterize the situation like this (although I can't speak for others, obviously). IMHO, it's perfectly fine on sequential / non-synchronizing code, because we know the difference isn't observable by a correct program. For synchronizing code, compilers just shouldn't do it, or they would have to truly prove that speculation is harmless. That will be hard, so I think it should just be avoided.
Synchronization code will likely have been tuned anyway (especially if it uses relaxed MO), so I don't see a large need for trying to optimize using speculative atomic stores.
Thus, I think there's an easy and practical solution.
Quoted 6 lines
Many of
them seem to hate dependency tracking with a purple passion. At least, such a hatred would go a long way towards explaining the incomplete and high-overhead implementations ofmemory_order_consume, the long and successful use of idioms based on thememory_order_consumepattern notwithstanding [*]. ;-)
I still think that's different because it blurs the difference between sequential code and synchronizing code (ie, atomic accesses). With consume MO, the simple solution above doesn't work anymore, because suddenly synchronizing code does affect optimizations in sequential code, even if that wouldn't reorder across the synchronizing code (which would be clearly "visible" to the implementation of the optimization).
Quoted 5 lines · Torvald Riegel
I wouldn't characterize the situation like this (although I can't speak for others, obviously). IMHO, it's perfectly fine on sequential / non-synchronizing code, because we know the difference isn't observable by a correct program.
What BS is that? If you use an "atomic_store_explicit()", by definition you're either
(a) f*cking insane
(b) not doing sequential non-synchronizing code
and a compiler that assumes that the programmer is insane may actually be correct more often than not, but it's still a shit compiler. Agreed?
So I don't see how any sane person can say that speculative writes are ok. They are clearly not ok.
Speculative stores are a bad idea in general. They are completely invalid for anything that says "atomic". This is not even worth discussing.
Linus
Quoted 10 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
I wouldn't characterize the situation like this (although I can't speak for others, obviously). IMHO, it's perfectly fine on sequential / non-synchronizing code, because we know the difference isn't observable by a correct program.
What BS is that? If you use an "
atomic_store_explicit()", by definition you're either
Quoted 6 lines
(a) f*cking insane
(b) not doing sequential non-synchronizing codeand a compiler that assumes that the programmer is insane may actually be correct more often than not, but it's still a shit compiler. Agreed?
Due to all the expletives, I can't really understand what you are saying. Nor does what I guess it might mean seem to relate to what I said.
(a) seems to say that you don't like requiring programmers to mark atomic accesses specially. Is that the case?
If so, I have to disagree. If you're writing concurrent code, marking the atomic accesses is the least of your problem. Instead, for the concurrent code I've written, it rather improved readability; it made code more verbose, but in turn it made the synchronization easier to see.
Beside this question of taste (and I don't care what the Kernel style guides are), there is a real technical argument here, though: Requiring all synchronizing memory accesses to be annotated makes the compiler aware what is sequential code, and what is not. Without it, one can't exploit data-race-freedom. So unless we want to slow down optimization of sequential code, we need to tell the compiler what is what. If every access is potentially synchronizing, then this doesn't just prevent speculative stores.
(b) seems as if you are saying that if there is a specially annotated atomic access in the code, that this isn't sequential/non-synchronizing code anymore. I don't agree with that, obviously.
Quoted 2 lines
So I don't see how any sane person can say that speculative writes are ok. They are clearly not ok.
We are discussing programs written against the C11/C++11 memory model here. At least that's the part that got forwarded to [email protected], and was subject of the nearest emails in the thread.
This memory model requires programs to be data-race-free. Thus, we can optimize accordingly. If you don't like that, then this isn't C11/C++11 anymore. Which would be fine, but then complain about that specifically.
Quoted 1 line
Speculative stores are a bad idea in general.
I disagree in the context of C11/C++11 programs. At least from a correctness point of view.
Quoted 2 lines
They are completely
invalid for anything that says "atomic".
I agree, as you will see when you read the other emails I posted as part of this thread. But those two things are separate.
Quoted 3 lines · Torvald Riegel
(a) seems to say that you don't like requiring programmers to mark atomic accesses specially. Is that the case?
In Paul's example, they were marked specially.
And you seemed to argue that Paul's example could possibly return anything but 0/0.
If you really think that, I hope to God that you have nothing to do with the C standard or any actual compiler I ever use.
Because such a standard or compiler would be shit. It's sadly not too uncommon.
Linus
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Torvald Riegel
(a) seems to say that you don't like requiring programmers to mark atomic accesses specially. Is that the case?
In Paul's example, they were marked specially.
And you seemed to argue that Paul's example could possibly return anything but 0/0.
Just read my reply to Paul again. Here's an excerpt:
Quoted 9 lines
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(42, y, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
IOW, I wrote that such a compiler transformation would be wrong in my opinion. Thus, it should *not* return 42.
If you still see how what I wrote could be misunderstood, please let me know because I really don't see it.
Yes, I don't do so by swearing or calling other insane, and I try to see the reasoning that those compilers' authors might have had, even if I don't agree with it. In my personal opinion, that's a good thing.
Quoted 4 lines
If you really think that, I hope to God that you have nothing to do with the C standard or any actual compiler I ever use.
Because such a standard or compiler would be shit. It's sadly not too uncommon.
Thanks for the kind words. Perhaps writing that was quicker than reading what I actually wrote.
Quoted 3 lines · Torvald Riegel
IOW, I wrote that such a compiler transformation would be wrong in my opinion. Thus, it should *not* return 42.
Ahh, I am happy to have misunderstood. The "intuitively" threw me, because I thought that was building up to a "but", and misread the rest.
I then react stronly, because I've seen so much total crap (the type-based C aliasing rules topping my list) etc coming out of standards groups because it allows them to generate wrong code that goes faster, that I just assume compiler people are out to do stupid things in the name of "..but the standard allows it".
Linus
Quoted 84 lines · Torvald Riegel
Quoted 70 lines · Paul E. McKenney
Quoted 13 lines · Will Deacon
Hi Paul,
Quoted 10 lines · Paul E. McKenney
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
Hopefully some discussion of out-of-thin-air values as well.
Yes, absolutely shoot store speculation in the head already. Then drive a wooden stake through its hart.
C11/C++11 should not be allowed to claim itself a memory model until that is sorted.
Quoted 8 lines
Quoted 4 lines
There actually is a proposal being put forward, but it might not make ARM and Power people happy because it involves adding a compare, a branch, and an ISB/isync after every relaxed load... Me, I agree with you, much preferring the no-store-speculation approach.
Can you elaborate a bit on this please? We don't permit speculative stores in the ARM architecture, so it seems counter-intuitive that GCC needs to emit any additional instructions to prevent that from happening.
Requiring a compare/branch/ISB after each relaxed load enables a simple(r) proof that out-of-thin-air values cannot be observed in the face of any compiler optimization that refrains from reordering a prior relaxed load with a subsequent relaxed store.
Quoted 2 lines
Stores can, of course, be observed out-of-order but that's a lot more reasonable :)
So let me try an example. I am sure that Torvald Riegel will jump in with any needed corrections or amplifications:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);One would intuitively expect r1 == r2 == 0 as the only possible outcome. But suppose that the compiler used specialization optimizations, as it would if there was a function that has a very lightweight implementation for some values and a very heavyweight one for other. In particular, suppose that the lightweight implementation was for the value 42. Then the compiler might do something like the following:
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); if (r1 == 42) atomic_store_explicit(42, y, memory_order_relaxed); else atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);Suddenly we have an explicit constant 42 showing up. Of course, if the compiler carefully avoided speculative stores (as both Peter and I believe that it should if its code generation is to be regarded as anything other than an act of vandalism, the words in the standard notwithstanding), there would be no problem. But currently, a number of compiler writers see absolutely nothing wrong with transforming the optimized-for-42 version above with something like this:
Initial state: x == y == 0
Quoted 7 lines
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(42, y, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Quoted 8 lines
For this to be correct, the compiler would actually have to prove that the speculative store is "as-if correct", which in turn would mean that it needs to be aware of all potential observers, and check whether those observers aren't actually affected by the speculative store.
I would guess that the compilers you have in mind don't really do that. If they do, then I don't see why this should be okay, unless you think out-of-thin-air values are something good (which I wouldn't agree with).
OK, we agree that pulling the atomic store to y out of its "if" statement is a bad thing. Very good! Now we just have to convince others on the committee. ;-)
Thanx, Paul
Quoted 2 lines · Torvald Riegel
Quoted 1 line · Paul E. McKenney
On Fri, Feb 07, 2014 at 04:55:48PM +0000, Will Deacon wrote:
[ . . . ]
Quoted 27 lines
Quoted 12 lines
And then it is a short and uncontroversial step to the following:
Initial state: x == y == 0
T1: atomic_store_explicit(42, y, memory_order_relaxed); r1 = atomic_load_explicit(x, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);Quoted 7 lines
This can of course result in r1 == r2 == 42, even though the constant 42 never appeared in the original code. This is one way to generate an out-of-thin-air value.
As near as I can tell, compiler writers hate the idea of prohibiting speculative-store optimizations because it requires them to introduce both control and data dependency tracking into their compilers.
I wouldn't characterize the situation like this (although I can't speak for others, obviously). IMHO, it's perfectly fine on sequential / non-synchronizing code, because we know the difference isn't observable by a correct program. For synchronizing code, compilers just shouldn't do it, or they would have to truly prove that speculation is harmless. That will be hard, so I think it should just be avoided.
Quoted 5 lines
Synchronization code will likely have been tuned anyway (especially if it uses relaxed MO), so I don't see a large need for trying to optimize using speculative atomic stores.
Thus, I think there's an easy and practical solution.
I like this approach, but there has been resistance to it in the past. Definitely worth a good try, though!
Quoted 7 lines
Quoted 6 lines
Many of
them seem to hate dependency tracking with a purple passion. At least, such a hatred would go a long way towards explaining the incomplete and high-overhead implementations ofmemory_order_consume, the long and successful use of idioms based on thememory_order_consumepattern notwithstanding [*]. ;-)
Quoted 6 lines
I still think that's different because it blurs the difference between sequential code and synchronizing code (ie, atomic accesses). With consume MO, the simple solution above doesn't work anymore, because suddenly synchronizing code does affect optimizations in sequential code, even if that wouldn't reorder across the synchronizing code (which would be clearly "visible" to the implementation of the optimization).
I understand that memory_order_consume is a bit harder on compiler writers than the other memory orders, but it is also pretty valuable.
Thanx, Paul
Peter Zijlstra
maintainerQuoted 7 lines · Torvald Riegel
Quoted 7 lines
Initial state: x == y == 0
T1: r1 = atomic_load_explicit(x, memory_order_relaxed); atomic_store_explicit(42, y, memory_order_relaxed); if (r1 != 42) atomic_store_explicit(r1, y, memory_order_relaxed);
Quoted 7 lines
Quoted 2 lines
T2: r2 = atomic_load_explicit(y, memory_order_relaxed); atomic_store_explicit(r2, x, memory_order_relaxed);Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Yeah, my bad for not being familiar with the atrocious crap C11 made of atomics :/
Quoted 5 lines · Torvald Riegel
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Btw, what is the definition of "observable" for the atomics?
Because I'm hoping that it's not the same as for volatiles, where "observable" is about the virtual machine itself, and as such volatile accesses cannot be combined or optimized at all.
Now, I claim that atomic accesses cannot be done speculatively for writes, and not re-done for reads (because the value could change), but *combining* them would be possible and good.
For example, we often have multiple independent atomic accesses that could certainly be combined: testing the individual bits of an atomic value with helper functions, causing things like "load atomic, test bit, load same atomic, test another bit". The two atomic loads could be done as a single load without possibly changing semantics on a real machine, but if "visibility" is defined in the same way it is for "volatile", that wouldn't be a valid transformation. Right now we use "volatile" semantics for these kinds of things, and they really can hurt.
Same goes for multiple writes (possibly due to setting bits):
combining multiple accesses into a single one is generally fine, it's *adding* write accesses speculatively that is broken by design..
At the same time, you can't combine atomic loads or stores infinitely
- "visibility" on a real machine definitely is about timeliness. Removing all but the last write when there are multiple consecutive writes is generally fine, even if you unroll a loop to generate those writes. But if what remains is a loop, it might be a busy-loop basically waiting for something, so it would be wrong ("untimely") to hoist a store in a loop entirely past the end of the loop, or hoist a load in a loop to before the loop.
Does the standard allow for that kind of behavior?
Linus
Quoted 40 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Btw, what is the definition of "observable" for the atomics?
Because I'm hoping that it's not the same as for volatiles, where "observable" is about the virtual machine itself, and as such volatile accesses cannot be combined or optimized at all.
Now, I claim that atomic accesses cannot be done speculatively for writes, and not re-done for reads (because the value could change), but *combining* them would be possible and good.
For example, we often have multiple independent atomic accesses that could certainly be combined: testing the individual bits of an atomic value with helper functions, causing things like "load atomic, test bit, load same atomic, test another bit". The two atomic loads could be done as a single load without possibly changing semantics on a real machine, but if "visibility" is defined in the same way it is for "volatile", that wouldn't be a valid transformation. Right now we use "volatile" semantics for these kinds of things, and they really can hurt.
Same goes for multiple writes (possibly due to setting bits):
combining multiple accesses into a single one is generally fine, it's *adding* write accesses speculatively that is broken by design..At the same time, you can't combine atomic loads or stores infinitely
- "visibility" on a real machine definitely is about timeliness. Removing all but the last write when there are multiple consecutive writes is generally fine, even if you unroll a loop to generate those writes. But if what remains is a loop, it might be a busy-loop basically waiting for something, so it would be wrong ("untimely") to hoist a store in a loop entirely past the end of the loop, or hoist a load in a loop to before the loop.
Quoted 1 line
Does the standard allow for that kind of behavior?
You asked! ;-)
So the current standard allows merging of both loads and stores, unless of course ordring constraints prevent the merging. Volatile semantics may be used to prevent this merging, if desired, for example, for real-time code. Infinite merging is intended to be prohibited, but I am not certain that the current wording is bullet-proof (1.10p24 and 1.10p25).
The only prohibition against speculative stores that I can see is in a non-normative note, and it can be argued to apply only to things that are not atomics (1.10p22). I don't see any prohibition against reordering a store to precede a load preceding a conditional branch -- which would not be speculative if the branch was know to be taken and the load hit in the store buffer. In a system where stores could be reordered, some other CPU might perceive the store as happening before the load that controlled the conditional branch. This needs to be addressed.
Why this hole? At the time, the current formalizations of popular CPU architectures did not exist, and it was not clear that all popular hardware avoided speculative stores.
There is also fun with "out of thin air" values, which everyone agrees should be prohibited, but where there is not agreement on how to prohibit them in a mathematically constructive manner. The current draft contains a clause simply stating that out-of-thin-air values are prohibited, which doesn't help someone constructing tools to analyze C++ code. One proposal requires that subsequent atomic stores never be reordered before prior atomic loads, which requires useless ordering code to be emitted on ARM and PowerPC (you may have seen Will Deacon's and Peter Zijlstra's reaction to this proposal a few days ago). Note that Itanium already pays this price in order to provide full single-variable cache coherence. This out-of-thin-air discussion is also happening in the Java community in preparation for a new rev of the Java memory model.
There will also be some discussions on memory_order_consume, which is intended to (eventually) implement rcu_dereference(). The compiler writers don't like tracking dependencies, but there may be some ways of constraining optimizations to preserve the common dependencies that, while providing some syntax to force preservation of dependencies that would normally be optimized out. One example of this is where you have an RCU-protected array that might sometimes contain only a single element. In the single-element case, the compiler knows a priori which element will be used, and will therefore optimize the dependency away, so that the reader might see pre-initialization state. But this is rare, so if added syntax needs to be added in this case, I believe we should be OK with it. (If syntax is needed for plain old dereferences, it is thumbs down all the way as far as I am concerned. Ditto for things like stripping the bottom bits off of a decorated pointer.)
No doubt other memory-model issues will come up, but those are the ones I know about at the moment.
As I said to begin with, hey, you asked!
That said, I would very much appreciate any thoughts or suggestions on handling these issues.
Thanx, Paul
Quoted 41 lines · Paul E. McKenney
Quoted 40 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Btw, what is the definition of "observable" for the atomics?
Because I'm hoping that it's not the same as for volatiles, where "observable" is about the virtual machine itself, and as such volatile accesses cannot be combined or optimized at all.
Now, I claim that atomic accesses cannot be done speculatively for writes, and not re-done for reads (because the value could change), but *combining* them would be possible and good.
For example, we often have multiple independent atomic accesses that could certainly be combined: testing the individual bits of an atomic value with helper functions, causing things like "load atomic, test bit, load same atomic, test another bit". The two atomic loads could be done as a single load without possibly changing semantics on a real machine, but if "visibility" is defined in the same way it is for "volatile", that wouldn't be a valid transformation. Right now we use "volatile" semantics for these kinds of things, and they really can hurt.
Same goes for multiple writes (possibly due to setting bits):
combining multiple accesses into a single one is generally fine, it's *adding* write accesses speculatively that is broken by design..At the same time, you can't combine atomic loads or stores infinitely
- "visibility" on a real machine definitely is about timeliness. Removing all but the last write when there are multiple consecutive writes is generally fine, even if you unroll a loop to generate those writes. But if what remains is a loop, it might be a busy-loop basically waiting for something, so it would be wrong ("untimely") to hoist a store in a loop entirely past the end of the loop, or hoist a load in a loop to before the loop.
Quoted 7 lines
Quoted 1 line
Does the standard allow for that kind of behavior?
You asked! ;-)
So the current standard allows merging of both loads and stores, unless of course ordring constraints prevent the merging. Volatile semantics may be used to prevent this merging, if desired, for example, for real-time code.
Agreed.
Quoted 2 lines
Infinite merging is intended to be prohibited, but I am not certain that the current wording is bullet-proof (1.10p24 and 1.10p25).
Yeah, maybe not. But it at least seems to rather clearly indicate the intent ;)
Quoted 3 lines
The only prohibition against speculative stores that I can see is in a non-normative note, and it can be argued to apply only to things that are not atomics (1.10p22).
I think this one is specifically about speculative stores that would affect memory locations that the abstract machine would not write to, and that might be observable or create data races. While a compiler could potentially prove that such stores aren't leading to a difference in the behavior of the program (e.g., by proving that there are no observers anywhere and this isn't overlapping with any volatile locations), I think that this is hard in general and most compilers will just not do such things. In GCC, bugs in that category were fixed after researchers doing fuzz-testing found them (IIRC, speculative stores by loops).
Quoted 6 lines
I don't see any prohibition against reordering
a store to precede a load preceding a conditional branch -- which would not be speculative if the branch was know to be taken and the load hit in the store buffer. In a system where stores could be reordered, some other CPU might perceive the store as happening before the load that controlled the conditional branch. This needs to be addressed.
I don't know the specifics of your example, but from how I understand it, I don't see a problem if the compiler can prove that the store will always happen.
To be more specific, if the compiler can prove that the store will happen anyway, and the region of code can be assumed to always run atomically (e.g., there's no loop or such in there), then it is known that we have one atomic region of code that will always perform the store, so we might as well do the stuff in the region in some order.
Now, if any of the memory accesses are atomic, then the whole region of code containing those accesses is often not atomic because other threads might observe intermediate results in a data-race-free way.
(I know that this isn't a very precise formulation, but I hope it brings my line of reasoning across.)
Quoted 3 lines
Why this hole? At the time, the current formalizations of popular CPU architectures did not exist, and it was not clear that all popular hardware avoided speculative stores.
I'm not quite sure which hole you see there. Can you elaborate?
Peter Zijlstra
maintainerQuoted 10 lines
I don't know the specifics of your example, but from how I understand it, I don't see a problem if the compiler can prove that the store will always happen.
To be more specific, if the compiler can prove that the store will happen anyway, and the region of code can be assumed to always run atomically (e.g., there's no loop or such in there), then it is known that we have one atomic region of code that will always perform the store, so we might as well do the stuff in the region in some order.
Quoted 6 lines
Now, if any of the memory accesses are atomic, then the whole region of code containing those accesses is often not atomic because other threads might observe intermediate results in a data-race-free way.
(I know that this isn't a very precise formulation, but I hope it brings my line of reasoning across.)
So given something like:
if (x) y = 3;
assuming both x and y are atomic (so don't gimme crap for now knowing the C11 atomic incantations); and you can prove x is always true; you don't see a problem with not emitting the conditional?
Avoiding the conditional changes the result; see that control dependency email from earlier. In the above example the load of X and the store to Y are strictly ordered, due to control dependencies. Not emitting the condition and maybe not even emitting the load completely wrecks this.
Its therefore an invalid optimization to take out the conditional or speculate the store, since it takes out the dependency.
Quoted 17 lines · Peter Zijlstra
Quoted 10 lines
I don't know the specifics of your example, but from how I understand it, I don't see a problem if the compiler can prove that the store will always happen.
To be more specific, if the compiler can prove that the store will happen anyway, and the region of code can be assumed to always run atomically (e.g., there's no loop or such in there), then it is known that we have one atomic region of code that will always perform the store, so we might as well do the stuff in the region in some order.
Quoted 6 lines
Now, if any of the memory accesses are atomic, then the whole region of code containing those accesses is often not atomic because other threads might observe intermediate results in a data-race-free way.
(I know that this isn't a very precise formulation, but I hope it brings my line of reasoning across.)
Quoted 8 lines
So given something like:
if (x) y = 3;assuming both x and y are atomic (so don't gimme crap for now knowing the C11 atomic incantations); and you can prove x is always true; you don't see a problem with not emitting the conditional?
You need volatile semantics to force the compiler to ignore any proofs it might otherwise attempt to construct. Hence all the ACCESS_ONCE() calls in my email to Torvald. (Hopefully I translated your example reasonably.)
Thanx, Paul
Quoted 8 lines
Avoiding the conditional changes the result; see that control dependency email from earlier. In the above example the load of X and the store to Y are strictly ordered, due to control dependencies. Not emitting the condition and maybe not even emitting the load completely wrecks this.
Its therefore an invalid optimization to take out the conditional or speculate the store, since it takes out the dependency.
Peter Zijlstra
maintainerQuoted 4 lines · Paul E. McKenney
You need volatile semantics to force the compiler to ignore any proofs it might otherwise attempt to construct. Hence all the
ACCESS_ONCE()calls in my email to Torvald. (Hopefully I translated your example reasonably.)
My brain gave out for today; but it did appear to have the right structure.
I would prefer it C11 would not require the volatile casts. It should simply _never_ speculate with atomic writes, volatile or not.
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Paul E. McKenney
You need volatile semantics to force the compiler to ignore any proofs it might otherwise attempt to construct. Hence all the
ACCESS_ONCE()calls in my email to Torvald. (Hopefully I translated your example reasonably.)My brain gave out for today; but it did appear to have the right structure.
I can relate. ;-)
Quoted 2 lines
I would prefer it C11 would not require the volatile casts. It should simply _never_ speculate with atomic writes, volatile or not.
I agree with not needing volatiles to prevent speculated writes. However, they will sometimes be needed to prevent excessive load/store combining. The compiler doesn't have the runtime feedback mechanisms that the hardware has, and thus will need help from the developer from time to time.
Or maybe the Linux kernel simply waits to transition to C11 relaxed atomics until the compiler has learned to be sufficiently conservative in its load-store combining decisions.
Thanx, Paul
Quoted 25 lines · Paul E. McKenney
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
You need volatile semantics to force the compiler to ignore any
proofs
Quoted 1 line
Quoted 1 line
it might otherwise attempt to construct. Hence all the
ACCESS_ONCE()Quoted 5 lines
Quoted 2 lines
calls in my email to Torvald. (Hopefully I translated your example reasonably.)
My brain gave out for today; but it did appear to have the right structure.
I can relate. ;-)
Quoted 2 lines
I would prefer it C11 would not require the volatile casts. It should simply _never_ speculate with atomic writes, volatile or not.
I agree with not needing volatiles to prevent speculated writes. However,
they will sometimes be needed to prevent excessive load/store combining.
The compiler doesn't have the runtime feedback mechanisms that the hardware has, and thus will need help from the developer from time to time.
Quoted 4 lines
Or maybe the Linux kernel simply waits to transition to C11 relaxed atomics
until the compiler has learned to be sufficiently conservative in its load-store combining decisions.
Sounds backwards. Currently the compiler does nothing to the atomics. I'm sure we'll eventually add something. But if testing coverage is zero outside then surely things get worse, not better with time.
Richard.
Quoted 1 line
Thanx, Paul
Quoted 26 lines · Richard Biener
Quoted 25 lines · Paul E. McKenney
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
You need volatile semantics to force the compiler to ignore any
proofs
Quoted 1 line
Quoted 1 line
it might otherwise attempt to construct. Hence all the
ACCESS_ONCE()Quoted 5 lines
Quoted 2 lines
calls in my email to Torvald. (Hopefully I translated your example reasonably.)
My brain gave out for today; but it did appear to have the right structure.
I can relate. ;-)
Quoted 2 lines
I would prefer it C11 would not require the volatile casts. It should simply _never_ speculate with atomic writes, volatile or not.
I agree with not needing volatiles to prevent speculated writes. However,
they will sometimes be needed to prevent excessive load/store combining.
The compiler doesn't have the runtime feedback mechanisms that the hardware has, and thus will need help from the developer from time to time.
Quoted 6 lines
Quoted 4 lines
Or maybe the Linux kernel simply waits to transition to C11 relaxed atomics
until the compiler has learned to be sufficiently conservative in its load-store combining decisions.Sounds backwards. Currently the compiler does nothing to the atomics. I'm sure we'll eventually add something. But if testing coverage is zero outside then surely things get worse, not better with time.
Perhaps we solve this chicken-and-egg problem by creating a test suite?
Thanx, Paul
Quoted 27 lines · Paul E. McKenney
Quoted 26 lines · Richard Biener
Quoted 25 lines · Paul E. McKenney
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney
You need volatile semantics to force the compiler to ignore any
proofs
Quoted 1 line
Quoted 1 line
it might otherwise attempt to construct. Hence all the
ACCESS_ONCE()Quoted 5 lines
Quoted 2 lines
calls in my email to Torvald. (Hopefully I translated your example reasonably.)
My brain gave out for today; but it did appear to have the right structure.
I can relate. ;-)
Quoted 2 lines
I would prefer it C11 would not require the volatile casts. It should simply _never_ speculate with atomic writes, volatile or not.
I agree with not needing volatiles to prevent speculated writes. However,
they will sometimes be needed to prevent excessive load/store combining.
The compiler doesn't have the runtime feedback mechanisms that the hardware has, and thus will need help from the developer from time to time.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Or maybe the Linux kernel simply waits to transition to C11 relaxed atomics
until the compiler has learned to be sufficiently conservative in its load-store combining decisions.Sounds backwards. Currently the compiler does nothing to the atomics. I'm sure we'll eventually add something. But if testing coverage is zero outside then surely things get worse, not better with time.
Perhaps we solve this chicken-and-egg problem by creating a test suite?
Perhaps. The test suite might also be a good set of examples showing which cases we expect to be optimized in a certain way, and which not. I suppose the uses of (the equivalent) of atomics in the kernel would be a good start.
Quoted 17 lines · Peter Zijlstra
Quoted 10 lines
I don't know the specifics of your example, but from how I understand it, I don't see a problem if the compiler can prove that the store will always happen.
To be more specific, if the compiler can prove that the store will happen anyway, and the region of code can be assumed to always run atomically (e.g., there's no loop or such in there), then it is known that we have one atomic region of code that will always perform the store, so we might as well do the stuff in the region in some order.
Quoted 6 lines
Now, if any of the memory accesses are atomic, then the whole region of code containing those accesses is often not atomic because other threads might observe intermediate results in a data-race-free way.
(I know that this isn't a very precise formulation, but I hope it brings my line of reasoning across.)
Quoted 8 lines
So given something like:
if (x) y = 3;assuming both x and y are atomic (so don't gimme crap for now knowing the C11 atomic incantations); and you can prove x is always true; you don't see a problem with not emitting the conditional?
That depends on what your goal is. It would be correct as far as the standard is specified; this makes sense if all you want is indeed a program that does what the abstract machine might do, and produces the same output / side effects.
If you're trying to preserve the branch in the code emitted / executed by the implementation, then it would not be correct. But those branches aren't specified as being part of the observable side effects. In the common case, this makes sense because it enables optimizations that are useful; this line of reasoning also allows the compiler to merge some atomic accesses in the way that Linus would like to see it.
Quoted 2 lines
Avoiding the conditional changes the result; see that control dependency email from earlier.
It does not regarding how the standard defines "result".
Quoted 3 lines
In the above example the load of X and the store to
Y are strictly ordered, due to control dependencies. Not emitting the condition and maybe not even emitting the load completely wrecks this.
I think you're trying to solve this backwards. You are looking at this with an implicit wishlist of what the compiler should do (or how you want to use the hardware), but this is not a viable specification that one can write a compiler against.
We do need clear rules for what the compiler is allowed to do or not (e.g., a memory model that models multi-threaded executions). Otherwise it's all hand-waving, and we're getting nowhere. Thus, the way to approach this is to propose a feature or change to the standard, make sure that this is consistent and has no unintended side effects for other aspects of compilation or other code, and then ask the compiler to implement it. IOW, we need a patch for where this all starts: in the rules and requirements for compilation.
Paul and I are at the C++ meeting currently, and we had sessions in which the concurrency study group talked about memory model issues like dependency tracking and memory_order_consume. Paul shared uses of atomics (or likewise) in the kernel, and we discussed how the memory model currently handles various cases and why, how one could express other requirements consistently, and what is actually implementable in practice. I can't speak for Paul, but I thought those discussions were productive.
Quoted 2 lines
Its therefore an invalid optimization to take out the conditional or speculate the store, since it takes out the dependency.
Peter Zijlstra
maintainerQuoted 1 line · Torvald Riegel
That depends on what your goal is.
A compiler that we don't need to fight in order to generate sane code would be nice. But as Linus said; we can continue to ignore you lot and go on as we've done.
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Torvald Riegel
That depends on what your goal is.
First, I don't know why you quoted that, but without the context, quoting it doesn't make sense. Let me repeat the point. The standard is the rule set for the compiler. Period. The compiler does not just serve the semantics that you might have in your head. It does have to do something meaningful for all of its users. Thus, the goal for the compiler is to properly compile programs in the language as specified.
If there is a deficiency in the standard (bug or missing feature) -- and thus the specification, we need to have a patch for the standard that fixes this deficiency. If you think that this is the case, that's where you fix it.
If your goal is to do wishful thinking, imagine some kind of semantics in your head, and then assume that magically, implementations will do just that, then that's bound to fail.
Quoted 3 lines
A compiler that we don't need to fight in order to generate sane code would be nice. But as Linus said; we can continue to ignore you lot and go on as we've done.
I don't see why it's so hard to understand that you need to specify semantics, and the place (or at least the base) for that is the standard. Aren't you guys the ones replying "send a patch"? :)
This isn't any different. If you're uncomfortable working with the standard, then say so, and reach out to people that aren't.
You can surely ignore the specification of the language(s) that you are depending on. But that won't help you. If you want a change, get involved. (Oh, and claiming that the other side doesn't get it doesn't count as getting involved.)
There's no fight between people here. It's just a technical problem that we have to solve in the right way.
Quoted 50 lines · Torvald Riegel
Quoted 41 lines · Paul E. McKenney
Quoted 40 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Btw, what is the definition of "observable" for the atomics?
Because I'm hoping that it's not the same as for volatiles, where "observable" is about the virtual machine itself, and as such volatile accesses cannot be combined or optimized at all.
Now, I claim that atomic accesses cannot be done speculatively for writes, and not re-done for reads (because the value could change), but *combining* them would be possible and good.
For example, we often have multiple independent atomic accesses that could certainly be combined: testing the individual bits of an atomic value with helper functions, causing things like "load atomic, test bit, load same atomic, test another bit". The two atomic loads could be done as a single load without possibly changing semantics on a real machine, but if "visibility" is defined in the same way it is for "volatile", that wouldn't be a valid transformation. Right now we use "volatile" semantics for these kinds of things, and they really can hurt.
Same goes for multiple writes (possibly due to setting bits):
combining multiple accesses into a single one is generally fine, it's *adding* write accesses speculatively that is broken by design..At the same time, you can't combine atomic loads or stores infinitely
- "visibility" on a real machine definitely is about timeliness. Removing all but the last write when there are multiple consecutive writes is generally fine, even if you unroll a loop to generate those writes. But if what remains is a loop, it might be a busy-loop basically waiting for something, so it would be wrong ("untimely") to hoist a store in a loop entirely past the end of the loop, or hoist a load in a loop to before the loop.Quoted 7 lines
Quoted 1 line
Does the standard allow for that kind of behavior?
You asked! ;-)
So the current standard allows merging of both loads and stores, unless of course ordring constraints prevent the merging. Volatile semantics may be used to prevent this merging, if desired, for example, for real-time code.
Quoted 7 lines
Agreed.
Quoted 2 lines
Infinite merging is intended to be prohibited, but I am not certain that the current wording is bullet-proof (1.10p24 and 1.10p25).
Yeah, maybe not. But it at least seems to rather clearly indicate the intent ;)
That is my hope. ;-)
Quoted 4 lines
Quoted 3 lines
The only prohibition against speculative stores that I can see is in a non-normative note, and it can be argued to apply only to things that are not atomics (1.10p22).
Quoted 10 lines
I think this one is specifically about speculative stores that would affect memory locations that the abstract machine would not write to, and that might be observable or create data races. While a compiler could potentially prove that such stores aren't leading to a difference in the behavior of the program (e.g., by proving that there are no observers anywhere and this isn't overlapping with any volatile locations), I think that this is hard in general and most compilers will just not do such things. In GCC, bugs in that category were fixed after researchers doing fuzz-testing found them (IIRC, speculative stores by loops).
And that is my fear. ;-)
Quoted 10 lines
Quoted 6 lines
I don't see any prohibition against reordering
a store to precede a load preceding a conditional branch -- which would not be speculative if the branch was know to be taken and the load hit in the store buffer. In a system where stores could be reordered, some other CPU might perceive the store as happening before the load that controlled the conditional branch. This needs to be addressed.I don't know the specifics of your example, but from how I understand it, I don't see a problem if the compiler can prove that the store will always happen.
The current Documentation/memory-barriers.txt formulation requires that both the load and the store have volatile semantics. Does that help?
Quoted 5 lines
To be more specific, if the compiler can prove that the store will happen anyway, and the region of code can be assumed to always run atomically (e.g., there's no loop or such in there), then it is known that we have one atomic region of code that will always perform the store, so we might as well do the stuff in the region in some order.
And it would be very hard to write a program that proved that the store had been reordered prior to the load in this case.
Quoted 4 lines
Now, if any of the memory accesses are atomic, then the whole region of code containing those accesses is often not atomic because other threads might observe intermediate results in a data-race-free way.
Quoted 8 lines
(I know that this isn't a very precise formulation, but I hope it brings my line of reasoning across.)
Quoted 3 lines
Why this hole? At the time, the current formalizations of popular CPU architectures did not exist, and it was not clear that all popular hardware avoided speculative stores.
I'm not quite sure which hole you see there. Can you elaborate?
Here is one attempt, based on Peter's example later in this thread:
#define ACCESS_ONCE(x) (*(volatile typeof(x) *)&(x))
atomic_int x, y; /* Default initialization to zero. */
int r1;
void T0(void)
{
if (atomic_load(&ACCESS_ONCE(x), memory_order_relaxed))
atomic_store(&ACCESS_ONCE(y), 1, memory_order_relaxed);
}
void T1(void)
{
r1 = atomic_load(y, memory_order_seq_cst);
/* Might also need an atomic_thread_fence() here... */
atomic_store(x, 1, memory_order_seq_cst);
}
assert(r1 == 0);Peter and I would like the assertion to never trigger in this case, but the current C11 does not seem to guarantee this. I believe that the volatile casts forbid the compiler from deciding to omit T0's "if" even in cases where it could prove that x was always zero. And given the store to x, it should not be able to prove constant x here, right?
Again, I believe that current C11 does not guarantee that the assertion will never trigger, but it would be good if one of the successors to C11 did make that guarantee. ;-)
Thanx, Paul
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Quoted 5 lines
Btw, what is the definition of "observable" for the atomics?
Because I'm hoping that it's not the same as for volatiles, where "observable" is about the virtual machine itself, and as such volatile accesses cannot be combined or optimized at all.
No, atomics aren't an observable behavior of the abstract machine (unless they are volatile). See 1.8.p8 (citing the C++ standard).
Quoted 2 lines
Now, I claim that atomic accesses cannot be done speculatively for writes, and not re-done for reads (because the value could change),
Agreed, unless the compiler can prove that this doesn't make a difference in the program at hand and it's not volatile atomics. In general, that will be hard and thus won't happen often I suppose, but if correctly proved it would fall under the as-if rule I think.
Quoted 1 line
but *combining* them would be possible and good.
Agreed.
Quoted 9 lines
For example, we often have multiple independent atomic accesses that could certainly be combined: testing the individual bits of an atomic value with helper functions, causing things like "load atomic, test bit, load same atomic, test another bit". The two atomic loads could be done as a single load without possibly changing semantics on a real machine, but if "visibility" is defined in the same way it is for "volatile", that wouldn't be a valid transformation. Right now we use "volatile" semantics for these kinds of things, and they really can hurt.
Agreed. In your example, the compiler would have to prove that the abstract machine would always be able to run the two loads atomically (ie, as one load) without running into impossible/disallowed behavior of the program. But if there's no loop or branch or such in-between, this should be straight-forward because any hardware oddity or similar could merge those loads and it wouldn't be disallowed by the standard (considering that we're talking about a finite number of loads), so the compiler would be allowed to do it as well.
Quoted 3 lines
Same goes for multiple writes (possibly due to setting bits):
combining multiple accesses into a single one is generally fine, it's *adding* write accesses speculatively that is broken by design..
Agreed. As Paul points out, this being correct assumes that there are no other ordering guarantees or memory accesses "interfering", but if the stores are to the same memory location and adjacent to each other in the program, then I don't see a reason why they wouldn't be combinable.
Quoted 8 lines
At the same time, you can't combine atomic loads or stores infinitely
- "visibility" on a real machine definitely is about timeliness. Removing all but the last write when there are multiple consecutive writes is generally fine, even if you unroll a loop to generate those writes. But if what remains is a loop, it might be a busy-loop basically waiting for something, so it would be wrong ("untimely") to hoist a store in a loop entirely past the end of the loop, or hoist a load in a loop to before the loop.
Agreed. That's what 1.10p24 and 1.10p25 are meant to specify for loads, although those might not be bullet-proof as Paul points out. Forward progress is rather vaguely specified in the standard, but at least parts of the committee (and people in ISO C++ SG1, in particular) are working on trying to improve this.
Quoted 1 line
Does the standard allow for that kind of behavior?
I think the standard requires (or intends to require) the behavior that you (and I) seem to prefer in these examples.
Quoted 8 lines · Torvald Riegel
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
Intuitively, this is wrong because this let's the program take a step the abstract machine wouldn't do. This is different to the sequential code that Peter posted because it uses atomics, and thus one can't easily assume that the difference is not observable.
Quoted 8 lines
Quoted 5 lines
Btw, what is the definition of "observable" for the atomics?
Because I'm hoping that it's not the same as for volatiles, where "observable" is about the virtual machine itself, and as such volatile accesses cannot be combined or optimized at all.
No, atomics aren't an observable behavior of the abstract machine (unless they are volatile). See 1.8.p8 (citing the C++ standard).
Us Linux-kernel hackers will often need to use volatile semantics in combination with C11 atomics in most cases. The C11 atomics do cover some of the reasons we currently use ACCESS_ONCE(), but not all of them -- in particular, it allows load/store merging.
Quoted 11 lines
Quoted 2 lines
Now, I claim that atomic accesses cannot be done speculatively for writes, and not re-done for reads (because the value could change),
Agreed, unless the compiler can prove that this doesn't make a difference in the program at hand and it's not volatile atomics. In general, that will be hard and thus won't happen often I suppose, but if correctly proved it would fall under the as-if rule I think.
Quoted 1 line
but *combining* them would be possible and good.
Agreed.
In some cases, agreed. But many uses in the Linux kernel will need volatile semantics in combination with C11 atomics. Which is OK, for the foreseeable future, anyway.
Quoted 10 lines
Quoted 9 lines
For example, we often have multiple independent atomic accesses that could certainly be combined: testing the individual bits of an atomic value with helper functions, causing things like "load atomic, test bit, load same atomic, test another bit". The two atomic loads could be done as a single load without possibly changing semantics on a real machine, but if "visibility" is defined in the same way it is for "volatile", that wouldn't be a valid transformation. Right now we use "volatile" semantics for these kinds of things, and they really can hurt.
Quoted 8 lines
Agreed. In your example, the compiler would have to prove that the abstract machine would always be able to run the two loads atomically (ie, as one load) without running into impossible/disallowed behavior of the program. But if there's no loop or branch or such in-between, this should be straight-forward because any hardware oddity or similar could merge those loads and it wouldn't be disallowed by the standard (considering that we're talking about a finite number of loads), so the compiler would be allowed to do it as well.
As long as they are not marked volatile, agreed.
Thanx, Paul
Quoted 30 lines
Quoted 3 lines
Same goes for multiple writes (possibly due to setting bits):
combining multiple accesses into a single one is generally fine, it's *adding* write accesses speculatively that is broken by design..Agreed. As Paul points out, this being correct assumes that there are no other ordering guarantees or memory accesses "interfering", but if the stores are to the same memory location and adjacent to each other in the program, then I don't see a reason why they wouldn't be combinable.
Quoted 8 lines
At the same time, you can't combine atomic loads or stores infinitely
- "visibility" on a real machine definitely is about timeliness. Removing all but the last write when there are multiple consecutive writes is generally fine, even if you unroll a loop to generate those writes. But if what remains is a loop, it might be a busy-loop basically waiting for something, so it would be wrong ("untimely") to hoist a store in a loop entirely past the end of the loop, or hoist a load in a loop to before the loop.Agreed. That's what 1.10p24 and 1.10p25 are meant to specify for loads, although those might not be bullet-proof as Paul points out. Forward progress is rather vaguely specified in the standard, but at least parts of the committee (and people in ISO C++ SG1, in particular) are working on trying to improve this.
Quoted 1 line
Does the standard allow for that kind of behavior?
I think the standard requires (or intends to require) the behavior that you (and I) seem to prefer in these examples.
Quoted 5 lines · Paul E. McKenney
Us Linux-kernel hackers will often need to use volatile semantics in combination with C11 atomics in most cases. The C11 atomics do cover some of the reasons we currently use
ACCESS_ONCE(), but not all of them -- in particular, it allows load/store merging.
I really disagree with the "will need to use volatile".
We should never need to use volatile (outside of whatever MMIO we do using C) if C11 defines atomics correctly.
Allowing load/store merging is *fine*. All sane CPU's do that anyway - it's called a cache - and there's no actual reason to think that "ACCESS_ONCE()" has to mean our current "volatile".
Now, it's possible that the C standards simply get atomics _wrong_, so that they create visible semantics that are different from what a CPU cache already does, but that's a plain bug in the standard if so.
But merging loads and stores is fine. And I *guarantee* it is fine, exactly because CPU's already do it, so claiming that the compiler couldn't do it is just insanity.
Now, there are things that are *not* fine, like speculative stores that could be visible to other threads. Those are *bugs* (either in the compiler or in the standard), and anybody who claims otherwise is not worth discussing with.
But I really really disagree with the "we might have to use 'volatile'". Because if we *ever* have to use 'volatile' with the standard C atomic types, then we're just better off ignoring the atomic types entirely, because they are obviously broken shit - and we're better off doing it ourselves the way we have forever.
Seriously. This is not even hyperbole. It really is as simple as that.
Linus
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Paul E. McKenney
Us Linux-kernel hackers will often need to use volatile semantics in combination with C11 atomics in most cases. The C11 atomics do cover some of the reasons we currently use
ACCESS_ONCE(), but not all of them -- in particular, it allows load/store merging.I really disagree with the "will need to use volatile".
We should never need to use volatile (outside of whatever MMIO we do using C) if C11 defines atomics correctly.
Allowing load/store merging is *fine*. All sane CPU's do that anyway - it's called a cache - and there's no actual reason to think that "
ACCESS_ONCE()" has to mean our current "volatile".
Quoted 7 lines
Now, it's possible that the C standards simply get atomics _wrong_, so that they create visible semantics that are different from what a CPU cache already does, but that's a plain bug in the standard if so.
But merging loads and stores is fine. And I *guarantee* it is fine, exactly because CPU's already do it, so claiming that the compiler couldn't do it is just insanity.
Agreed, both CPUs and compilers can merge loads and stores. But CPUs normally get their stores pushed through the store buffer in reasonable time, and CPUs also use things like invalidations to ensure that a store is seen in reasonable time by readers. Compilers don't always have these two properties, so we do need to be more careful of load and store merging by compilers.
Quoted 4 lines
Now, there are things that are *not* fine, like speculative stores that could be visible to other threads. Those are *bugs* (either in the compiler or in the standard), and anybody who claims otherwise is not worth discussing with.
And as near as I can tell, volatile semantics are required in C11 to avoid speculative stores. I might be wrong about this, and hope that I am wrong. But I am currently not seeing it in the current standard. (Though I expect that most compilers would avoid speculating stores, especially in the near term.
Quoted 7 lines
But I really really disagree with the "we might have to use 'volatile'". Because if we *ever* have to use 'volatile' with the standard C atomic types, then we're just better off ignoring the atomic types entirely, because they are obviously broken shit - and we're better off doing it ourselves the way we have forever.
Seriously. This is not even hyperbole. It really is as simple as that.
Agreed, if we are talking about replacing ACCESS_ONCE() with C11 relaxed atomics any time soon. But someone porting Linux to a new CPU architecture might use a carefully chosen subset of C11 atomics to implement some of the Linux atomic operations, especially non-value-returning atomics such as atomic_inc().
Thanx, Paul
Quoted 26 lines · Paul E. McKenney
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Paul E. McKenney
Us Linux-kernel hackers will often need to use volatile semantics in combination with C11 atomics in most cases. The C11 atomics do cover some of the reasons we currently use
ACCESS_ONCE(), but not all of them -- in particular, it allows load/store merging.I really disagree with the "will need to use volatile".
We should never need to use volatile (outside of whatever MMIO we do using C) if C11 defines atomics correctly.
Allowing load/store merging is *fine*. All sane CPU's do that anyway - it's called a cache - and there's no actual reason to think that "
ACCESS_ONCE()" has to mean our current "volatile".Quoted 7 lines
Now, it's possible that the C standards simply get atomics _wrong_, so that they create visible semantics that are different from what a CPU cache already does, but that's a plain bug in the standard if so.
But merging loads and stores is fine. And I *guarantee* it is fine, exactly because CPU's already do it, so claiming that the compiler couldn't do it is just insanity.
Quoted 6 lines
Agreed, both CPUs and compilers can merge loads and stores. But CPUs normally get their stores pushed through the store buffer in reasonable time, and CPUs also use things like invalidations to ensure that a store is seen in reasonable time by readers. Compilers don't always have these two properties, so we do need to be more careful of load and store merging by compilers.
The standard's _wording_ is a little vague about forward-progress guarantees, but I believe the vast majority of the people involved do want compilers to not prevent forward progress. There is of course a difference whether a compiler establishes _eventual_ forward progress in the sense of after 10 years or forward progress in a small bounded interval of time, but this is a QoI issue, and good compilers won't want to introduce unnecessary latencies. I believe that it is fine if the standard merely talks about eventual forward progress.
Quoted 10 lines
Quoted 4 lines
Now, there are things that are *not* fine, like speculative stores that could be visible to other threads. Those are *bugs* (either in the compiler or in the standard), and anybody who claims otherwise is not worth discussing with.
And as near as I can tell, volatile semantics are required in C11 to avoid speculative stores. I might be wrong about this, and hope that I am wrong. But I am currently not seeing it in the current standard. (Though I expect that most compilers would avoid speculating stores, especially in the near term.
This really depends on how we define speculative stores. The memory model is absolutely clear that programs have to behave as if executed by the virtual machine, and that rules out speculative stores to volatiles and other locations. Under certain circumstances, there will be "speculative" stores in the sense that they will happen at different times as if you had a trivial implementation of the abstract machine. But to be allowed to do that, the compiler has to prove that such a transformation still fulfills the as-if rule.
IOW, the abstract machine is what currently defines disallowed speculative stores. If you want to put *further* constraints on what implementations are allowed to do, I suppose it is best to talk about those and see how we can add rules that allow programmers to express those constraints. For example, control dependencies might be such a case. I don't have a specific suggestion -- maybe the control dependencies are best tackled similar to consume dependencies (even though we don't have a good solution for those yets). But using volatile accesses for that seems to be a big hammer, or even the wrong one.
Quoted 34 lines · Torvald Riegel
Quoted 26 lines · Paul E. McKenney
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Paul E. McKenney
Us Linux-kernel hackers will often need to use volatile semantics in combination with C11 atomics in most cases. The C11 atomics do cover some of the reasons we currently use
ACCESS_ONCE(), but not all of them -- in particular, it allows load/store merging.I really disagree with the "will need to use volatile".
We should never need to use volatile (outside of whatever MMIO we do using C) if C11 defines atomics correctly.
Allowing load/store merging is *fine*. All sane CPU's do that anyway - it's called a cache - and there's no actual reason to think that "
ACCESS_ONCE()" has to mean our current "volatile".Quoted 7 lines
Now, it's possible that the C standards simply get atomics _wrong_, so that they create visible semantics that are different from what a CPU cache already does, but that's a plain bug in the standard if so.
But merging loads and stores is fine. And I *guarantee* it is fine, exactly because CPU's already do it, so claiming that the compiler couldn't do it is just insanity.
Quoted 6 lines
Agreed, both CPUs and compilers can merge loads and stores. But CPUs normally get their stores pushed through the store buffer in reasonable time, and CPUs also use things like invalidations to ensure that a store is seen in reasonable time by readers. Compilers don't always have these two properties, so we do need to be more careful of load and store merging by compilers.
Quoted 8 lines
The standard's _wording_ is a little vague about forward-progress guarantees, but I believe the vast majority of the people involved do want compilers to not prevent forward progress. There is of course a difference whether a compiler establishes _eventual_ forward progress in the sense of after 10 years or forward progress in a small bounded interval of time, but this is a QoI issue, and good compilers won't want to introduce unnecessary latencies. I believe that it is fine if the standard merely talks about eventual forward progress.
The compiler will need to earn my trust on this one. ;-)
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Now, there are things that are *not* fine, like speculative stores that could be visible to other threads. Those are *bugs* (either in the compiler or in the standard), and anybody who claims otherwise is not worth discussing with.
And as near as I can tell, volatile semantics are required in C11 to avoid speculative stores. I might be wrong about this, and hope that I am wrong. But I am currently not seeing it in the current standard. (Though I expect that most compilers would avoid speculating stores, especially in the near term.
Quoted 8 lines
This really depends on how we define speculative stores. The memory model is absolutely clear that programs have to behave as if executed by the virtual machine, and that rules out speculative stores to volatiles and other locations. Under certain circumstances, there will be "speculative" stores in the sense that they will happen at different times as if you had a trivial implementation of the abstract machine. But to be allowed to do that, the compiler has to prove that such a transformation still fulfills the as-if rule.
Agreed, although the as-if rule would ignore control dependencies, since these are not yet part of the standard (as you in fact note below). I nevertheless consider myself at least somewhat reassured that current C11 won't speculate stores. My remaining concerns involve the compiler proving to itself that a given branch is always taken, thus motivating it to optimize the branch away -- though this is more properly a control-dependency concern.
Quoted 10 lines
IOW, the abstract machine is what currently defines disallowed speculative stores. If you want to put *further* constraints on what implementations are allowed to do, I suppose it is best to talk about those and see how we can add rules that allow programmers to express those constraints. For example, control dependencies might be such a case. I don't have a specific suggestion -- maybe the control dependencies are best tackled similar to consume dependencies (even though we don't have a good solution for those yets). But using volatile accesses for that seems to be a big hammer, or even the wrong one.
In current compilers, the two hammers we have are volatile and barrier(). But yes, it would be good to have something more focused. One option would be to propose memory_order_control loads to see how loudly the committee screams. One use case might be as follows:
if (atomic_load(x, memory_order_control)) atomic_store(y, memory_order_relaxed);
This could also be written:
r1 = atomic_load(x, memory_order_control); if (r1) atomic_store(y, memory_order_relaxed);
A branch depending on the memory_order_control load could not be optimized out, though I suppose that the compiler could substitute a memory-barrier instruction for the branch. Seems like it would take a very large number of branches to equal the overhead of the memory barrier, though.
Another option would be to flag the conditional expression, prohibiting the compiler from optimizing out any conditional branches. Perhaps something like this:
r1 = atomic_load(x, memory_order_control); if (control_dependency(r1)) atomic_store(y, memory_order_relaxed);
Other thoughts?
Thanx, Paul
Quoted 104 lines · Paul E. McKenney
Quoted 34 lines · Torvald Riegel
Quoted 26 lines · Paul E. McKenney
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Paul E. McKenney
Us Linux-kernel hackers will often need to use volatile semantics in combination with C11 atomics in most cases. The C11 atomics do cover some of the reasons we currently use
ACCESS_ONCE(), but not all of them -- in particular, it allows load/store merging.I really disagree with the "will need to use volatile".
We should never need to use volatile (outside of whatever MMIO we do using C) if C11 defines atomics correctly.
Allowing load/store merging is *fine*. All sane CPU's do that anyway - it's called a cache - and there's no actual reason to think that "
ACCESS_ONCE()" has to mean our current "volatile".Quoted 7 lines
Now, it's possible that the C standards simply get atomics _wrong_, so that they create visible semantics that are different from what a CPU cache already does, but that's a plain bug in the standard if so.
But merging loads and stores is fine. And I *guarantee* it is fine, exactly because CPU's already do it, so claiming that the compiler couldn't do it is just insanity.
Quoted 6 lines
Agreed, both CPUs and compilers can merge loads and stores. But CPUs normally get their stores pushed through the store buffer in reasonable time, and CPUs also use things like invalidations to ensure that a store is seen in reasonable time by readers. Compilers don't always have these two properties, so we do need to be more careful of load and store merging by compilers.
Quoted 8 lines
The standard's _wording_ is a little vague about forward-progress guarantees, but I believe the vast majority of the people involved do want compilers to not prevent forward progress. There is of course a difference whether a compiler establishes _eventual_ forward progress in the sense of after 10 years or forward progress in a small bounded interval of time, but this is a QoI issue, and good compilers won't want to introduce unnecessary latencies. I believe that it is fine if the standard merely talks about eventual forward progress.
The compiler will need to earn my trust on this one. ;-)
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Now, there are things that are *not* fine, like speculative stores that could be visible to other threads. Those are *bugs* (either in the compiler or in the standard), and anybody who claims otherwise is not worth discussing with.
And as near as I can tell, volatile semantics are required in C11 to avoid speculative stores. I might be wrong about this, and hope that I am wrong. But I am currently not seeing it in the current standard. (Though I expect that most compilers would avoid speculating stores, especially in the near term.
Quoted 8 lines
This really depends on how we define speculative stores. The memory model is absolutely clear that programs have to behave as if executed by the virtual machine, and that rules out speculative stores to volatiles and other locations. Under certain circumstances, there will be "speculative" stores in the sense that they will happen at different times as if you had a trivial implementation of the abstract machine. But to be allowed to do that, the compiler has to prove that such a transformation still fulfills the as-if rule.
Agreed, although the as-if rule would ignore control dependencies, since these are not yet part of the standard (as you in fact note below). I nevertheless consider myself at least somewhat reassured that current C11 won't speculate stores. My remaining concerns involve the compiler proving to itself that a given branch is always taken, thus motivating it to optimize the branch away -- though this is more properly a control-dependency concern.
Quoted 10 lines
IOW, the abstract machine is what currently defines disallowed speculative stores. If you want to put *further* constraints on what implementations are allowed to do, I suppose it is best to talk about those and see how we can add rules that allow programmers to express those constraints. For example, control dependencies might be such a case. I don't have a specific suggestion -- maybe the control dependencies are best tackled similar to consume dependencies (even though we don't have a good solution for those yets). But using volatile accesses for that seems to be a big hammer, or even the wrong one.
In current compilers, the two hammers we have are volatile and
barrier(). But yes, it would be good to have something more focused. One option would be to proposememory_order_controlloads to see how loudly the committee screams. One use case might be as follows:if (atomic_load(x, memory_order_control)) atomic_store(y, memory_order_relaxed);This could also be written:
r1 = atomic_load(x, memory_order_control); if (r1) atomic_store(y, memory_order_relaxed);A branch depending on the
memory_order_controlload could not be optimized out, though I suppose that the compiler could substitute a memory-barrier instruction for the branch. Seems like it would take a very large number of branches to equal the overhead of the memory barrier, though.
Quoted 7 lines
Another option would be to flag the conditional expression, prohibiting the compiler from optimizing out any conditional branches. Perhaps something like this:
r1 = atomic_load(x, memory_order_control); if (control_dependency(r1)) atomic_store(y, memory_order_relaxed);
That's the one I had in mind and talked to you about earlier today. My gut feeling is that this is preferably over the other because it "marks" the if-statement, so the compiler knows exactly which branches matter. I'm not sure one would need the other memory order for that, if indeed all you want is relaxed -> branch -> relaxed. But maybe there are corner cases (see the weaker-than-relaxed discussion in SG1 today).
Quoted 1 line · Torvald Riegel
On Thu, 2014-02-13 at 18:01 -0800, Paul E. McKenney wrote:
[ . . . ]
Quoted 8 lines
Quoted 7 lines
Another option would be to flag the conditional expression, prohibiting the compiler from optimizing out any conditional branches. Perhaps something like this:
r1 = atomic_load(x, memory_order_control); if (control_dependency(r1)) atomic_store(y, memory_order_relaxed);
Quoted 6 lines
That's the one I had in mind and talked to you about earlier today. My gut feeling is that this is preferably over the other because it "marks" the if-statement, so the compiler knows exactly which branches matter. I'm not sure one would need the other memory order for that, if indeed all you want is relaxed -> branch -> relaxed. But maybe there are corner cases (see the weaker-than-relaxed discussion in SG1 today).
Linus, Peter, any objections to marking places where we are relying on ordering from control dependencies against later stores? This approach seems to me to have significant documentation benefits.
Thanx, Paul
Quoted 20 lines · Paul E. McKenney
Quoted 1 line · Torvald Riegel
On Thu, 2014-02-13 at 18:01 -0800, Paul E. McKenney wrote:
[ . . . ]
Quoted 8 lines
Quoted 7 lines
Another option would be to flag the conditional expression, prohibiting the compiler from optimizing out any conditional branches. Perhaps something like this:
r1 = atomic_load(x, memory_order_control); if (control_dependency(r1)) atomic_store(y, memory_order_relaxed);Quoted 6 lines
That's the one I had in mind and talked to you about earlier today. My gut feeling is that this is preferably over the other because it "marks" the if-statement, so the compiler knows exactly which branches matter. I'm not sure one would need the other memory order for that, if indeed all you want is relaxed -> branch -> relaxed. But maybe there are corner cases (see the weaker-than-relaxed discussion in SG1 today).
Quoted 3 lines
Linus, Peter, any objections to marking places where we are relying on ordering from control dependencies against later stores? This approach seems to me to have significant documentation benefits.
Let me note that at least as I'm concerned, that's just a quick idea. At least I haven't looked at (1) how to properly specify the semantics of this, (2) whether it has any bad effects on unrelated code, (3) and whether there are pitfalls for compiler implementations. It looks not too bad at first glance, though.
Quoted 4 lines · Paul E. McKenney
Linus, Peter, any objections to marking places where we are relying on ordering from control dependencies against later stores? This approach seems to me to have significant documentation benefits.
Quite frankly, I think it's stupid, and the "documentation" is not a benefit, it's just wrong.
How would you figure out whether your added "documentation" holds true for particular branches but not others?
How could you *ever* trust a compiler that makes the dependency meaningful?
Again, let's keep this simple and sane:
- if a compiler ever generates code where an atomic store movement is
"visible" in any way, then that compiler is broken shit.
I don't understand why you even argue this. Seriously, Paul, you seem to *want* to think that "broken shit" is acceptable, and that we should then add magic markers to say "now you need to *not* be broken shit".
Here's a magic marker for you: DON'T USE THAT BROKEN COMPILER.
And if a compiler can *prove* that whatever code movement it does cannot make a difference, then let it do so. No amount of "documentation" should matter.
Seriously, this whole discussion has been completely moronic. I don't understand why you even bring shit like this up:
Quoted 3 lines
Quoted 3 lines
r1 = atomic_load(x, memory_order_control); if (control_dependency(r1)) atomic_store(y, memory_order_relaxed);
I mean, really? Anybody who writes code like that, or any compiler where that "control_dependency()" marker makes any difference what-so-ever for code generation should just be retroactively aborted.
There is absolutely *zero* reason for that "control_dependency()" crap. If you ever find a reason for it, it is either because the compiler is buggy, or because the standard is so shit that we should never *ever* use the atomics.
Seriously. This thread has devolved into some kind of "just what kind of idiotic compiler cesspool crap could we accept". Get away from that f*cking mindset. We don't accept *any* crap.
Why are we still discussing this idiocy? It's irrelevant. If the standard really allows random store speculation, the standard doesn't matter, and sane people shouldn't waste their time arguing about it.
Linus
Quoted 4 lines · Linus Torvalds
Why are we still discussing this idiocy? It's irrelevant. If the standard really allows random store speculation, the standard doesn't matter, and sane people shouldn't waste their time arguing about it.
Btw, the other part of this coin is that our manual types (using volatile and various architecture-specific stuff) and our manual barriers and inline asm accesses are generally *fine*.
The C11 stuff doesn't buy us anything. The argument that "new architectures might want to use it" is prue and utter bollocks, since unless the standard gets the thing *right*, nobody sane would ever use it for some new architecture, when the sane thing to do is to just fill in the normal barriers and inline asms.
So I'm very very serious: either the compiler and the standard gets things right, or we don't use it. There is no middle ground where "we might use it for one or two architectures and add random hints". That's just stupid.
The only "middle ground" is about which compiler version we end up trusting _if_ it turns out that the compiler and standard do get things right. From Torvald's explanations (once I don't mis-read them ;), my take-away so far has actually been that the standard *does* get things right, but I do know from over-long personal experience that compiler people sometimes want to be legalistic and twist the documentation to the breaking point, at which point we just go "we'd be crazy do use that".
See our use of "-fno-strict-aliasing", for example. The C standard aliasing rules are a mistake, stupid, and wrong, and gcc uses those stupid type-based alias rules even when statically *proving* the aliasing gives the opposite result. End result: we turn the shit off.
Exact same deal wrt atomics. We are *not* going to add crazy "this here is a control dependency" crap. There's a test, the compiler *sees* the control dependency for chrissake, and it still generates crap, we turn that broken "optimization" off. It really is that simple.
Linus
Quoted 36 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds
Why are we still discussing this idiocy? It's irrelevant. If the standard really allows random store speculation, the standard doesn't matter, and sane people shouldn't waste their time arguing about it.
Btw, the other part of this coin is that our manual types (using volatile and various architecture-specific stuff) and our manual barriers and inline asm accesses are generally *fine*.
The C11 stuff doesn't buy us anything. The argument that "new architectures might want to use it" is prue and utter bollocks, since unless the standard gets the thing *right*, nobody sane would ever use it for some new architecture, when the sane thing to do is to just fill in the normal barriers and inline asms.
So I'm very very serious: either the compiler and the standard gets things right, or we don't use it. There is no middle ground where "we might use it for one or two architectures and add random hints". That's just stupid.
The only "middle ground" is about which compiler version we end up trusting _if_ it turns out that the compiler and standard do get things right. From Torvald's explanations (once I don't mis-read them ;), my take-away so far has actually been that the standard *does* get things right, but I do know from over-long personal experience that compiler people sometimes want to be legalistic and twist the documentation to the breaking point, at which point we just go "we'd be crazy do use that".
See our use of "-fno-strict-aliasing", for example. The C standard aliasing rules are a mistake, stupid, and wrong, and gcc uses those stupid type-based alias rules even when statically *proving* the aliasing gives the opposite result. End result: we turn the shit off.
Quoted 5 lines
Exact same deal wrt atomics. We are *not* going to add crazy "this here is a control dependency" crap. There's a test, the compiler *sees* the control dependency for chrissake, and it still generates crap, we turn that broken "optimization" off. It really is that simple.
From what I can see at the moment, the standard -generally- avoids speculative stores, but there are a few corner cases where it might allow them. I will be working with the committee to see exactly what the situation is. Might be that I am confused and that everything really is OK, might be that I am right but the corner cases are things that no sane kernel developer would do anyway, it might be that the standard needs a bit of repair, or it might be that the corner cases are somehow inherent and problematic (but I hope not!). I will let you know what I find, but it will probably be a few months.
In the meantime, agreed, we keep doing what we have been doing. And maybe in the long term as well, for that matter.
One way of looking at the discussion between Torvald and myself would be as a seller (Torvald) and a buyer (me) haggling over the fine print in a proposed contract (the standard). Whether that makes you feel better or worse about the situation I cannot say. ;-)
Thanx, Paul
Quoted 5 lines · Paul E. McKenney
One way of looking at the discussion between Torvald and myself would be as a seller (Torvald) and a buyer (me) haggling over the fine print in a proposed contract (the standard). Whether that makes you feel better or worse about the situation I cannot say. ;-)
Oh, I'm perfectly fine with that. But we're not desperate to buy, and I actually think the C11 people are - or at least should be - *way* more desperate to sell their model to us than we are to take it.
Which means that as a buyer you should say "this is what we want, if you don't give us this, we'll just walk away". Not try to see how much we can pay for it. Because there is very little upside for us, and _unless_ the C11 standard gets things right it's just extra complexity for us, coupled with new compiler fragility and years of code generation bugs.
Why would we want that extra complexity and inevitable compiler bugs? If we then have to fight compiler writers that point to the standard and say "..but look, the standard says we can do this", then at that point it went from "extra complexity and compiler bugs" to a whole 'nother level of frustration and pain.
So just walk away unless the C11 standard gives us exactly what we want. Not "something kind of like what we'd use". EXACTLY. Because I'm not in the least interested in fighting compiler people that have a crappy standard they can point to. Been there, done that, got the T-shirt and learnt my lesson.
And the thing is, I suspect that the Linux kernel is the most complete
- and most serious - user of true atomics that the C11 people can sell their solution to.
If we don't buy it, they have no serious user. Sure, they'll have lots of random other one-off users for their atomics, where each user wants one particular thing, but I suspect that we'll have the only really unified portable code base that handles pretty much *all* the serious odd cases that the C11 atomics can actually talk about to each other.
Oh, they'll push things through with or without us, and it will be a collection of random stuff, where they tried to please everybody, with particularly compiler/architecture people who have no f*cking clue about how their stuff is used pushing to make it easy/efficient for their particular compiler/architecture.
But we have real optimized uses of pretty much all relevant cases that people actually care about.
We can walk away from them, and not really lose anything but a small convenience (and it's a convenience *only* if the standard gets things right).
And conversely, the C11 people can walk away from us too. But if they can't make us happy (and by "make us happy", I really mean no stupid games on our part) I personally think they'll have a stronger standard, and a real use case, and real arguments. I'm assuming they want that.
That's why I complain when you talk about things like marking control dependencies explicitly. That's *us* bending over backwards. And as a buyer, we have absolutely zero reason to do that.
Tell the C11 people: "no speculative writes". Full stop. End of story. Because we're not buying anything else.
Similarly, if we need to mark atomics "volatile", then now the C11 atomics are no longer even a "small convenience", now they are just extra complexity wrt what we already have. So just make it clear that if the C11 standard needs to mark atomics volatile in order to get non-speculative and non-reloading behavior, then the C11 atomics are useless to us, and we're not buying.
Remember: a compiler can *always* do "as if" optimizations - if a compiler writer can prove that the end result acts 100% the same using an optimized sequence, then they can do whatever the hell they want. That's not the issue. But if we can *ever* see semantic impact of speculative writes, the compiler is buggy, and the compiler writers need to be aware that it is buggy. No ifs, buts, maybes about it.
So I'm perfectly fine with you seeing yourself as a buyer. But I want you to be a really *picky* and anal buyer - one that knows he has the upper hand, and can walk away with no downside.
Linus
Quoted 6 lines · Linus Torvalds
And conversely, the C11 people can walk away from us too. But if they can't make us happy (and by "make us happy", I really mean no stupid games on our part) I personally think they'll have a stronger standard, and a real use case, and real arguments. I'm assuming they want that.
I should have somebody who proof-reads my emails before I send them out.
I obviously meant "if they *can* make us happy" (not "can't").
Linus
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Linus Torvalds
And conversely, the C11 people can walk away from us too. But if they can't make us happy (and by "make us happy", I really mean no stupid games on our part) I personally think they'll have a stronger standard, and a real use case, and real arguments. I'm assuming they want that.
Quoted 3 lines
I should have somebody who proof-reads my emails before I send them out.
I obviously meant "if they *can* make us happy" (not "can't").
Understood. My next step is to take a more detailed look at the piece of the standard that should support RCU. Depending on how that turns out, I might look at other parts of the standard vs. Linux's atomics and memory-ordering needs. Should be interesting. ;-)
Thanx, Paul
Quoted 10 lines · Paul E. McKenney
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Linus Torvalds
And conversely, the C11 people can walk away from us too. But if they can't make us happy (and by "make us happy", I really mean no stupid games on our part) I personally think they'll have a stronger standard, and a real use case, and real arguments. I'm assuming they want that.
Quoted 8 lines
Quoted 3 lines
I should have somebody who proof-reads my emails before I send them out.
I obviously meant "if they *can* make us happy" (not "can't").
Understood. My next step is to take a more detailed look at the piece of the standard that should support RCU. Depending on how that turns out, I might look at other parts of the standard vs. Linux's atomics and memory-ordering needs. Should be interesting. ;-)
And perhaps a better way to represent the roles is that I am not the buyer, but rather the purchasing agent for the -potential- buyer. -You- are of course the potential buyer.
If I were to see myself as the buyer, then I must confess that the concerns you implicitly expressed in your prior email would be all too well-founded!
Thanx, Paul
Quoted 12 lines · Linus Torvalds
Quoted 5 lines · Paul E. McKenney
One way of looking at the discussion between Torvald and myself would be as a seller (Torvald) and a buyer (me) haggling over the fine print in a proposed contract (the standard). Whether that makes you feel better or worse about the situation I cannot say. ;-)
Oh, I'm perfectly fine with that. But we're not desperate to buy, and I actually think the C11 people are - or at least should be - *way* more desperate to sell their model to us than we are to take it.
Quoted 6 lines
Which means that as a buyer you should say "this is what we want, if you don't give us this, we'll just walk away". Not try to see how much we can pay for it. Because there is very little upside for us, and _unless_ the C11 standard gets things right it's just extra complexity for us, coupled with new compiler fragility and years of code generation bugs.
I think there is an upside to you, mainly in that it allows compiler testing tools, potentially verification tools for atomics, and tools like cppmem that show allowed executions of code.
I agree that the Linux community has been working well without this, and it's big enough to make running it's own show viable. This will be different for smaller projects, though.
Quoted 5 lines
Why would we want that extra complexity and inevitable compiler bugs? If we then have to fight compiler writers that point to the standard and say "..but look, the standard says we can do this", then at that point it went from "extra complexity and compiler bugs" to a whole 'nother level of frustration and pain.
I see your point, but flip side of the coin is that if you get the standard to say what you want, then you can tell the compiler writers to look at the standard. Or show them bugs revealed by fuzz testing and such.
Quoted 9 lines
So just walk away unless the C11 standard gives us exactly what we want. Not "something kind of like what we'd use". EXACTLY. Because I'm not in the least interested in fighting compiler people that have a crappy standard they can point to. Been there, done that, got the T-shirt and learnt my lesson.
And the thing is, I suspect that the Linux kernel is the most complete
- and most serious - user of true atomics that the C11 people can sell their solution to.
I agree, but there are likely also other big projects that could make use of C11 atomics on the userspace side (e.g., certain databases, ...).
Quoted 1 line
If we don't buy it, they have no serious user.
I disagree with that. That obviously depends on one's definition of "serious", but if you combine all C/C++ programs that use low-level atomics, then this is serious use as well. There's lots of shared-memory synchronization in userspace as well.
Quoted 4 lines
Sure, they'll have lots
of random other one-off users for their atomics, where each user wants one particular thing, but I suspect that we'll have the only really unified portable code base
glibc is a counterexample that comes to mind, although it's a smaller code base. (It's currently not using C11 atomics, but transitioning there makes sense, and some thing I want to get to eventually.)
Quoted 2 lines
that handles pretty much *all* the serious
odd cases that the C11 atomics can actually talk about to each other.
You certainly have lots of odd cases, but I would disagree with the assumption that only the Linux kernel will do full "testing" of the implementations. If you have plenty of userspace programs using the atomics, that's a pretty big test suite, and one that should help bring the compilers up to speed. So that might be a benefit even to the Linux kernel if it would use the C11 atomics.
Quoted 5 lines
Oh, they'll push things through with or without us, and it will be a collection of random stuff, where they tried to please everybody, with particularly compiler/architecture people who have no f*cking clue about how their stuff is used pushing to make it easy/efficient for their particular compiler/architecture.
I'll ignore this... :)
Quoted 2 lines
But we have real optimized uses of pretty much all relevant cases that people actually care about.
You certainly cover a lot of cases. Finding out whether you cover all that "people care about" would require you to actually ask all people, which I'm sure you've done ;)
Quoted 9 lines
We can walk away from them, and not really lose anything but a small convenience (and it's a convenience *only* if the standard gets things right).
And conversely, the C11 people can walk away from us too. But if they can't make us happy (and by "make us happy", I really mean no stupid games on our part) I personally think they'll have a stronger standard, and a real use case, and real arguments. I'm assuming they want that.
I agree.
Quoted 3 lines
That's why I complain when you talk about things like marking control dependencies explicitly. That's *us* bending over backwards. And as a buyer, we have absolutely zero reason to do that.
As I understood the situation, it was rather like the buyer trying to wrestle in an additional feature without having a proper specification of what the feature should actually do. That's why we've been talking about what it should actually do, and how ...
Quoted 9 lines
Tell the C11 people: "no speculative writes". Full stop. End of story. Because we're not buying anything else.
Similarly, if we need to mark atomics "volatile", then now the C11 atomics are no longer even a "small convenience", now they are just extra complexity wrt what we already have. So just make it clear that if the C11 standard needs to mark atomics volatile in order to get non-speculative and non-reloading behavior, then the C11 atomics are useless to us, and we're not buying.
I hope what I wrote previously removes those concerns.
Quoted 4 lines
Remember: a compiler can *always* do "as if" optimizations - if a compiler writer can prove that the end result acts 100% the same using an optimized sequence, then they can do whatever the hell they want. That's not the issue.
Good to know that we agree on that.
Quoted 3 lines
But if we can *ever* see semantic impact of
speculative writes, the compiler is buggy, and the compiler writers need to be aware that it is buggy. No ifs, buts, maybes about it.
Agreed.
Quoted 3 lines
So I'm perfectly fine with you seeing yourself as a buyer. But I want you to be a really *picky* and anal buyer - one that knows he has the upper hand, and can walk away with no downside.
FWIW, I don't see myself as being the seller. Instead, I'm hope we can improve the situation for everyone involved, whether that involves C11 or something else. I see a potential mutual benefit, and I want to try to exploit it. Regarding the distributions I'm concerned about, the Linux kernel and GCC are very much in the same boat.
Quoted 3 lines · Torvald Riegel
glibc is a counterexample that comes to mind, although it's a smaller code base. (It's currently not using C11 atomics, but transitioning there makes sense, and some thing I want to get to eventually.)
glibc is using C11 atomics (GCC builtins rather than _Atomic / <stdatomic.h>, but using __atomic_* with explicitly specified memory model rather than the older __sync_*) on AArch64, plus in certain cases on ARM and MIPS.
Signature · 2 lines
Joseph S. Myers [email protected]
Quoted 10 lines · Joseph S. Myers
Quoted 3 lines · Torvald Riegel
glibc is a counterexample that comes to mind, although it's a smaller code base. (It's currently not using C11 atomics, but transitioning there makes sense, and some thing I want to get to eventually.)
glibc is using C11 atomics (GCC builtins rather than
_Atomic/ <stdatomic.h>, but using __atomic_* with explicitly specified memory model rather than the older __sync_*) on AArch64, plus in certain cases on ARM and MIPS.
Hmm, actually that results in a change in behaviour for the __sync_* primitives on AArch64. The documentation for those states that:
`In most cases, these built-in functions are considered a full barrier. That is, no memory operand is moved across the operation, either forward or backward. Further, instructions are issued as necessary to prevent the processor from speculating loads across the operation and from queuing stores after the operation.'
which is stronger than simply mapping them to memory_model_seq_cst, which seems to be what the AArch64 compiler is doing (so you get acquire + release instead of a full fence).
Will
Quoted 10 lines · Joseph S. Myers
Quoted 3 lines · Torvald Riegel
glibc is a counterexample that comes to mind, although it's a smaller code base. (It's currently not using C11 atomics, but transitioning there makes sense, and some thing I want to get to eventually.)
glibc is using C11 atomics (GCC builtins rather than
_Atomic/ <stdatomic.h>, but using __atomic_* with explicitly specified memory model rather than the older __sync_*) on AArch64, plus in certain cases on ARM and MIPS.
I think the major steps remaining is moving the other architectures over, and rechecking concurrent code (e.g., for the code that I have seen, it was either asm variants (eg, on x86), or built before C11; ARM pthread_once was lacking memory_barriers (see "pthread_once unification" patches I posted)). We also need/should to move towards using relaxed-MO atomic loads instead of plain loads.
Quoted 7 lines · Torvald Riegel
Quoted 6 lines · Joseph S. Myers
Quoted 3 lines · Torvald Riegel
glibc is a counterexample that comes to mind, although it's a smaller code base. (It's currently not using C11 atomics, but transitioning there makes sense, and some thing I want to get to eventually.)
Quoted 7 lines
Quoted 4 lines
glibc is using C11 atomics (GCC builtins rather than
_Atomic/ <stdatomic.h>, but using __atomic_* with explicitly specified memory model rather than the older __sync_*) on AArch64, plus in certain cases on ARM and MIPS.I think the major steps remaining is moving the other architectures over, and rechecking concurrent code (e.g., for the code that I have
I don't think we'll be ready to require GCC >= 4.7 to build glibc for another year or two, although probably we could move the requirement up from 4.4 to 4.6. (And some platforms only had the C11 atomics optimized later than 4.7.)
Signature · 2 lines
Joseph S. Myers [email protected]
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds
Why are we still discussing this idiocy? It's irrelevant. If the standard really allows random store speculation, the standard doesn't matter, and sane people shouldn't waste their time arguing about it.
Btw, the other part of this coin is that our manual types (using volatile and various architecture-specific stuff) and our manual barriers and inline asm accesses are generally *fine*.
AFAICT, it does work for you, but hasn't been exactly pain-free.
I think a major benefit of C11's memory model is that it gives a *precise* specification for how a compiler is allowed to optimize. There is a formalization of the model, which allows things like the cppmem tool by the Cambridge group. It also allows meaningful fuzz
testing: http://www.di.ens.fr/~zappa/projects/cmmtest/ ; this did reveal several GCC compiler bugs.
I also think that reasoning about this model is easier than reasoning about how lots of different, concrete compiler optimizations would interact.
Quoted 11 lines
The C11 stuff doesn't buy us anything. The argument that "new architectures might want to use it" is prue and utter bollocks, since unless the standard gets the thing *right*, nobody sane would ever use it for some new architecture, when the sane thing to do is to just fill in the normal barriers and inline asms.
So I'm very very serious: either the compiler and the standard gets things right, or we don't use it. There is no middle ground where "we might use it for one or two architectures and add random hints". That's just stupid.
Quoted 8 lines
The only "middle ground" is about which compiler version we end up trusting _if_ it turns out that the compiler and standard do get things right. From Torvald's explanations (once I don't mis-read them ;), my take-away so far has actually been that the standard *does* get things right, but I do know from over-long personal experience that compiler people sometimes want to be legalistic and twist the documentation to the breaking point, at which point we just go "we'd be crazy do use that".
I agree that compilers want to optimize, and sometimes there's probably a little too much emphasis on applying an optimization vs. not surprising users. But we have to draw a line (e.g., what is undefined behavior and what is not), because we need this to be actually able to optimize.
Therefore, we need to get the rules into a shape that both allows optimizations and isn't full of surprising corner cases. The rules are the standard, so it's the standard we have to get right. According to my experience, a lot of thought goes into how to design the standard's language and library so that they are intuitive yet efficient.
If you see issues in the standard, please bring them up. Either report the defects directly and get involved yourself, or reach out to somebody that is participating in the standards process.
The standard certainly isn't perfect, so there is room to contribute.
Quoted 3 lines · Torvald Riegel
I think a major benefit of C11's memory model is that it gives a *precise* specification for how a compiler is allowed to optimize.
Clearly it does *not*. This whole discussion is proof of that. It's not at all clear, and the standard apparently is at least debatably allowing things that shouldn't be allowed. It's also a whole lot more complicated than "volatile", so the likelihood of a compiler writer actually getting it right - even if the standard does - is lower. They've gotten "volatile" wrong too, after all (particularly in C++).
Linus
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Torvald Riegel
I think a major benefit of C11's memory model is that it gives a *precise* specification for how a compiler is allowed to optimize.
Clearly it does *not*. This whole discussion is proof of that. It's not at all clear,
It might not be an easy-to-understand specification, but as far as I'm aware it is precise. The Cambridge group's formalization certainly is precise. From that, one can derive (together with the usual rules for as-if etc.) what a compiler is allowed to do (assuming that the standard is indeed precise). My replies in this discussion have been based on reasoning about the standard, and not secret knowledge (with the exception of no-out-of-thin-air, which is required in the standard's prose but not yet formalized).
I agree that I'm using the formalization as a kind of placeholder for the standard's prose (which isn't all that easy to follow for me either), but I guess there's no way around an ISO standard using prose.
If you see a case in which the standard isn't precise, please bring it up or open a C++ CWG issue for it.
Quoted 2 lines
and the standard apparently is at least debatably
allowing things that shouldn't be allowed.
Which example do you have in mind here? Haven't we resolved all the debated examples, or did I miss any?
Quoted 3 lines
It's also a whole lot more
complicated than "volatile", so the likelihood of a compiler writer actually getting it right - even if the standard does - is lower.
It's not easy, that's for sure, but none of the high-performance alternatives are easy either. There are testing tools out there based on the formalization of the model, and we've found bugs with them.
And the alternative of using something not specified by the standard is even worse, I think, because then you have to guess what a compiler might do, without having any constraints; IOW, one is resorting to "no sane compiler would do that", and that doesn't seem to very robust either.
Quoted 3 lines · Torvald Riegel
Which example do you have in mind here? Haven't we resolved all the debated examples, or did I miss any?
Well, Paul seems to still think that the standard possibly allows speculative writes or possibly value speculation in ways that break the hardware-guaranteed orderings.
And personally, I can't read standards paperwork. It is invariably written in some basically impossible-to-understand lawyeristic mode, and then it is read by people (compiler writers) that intentionally try to mis-use the words and do language-lawyering ("that depends on what the meaning of 'is' is"). The whole "lvalue vs rvalue expression vs 'what is a volatile access'" thing for C++ was/is a great example of that.
So quite frankly, as a result I refuse to have anything to do with the process directly.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Torvald Riegel
Which example do you have in mind here? Haven't we resolved all the debated examples, or did I miss any?
Well, Paul seems to still think that the standard possibly allows speculative writes or possibly value speculation in ways that break the hardware-guaranteed orderings.
That's true, I just didn't see any specific examples so far.
Quoted 2 lines
And personally, I can't read standards paperwork. It is invariably written in some basically impossible-to-understand lawyeristic mode,
Yeah, it's not the most intuitive form for things like the memory model.
Quoted 3 lines
and then it is read by people (compiler writers) that intentionally try to mis-use the words and do language-lawyering ("that depends on what the meaning of 'is' is").
That assumption about people working on compilers is a little too broad, don't you think?
I think that it is important to stick to a specification, in the same way that one wouldn't expect a program with undefined behavior make any sense of it, magically, in cases where stuff is undefined.
However, that of course doesn't include trying to exploit weasel-wording (BTW, both users and compiler writers try to do it). IMHO, weasel-wording in a standard is a problem in itself even if not exploited, and often it indicates that there is a real issue. There might be reasons to have weasel-wording (e.g., because there's no known better way to express it like in case of the not really precise no-out-of-thin-air rule today), but nonetheless those aren't ideal.
Quoted 3 lines
The whole "lvalue vs rvalue expression
vs 'what is a volatile access'" thing for C++ was/is a great example of that.
I'm not aware of the details of this.
Quoted 2 lines
So quite frankly, as a result I refuse to have anything to do with the process directly.
That's unfortunate. Then please work with somebody that isn't uncomfortable with participating directly in the process. But be warned, it may very well be a person working on compilers :)
Have you looked at the formalization of the model by Batty et al.? The overview of this is prose, but the formalized model itself is all formal relations and logic. So there should be no language-lawyering issues with that form. (For me, the formalized model is much easier to reason about.)
Quoted 7 lines · Torvald Riegel
Quoted 3 lines · Linus Torvalds
and then it is read by people (compiler writers) that intentionally try to mis-use the words and do language-lawyering ("that depends on what the meaning of 'is' is").
That assumption about people working on compilers is a little too broad, don't you think?
Let's just say that *some* are that way, and those are the ones that I end up butting heads with.
The sane ones I never have to argue with - point them at a bug, and they just say "yup, bug". The insane ones say "we don't need to fix that, because if you read this copy of the standards that have been translated to chinese and back, it clearly says that this is acceptable".
Quoted 5 lines
Quoted 3 lines
The whole "lvalue vs rvalue expression
vs 'what is a volatile access'" thing for C++ was/is a great example of that.I'm not aware of the details of this.
The argument was that an lvalue doesn't actually "access" the memory (an rvalue does), so this:
volatile int *p = ...;
*p;
doesn't need to generate a load from memory, because "*p" is still an lvalue (since you could assign things to it).
This isn't an issue in C, because in C, expression statements are always rvalues, but C++ changed that. The people involved with the C++ standards have generally been totally clueless about their subtle changes.
I may have misstated something, but basically some C++ people tried very hard to make "volatile" useless.
We had other issues too. Like C compiler people who felt that the type-based aliasing should always override anything else, even if the variable accessed (through different types) was statically clearly aliasing and used the exact same pointer. That made it impossible to do a syntactically clean model of "this aliases", since the _only_ exception to the type-based aliasing rule was to generate a union for every possible access pairing.
We turned off type-based aliasing (as I've mentioned before, I think it's a fundamentally broken feature to begin with, and a horrible horrible hack that adds no value for anybody but the HPC people).
Gcc eventually ended up having some sane syntax for overriding it, but by then I was too disgusted with the people involved to even care.
Linus
Quoted 29 lines · Linus Torvalds
Quoted 7 lines · Torvald Riegel
Quoted 3 lines · Linus Torvalds
and then it is read by people (compiler writers) that intentionally try to mis-use the words and do language-lawyering ("that depends on what the meaning of 'is' is").
That assumption about people working on compilers is a little too broad, don't you think?
Let's just say that *some* are that way, and those are the ones that I end up butting heads with.
The sane ones I never have to argue with - point them at a bug, and they just say "yup, bug". The insane ones say "we don't need to fix that, because if you read this copy of the standards that have been translated to chinese and back, it clearly says that this is acceptable".
Quoted 5 lines
Quoted 3 lines
The whole "lvalue vs rvalue expression
vs 'what is a volatile access'" thing for C++ was/is a great example of that.I'm not aware of the details of this.
The argument was that an lvalue doesn't actually "access" the memory (an rvalue does), so this:
volatile int *p = ...;
Quoted 7 lines
*p;doesn't need to generate a load from memory, because "*p" is still an lvalue (since you could assign things to it).
This isn't an issue in C, because in C, expression statements are always rvalues, but C++ changed that.
Huhh. I can see the problems that this creates in terms of C/C++ compatibility.
Quoted 3 lines
The people involved with the C++
standards have generally been totally clueless about their subtle changes.
This isn't a fair characterization. There are many people that do care, and certainly not all are clueless. But it's a limited set of people, bugs happen, and not all of them will have the same goals.
I think one way to prevent such problems in the future could be to have someone in the kernel community volunteer to look through standard revisions before they are published. The standard needs to be fixed, because compilers need to conform to the standard (e.g., a compiler's extension "fixing" the above wouldn't be conforming anymore because it emits more volatile reads than specified).
Or maybe those of us working on the standard need to flag potential changes of interest to the kernel folks. But that may be less reliable than someone from the kernel side looking at them; I don't know.
Quoted 9 lines · Torvald Riegel
Quoted 8 lines · Linus Torvalds
The argument was that an lvalue doesn't actually "access" the memory (an rvalue does), so this:
volatile int *p = ...;*p;
Quoted 8 lines
Quoted 5 lines
doesn't need to generate a load from memory, because "*p" is still an lvalue (since you could assign things to it).
This isn't an issue in C, because in C, expression statements are always rvalues, but C++ changed that.
Huhh. I can see the problems that this creates in terms of C/C++ compatibility.
That's not the biggest problem.
The biggest problem is that you have compiler writers that don't care about sane *use* of the features they write a compiler for, they just care about the standard.
So they don't care about C vs C++ compatibility. Even more importantly, they don't care about the *user* that uses only C++ and the fact that their reading of the standard results in *meaningless* behavior. They point to the standard and say "that's what the standard says, suck it", and silently generate code (or in this case, avoid generating code) that makes no sense.
So it's not about C++ being incompatible with C, it's about C++ having insane and bad semantics unless you just admit that "oh, ok, I need to not just read the standard, I also need to use my brain, and admit that a C++ statement expression needs to act as if it is an "access" wrt volatile variables".
In other words, as a compiler person, you do need to read more than the paper of standard. You need to also take into account what is reasonable behavior even when the standard could possibly be read some other way. And some compiler people don't.
The "volatile access in statement expression" did get resolved, sanely, at least in gcc. I think gcc warns about some remaining cases.
Btw, afaik, C++11 actually clarifies the standard to require the reads, because everybody *knew* that not requiring the read was insane and meaningless behavior, and clearly against the intent of "volatile".
But that didn't stop compiler writers from saying "hey, the standard allows my insane and meaningless behavior, so I'll implement it and not consider it a bug".
Linus
Quoted 25 lines · Linus Torvalds
Quoted 9 lines · Torvald Riegel
Quoted 8 lines · Linus Torvalds
The argument was that an lvalue doesn't actually "access" the memory (an rvalue does), so this:
volatile int *p = ...;*p;Quoted 8 lines
Quoted 5 lines
doesn't need to generate a load from memory, because "*p" is still an lvalue (since you could assign things to it).
This isn't an issue in C, because in C, expression statements are always rvalues, but C++ changed that.
Huhh. I can see the problems that this creates in terms of C/C++ compatibility.
That's not the biggest problem.
The biggest problem is that you have compiler writers that don't care about sane *use* of the features they write a compiler for, they just care about the standard.
Quoted 6 lines
So they don't care about C vs C++ compatibility. Even more importantly, they don't care about the *user* that uses only C++ and the fact that their reading of the standard results in *meaningless* behavior. They point to the standard and say "that's what the standard says, suck it", and silently generate code (or in this case, avoid generating code) that makes no sense.
There's an underlying problem here that's independent from the actual instance that you're worried about here: "no sense" is a ultimately a matter of taste/objectives/priorities as long as the respective specification is logically consistent.
If you want to be independent of your sanity being different from other people's sanity (e.g., compiler writers), you need to make sure that the specification is precise and says what you want. IOW, think about the specification being the program, and the people being computers; you better want a well-defined program in this case.
Quoted 5 lines
So it's not about C++ being incompatible with C, it's about C++ having insane and bad semantics unless you just admit that "oh, ok, I need to not just read the standard, I also need to use my brain, and admit that a C++ statement expression needs to act as if it is an "access" wrt volatile variables".
1) I agree that (IMO) a good standard strives for being easy to understand.
2) In practice, there is a trade-off between "Easy to understand" and actually producing a specification. A standard is not a tutorial. And that's for good reason, because (a) there might be more than one way to teach something and that should be allowed and (b) that the standard should carry the full precision but still be compact enough to be manageable.
3) Implementations can try to be nice to users by helping them avoiding error-prone corner cases or such. A warning for common problems is such a case. But an implementation has to draw a line somewhere, demarcating cases where it fully exploits what the standard says (eg, to allow optimizations) from cases where it is more conservative and does what the standard allows but in a potentially more intuitive way. That's especially the case if it's being asked to produce high-performance code.
4) There will be arguments for where the line actually is, simply because different users will have different goals.
5) The way to reduce 4) is to either make the standard more specific, or to provide better user documentation. If the standard has strict requirements, then there will be less misunderstanding.
6) To achieve 5), one way is to get involved in the standards process.
Quoted 5 lines · Torvald Riegel
There's an underlying problem here that's independent from the actual instance that you're worried about here: "no sense" is a ultimately a matter of taste/objectives/priorities as long as the respective specification is logically consistent.
Yes. But I don't think it's "independent".
Exactly *because* some people will read standards without applying "does the resulting code generation actually make sense for the programmer that wrote the code", the standard has to be pretty clear.
The standard often *isn't* pretty clear. It wasn't clear enough when it came to "volatile", and yet that was a *much* simpler concept than atomic accesses and memory ordering.
And most of the time it's not a big deal. But because the C standard generally tries to be very portable, and cover different machines, there tends to be a mindset that anything inherently unportable is "undefined" or "implementation defined", and then the compiler writer is basically given free reign to do anything they want (with "implementation defined" at least requiring that it is reliably the same thing).
And when it comes to memory ordering, *everything* is basically non-portable, because different CPU's very much have different rules. I worry that that means that the standard then takes the stance that "well, compiler re-ordering is no worse than CPU re-ordering, so we let the compiler do anything". And then we have to either add "volatile" to make sure the compiler doesn't do that, or use an overly strict memory model at the compiler level that makes it all pointless.
So I really really hope that the standard doesn't give compiler writers free hands to do anything that they can prove is "equivalent" in the virtual C machine model. That's not how you get reliable results.
Linus
Quoted 25 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
There's an underlying problem here that's independent from the actual instance that you're worried about here: "no sense" is a ultimately a matter of taste/objectives/priorities as long as the respective specification is logically consistent.
Yes. But I don't think it's "independent".
Exactly *because* some people will read standards without applying "does the resulting code generation actually make sense for the programmer that wrote the code", the standard has to be pretty clear.
The standard often *isn't* pretty clear. It wasn't clear enough when it came to "volatile", and yet that was a *much* simpler concept than atomic accesses and memory ordering.
And most of the time it's not a big deal. But because the C standard generally tries to be very portable, and cover different machines, there tends to be a mindset that anything inherently unportable is "undefined" or "implementation defined", and then the compiler writer is basically given free reign to do anything they want (with "implementation defined" at least requiring that it is reliably the same thing).
Quoted 7 lines
And when it comes to memory ordering, *everything* is basically non-portable, because different CPU's very much have different rules. I worry that that means that the standard then takes the stance that "well, compiler re-ordering is no worse than CPU re-ordering, so we let the compiler do anything". And then we have to either add "volatile" to make sure the compiler doesn't do that, or use an overly strict memory model at the compiler level that makes it all pointless.
For whatever it is worth, this line of reasoning has been one reason why I have been objecting strenuously every time someone on the committee suggests eliminating "volatile" from the standard.
Thanx, Paul
Quoted 7 lines
So I really really hope that the standard doesn't give compiler writers free hands to do anything that they can prove is "equivalent" in the virtual C machine model. That's not how you get reliable results.
Linus
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
There's an underlying problem here that's independent from the actual instance that you're worried about here: "no sense" is a ultimately a matter of taste/objectives/priorities as long as the respective specification is logically consistent.
Yes. But I don't think it's "independent".
Exactly *because* some people will read standards without applying "does the resulting code generation actually make sense for the programmer that wrote the code", the standard has to be pretty clear.
The standard often *isn't* pretty clear. It wasn't clear enough when it came to "volatile", and yet that was a *much* simpler concept than atomic accesses and memory ordering.
Quoted 7 lines
And most of the time it's not a big deal. But because the C standard generally tries to be very portable, and cover different machines, there tends to be a mindset that anything inherently unportable is "undefined" or "implementation defined", and then the compiler writer is basically given free reign to do anything they want (with "implementation defined" at least requiring that it is reliably the same thing).
Yes, that's how it works in general. And this makes sense, because all optimizations rely on that. Second, you can't keep something consistent (eg, between compilers) if it isn't specified. So if we want stricter rules, those need to be specified somewhere.
Quoted 2 lines
And when it comes to memory ordering, *everything* is basically non-portable, because different CPU's very much have different rules.
Well, the current set of memory orders (and the memory model as a whole) is portable, even though it might not allow to exploit all hardware properties, and thus might perform sub-optimally in some cases.
Quoted 5 lines
I worry that that means that the standard then takes the stance that "well, compiler re-ordering is no worse than CPU re-ordering, so we let the compiler do anything". And then we have to either add "volatile" to make sure the compiler doesn't do that, or use an overly strict memory model at the compiler level that makes it all pointless.
Using "volatile" is not a good option, I think, because synchronization between threads should be orthogonal to observable output of the abstract machine.
The current memory model might not allow to exploit all hardware properties, I agree.
But then why don't we work on how to extend it to do so? We need to specify the behavior we want anyway, and this can't be independent of the language semantics, so it has to be conceptually integrated with the standard anyway.
Quoted 3 lines
So I really really hope that the standard doesn't give compiler writers free hands to do anything that they can prove is "equivalent" in the virtual C machine model.
It does, but it also doesn't mean this can't be extended. So let's focus on whether we can find an extension.
Quoted 2 lines
That's not how you get reliable
results.
In this general form, that's obviously a false claim.
Quoted 44 lines · Torvald Riegel
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
There's an underlying problem here that's independent from the actual instance that you're worried about here: "no sense" is a ultimately a matter of taste/objectives/priorities as long as the respective specification is logically consistent.
Yes. But I don't think it's "independent".
Exactly *because* some people will read standards without applying "does the resulting code generation actually make sense for the programmer that wrote the code", the standard has to be pretty clear.
The standard often *isn't* pretty clear. It wasn't clear enough when it came to "volatile", and yet that was a *much* simpler concept than atomic accesses and memory ordering.
Quoted 7 lines
And most of the time it's not a big deal. But because the C standard generally tries to be very portable, and cover different machines, there tends to be a mindset that anything inherently unportable is "undefined" or "implementation defined", and then the compiler writer is basically given free reign to do anything they want (with "implementation defined" at least requiring that it is reliably the same thing).
Yes, that's how it works in general. And this makes sense, because all optimizations rely on that. Second, you can't keep something consistent (eg, between compilers) if it isn't specified. So if we want stricter rules, those need to be specified somewhere.
Quoted 2 lines
And when it comes to memory ordering, *everything* is basically non-portable, because different CPU's very much have different rules.
Well, the current set of memory orders (and the memory model as a whole) is portable, even though it might not allow to exploit all hardware properties, and thus might perform sub-optimally in some cases.
Quoted 5 lines
I worry that that means that the standard then takes the stance that "well, compiler re-ordering is no worse than CPU re-ordering, so we let the compiler do anything". And then we have to either add "volatile" to make sure the compiler doesn't do that, or use an overly strict memory model at the compiler level that makes it all pointless.
Quoted 3 lines
Using "volatile" is not a good option, I think, because synchronization between threads should be orthogonal to observable output of the abstract machine.
Are you thinking of "volatile" -instead- of atomics? My belief is that given the current standard there will be times that we need to use "volatile" -in- -addition- to atomics.
Quoted 12 lines
The current memory model might not allow to exploit all hardware properties, I agree.
But then why don't we work on how to extend it to do so? We need to specify the behavior we want anyway, and this can't be independent of the language semantics, so it has to be conceptually integrated with the standard anyway.
Quoted 3 lines
So I really really hope that the standard doesn't give compiler writers free hands to do anything that they can prove is "equivalent" in the virtual C machine model.
Quoted 7 lines
It does, but it also doesn't mean this can't be extended. So let's focus on whether we can find an extension.
Quoted 2 lines
That's not how you get reliable
results.In this general form, that's obviously a false claim.
These two sentences starkly illustrate the difference in perspective between you two. You are talking past each other. Not sure how to fix this at the moment, but what else is new? ;-)
Thanx, Paul
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Torvald Riegel
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Torvald Riegel
There's an underlying problem here that's independent from the actual instance that you're worried about here: "no sense" is a ultimately a matter of taste/objectives/priorities as long as the respective specification is logically consistent.
Yes. But I don't think it's "independent".
Exactly *because* some people will read standards without applying "does the resulting code generation actually make sense for the programmer that wrote the code", the standard has to be pretty clear.
The standard often *isn't* pretty clear. It wasn't clear enough when it came to "volatile", and yet that was a *much* simpler concept than atomic accesses and memory ordering.
Quoted 7 lines
And most of the time it's not a big deal. But because the C standard generally tries to be very portable, and cover different machines, there tends to be a mindset that anything inherently unportable is "undefined" or "implementation defined", and then the compiler writer is basically given free reign to do anything they want (with "implementation defined" at least requiring that it is reliably the same thing).
Yes, that's how it works in general. And this makes sense, because all optimizations rely on that. Second, you can't keep something consistent (eg, between compilers) if it isn't specified. So if we want stricter rules, those need to be specified somewhere.
Quoted 2 lines
And when it comes to memory ordering, *everything* is basically non-portable, because different CPU's very much have different rules.
Well, the current set of memory orders (and the memory model as a whole) is portable, even though it might not allow to exploit all hardware properties, and thus might perform sub-optimally in some cases.
Quoted 5 lines
I worry that that means that the standard then takes the stance that "well, compiler re-ordering is no worse than CPU re-ordering, so we let the compiler do anything". And then we have to either add "volatile" to make sure the compiler doesn't do that, or use an overly strict memory model at the compiler level that makes it all pointless.
Quoted 7 lines
Quoted 3 lines
Using "volatile" is not a good option, I think, because synchronization between threads should be orthogonal to observable output of the abstract machine.
Are you thinking of "volatile" -instead- of atomics? My belief is that given the current standard there will be times that we need to use "volatile" -in- -addition- to atomics.
No, I was talking about having to use "volatile" in addition to atomics to get synchronization properties for the atomics. ISTM that this would be unfortunate because the objective for "volatile" (ie, declaring what is output of the abstract machine) is orthogonal to synchronization between threads. So if we want to preserve control dependencies and get ordering guarantees through that, I wouldn't prefer it through hacks based on volatile.
For example, if we have a macro or helper function that contains synchronization code, but the compiler can prove that the macro/function is used just on data only accessible to a single thread, then the "volatile" notation on it would prevent optimizations.
Quoted 15 lines
Quoted 15 lines
The current memory model might not allow to exploit all hardware properties, I agree.
But then why don't we work on how to extend it to do so? We need to specify the behavior we want anyway, and this can't be independent of the language semantics, so it has to be conceptually integrated with the standard anyway.
Quoted 3 lines
So I really really hope that the standard doesn't give compiler writers free hands to do anything that they can prove is "equivalent" in the virtual C machine model.
It does, but it also doesn't mean this can't be extended. So let's focus on whether we can find an extension.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
That's not how you get reliable
results.In this general form, that's obviously a false claim.
These two sentences starkly illustrate the difference in perspective between you two. You are talking past each other. Not sure how to fix this at the moment, but what else is new? ;-)
Well, we're still talking, and we are making little steps in clarifying things, so there is progress :)
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Torvald Riegel
Which example do you have in mind here? Haven't we resolved all the debated examples, or did I miss any?
Well, Paul seems to still think that the standard possibly allows speculative writes or possibly value speculation in ways that break the hardware-guaranteed orderings.
And personally, I can't read standards paperwork. It is invariably
Can't => Don't - evidently.
Quoted 1 line
written in some basically impossible-to-understand lawyeristic mode,
You mean "unambiguous" - try reading a patent (Apple have 1000s of trivial ones, I tried reading one once thinking "how could they have phrased it so this got approved", their technique was to make the reader want to start cutting themselves to prove they wern't numb to everything)
Quoted 5 lines
and then it is read by people (compiler writers) that intentionally try to mis-use the words and do language-lawyering ("that depends on what the meaning of 'is' is"). The whole "lvalue vs rvalue expression vs 'what is a volatile access'" thing for C++ was/is a great example of that.
I'm not going to teach you what rvalues and lvalues, but! http://lmgtfy.com/?q=what+are+rvalues might help.
Quoted 3 lines
So quite frankly, as a result I refuse to have anything to do with the process directly.
Is this goodbye?
Quoted 2 lines
Linus
That aside, what is the problem? If the compiler has created code that that has different program states than what would be created without optimisation please file a bug report and/or send something to the mailing list USING A CIVIL TONE, there's no need for swear-words and profanities all the time - use them when you want to emphasise something. Additionally if you are always angry, start calling that state "normal" then reserve such words for when you are outraged.
There are so many emails from you bitching about stuff, I've lost track of what you're bitching about you bitch that much about it. Like this standards stuff above (notice I said stuff, not "crap" or "shit").
What exactly is your problem, if the compiler is doing something the standard does not permit, or optimising something wrongly (read: "puts the program in a different state than if the optimisation was not applied") that is REALLY serious, you are right to report it; but whining like a n00b on Stack-overflow when a question gets closed is not helping.
I tried reading back though the emails (I dismissed them previously) but there's just so much ranting, and rants about the standard too (I would trash this if I deemed the effort required to delete was less than the storage of the bytes the message takes up) standardised behaviour is VERY important.
So start again, what is the serious problem, have you got any code that would let me replicate it, what is your version of GCC?
Oh and lastly! Optimisations are not as casual as "oh, we could do this and it'd work better" unlike kernel work or any other software that is being improved, it is very formal (and rightfully so). I seriously recommend you read the first 40 pages at least of a book called "Compiler Design, Analysis and Transformation" it's not about the parsing phases or anything, but it develops a good introduction and later a good foundation for exploring the field further. Compilers do not operate on what I call "A-level logic" and to show what I mean I use the shovel-to-the-face of real analysis, "of course 1/x tends towards 0, it's not gonna be 5!!" = A-level logic. "Let epsilon > 0 be given, then there exists an N...." - formal proof. So when one says "the compiler can prove" it's not some silly thing powered by A-level logic, it is the implementation of something that can be proven to be correct (in the sense of the program states mentioned before)
So yeah, calm down and explain - no lashing out at standards bodies, what is the problem?
Alec
Quoted 5 lines · Alec Teal
You mean "unambiguous" - try reading a patent (Apple have 1000s of trivial ones, I tried reading one once thinking "how could they have phrased it so this got approved", their technique was to make the reader want to start cutting themselves to prove they wern't numb to everything)
Oh, I agree, patent language is worse.
Quoted 1 line
I'm not going to teach you what rvalues and lvalues, but!
I know what lvalues and rvalues are. I *understand* the thinking that goes on behind the "let's not do the access, because it's not an rvalue, so there is no 'access' to the object".
I understand it from a technical perspective.
I don't understand the compiler writer that uses a *technicality* to argue against generating sane code that is obviously what the user actually asked for.
See the difference?
Quoted 2 lines
So start again, what is the serious problem, have you got any code that would let me replicate it, what is your version of GCC?
The volatile problem is long fixed. The people who argued for the "legalistically correct", but insane behavior lost (and as mentioned, I think C++11 actually fixed the legalistic reading too).
I'm bringing it up because I've had too many cases where compiler writers pointed to standard and said "that is ambiguous or undefined, so we can do whatever the hell we want, regardless of whether that's sensible, or regardless of whether there is a sensible way to get the behavior you want or not".
Quoted 3 lines
Oh and lastly! Optimisations are not as casual as "oh, we could do this and it'd work better" unlike kernel work or any other software that is being improved, it is very formal (and rightfully so)
Alec, I know compilers. I don't do code generation (quite frankly, register allocation and instruction choice is when I give up), but I did actually write my own for static analysis, including turning things into SSA etc.
No, I'm not a "compiler person", but I actually do know enough that I understand what goes on.
And exactly because I know enough, I would *really* like atomics to be well-defined, and have very clear - and *local* - rules about how they can be combined and optimized.
None of this "if you can prove that the read has value X" stuff. And things like value speculation should simply not be allowed, because that actually breaks the dependency chain that the CPU architects give guarantees for. Instead, make the rules be very clear, and very simple, like my suggestion. You can never remove a load because you can "prove" it has some value, but you can combine two consecutive atomic accesses/
For example, CPU people actually do tend to give guarantees for certain things, like stores that are causally related being visible in a particular order. If the compiler starts doing value speculation on atomic accesses, you are quite possibly breaking things like that. It's just not a good idea. Don't do it. Write the standard so that it clearly is disallowed.
Because you may think that a C standard is machine-independent, but that isn't really the case. The people who write code still write code for a particular machine. Our code works (in the general case) on different byte orderings, different register sizes, different memory ordering models. But in each *instance* we still end up actually coding for each machine.
So the rules for atomics should be simple and *specific* enough that when you write code for a particular architecture, you can take the architecture memory ordering *and* the C atomics orderings into account, and do the right thing for that architecture.
And that very much means that doing things like value speculation MUST NOT HAPPEN. See? Even if you can prove that your code is "equivalent", it isn't.
So for example, let's say that you have a pointer, and you have some reason to believe that the pointer has a particular value. So you rewrite following the pointer from this:
value = ptr->val;
into
value = speculated->value;
tmp = ptr;
if (unlikely(tmp != speculated))
value = tmp->value;and maybe you can now make the critical code-path for the speculated case go faster (since now there is no data dependency for the speculated case, and the actual pointer chasing load is now no longer in the critical path), and you made things faster because your profiling showed that the speculated case was true 99% of the time. Wonderful, right? And clearly, the code "provably" does the same thing.
EXCEPT THAT IS NOT TRUE AT ALL.
It very much does not do the same thing at all, and by doing value speculation and "proving" something was true, the only thing you did was to make incorrect code run faster. Because now the causally related load of value from the pointer isn't actually causally related at all, and you broke the memory ordering.
This is why I don't like it when I see Torvald talk about "proving" things. It's bullshit. You can "prove" pretty much anything, and in the process lose sight of the bigger issue, namely that there is code that depends on
When it comes to atomic accesses, you don't play those kinds of games, exactly because the ordering of the accesses matter in ways that are not really sanely describable at a source code level. The *only* kinds of games you play are like the ones I described - combining accesses under very strict local rules.
And the strict local rules really should be of the type "a store followed by a load to the same location with the same memory ordering constraints can be combined". Never *ever* of the kind "if you can prove X".
I hope my example made it clear *why* I react so strongly when Torvald starts talking about "if you can prove the value is 1".
Linus