Skip to contentSkip to conversation

locking changes for v4.8

GIT PULL Ingo Molnar 1 participant 1 message Jul 25, 2016

Subsystems (28)
Networking Documentation Arm (32-bit) x86 +24 more subsystems
lore
Share & tools
lore

Ingo Molnar

author
#0 lore raw

Linus,

Please pull the latest locking-core-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking-core-for-linus

# HEAD: f06628638cf6e75f179742b6c1b35076965b9fdd locking/atomic: Introduce inc/dec variants for the atomic_fetch_$op() API

The locking tree was busier in this cycle than the usual pattern - a couple of major projects happened to coincide. The main changes are:

- Implement the atomic_fetch_{add,sub,and,or,xor}() API natively across all SMP architectures (Peter Zijlstra)

- Add atomic_fetch_{inc/dec}() as well, using the generic primitives (Davidlohr Bueso)

- Optimize various aspects of rwsems (Jason Low, Davidlohr Bueso, Waiman Long)

- Optimize smp_cond_load_acquire() on arm64 and implement LSE based atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() on arm64 (Will Deacon)

- Introduce smp_acquire__after_ctrl_dep() and fix various barrier mis-uses and bugs (Peter Zijlstra)

- After discovering ancient spin_unlock_wait() barrier bugs in its implementation and usage, strengthen its semantics and update/fix usage sites (Peter Zijlstra)

- Optimize mutex_trylock() fastpath (Peter Zijlstra)

- ... misc fixes and cleanups

Thanks,

Ingo

------------------>
Davidlohr Bueso (3):
locking/rwsem: Enable lockless waiter wakeup(s) locking/rwsem: Rework zeroing reader waiter->task locking/atomic: Introduce inc/dec variants for the atomic_fetch_$op() API

Ingo Molnar (1):
MAINTAINERS: Update locking tree description and file patterns

Jason Low (4):
locking/rwsem: Optimize write lock by reducing operations in slowpath locking/mutex: Set and clear owner using WRITE_ONCE() locking/rwsem: Convert sem->count to 'atomic_long_t' locking/rwsem: Remove rwsem_atomic_add() and rwsem_atomic_update()

Pan Xinhui (2):
locking/qspinlock: Use atomic_sub_return_release() in queued_spin_unlock() locking/qspinlock: Use __this_cpu_dec() instead of full-blown this_cpu_dec()

Paul E. McKenney (1):
locking/Documentation: Clarify limited control-dependency scope

Peter Zijlstra (46):
locking/lockdep: Use __jhash_mix() for iterate_chain_key()
locking/barriers: Validate lockless_dereference() is used on a pointer type
locking/qspinlock: Clarify xchg_tail() ordering
locking/qspinlock: Add comments
locking/mutex: Optimize mutex_trylock() fast-path
locking/barriers: Replace smp_cond_acquire() with smp_cond_load_acquire()
locking/barriers: Introduce smp_acquire__after_ctrl_dep()
locking/barriers: Move smp_cond_load_acquire() to asm-generic/barrier.h
locking/barriers, tile: Provide TILE specific smp_acquire__after_ctrl_dep()
locking/spinlock, arch: Update and fix spin_unlock_wait() implementations
locking/spinlock: Update spin_unlock_wait() users
locking/spinlock, netfilter: Fix nf_conntrack_lock() barriers
locking/atomic, arch/alpha: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}() locking/atomic, arch/arc: Implement atomic_fetch_{add,sub,and,andnot,or,xor}()
locking/atomic, arch/arm: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() locking/atomic, arch/arm64: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() locking/atomic, arch/avr32: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/blackfin: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/frv: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/h8300: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/hexagon: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/ia64: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/m32r: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/m68k: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/metag: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/mips: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/mn10300: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/parisc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}() locking/atomic, arch/powerpc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}{,_relaxed,_acquire,_release}() locking/atomic, arch/s390: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/sh: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/sparc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/tile: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/x86: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
locking/atomic, arch/xtensa: Implement atomic_fetch_{add,sub,and,or,xor}()
locking/atomic: Fix atomic64_relaxed() bits
locking/atomic: Implement atomic{,64,_long}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() locking/atomic: Remove linux/atomic.h:atomic_fetch_or()
locking/atomic: Remove the deprecated atomic_{set,clear}_mask() functions
locking/atomic, arch/alpha: Convert to _relaxed atomics
locking/atomic, arch/mips: Convert to _relaxed atomics
locking/atomic, arch/qrwlock: Employ atomic_fetch_add_acquire()
locking/atomic, arch/rwsem: Employ atomic_long_fetch_add()
locking/atomic, arch/arc: Fix build
locking/atomic, arch/m68k: Remove comment
locking/atomic, arch/tile: Fix tilepro build

Sebastian Andrzej Siewior (1):
locking/rtmutex: Only warn once on a trylock from bad context

Tejun Heo (1):
percpu, locking: Revert ("percpu: Replace smp_read_barrier_depends() with lockless_dereference()")

Waiman Long (4):
locking/rwsem: Add reader-owned state to the owner field locking/rwsem: Protect all writes to owner by WRITE_ONCE() locking/rwsem: Improve reader wakeup code locking/rwsem: Streamline the rwsem_optimistic_spin() code

Wei Yongjun (1):
locking/static_keys: Fix non static symbol Sparse warning

Will Deacon (3):
locking/atomic, arch/arm64: Generate LSE non-return cases using common macros
locking/atomic, arch/arm64: Implement atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() for LSE instructions locking/barriers, arch/arm64: Implement LDXR+WFE based smp_cond_load_acquire()

 Documentation/memory-barriers.txt      |  41 ++
 MAINTAINERS                            |  16 +-
 arch/alpha/include/asm/atomic.h        |  87 +++-
 arch/alpha/include/asm/rwsem.h         |  68 +--
 arch/alpha/include/asm/spinlock.h      |   9 +-
 arch/arc/include/asm/atomic.h          |  99 ++++-
 arch/arc/include/asm/spinlock.h        |   7 +-
 arch/arm/include/asm/atomic.h          | 106 ++++-
 arch/arm/include/asm/spinlock.h        |  19 +-
 arch/arm64/include/asm/atomic.h        |  60 +++
 arch/arm64/include/asm/atomic_ll_sc.h  | 110 +++--
 arch/arm64/include/asm/atomic_lse.h    | 278 ++++++++----
 arch/arm64/include/asm/barrier.h       |  13 +
 arch/arm64/include/asm/cmpxchg.h       |  51 +++
 arch/avr32/include/asm/atomic.h        |  54 ++-
 arch/blackfin/include/asm/atomic.h     |   8 +
 arch/blackfin/include/asm/spinlock.h   |   5 +-
 arch/blackfin/kernel/bfin_ksyms.c      |   1 +
 arch/blackfin/mach-bf561/atomic.S      |  43 +-
 arch/frv/include/asm/atomic.h          |  30 +-
 arch/frv/include/asm/atomic_defs.h     |   2 +
 arch/h8300/include/asm/atomic.h        |  29 +-
 arch/hexagon/include/asm/atomic.h      |  31 +-
 arch/hexagon/include/asm/spinlock.h    |  10 +-
 arch/ia64/include/asm/atomic.h         | 130 +++++-
 arch/ia64/include/asm/mutex.h          |   2 +-
 arch/ia64/include/asm/rwsem.h          |  31 +-
 arch/ia64/include/asm/spinlock.h       |   4 +
 arch/m32r/include/asm/atomic.h         |  36 +-
 arch/m32r/include/asm/spinlock.h       |   9 +-
 arch/m68k/include/asm/atomic.h         |  44 +-
 arch/metag/include/asm/atomic_lnkget.h |  36 +-
 arch/metag/include/asm/atomic_lock1.h  |  33 +-
 arch/metag/include/asm/spinlock.h      |  14 +-
 arch/mips/include/asm/atomic.h         | 154 ++++++-
 arch/mips/include/asm/spinlock.h       |  19 +-
 arch/mn10300/include/asm/atomic.h      |  33 +-
 arch/mn10300/include/asm/spinlock.h    |   8 +-
 arch/parisc/include/asm/atomic.h       |  63 ++-
 arch/parisc/include/asm/spinlock.h     |   9 +-
 arch/powerpc/include/asm/atomic.h      |  83 +++-
 arch/powerpc/include/asm/mutex.h       |   2 +-
 arch/s390/include/asm/atomic.h         |  40 +-
 arch/s390/include/asm/rwsem.h          |  37 --
 arch/s390/include/asm/spinlock.h       |   3 +
 arch/sh/include/asm/atomic-grb.h       |  34 +-
 arch/sh/include/asm/atomic-irq.h       |  31 +-
 arch/sh/include/asm/atomic-llsc.h      |  32 +-
 arch/sh/include/asm/spinlock.h         |  10 +-
 arch/sparc/include/asm/atomic_32.h     |  13 +-
 arch/sparc/include/asm/atomic_64.h     |  16 +-
 arch/sparc/include/asm/spinlock_32.h   |   7 +-
 arch/sparc/include/asm/spinlock_64.h   |  10 +-
 arch/sparc/lib/atomic32.c              |  29 +-
 arch/sparc/lib/atomic_64.S             |  61 ++-
 arch/sparc/lib/ksyms.c                 |  17 +-
 arch/tile/include/asm/atomic.h         |   2 +
 arch/tile/include/asm/atomic_32.h      |  74 ++--
 arch/tile/include/asm/atomic_64.h      | 115 +++--
 arch/tile/include/asm/barrier.h        |   7 +
 arch/tile/include/asm/bitops_32.h      |  18 +-
 arch/tile/include/asm/futex.h          |  14 +-
 arch/tile/lib/atomic_32.c              |  50 +--
 arch/tile/lib/atomic_asm_32.S          |  27 +-
 arch/tile/lib/spinlock_32.c            |   6 +
 arch/tile/lib/spinlock_64.c            |   6 +
 arch/x86/include/asm/atomic.h          |  35 +-
 arch/x86/include/asm/atomic64_32.h     |  25 +-
 arch/x86/include/asm/atomic64_64.h     |  35 +-
 arch/x86/include/asm/mutex_32.h        |   2 +-
 arch/x86/include/asm/mutex_64.h        |   6 +-
 arch/x86/include/asm/rwsem.h           |  18 -
 arch/xtensa/include/asm/atomic.h       |  52 ++-
 arch/xtensa/include/asm/spinlock.h     |  10 +-
 include/asm-generic/atomic-long.h      |  58 ++-
 include/asm-generic/atomic.h           |  47 +++
 include/asm-generic/atomic64.h         |  15 +-
 include/asm-generic/barrier.h          |  41 +-
 include/asm-generic/mutex-dec.h        |   2 +-
 include/asm-generic/mutex-xchg.h       |   6 +-
 include/asm-generic/qspinlock.h        |   5 +-
 include/asm-generic/rwsem.h            |  22 +-
 include/linux/atomic.h                 | 747 ++++++++++++++++++++++++++-------
 include/linux/compiler.h               |  21 +-
 include/linux/percpu-refcount.h        |  12 +-
 include/linux/rwsem.h                  |   8 +-
 include/linux/spinlock_up.h            |  10 +-
 ipc/sem.c                              |  13 +-
 kernel/exit.c                          |   8 +-
 kernel/jump_label.c                    |   2 +-
 kernel/locking/lockdep.c               |  13 +-
 kernel/locking/mutex-debug.h           |   4 +-
 kernel/locking/mutex.h                 |  10 +-
 kernel/locking/qrwlock.c               |   2 +-
 kernel/locking/qspinlock.c             |  88 +++-
 kernel/locking/qspinlock_paravirt.h    |   4 +-
 kernel/locking/rtmutex.c               |   2 +-
 kernel/locking/rwsem-xadd.c            | 194 +++++----
 kernel/locking/rwsem.c                 |   8 +-
 kernel/locking/rwsem.h                 |  52 ++-
 kernel/rcu/tree.c                      |   1 -
 kernel/sched/core.c                    |   8 +-
 kernel/sched/sched.h                   |   2 +-
 kernel/smp.c                           |   2 +-
 kernel/task_work.c                     |   1 -
 lib/atomic64.c                         |  32 +-
 lib/atomic64_test.c                    |  34 ++
 net/netfilter/nf_conntrack_core.c      |  23 +-
 108 files changed, 3268 insertions(+), 958 deletions(-)
Documentation/memory-barriers.txt +41 -0
index 147ae8ec836f..a4d0a99de04d 100644
@@ -806,6 +806,41 @@ out-guess your code.  More generally, although READ_ONCE() does force
 the compiler to actually emit code for a given load, it does not force
 the compiler to use the results.
 
+In addition, control dependencies apply only to the then-clause and
+else-clause of the if-statement in question.  In particular, it does
+not necessarily apply to code following the if-statement:
+
+	q = READ_ONCE(a);
+	if (q) {
+		WRITE_ONCE(b, p);
+	} else {
+		WRITE_ONCE(b, r);
+	}
+	WRITE_ONCE(c, 1);  /* BUG: No ordering against the read from "a". */
+
+It is tempting to argue that there in fact is ordering because the
+compiler cannot reorder volatile accesses and also cannot reorder
+the writes to "b" with the condition.  Unfortunately for this line
+of reasoning, the compiler might compile the two writes to "b" as
+conditional-move instructions, as in this fanciful pseudo-assembly
+language:
+
+	ld r1,a
+	ld r2,p
+	ld r3,r
+	cmp r1,$0
+	cmov,ne r4,r2
+	cmov,eq r4,r3
+	st r4,b
+	st $1,c
+
+A weakly ordered CPU would have no dependency of any sort between the load
+from "a" and the store to "c".  The control dependencies would extend
+only to the pair of cmov instructions and the store depending on them.
+In short, control dependencies apply only to the stores in the then-clause
+and else-clause of the if-statement in question (including functions
+invoked by those two clauses), not to code following that if-statement.
+
 Finally, control dependencies do -not- provide transitivity.  This is
 demonstrated by two related examples, with the initial values of
 x and y both being zero:

@@ -869,6 +904,12 @@ site: https://www.cl.cam.ac.uk/~pes20/ppcmem/index.html.
       atomic{,64}_read() can help to preserve your control dependency.
       Please see the COMPILER BARRIER section for more information.
 
+  (*) Control dependencies apply only to the then-clause and else-clause
+      of the if-statement containing the control dependency, including
+      any functions that these two clauses call.  Control dependencies
+      do -not- apply to code following the if-statement containing the
+      control dependency.
+
   (*) Control dependencies pair normally with other types of barriers.
 
   (*) Control dependencies do -not- provide transitivity.  If you
MAINTAINERS +12 -4
index 16700e4fcc4a..2223d5b02053 100644
@@ -7007,15 +7007,23 @@ Q:	http://patchwork.linuxtv.org/project/linux-media/list/
 S:	Maintained
 F:	drivers/media/usb/dvb-usb-v2/lmedm04*
 
-LOCKDEP AND LOCKSTAT
+LOCKING PRIMITIVES
 M:	Peter Zijlstra <[email protected]>
 M:	Ingo Molnar <[email protected]>
 L:	[email protected]
-T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/locking
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core
 S:	Maintained
-F:	Documentation/locking/lockdep*.txt
-F:	Documentation/locking/lockstat.txt
+F:	Documentation/locking/
 F:	include/linux/lockdep.h
+F:	include/linux/spinlock*.h
+F:	arch/*/include/asm/spinlock*.h
+F:	include/linux/rwlock*.h
+F:	include/linux/mutex*.h
+F:	arch/*/include/asm/mutex*.h
+F:	include/linux/rwsem*.h
+F:	arch/*/include/asm/rwsem.h
+F:	include/linux/seqlock.h
+F:	lib/locking*.[ch]
 F:	kernel/locking/
 
 LOGICAL DISK MANAGER SUPPORT (LDM, Windows 2000/XP/Vista Dynamic Disks)
arch/alpha/include/asm/atomic.h +72 -15

133 lines not on this page · raw message

arch/alpha/include/asm/rwsem.h +13 -55

120 lines not on this page · raw message

arch/alpha/include/asm/spinlock.h +7 -2
index fed9c6f44c19..a40b9fc0c6c3 100644
@@ -3,6 +3,8 @@
 
 #include <linux/kernel.h>
 #include <asm/current.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 
 /*
  * Simple spin lock operations.  There are two variants, one clears IRQ's

@@ -13,8 +15,11 @@
 
 #define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)
 #define arch_spin_is_locked(x)	((x)->lock != 0)
-#define arch_spin_unlock_wait(x) \
-		do { cpu_relax(); } while ((x)->lock)
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	smp_cond_load_acquire(&lock->lock, !VAL);
+}
 
 static inline int arch_spin_value_unlocked(arch_spinlock_t lock)
 {
arch/arc/include/asm/atomic.h +87 -12

140 lines not on this page · raw message

arch/arc/include/asm/spinlock.h +5 -2
index cded4a9b5438..233d5ffe6ec7 100644
@@ -15,8 +15,11 @@
 
 #define arch_spin_is_locked(x)	((x)->slock != __ARCH_SPIN_LOCK_UNLOCKED__)
 #define arch_spin_lock_flags(lock, flags)	arch_spin_lock(lock)
-#define arch_spin_unlock_wait(x) \
-	do { while (arch_spin_is_locked(x)) cpu_relax(); } while (0)
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	smp_cond_load_acquire(&lock->slock, !VAL);
+}
 
 #ifdef CONFIG_ARC_HAS_LLSC
arch/arm/include/asm/atomic.h +96 -10

153 lines not on this page · raw message

arch/arm/include/asm/spinlock.h +17 -2
index 0fa418463f49..4bec45442072 100644
@@ -6,6 +6,8 @@
 #endif
 
 #include <linux/prefetch.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 
 /*
  * sev and wfe are ARMv6K extensions.  Uniprocessor ARMv6 may not have the K

@@ -50,8 +52,21 @@ static inline void dsb_sev(void)
  * memory.
  */
 
-#define arch_spin_unlock_wait(lock) \
-	do { while (arch_spin_is_locked(lock)) cpu_relax(); } while (0)
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	u16 owner = READ_ONCE(lock->tickets.owner);
+
+	for (;;) {
+		arch_spinlock_t tmp = READ_ONCE(*lock);
+
+		if (tmp.tickets.owner == tmp.tickets.next ||
+		    tmp.tickets.owner != owner)
+			break;
+
+		wfe();
+	}
+	smp_acquire__after_ctrl_dep();
+}
 
 #define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)
arch/arm64/include/asm/atomic.h +60 -0

72 lines not on this page · raw message

arch/arm64/include/asm/atomic_ll_sc.h +86 -24

138 lines not on this page · raw message

arch/arm64/include/asm/atomic_lse.h +196 -82

325 lines not on this page · raw message

arch/arm64/include/asm/barrier.h +13 -0
index dae5c49618db..4eea7f618dce 100644
@@ -91,6 +91,19 @@ do {									\
 	__u.__val;							\
 })
 
+#define smp_cond_load_acquire(ptr, cond_expr)				\
+({									\
+	typeof(ptr) __PTR = (ptr);					\
+	typeof(*ptr) VAL;						\
+	for (;;) {							\
+		VAL = smp_load_acquire(__PTR);				\
+		if (cond_expr)						\
+			break;						\
+		__cmpwait_relaxed(__PTR, VAL);				\
+	}								\
+	VAL;								\
+})
+
 #include <asm-generic/barrier.h>
 
 #endif	/* __ASSEMBLY__ */
arch/arm64/include/asm/cmpxchg.h +51 -0

55 lines not on this page · raw message

arch/avr32/include/asm/atomic.h +49 -5

82 lines not on this page · raw message

arch/blackfin/include/asm/atomic.h +8 -0
index 1c1c42330c99..63c7deceeeb6 100644
@@ -17,6 +17,7 @@
 
 asmlinkage int __raw_uncached_fetch_asm(const volatile int *ptr);
 asmlinkage int __raw_atomic_add_asm(volatile int *ptr, int value);
+asmlinkage int __raw_atomic_xadd_asm(volatile int *ptr, int value);
 
 asmlinkage int __raw_atomic_and_asm(volatile int *ptr, int value);
 asmlinkage int __raw_atomic_or_asm(volatile int *ptr, int value);

@@ -28,10 +29,17 @@ asmlinkage int __raw_atomic_test_asm(const volatile int *ptr, int value);
 #define atomic_add_return(i, v) __raw_atomic_add_asm(&(v)->counter, i)
 #define atomic_sub_return(i, v) __raw_atomic_add_asm(&(v)->counter, -(i))
 
+#define atomic_fetch_add(i, v) __raw_atomic_xadd_asm(&(v)->counter, i)
+#define atomic_fetch_sub(i, v) __raw_atomic_xadd_asm(&(v)->counter, -(i))
+
 #define atomic_or(i, v)  (void)__raw_atomic_or_asm(&(v)->counter, i)
 #define atomic_and(i, v) (void)__raw_atomic_and_asm(&(v)->counter, i)
 #define atomic_xor(i, v) (void)__raw_atomic_xor_asm(&(v)->counter, i)
 
+#define atomic_fetch_or(i, v)  __raw_atomic_or_asm(&(v)->counter, i)
+#define atomic_fetch_and(i, v) __raw_atomic_and_asm(&(v)->counter, i)
+#define atomic_fetch_xor(i, v) __raw_atomic_xor_asm(&(v)->counter, i)
+
 #endif
 
 #include <asm-generic/atomic.h>
arch/blackfin/include/asm/spinlock.h +3 -2
index 490c7caa02d9..c58f4a83ed6f 100644
@@ -12,6 +12,8 @@
 #else
 
 #include <linux/atomic.h>
+#include <asm/processor.h>
+#include <asm/barrier.h>
 
 asmlinkage int __raw_spin_is_locked_asm(volatile int *ptr);
 asmlinkage void __raw_spin_lock_asm(volatile int *ptr);

@@ -48,8 +50,7 @@ static inline void arch_spin_unlock(arch_spinlock_t *lock)
 
 static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
 {
-	while (arch_spin_is_locked(lock))
-		cpu_relax();
+	smp_cond_load_acquire(&lock->lock, !VAL);
 }
 
 static inline int arch_read_can_lock(arch_rwlock_t *rw)
arch/blackfin/kernel/bfin_ksyms.c +1 -0
index a401c27b69b4..68096e8f787f 100644
@@ -84,6 +84,7 @@ EXPORT_SYMBOL(insl_16);
 
 #ifdef CONFIG_SMP
 EXPORT_SYMBOL(__raw_atomic_add_asm);
+EXPORT_SYMBOL(__raw_atomic_xadd_asm);
 EXPORT_SYMBOL(__raw_atomic_and_asm);
 EXPORT_SYMBOL(__raw_atomic_or_asm);
 EXPORT_SYMBOL(__raw_atomic_xor_asm);
arch/blackfin/mach-bf561/atomic.S +31 -12

67 lines not on this page · raw message

arch/frv/include/asm/atomic.h +10 -20
index 64f02d451aa8..1c2a5e264fc7 100644
@@ -60,16 +60,6 @@ static inline int atomic_add_negative(int i, atomic_t *v)
 	return atomic_add_return(i, v) < 0;
 }
 
-static inline void atomic_add(int i, atomic_t *v)
-{
-	atomic_add_return(i, v);
-}
-
-static inline void atomic_sub(int i, atomic_t *v)
-{
-	atomic_sub_return(i, v);
-}
-
 static inline void atomic_inc(atomic_t *v)
 {
 	atomic_inc_return(v);

@@ -136,16 +126,6 @@ static inline long long atomic64_add_negative(long long i, atomic64_t *v)
 	return atomic64_add_return(i, v) < 0;
 }
 
-static inline void atomic64_add(long long i, atomic64_t *v)
-{
-	atomic64_add_return(i, v);
-}
-
-static inline void atomic64_sub(long long i, atomic64_t *v)
-{
-	atomic64_sub_return(i, v);
-}
-
 static inline void atomic64_inc(atomic64_t *v)
 {
 	atomic64_inc_return(v);

@@ -182,11 +162,19 @@ static __inline__ int __atomic_add_unless(atomic_t *v, int a, int u)
 }
 
 #define ATOMIC_OP(op)							\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	return __atomic32_fetch_##op(i, &v->counter);			\
+}									\
 static inline void atomic_##op(int i, atomic_t *v)			\
 {									\
 	(void)__atomic32_fetch_##op(i, &v->counter);			\
 }									\
 									\
+static inline long long atomic64_fetch_##op(long long i, atomic64_t *v)	\
+{									\
+	return __atomic64_fetch_##op(i, &v->counter);			\
+}									\
 static inline void atomic64_##op(long long i, atomic64_t *v)		\
 {									\
 	(void)__atomic64_fetch_##op(i, &v->counter);			\

@@ -195,6 +183,8 @@ static inline void atomic64_##op(long long i, atomic64_t *v)		\
 ATOMIC_OP(or)
 ATOMIC_OP(and)
 ATOMIC_OP(xor)
+ATOMIC_OP(add)
+ATOMIC_OP(sub)
 
 #undef ATOMIC_OP
arch/frv/include/asm/atomic_defs.h +2 -0
index 36e126d2f801..d4912c88b829 100644
@@ -162,6 +162,8 @@ ATOMIC_EXPORT(__atomic64_fetch_##op);
 ATOMIC_FETCH_OP(or)
 ATOMIC_FETCH_OP(and)
 ATOMIC_FETCH_OP(xor)
+ATOMIC_FETCH_OP(add)
+ATOMIC_FETCH_OP(sub)
 
 ATOMIC_OP_RETURN(add)
 ATOMIC_OP_RETURN(sub)
arch/h8300/include/asm/atomic.h +23 -6
index 4435a445ae7e..349a47a918db 100644
@@ -28,6 +28,19 @@ static inline int atomic_##op##_return(int i, atomic_t *v)	\
 	return ret;						\
 }
 
+#define ATOMIC_FETCH_OP(op, c_op)				\
+static inline int atomic_fetch_##op(int i, atomic_t *v)		\
+{								\
+	h8300flags flags;					\
+	int ret;						\
+								\
+	flags = arch_local_irq_save();				\
+	ret = v->counter;					\
+	v->counter c_op i;					\
+	arch_local_irq_restore(flags);				\
+	return ret;						\
+}
+
 #define ATOMIC_OP(op, c_op)					\
 static inline void atomic_##op(int i, atomic_t *v)		\
 {								\

@@ -41,17 +54,21 @@ static inline void atomic_##op(int i, atomic_t *v)		\
 ATOMIC_OP_RETURN(add, +=)
 ATOMIC_OP_RETURN(sub, -=)
 
-ATOMIC_OP(and, &=)
-ATOMIC_OP(or,  |=)
-ATOMIC_OP(xor, ^=)
+#define ATOMIC_OPS(op, c_op)					\
+	ATOMIC_OP(op, c_op)					\
+	ATOMIC_FETCH_OP(op, c_op)
+
+ATOMIC_OPS(and, &=)
+ATOMIC_OPS(or,  |=)
+ATOMIC_OPS(xor, ^=)
+ATOMIC_OPS(add, +=)
+ATOMIC_OPS(sub, -=)
 
+#undef ATOMIC_OPS
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
 
-#define atomic_add(i, v)		(void)atomic_add_return(i, v)
 #define atomic_add_negative(a, v)	(atomic_add_return((a), (v)) < 0)
-
-#define atomic_sub(i, v)		(void)atomic_sub_return(i, v)
 #define atomic_sub_and_test(i, v)	(atomic_sub_return(i, v) == 0)
 
 #define atomic_inc_return(v)		atomic_add_return(1, v)
arch/hexagon/include/asm/atomic.h +26 -5
index 55696c4100d4..a62ba368b27d 100644
@@ -110,7 +110,7 @@ static inline void atomic_##op(int i, atomic_t *v)			\
 	);								\
 }									\
 
-#define ATOMIC_OP_RETURN(op)							\
+#define ATOMIC_OP_RETURN(op)						\
 static inline int atomic_##op##_return(int i, atomic_t *v)		\
 {									\
 	int output;							\

@@ -127,16 +127,37 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return output;							\
 }
 
-#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op)
+#define ATOMIC_FETCH_OP(op)						\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	int output, val;						\
+									\
+	__asm__ __volatile__ (						\
+		"1:	%0 = memw_locked(%2);\n"			\
+		"	%1 = "#op "(%0,%3);\n"				\
+		"	memw_locked(%2,P3)=%1;\n"			\
+		"	if !P3 jump 1b;\n"				\
+		: "=&r" (output), "=&r" (val)				\
+		: "r" (&v->counter), "r" (i)				\
+		: "memory", "p3"					\
+	);								\
+	return output;							\
+}
+
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op) ATOMIC_FETCH_OP(op)
 
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
-ATOMIC_OP(and)
-ATOMIC_OP(or)
-ATOMIC_OP(xor)
+#undef ATOMIC_OPS
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
+
+ATOMIC_OPS(and)
+ATOMIC_OPS(or)
+ATOMIC_OPS(xor)
 
 #undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/hexagon/include/asm/spinlock.h +8 -2
index 12ca4ebc0338..a1c55788c5d6 100644
@@ -23,6 +23,8 @@
 #define _ASM_SPINLOCK_H
 
 #include <asm/irqflags.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 
 /*
  * This file is pulled in for SMP builds.

@@ -176,8 +178,12 @@ static inline unsigned int arch_spin_trylock(arch_spinlock_t *lock)
  * SMP spinlocks are intended to allow only a single CPU at the lock
  */
 #define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)
-#define arch_spin_unlock_wait(lock) \
-	do {while (arch_spin_is_locked(lock)) cpu_relax(); } while (0)
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	smp_cond_load_acquire(&lock->lock, !VAL);
+}
+
 #define arch_spin_is_locked(x) ((x)->lock != 0)
 
 #define arch_read_lock_flags(lock, flags) arch_read_lock(lock)
arch/ia64/include/asm/atomic.h +114 -16

158 lines not on this page · raw message

arch/ia64/include/asm/mutex.h +1 -1
index f41e66d65e31..28cb819e0ff9 100644
@@ -82,7 +82,7 @@ __mutex_fastpath_unlock(atomic_t *count, void (*fail_fn)(atomic_t *))
 static inline int
 __mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))
 {
-	if (cmpxchg_acq(count, 1, 0) == 1)
+	if (atomic_read(count) == 1 && cmpxchg_acq(count, 1, 0) == 1)
 		return 1;
 	return 0;
 }
arch/ia64/include/asm/rwsem.h +12 -19
index 8b23e070b844..8fa98dd303b4 100644
@@ -40,7 +40,7 @@
 static inline void
 __down_read (struct rw_semaphore *sem)
 {
-	long result = ia64_fetchadd8_acq((unsigned long *)&sem->count, 1);
+	long result = ia64_fetchadd8_acq((unsigned long *)&sem->count.counter, 1);
 
 	if (result < 0)
 		rwsem_down_read_failed(sem);

@@ -55,9 +55,9 @@ ___down_write (struct rw_semaphore *sem)
 	long old, new;
 
 	do {
-		old = sem->count;
+		old = atomic_long_read(&sem->count);
 		new = old + RWSEM_ACTIVE_WRITE_BIAS;
-	} while (cmpxchg_acq(&sem->count, old, new) != old);
+	} while (atomic_long_cmpxchg_acquire(&sem->count, old, new) != old);
 
 	return old;
 }

@@ -85,7 +85,7 @@ __down_write_killable (struct rw_semaphore *sem)
 static inline void
 __up_read (struct rw_semaphore *sem)
 {
-	long result = ia64_fetchadd8_rel((unsigned long *)&sem->count, -1);
+	long result = ia64_fetchadd8_rel((unsigned long *)&sem->count.counter, -1);
 
 	if (result < 0 && (--result & RWSEM_ACTIVE_MASK) == 0)
 		rwsem_wake(sem);

@@ -100,9 +100,9 @@ __up_write (struct rw_semaphore *sem)
 	long old, new;
 
 	do {
-		old = sem->count;
+		old = atomic_long_read(&sem->count);
 		new = old - RWSEM_ACTIVE_WRITE_BIAS;
-	} while (cmpxchg_rel(&sem->count, old, new) != old);
+	} while (atomic_long_cmpxchg_release(&sem->count, old, new) != old);
 
 	if (new < 0 && (new & RWSEM_ACTIVE_MASK) == 0)
 		rwsem_wake(sem);

@@ -115,8 +115,8 @@ static inline int
 __down_read_trylock (struct rw_semaphore *sem)
 {
 	long tmp;
-	while ((tmp = sem->count) >= 0) {
-		if (tmp == cmpxchg_acq(&sem->count, tmp, tmp+1)) {
+	while ((tmp = atomic_long_read(&sem->count)) >= 0) {
+		if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, tmp+1)) {
 			return 1;
 		}
 	}

@@ -129,8 +129,8 @@ __down_read_trylock (struct rw_semaphore *sem)
 static inline int
 __down_write_trylock (struct rw_semaphore *sem)
 {
-	long tmp = cmpxchg_acq(&sem->count, RWSEM_UNLOCKED_VALUE,
-			      RWSEM_ACTIVE_WRITE_BIAS);
+	long tmp = atomic_long_cmpxchg_acquire(&sem->count,
+			RWSEM_UNLOCKED_VALUE, RWSEM_ACTIVE_WRITE_BIAS);
 	return tmp == RWSEM_UNLOCKED_VALUE;
 }
 

@@ -143,19 +143,12 @@ __downgrade_write (struct rw_semaphore *sem)
 	long old, new;
 
 	do {
-		old = sem->count;
+		old = atomic_long_read(&sem->count);
 		new = old - RWSEM_WAITING_BIAS;
-	} while (cmpxchg_rel(&sem->count, old, new) != old);
+	} while (atomic_long_cmpxchg_release(&sem->count, old, new) != old);
 
 	if (old < 0)
 		rwsem_downgrade_wake(sem);
 }
 
-/*
- * Implement atomic add functionality.  These used to be "inline" functions, but GCC v3.1
- * doesn't quite optimize this stuff right and ends up with bad calls to fetchandadd.
- */
-#define rwsem_atomic_add(delta, sem)	atomic64_add(delta, (atomic64_t *)(&(sem)->count))
-#define rwsem_atomic_update(delta, sem)	atomic64_add_return(delta, (atomic64_t *)(&(sem)->count))
-
 #endif /* _ASM_IA64_RWSEM_H */
arch/ia64/include/asm/spinlock.h +4 -0
index 45698cd15b7b..ca9e76149a4a 100644
@@ -15,6 +15,8 @@
 
 #include <linux/atomic.h>
 #include <asm/intrinsics.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 
 #define arch_spin_lock_init(x)			((x)->lock = 0)
 

@@ -86,6 +88,8 @@ static __always_inline void __ticket_spin_unlock_wait(arch_spinlock_t *lock)
 			return;
 		cpu_relax();
 	}
+
+	smp_acquire__after_ctrl_dep();
 }
 
 static inline int __ticket_spin_is_locked(arch_spinlock_t *lock)
arch/m32r/include/asm/atomic.h +32 -4
index ea35160d632b..640cc1c7099f 100644
@@ -89,16 +89,44 @@ static __inline__ int atomic_##op##_return(int i, atomic_t *v)		\
 	return result;							\
 }
 
-#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op)
+#define ATOMIC_FETCH_OP(op)						\
+static __inline__ int atomic_fetch_##op(int i, atomic_t *v)		\
+{									\
+	unsigned long flags;						\
+	int result, val;						\
+									\
+	local_irq_save(flags);						\
+	__asm__ __volatile__ (						\
+		"# atomic_fetch_" #op "		\n\t"			\
+		DCACHE_CLEAR("%0", "r4", "%2")				\
+		M32R_LOCK" %1, @%2;		\n\t"			\
+		"mv %0, %1			\n\t" 			\
+		#op " %1, %3;			\n\t"			\
+		M32R_UNLOCK" %1, @%2;		\n\t"			\
+		: "=&r" (result), "=&r" (val)				\
+		: "r" (&v->counter), "r" (i)				\
+		: "memory"						\
+		__ATOMIC_CLOBBER					\
+	);								\
+	local_irq_restore(flags);					\
+									\
+	return result;							\
+}
+
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op) ATOMIC_FETCH_OP(op)
 
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
-ATOMIC_OP(and)
-ATOMIC_OP(or)
-ATOMIC_OP(xor)
+#undef ATOMIC_OPS
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
+
+ATOMIC_OPS(and)
+ATOMIC_OPS(or)
+ATOMIC_OPS(xor)
 
 #undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/m32r/include/asm/spinlock.h +7 -2
index fa13694eaae3..323c7fc953cd 100644
@@ -13,6 +13,8 @@
 #include <linux/atomic.h>
 #include <asm/dcache_clear.h>
 #include <asm/page.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 
 /*
  * Your basic SMP spinlocks, allowing only a single CPU anywhere

@@ -27,8 +29,11 @@
 
 #define arch_spin_is_locked(x)		(*(volatile int *)(&(x)->slock) <= 0)
 #define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)
-#define arch_spin_unlock_wait(x) \
-		do { cpu_relax(); } while (arch_spin_is_locked(x))
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	smp_cond_load_acquire(&lock->slock, VAL > 0);
+}
 
 /**
  * arch_spin_trylock - Try spin lock and return a result
arch/m68k/include/asm/atomic.h +40 -4

66 lines not on this page · raw message

arch/metag/include/asm/atomic_lnkget.h +32 -4
index 88fa25fae8bd..def2c642f053 100644
@@ -69,16 +69,44 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return result;							\
 }
 
-#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op)
+#define ATOMIC_FETCH_OP(op)						\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	int result, temp;						\
+									\
+	smp_mb();							\
+									\
+	asm volatile (							\
+		"1:	LNKGETD %1, [%2]\n"				\
+		"	" #op "	%0, %1, %3\n"				\
+		"	LNKSETD [%2], %0\n"				\
+		"	DEFR	%0, TXSTAT\n"				\
+		"	ANDT	%0, %0, #HI(0x3f000000)\n"		\
+		"	CMPT	%0, #HI(0x02000000)\n"			\
+		"	BNZ 1b\n"					\
+		: "=&d" (temp), "=&d" (result)				\
+		: "da" (&v->counter), "bd" (i)				\
+		: "cc");						\
+									\
+	smp_mb();							\
+									\
+	return result;							\
+}
+
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op) ATOMIC_FETCH_OP(op)
 
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
-ATOMIC_OP(and)
-ATOMIC_OP(or)
-ATOMIC_OP(xor)
+#undef ATOMIC_OPS
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
+
+ATOMIC_OPS(and)
+ATOMIC_OPS(or)
+ATOMIC_OPS(xor)
 
 #undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/metag/include/asm/atomic_lock1.h +29 -4
index 0295d9b8d5bf..6c1380a8a0d4 100644
@@ -64,15 +64,40 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return result;							\
 }
 
-#define ATOMIC_OPS(op, c_op) ATOMIC_OP(op, c_op) ATOMIC_OP_RETURN(op, c_op)
+#define ATOMIC_FETCH_OP(op, c_op)					\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	unsigned long result;						\
+	unsigned long flags;						\
+									\
+	__global_lock1(flags);						\
+	result = v->counter;						\
+	fence();							\
+	v->counter c_op i;						\
+	__global_unlock1(flags);					\
+									\
+	return result;							\
+}
+
+#define ATOMIC_OPS(op, c_op)						\
+	ATOMIC_OP(op, c_op)						\
+	ATOMIC_OP_RETURN(op, c_op)					\
+	ATOMIC_FETCH_OP(op, c_op)
 
 ATOMIC_OPS(add, +=)
 ATOMIC_OPS(sub, -=)
-ATOMIC_OP(and, &=)
-ATOMIC_OP(or, |=)
-ATOMIC_OP(xor, ^=)
 
 #undef ATOMIC_OPS
+#define ATOMIC_OPS(op, c_op)						\
+	ATOMIC_OP(op, c_op)						\
+	ATOMIC_FETCH_OP(op, c_op)
+
+ATOMIC_OPS(and, &=)
+ATOMIC_OPS(or, |=)
+ATOMIC_OPS(xor, ^=)
+
+#undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/metag/include/asm/spinlock.h +12 -2
index 86a7cf3d1386..c0c7a22be1ae 100644
@@ -1,14 +1,24 @@
 #ifndef __ASM_SPINLOCK_H
 #define __ASM_SPINLOCK_H
 
+#include <asm/barrier.h>
+#include <asm/processor.h>
+
 #ifdef CONFIG_METAG_ATOMICITY_LOCK1
 #include <asm/spinlock_lock1.h>
 #else
 #include <asm/spinlock_lnkget.h>
 #endif
 
-#define arch_spin_unlock_wait(lock) \
-	do { while (arch_spin_is_locked(lock)) cpu_relax(); } while (0)
+/*
+ * both lock1 and lnkget are test-and-set spinlocks with 0 unlocked and 1
+ * locked.
+ */
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	smp_cond_load_acquire(&lock->lock, !VAL);
+}
 
 #define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)
arch/mips/include/asm/atomic.h +137 -17

213 lines not on this page · raw message

arch/mips/include/asm/spinlock.h +17 -2
index 40196bebe849..f485afe51514 100644
@@ -12,6 +12,7 @@
 #include <linux/compiler.h>
 
 #include <asm/barrier.h>
+#include <asm/processor.h>
 #include <asm/compiler.h>
 #include <asm/war.h>
 

@@ -48,8 +49,22 @@ static inline int arch_spin_value_unlocked(arch_spinlock_t lock)
 }
 
 #define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)
-#define arch_spin_unlock_wait(x) \
-	while (arch_spin_is_locked(x)) { cpu_relax(); }
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	u16 owner = READ_ONCE(lock->h.serving_now);
+	smp_rmb();
+	for (;;) {
+		arch_spinlock_t tmp = READ_ONCE(*lock);
+
+		if (tmp.h.serving_now == tmp.h.ticket ||
+		    tmp.h.serving_now != owner)
+			break;
+
+		cpu_relax();
+	}
+	smp_acquire__after_ctrl_dep();
+}
 
 static inline int arch_spin_is_contended(arch_spinlock_t *lock)
 {
arch/mn10300/include/asm/atomic.h +29 -4
index ce318d5ab23b..36389efd45e8 100644
@@ -84,16 +84,41 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return retval;							\
 }
 
-#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op)
+#define ATOMIC_FETCH_OP(op)						\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	int retval, status;						\
+									\
+	asm volatile(							\
+		"1:	mov	%4,(_AAR,%3)	\n"			\
+		"	mov	(_ADR,%3),%1	\n"			\
+		"	mov	%1,%0		\n"			\
+		"	" #op "	%5,%0		\n"			\
+		"	mov	%0,(_ADR,%3)	\n"			\
+		"	mov	(_ADR,%3),%0	\n"	/* flush */	\
+		"	mov	(_ASR,%3),%0	\n"			\
+		"	or	%0,%0		\n"			\
+		"	bne	1b		\n"			\
+		: "=&r"(status), "=&r"(retval), "=m"(v->counter)	\
+		: "a"(ATOMIC_OPS_BASE_ADDR), "r"(&v->counter), "r"(i)	\
+		: "memory", "cc");					\
+	return retval;							\
+}
+
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op) ATOMIC_FETCH_OP(op)
 
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
-ATOMIC_OP(and)
-ATOMIC_OP(or)
-ATOMIC_OP(xor)
+#undef ATOMIC_OPS
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
+
+ATOMIC_OPS(and)
+ATOMIC_OPS(or)
+ATOMIC_OPS(xor)
 
 #undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/mn10300/include/asm/spinlock.h +7 -1
index 1ae580f38933..9c7b8f7942d8 100644
@@ -12,6 +12,8 @@
 #define _ASM_SPINLOCK_H
 
 #include <linux/atomic.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 #include <asm/rwlock.h>
 #include <asm/page.h>
 

@@ -23,7 +25,11 @@
  */
 
 #define arch_spin_is_locked(x)	(*(volatile signed char *)(&(x)->slock) != 0)
-#define arch_spin_unlock_wait(x) do { barrier(); } while (arch_spin_is_locked(x))
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
+{
+	smp_cond_load_acquire(&lock->slock, !VAL);
+}
 
 static inline void arch_spin_unlock(arch_spinlock_t *lock)
 {
arch/parisc/include/asm/atomic.h +55 -8

86 lines not on this page · raw message

arch/parisc/include/asm/spinlock.h +7 -2
index 64f2992e439f..e32936cd7f10 100644
@@ -13,8 +13,13 @@ static inline int arch_spin_is_locked(arch_spinlock_t *x)
 }
 
 #define arch_spin_lock(lock) arch_spin_lock_flags(lock, 0)
-#define arch_spin_unlock_wait(x) \
-		do { cpu_relax(); } while (arch_spin_is_locked(x))
+
+static inline void arch_spin_unlock_wait(arch_spinlock_t *x)
+{
+	volatile unsigned int *a = __ldcw_align(x);
+
+	smp_cond_load_acquire(a, VAL);
+}
 
 static inline void arch_spin_lock_flags(arch_spinlock_t *x,
 					 unsigned long flags)
arch/powerpc/include/asm/atomic.h +74 -9

115 lines not on this page · raw message

arch/powerpc/include/asm/mutex.h +1 -1
index 127ab23e1f6c..078155fa1189 100644
@@ -124,7 +124,7 @@ __mutex_fastpath_unlock(atomic_t *count, void (*fail_fn)(atomic_t *))
 static inline int
 __mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))
 {
-	if (likely(__mutex_cmpxchg_lock(count, 1, 0) == 1))
+	if (likely(atomic_read(count) == 1 && __mutex_cmpxchg_lock(count, 1, 0) == 1))
 		return 1;
 	return 0;
 }
arch/s390/include/asm/atomic.h +30 -10
index 911064aa59b2..d28cc2f5b7b2 100644
@@ -93,6 +93,11 @@ static inline int atomic_add_return(int i, atomic_t *v)
 	return __ATOMIC_LOOP(v, i, __ATOMIC_ADD, __ATOMIC_BARRIER) + i;
 }
 
+static inline int atomic_fetch_add(int i, atomic_t *v)
+{
+	return __ATOMIC_LOOP(v, i, __ATOMIC_ADD, __ATOMIC_BARRIER);
+}
+
 static inline void atomic_add(int i, atomic_t *v)
 {
 #ifdef CONFIG_HAVE_MARCH_Z196_FEATURES

@@ -114,22 +119,27 @@ static inline void atomic_add(int i, atomic_t *v)
 #define atomic_inc_and_test(_v)		(atomic_add_return(1, _v) == 0)
 #define atomic_sub(_i, _v)		atomic_add(-(int)(_i), _v)
 #define atomic_sub_return(_i, _v)	atomic_add_return(-(int)(_i), _v)
+#define atomic_fetch_sub(_i, _v)	atomic_fetch_add(-(int)(_i), _v)
 #define atomic_sub_and_test(_i, _v)	(atomic_sub_return(_i, _v) == 0)
 #define atomic_dec(_v)			atomic_sub(1, _v)
 #define atomic_dec_return(_v)		atomic_sub_return(1, _v)
 #define atomic_dec_and_test(_v)		(atomic_sub_return(1, _v) == 0)
 
-#define ATOMIC_OP(op, OP)						\
+#define ATOMIC_OPS(op, OP)						\
 static inline void atomic_##op(int i, atomic_t *v)			\
 {									\
 	__ATOMIC_LOOP(v, i, __ATOMIC_##OP, __ATOMIC_NO_BARRIER);	\
+}									\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	return __ATOMIC_LOOP(v, i, __ATOMIC_##OP, __ATOMIC_BARRIER);	\
 }
 
-ATOMIC_OP(and, AND)
-ATOMIC_OP(or, OR)
-ATOMIC_OP(xor, XOR)
+ATOMIC_OPS(and, AND)
+ATOMIC_OPS(or, OR)
+ATOMIC_OPS(xor, XOR)
 
-#undef ATOMIC_OP
+#undef ATOMIC_OPS
 
 #define atomic_xchg(v, new) (xchg(&((v)->counter), new))
 

@@ -236,6 +246,11 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v)
 	return __ATOMIC64_LOOP(v, i, __ATOMIC64_ADD, __ATOMIC64_BARRIER) + i;
 }
 
+static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+{
+	return __ATOMIC64_LOOP(v, i, __ATOMIC64_ADD, __ATOMIC64_BARRIER);
+}
+
 static inline void atomic64_add(long long i, atomic64_t *v)
 {
 #ifdef CONFIG_HAVE_MARCH_Z196_FEATURES

@@ -264,17 +279,21 @@ static inline long long atomic64_cmpxchg(atomic64_t *v,
 	return old;
 }
 
-#define ATOMIC64_OP(op, OP)						\
+#define ATOMIC64_OPS(op, OP)						\
 static inline void atomic64_##op(long i, atomic64_t *v)			\
 {									\
 	__ATOMIC64_LOOP(v, i, __ATOMIC64_##OP, __ATOMIC64_NO_BARRIER);	\
+}									\
+static inline long atomic64_fetch_##op(long i, atomic64_t *v)		\
+{									\
+	return __ATOMIC64_LOOP(v, i, __ATOMIC64_##OP, __ATOMIC64_BARRIER); \
 }
 
-ATOMIC64_OP(and, AND)
-ATOMIC64_OP(or, OR)
-ATOMIC64_OP(xor, XOR)
+ATOMIC64_OPS(and, AND)
+ATOMIC64_OPS(or, OR)
+ATOMIC64_OPS(xor, XOR)
 
-#undef ATOMIC64_OP
+#undef ATOMIC64_OPS
 #undef __ATOMIC64_LOOP
 
 static inline int atomic64_add_unless(atomic64_t *v, long long i, long long u)

@@ -315,6 +334,7 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
 #define atomic64_inc_return(_v)		atomic64_add_return(1, _v)
 #define atomic64_inc_and_test(_v)	(atomic64_add_return(1, _v) == 0)
 #define atomic64_sub_return(_i, _v)	atomic64_add_return(-(long long)(_i), _v)
+#define atomic64_fetch_sub(_i, _v)	atomic64_fetch_add(-(long long)(_i), _v)
 #define atomic64_sub(_i, _v)		atomic64_add(-(long long)(_i), _v)
 #define atomic64_sub_and_test(_i, _v)	(atomic64_sub_return(_i, _v) == 0)
 #define atomic64_dec(_v)		atomic64_sub(1, _v)
arch/s390/include/asm/rwsem.h +0 -37
index c75e4471e618..597e7e96b59e 100644
@@ -207,41 +207,4 @@ static inline void __downgrade_write(struct rw_semaphore *sem)
 		rwsem_downgrade_wake(sem);
 }
 
-/*
- * implement atomic add functionality
- */
-static inline void rwsem_atomic_add(long delta, struct rw_semaphore *sem)
-{
-	signed long old, new;
-
-	asm volatile(
-		"	lg	%0,%2\n"
-		"0:	lgr	%1,%0\n"
-		"	agr	%1,%4\n"
-		"	csg	%0,%1,%2\n"
-		"	jl	0b"
-		: "=&d" (old), "=&d" (new), "=Q" (sem->count)
-		: "Q" (sem->count), "d" (delta)
-		: "cc", "memory");
-}
-
-/*
- * implement exchange and add functionality
- */
-static inline long rwsem_atomic_update(long delta, struct rw_semaphore *sem)
-{
-	signed long old, new;
-
-	asm volatile(
-		"	lg	%0,%2\n"
-		"0:	lgr	%1,%0\n"
-		"	agr	%1,%4\n"
-		"	csg	%0,%1,%2\n"
-		"	jl	0b"
-		: "=&d" (old), "=&d" (new), "=Q" (sem->count)
-		: "Q" (sem->count), "d" (delta)
-		: "cc", "memory");
-	return new;
-}
-
 #endif /* _S390_RWSEM_H */
arch/s390/include/asm/spinlock.h +3 -0
index 63ebf37d3143..7e9e09f600fa 100644
@@ -10,6 +10,8 @@
 #define __ASM_SPINLOCK_H
 
 #include <linux/smp.h>
+#include <asm/barrier.h>
+#include <asm/processor.h>
 
 #define SPINLOCK_LOCKVAL (S390_lowcore.spinlock_lockval)
 

@@ -97,6 +99,7 @@ static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
 {
 	while (arch_spin_is_locked(lock))
 		arch_spin_relax(lock);
+	smp_acquire__after_ctrl_dep();
 }
 
 /*
arch/sh/include/asm/atomic-grb.h +30 -4
index b94df40e5f2d..d755e96c3064 100644
@@ -43,16 +43,42 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return tmp;							\
 }
 
-#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op)
+#define ATOMIC_FETCH_OP(op)						\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	int res, tmp;							\
+									\
+	__asm__ __volatile__ (						\
+		"   .align 2              \n\t"				\
+		"   mova    1f,   r0      \n\t" /* r0 = end point */	\
+		"   mov    r15,   r1      \n\t" /* r1 = saved sp */	\
+		"   mov    #-6,   r15     \n\t" /* LOGIN: r15 = size */	\
+		"   mov.l  @%2,   %0      \n\t" /* load old value */	\
+		"   mov     %0,   %1      \n\t" /* save old value */	\
+		" " #op "   %3,   %0      \n\t" /* $op */		\
+		"   mov.l   %0,   @%2     \n\t" /* store new value */	\
+		"1: mov     r1,   r15     \n\t" /* LOGOUT */		\
+		: "=&r" (tmp), "=&r" (res), "+r"  (v)			\
+		: "r"   (i)						\
+		: "memory" , "r0", "r1");				\
+									\
+	return res;							\
+}
+
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op) ATOMIC_FETCH_OP(op)
 
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
 
-ATOMIC_OP(and)
-ATOMIC_OP(or)
-ATOMIC_OP(xor)
+#undef ATOMIC_OPS
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
+
+ATOMIC_OPS(and)
+ATOMIC_OPS(or)
+ATOMIC_OPS(xor)
 
 #undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/sh/include/asm/atomic-irq.h +27 -4
index 23fcdad5773e..8e2da5fa0178 100644
@@ -33,15 +33,38 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return temp;							\
 }
 
-#define ATOMIC_OPS(op, c_op) ATOMIC_OP(op, c_op) ATOMIC_OP_RETURN(op, c_op)
+#define ATOMIC_FETCH_OP(op, c_op)					\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	unsigned long temp, flags;					\
+									\
+	raw_local_irq_save(flags);					\
+	temp = v->counter;						\
+	v->counter c_op i;						\
+	raw_local_irq_restore(flags);					\
+									\
+	return temp;							\
+}
+
+#define ATOMIC_OPS(op, c_op)						\
+	ATOMIC_OP(op, c_op)						\
+	ATOMIC_OP_RETURN(op, c_op)					\
+	ATOMIC_FETCH_OP(op, c_op)
 
 ATOMIC_OPS(add, +=)
 ATOMIC_OPS(sub, -=)
-ATOMIC_OP(and, &=)
-ATOMIC_OP(or, |=)
-ATOMIC_OP(xor, ^=)
 
 #undef ATOMIC_OPS
+#define ATOMIC_OPS(op, c_op)						\
+	ATOMIC_OP(op, c_op)						\
+	ATOMIC_FETCH_OP(op, c_op)
+
+ATOMIC_OPS(and, &=)
+ATOMIC_OPS(or, |=)
+ATOMIC_OPS(xor, ^=)
+
+#undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
 #undef ATOMIC_OP
arch/sh/include/asm/atomic-llsc.h +28 -4
index 33d34b16d4d6..caea2c45f6c2 100644
@@ -48,15 +48,39 @@ static inline int atomic_##op##_return(int i, atomic_t *v)		\
 	return temp;							\
 }
 
-#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op)
+#define ATOMIC_FETCH_OP(op)						\
+static inline int atomic_fetch_##op(int i, atomic_t *v)			\
+{									\
+	unsigned long res, temp;					\
+									\
+	__asm__ __volatile__ (						\
+"1:	movli.l @%3, %0		! atomic_fetch_" #op "	\n"		\
+"	mov %0, %1					\n"		\
+"	" #op "	%2, %0					\n"		\
+"	movco.l	%0, @%3					\n"		\
+"	bf	1b					\n"		\
+"	synco						\n"		\
+	: "=&z" (temp), "=&z" (res)					\
+	: "r" (i), "r" (&v->counter)					\
+	: "t");								\
+									\
+	return res;							\
+}
+
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_OP_RETURN(op) ATOMIC_FETCH_OP(op)
 
 ATOMIC_OPS(add)
 ATOMIC_OPS(sub)
-ATOMIC_OP(and)
-ATOMIC_OP(or)
-ATOMIC_OP(xor)
 
 #undef ATOMIC_OPS
+#define ATOMIC_OPS(op) ATOMIC_OP(op) ATOMIC_FETCH_OP(op)
+
+ATOMIC_OPS(and)
+ATOMIC_OPS(or)
+ATOMIC_OPS(xor)
+
+#undef ATOMIC_OPS
+#undef ATOMIC_FETCH_OP
 #undef ATOMIC_OP_RETURN
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

arch/sh/include/asm/spinlock.h +8 -2
index bdc0f3b6c56a..416834b60ad0 100644

Lines of this file are not shown. View the raw message.

arch/sparc/include/asm/atomic_32.h +10 -3
index 7dcbebbcaec6..ee3f11c43cda 100644

Lines of this file are not shown. View the raw message.

arch/sparc/include/asm/atomic_64.h +12 -4
index f2fbf9e16faf..24827a3f733a 100644

Lines of this file are not shown. View the raw message.

arch/sparc/include/asm/spinlock_32.h +5 -2
index bcc98fc35281..d9c5876c6121 100644

Lines of this file are not shown. View the raw message.

arch/sparc/include/asm/spinlock_64.h +7 -3
index 968917694978..87990b7c6b0d 100644

Lines of this file are not shown. View the raw message.

arch/sparc/lib/atomic32.c +17 -12
index b9d63c0a7aab..2c373329d5cb 100644

Lines of this file are not shown. View the raw message.

arch/sparc/lib/atomic_64.S +50 -11

61 lines not on this page · raw message

arch/sparc/lib/ksyms.c +13 -4
index 8eb454cfe05c..de5e97817bdb 100644

Lines of this file are not shown. View the raw message.

arch/tile/include/asm/atomic.h +2 -0
index 9fc0107a9c5e..8dda3c8ff5ab 100644

Lines of this file are not shown. View the raw message.

arch/tile/include/asm/atomic_32.h +49 -25

74 lines not on this page · raw message

arch/tile/include/asm/atomic_64.h +76 -39

115 lines not on this page · raw message

arch/tile/include/asm/barrier.h +7 -0
index d55222806c2f..4c419ab95ab7 100644

Lines of this file are not shown. View the raw message.

arch/tile/include/asm/bitops_32.h +9 -9
index bbf7b666f21d..d1406a95f6b7 100644

Lines of this file are not shown. View the raw message.

arch/tile/include/asm/futex.h +7 -7
index 1a6ef1b69cb1..e64a1b75fc38 100644

Lines of this file are not shown. View the raw message.

arch/tile/lib/atomic_32.c +25 -25

50 lines not on this page · raw message

arch/tile/lib/atomic_asm_32.S +16 -11
index f611265633d6..1a70e6c0f259 100644

Lines of this file are not shown. View the raw message.

arch/tile/lib/spinlock_32.c +6 -0
index 88c2a53362e7..076c6cc43113 100644

Lines of this file are not shown. View the raw message.

arch/tile/lib/spinlock_64.c +6 -0
index c8d1f94ff1fe..a4b5b2cbce93 100644

Lines of this file are not shown. View the raw message.

arch/x86/include/asm/atomic.h +32 -3
index 3e8674288198..a58b99811105 100644

Lines of this file are not shown. View the raw message.

arch/x86/include/asm/atomic64_32.h +22 -3
index a984111135b1..71d7705fb303 100644

Lines of this file are not shown. View the raw message.

arch/x86/include/asm/atomic64_64.h +32 -3
index 037351022f54..70eed0e14553 100644

Lines of this file are not shown. View the raw message.

arch/x86/include/asm/mutex_32.h +1 -1
index 85e6cda45a02..e9355a84fc67 100644

Lines of this file are not shown. View the raw message.

arch/x86/include/asm/mutex_64.h +3 -3
index 07537a44216e..d9850758464e 100644

Lines of this file are not shown. View the raw message.

arch/x86/include/asm/rwsem.h +0 -18
index 453744c1d347..089ced4edbbc 100644

Lines of this file are not shown. View the raw message.

arch/xtensa/include/asm/atomic.h +48 -4

52 lines not on this page · raw message

arch/xtensa/include/asm/spinlock.h +8 -2
index 1d95fa5dcd10..a36221cf6363 100644

Lines of this file are not shown. View the raw message.

include/asm-generic/atomic-long.h +57 -1

58 lines not on this page · raw message

include/asm-generic/atomic.h +47 -0

47 lines not on this page · raw message

include/asm-generic/atomic64.h +11 -4
index d48e78ccad3d..dad68bf46c77 100644

Lines of this file are not shown. View the raw message.

include/asm-generic/barrier.h +40 -1

41 lines not on this page · raw message

include/asm-generic/mutex-dec.h +1 -1
index fd694cfd678a..c54829d3de37 100644

Lines of this file are not shown. View the raw message.

include/asm-generic/mutex-xchg.h +5 -1
index a6b4a7bd6ac9..3269ec4e195f 100644

Lines of this file are not shown. View the raw message.

include/asm-generic/qspinlock.h +2 -3
index 05f05f17a7c2..9f0681bf1e87 100644

Lines of this file are not shown. View the raw message.

include/asm-generic/rwsem.h +3 -19
index 3fc94a046bf5..5be122e3d326 100644

Lines of this file are not shown. View the raw message.

include/linux/atomic.h +590 -157

747 lines not on this page · raw message

include/linux/compiler.h +4 -17
index 793c0829e3a3..2e853b679a5d 100644

Lines of this file are not shown. View the raw message.

include/linux/percpu-refcount.h +5 -7
index 84f542df7ff5..1c7eec09e5eb 100644

Lines of this file are not shown. View the raw message.

include/linux/rwsem.h +5 -3
index d37fbb34d06f..dd1d14250340 100644

Lines of this file are not shown. View the raw message.

include/linux/spinlock_up.h +7 -3
index 8b3ac0d718eb..0d9848de677d 100644

Lines of this file are not shown. View the raw message.

ipc/sem.c +1 -12
index b3757ea0694b..ae72b3cddc8d 100644

Lines of this file are not shown. View the raw message.

kernel/exit.c +6 -2
index 9e6e1356e6bb..0b40791b9e70 100644

Lines of this file are not shown. View the raw message.

kernel/jump_label.c +1 -1
index 05254eeb4b4e..ac4ab953b49c 100644

Lines of this file are not shown. View the raw message.

kernel/locking/lockdep.c +9 -4
index 81f1a7107c0e..589d763a49b3 100644

Lines of this file are not shown. View the raw message.

kernel/locking/mutex-debug.h +2 -2
index 0799fd3e4cfa..372e6530180d 100644

Lines of this file are not shown. View the raw message.

kernel/locking/mutex.h +8 -2
index 5cda397607f2..12f96199441c 100644

Lines of this file are not shown. View the raw message.

kernel/locking/qrwlock.c +1 -1
index fec082338668..19248ddf37ce 100644

Lines of this file are not shown. View the raw message.

kernel/locking/qspinlock.c +78 -10

88 lines not on this page · raw message

kernel/locking/qspinlock_paravirt.h +2 -2
index 21ede57f68b3..37649e69056c 100644

Lines of this file are not shown. View the raw message.

kernel/locking/rtmutex.c +1 -1
index 3e746607abe5..1ec0f48962b3 100644

Lines of this file are not shown. View the raw message.

kernel/locking/rwsem-xadd.c +120 -74

194 lines not on this page · raw message

kernel/locking/rwsem.c +6 -2
index 2e853ad93a3a..45ba475d4be3 100644

Lines of this file are not shown. View the raw message.

kernel/locking/rwsem.h +50 -2

52 lines not on this page · raw message

kernel/rcu/tree.c +0 -1
index c7f1bc4f817c..b7326893221f 100644

Lines of this file are not shown. View the raw message.

kernel/sched/core.c +4 -4
index 017d5394f5dc..5cd6931cb2cb 100644

Lines of this file are not shown. View the raw message.

kernel/sched/sched.h +1 -1
index 72f1f3087b04..425bf5ddaa5a 100644

Lines of this file are not shown. View the raw message.

kernel/smp.c +1 -1
index 74165443c240..36552beed397 100644

Lines of this file are not shown. View the raw message.

kernel/task_work.c +0 -1
index 53fa971d000d..6ab4842b00e8 100644

Lines of this file are not shown. View the raw message.

lib/atomic64.c +28 -4
index 2886ebac6567..53c2d5edc826 100644

Lines of this file are not shown. View the raw message.

lib/atomic64_test.c +34 -0
index 123481814320..dbb369145dda 100644

Lines of this file are not shown. View the raw message.

net/netfilter/nf_conntrack_core.c +22 -1
index db2312eeb2a4..b8c5501d3872 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.