Skip to contentSkip to conversation

Question on smp_mb__before_spinlock

PATCH Peter Zijlstra 5 participants 18 messages Sep 5, 2016 – Sep 13, 2016

Subsystems (4)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

Hi all,

So recently I've had two separate issues that touched upon smp_mb__before_spinlock().

Since its inception, our understanding of ACQUIRE, esp. as applied to spinlocks, has changed somewhat. Also, I wonder if, with a simple change, we cannot make it provide more.

The problem with the comment is that the STORE done by spin_lock isn't itself ordered by the ACQUIRE, and therefore a later LOAD can pass over it and cross with any prior STORE, rendering the default WMB insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, the former of which already defined smp_mb__before_spinlock() as a smp_mb(), the latter does not, Will?

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

That would make all schedule() calls fully transitive against one another.

That is, would something like the below make sense?

(does not deal with mm_types.h and overlayfs using smp_mb__before_spnlock).

---
 arch/arm64/include/asm/barrier.h   |  2 ++
 arch/powerpc/include/asm/barrier.h |  2 +-
 include/linux/spinlock.h           | 41 +++++++++++++++++++++++++++++---------
 kernel/sched/core.c                |  5 +++--
 4 files changed, 38 insertions(+), 12 deletions(-)
arch/arm64/include/asm/barrier.h +2 -0
index 4eea7f618dce..d5cc8b58f942 100644
@@ -104,6 +104,8 @@ do {									\
 	VAL;								\
 })
 
+#define smp_mb__after_spinlock()	smp_mb()
+
 #include <asm-generic/barrier.h>
 
 #endif	/* __ASSEMBLY__ */
arch/powerpc/include/asm/barrier.h +1 -1
index c0deafc212b8..23d64d7196b7 100644
@@ -74,7 +74,7 @@ do {									\
 	___p1;								\
 })
 
-#define smp_mb__before_spinlock()   smp_mb()
+#define smp_mb__after_spinlock()   smp_mb()
 
 #include <asm-generic/barrier.h>
 
include/linux/spinlock.h +32 -9

48 lines not on this page · raw message

kernel/sched/core.c +3 -2
index 556cb07ab1cf..b151a33d393b 100644
@@ -2006,8 +2006,8 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 	 * reordered with p->state check below. This pairs with mb() in
 	 * set_current_state() the waiting thread does.
 	 */
-	smp_mb__before_spinlock();
 	raw_spin_lock_irqsave(&p->pi_lock, flags);
+	smp_mb__after_spinlock();
 	if (!(p->state & state))
 		goto out;
 
@@ -3332,8 +3332,9 @@ static void __sched notrace __schedule(bool preempt)
 	 * can't be reordered with __set_current_state(TASK_INTERRUPTIBLE)
 	 * done by the caller to avoid the race with signal_wake_up().
 	 */
-	smp_mb__before_spinlock();
 	raw_spin_lock(&rq->lock);
+	smp_mb__after_spinlock();
+
 	cookie = lockdep_pin_lock(&rq->lock);
 
 	rq->clock_skip_update <<= 1; /* promote REQ to ACT */
#1 lore raw
url: https://github.com/0day-ci/linux/commits/Peter-Zijlstra/Question-on-smp_mb__before_spinlock/20160905-174026 config: x86_64-randconfig-x013-201636 (attached as .config) compiler: gcc-6 (Debian…

Hi Peter,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.8-rc5 next-20160825]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
[Suggest to use git(>=2.9.0) format-patch --base=<commit> (or --base=auto for convenience) to record what (public, well-known) commit your patch series was built on] [Check https://git-scm.com/docs/git-format-patch for more information]

url: https://github.com/0day-ci/linux/commits/Peter-Zijlstra/Question-on-smp_mb__before_spinlock/20160905-174026
config: x86_64-randconfig-x013-201636 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-9) 6.1.1 20160705 reproduce:

        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

In file included from include/linux/sched.h:27:0,

                    from include/linux/kasan.h:4,
                    from include/linux/slab.h:118,
                    from include/linux/crypto.h:24,
                    from arch/x86/kernel/asm-offsets.c:8:
   include/linux/mm_types.h: In function 'set_tlb_flush_pending':
>> include/linux/mm_types.h:557:2: error: implicit declaration of function 'smp_mb__before_spinlock' [-Werror=implicit-function-declaration]
     smp_mb__before_spinlock();
     ^~~~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors make[2]: *** [arch/x86/kernel/asm-offsets.s] Error 1 make[2]: Target '__build' not remade because of errors. make[1]: *** [prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [sub-make] Error 2

vim +/smp_mb__before_spinlock +557 include/linux/mm_types.h

20841405 Rik van Riel 2013-12-18  551  	mm->tlb_flush_pending = true;
af2c1401 Mel Gorman   2013-12-18  552  
af2c1401 Mel Gorman   2013-12-18  553  	/*
af2c1401 Mel Gorman   2013-12-18  554  	 * Guarantee that the tlb_flush_pending store does not leak into the
af2c1401 Mel Gorman   2013-12-18  555  	 * critical section updating the page tables
af2c1401 Mel Gorman   2013-12-18  556  	 */
af2c1401 Mel Gorman   2013-12-18 @557  	smp_mb__before_spinlock();
20841405 Rik van Riel 2013-12-18  558  }
20841405 Rik van Riel 2013-12-18  559  /* Clearing is done after a TLB flush, which also provides a barrier. */
20841405 Rik van Riel 2013-12-18  560  static inline void clear_tlb_flush_pending(struct mm_struct *mm)

:::::: The code at line 557 was first introduced by commit
:::::: af2c1401e6f9177483be4fad876d0073669df9df mm: numa: guarantee that tlb_flush_pending updates are visible before page table updates

:::::: TO: Mel Gorman <[email protected]>
:::::: CC: Linus Torvalds <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

author maintainer
#2 lore raw
Quoted 7 lines · kbuild test robot

Hi Peter,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.8-rc5 next-20160825]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
[Suggest to use git(>=2.9.0) format-patch --base=<commit> (or --base=auto for convenience) to record what (public, well-known) commit your patch series was built on] [Check https://git-scm.com/docs/git-format-patch for more information]

What happend to not applying patches that lack a SoB and the subject doesn't even include [PATCH] either.

build bot
#3 lore raw
Sorry the current logic assumes "Re: " in subject. Just fixed it to ignore all patches w/o SOB.

Hi Peter,

On Mon, Sep 05, 2016 at 12:19:37PM +0200, Peter Zijlstra wrote:

>On Mon, Sep 05, 2016 at 05:56:46PM +0800, kbuild test robot wrote:
>> Hi Peter,
>>
>> [auto build test ERROR on linus/master]
>> [also build test ERROR on v4.8-rc5 next-20160825]
>> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
>> [Suggest to use git(>=2.9.0) format-patch --base=<commit> (or --base=auto for convenience) to record what (public, well-known) commit your patch series was built on]
>> [Check https://git-scm.com/docs/git-format-patch for more information]
>
>What happend to not applying patches that lack a SoB and the subject
>doesn't even include [PATCH] either.

Sorry the current logic assumes "Re: " in subject. Just fixed it to ignore all patches w/o SOB.

Regards,
Fengguang

#4 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra

So recently I've had two separate issues that touched upon smp_mb__before_spinlock().

Since its inception, our understanding of ACQUIRE, esp. as applied to spinlocks, has changed somewhat. Also, I wonder if, with a simple change, we cannot make it provide more.

Quoted 8 lines

The problem with the comment is that the STORE done by spin_lock isn't itself ordered by the ACQUIRE, and therefore a later LOAD can pass over it and cross with any prior STORE, rendering the default WMB insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, the former of which already defined smp_mb__before_spinlock() as a smp_mb(), the latter does not, Will?

I just replied to that thread and, assuming I've groked the sched/core.c usage correctly, then it does look like we need to make that an smp_mb() with the current code.

Quoted 7 lines

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

That would make all schedule() calls fully transitive against one another.

It would also match the way in which the arm64 atomic_*_return ops are implemented, since full barrier semantics are required there.

Quoted 1 line

That is, would something like the below make sense?

Works for me, but I'll do a fix to smp_mb__before_spinlock anyway for the stable tree.

The only slight annoyance is that, on arm64 anyway, a store-release appearing in program order before the LOCK operation will be observed in order, so if the write of CONDITION=1 in the try_to_wake_up case used smp_store_release, we wouldn't need this barrier at all.

Will

author maintainer
#5 lore raw
Quoted 10 lines · Will Deacon
Quoted 7 lines

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

That would make all schedule() calls fully transitive against one another.

It would also match the way in which the arm64 atomic_*_return ops are implemented, since full barrier semantics are required there.

Hmm, are you sure; the way I read arch/arm64/include/asm/atomic_ll_sc.h is that you do ll/sc-rel + mb.

Quoted 4 lines
Quoted 1 line

That is, would something like the below make sense?

Works for me, but I'll do a fix to smp_mb__before_spinlock anyway for the stable tree.

Indeed, thanks!

Quoted 5 lines

The only slight annoyance is that, on arm64 anyway, a store-release appearing in program order before the LOCK operation will be observed in order, so if the write of CONDITION=1 in the try_to_wake_up case used smp_store_release, we wouldn't need this barrier at all.

Right, but this is because your load-acquire and store-release are much stronger than Linux's. Not only are they RCsc, they are also globally ordered irrespective of the variable (iirc).

This wouldn't work for PPC (even if we could find all such prior stores).

OK, I suppose I'll go stare what we can do about the mm_types.h use and spin a patch with Changelog.

#6 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Will Deacon
Quoted 5 lines

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

That would make all schedule() calls fully transitive against one another.

It would also match the way in which the arm64 atomic_*_return ops are implemented, since full barrier semantics are required there.

Hmm, are you sure; the way I read arch/arm64/include/asm/atomic_ll_sc.h is that you do ll/sc-rel + mb.

Yes, all I meant was that we put the full barrier at the end, but the two things are indeed different sequences.

Will

#7 lore raw
In reply to Peter Zijlstra
Quoted 24 lines · Peter Zijlstra

Hi all,

So recently I've had two separate issues that touched upon smp_mb__before_spinlock().

Since its inception, our understanding of ACQUIRE, esp. as applied to spinlocks, has changed somewhat. Also, I wonder if, with a simple change, we cannot make it provide more.

The problem with the comment is that the STORE done by spin_lock isn't itself ordered by the ACQUIRE, and therefore a later LOAD can pass over it and cross with any prior STORE, rendering the default WMB insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, the former of which already defined smp_mb__before_spinlock() as a smp_mb(), the latter does not, Will?

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

Quoted 5 lines

That would make all schedule() calls fully transitive against one another.

That is, would something like the below make sense?

Looks to me like you have reinvented smp_mb__after_unlock_lock()...

Thanx, Paul

Quoted 115 lines

(does not deal with mm_types.h and overlayfs using smp_mb__before_spnlock).

---
 arch/arm64/include/asm/barrier.h   |  2 ++
 arch/powerpc/include/asm/barrier.h |  2 +-
 include/linux/spinlock.h           | 41 +++++++++++++++++++++++++++++---------
 kernel/sched/core.c                |  5 +++--
 4 files changed, 38 insertions(+), 12 deletions(-)
arch/arm64/include/asm/barrier.h +2 -0
index 4eea7f618dce..d5cc8b58f942 100644
@@ -104,6 +104,8 @@ do {									\
 	VAL;								\
 })
 
+#define smp_mb__after_spinlock()	smp_mb()
+
 #include <asm-generic/barrier.h>
 
 #endif	/* __ASSEMBLY__ */
arch/powerpc/include/asm/barrier.h +1 -1
index c0deafc212b8..23d64d7196b7 100644
@@ -74,7 +74,7 @@ do {									\
 	___p1;								\
 })
 
-#define smp_mb__before_spinlock()   smp_mb()
+#define smp_mb__after_spinlock()   smp_mb()
 
 #include <asm-generic/barrier.h>
 
include/linux/spinlock.h +32 -9
index 47dd0cebd204..284616dad607 100644
@@ -118,16 +118,39 @@ do {								\
 #endif
 
 /*
- * Despite its name it doesn't necessarily has to be a full barrier.
- * It should only guarantee that a STORE before the critical section
- * can not be reordered with LOADs and STOREs inside this section.
- * spin_lock() is the one-way barrier, this LOAD can not escape out
- * of the region. So the default implementation simply ensures that
- * a STORE can not move into the critical section, smp_wmb() should
- * serialize it with another STORE done by spin_lock().
+ * This barrier must provide two things:
+ *
+ *   - it must guarantee a STORE before the spin_lock() is ordered against a
+ *     LOAD after it, see the comments at its two usage sites.
+ *
+ *   - it must ensure the critical section is RCsc.
+ *
+ * The latter is important for cases where we observe values written by other
+ * CPUs in spin-loops, without barriers, while being subject to scheduling.
+ *
+ * CPU0			CPU1			CPU2
+ * 
+ * 			for (;;) {
+ * 			  if (READ_ONCE(X))
+ * 			  	break;
+ * 			}
+ * X=1
+ * 			<sched-out>
+ * 						<sched-in>
+ * 						r = X;
+ *
+ * without transitivity it could be that CPU1 observes X!=0 breaks the loop,
+ * we get migrated and CPU2 sees X==0.
+ *
+ * Since most load-store architectures implement ACQUIRE with an smp_mb() after
+ * the LL/SC loop, they need no further barriers. Similarly all our TSO
+ * architectures imlpy an smp_mb() for each atomic instruction and equally don't
+ * need more.
+ *
+ * Architectures that can implement ACQUIRE better need to take care.
  */
-#ifndef smp_mb__before_spinlock
-#define smp_mb__before_spinlock()	smp_wmb()
+#ifndef smp_mb__after_spinlock
+#define smp_mb__after_spinlock()	do { } while (0)
 #endif
 
 /**
kernel/sched/core.c +3 -2
index 556cb07ab1cf..b151a33d393b 100644
@@ -2006,8 +2006,8 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 	 * reordered with p->state check below. This pairs with mb() in
 	 * set_current_state() the waiting thread does.
 	 */
-	smp_mb__before_spinlock();
 	raw_spin_lock_irqsave(&p->pi_lock, flags);
+	smp_mb__after_spinlock();
 	if (!(p->state & state))
 		goto out;
 
@@ -3332,8 +3332,9 @@ static void __sched notrace __schedule(bool preempt)
 	 * can't be reordered with __set_current_state(TASK_INTERRUPTIBLE)
 	 * done by the caller to avoid the race with signal_wake_up().
 	 */
-	smp_mb__before_spinlock();
 	raw_spin_lock(&rq->lock);
+	smp_mb__after_spinlock();
+
 	cookie = lockdep_pin_lock(&rq->lock);
 
 	rq->clock_skip_update <<= 1; /* promote REQ to ACT */
author maintainer
#8 lore raw
Quoted 25 lines · Paul E. McKenney
Quoted 24 lines · Peter Zijlstra

Hi all,

So recently I've had two separate issues that touched upon smp_mb__before_spinlock().

Since its inception, our understanding of ACQUIRE, esp. as applied to spinlocks, has changed somewhat. Also, I wonder if, with a simple change, we cannot make it provide more.

The problem with the comment is that the STORE done by spin_lock isn't itself ordered by the ACQUIRE, and therefore a later LOAD can pass over it and cross with any prior STORE, rendering the default WMB insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, the former of which already defined smp_mb__before_spinlock() as a smp_mb(), the latter does not, Will?

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

Quoted 7 lines
Quoted 5 lines

That would make all schedule() calls fully transitive against one another.

That is, would something like the below make sense?

Looks to me like you have reinvented smp_mb__after_unlock_lock()...

Will said the same, but that one doesn't in fact do the first bit, as ARM64 also needs a full barrier for that, while it doesn't need that to upgrade to RCsc.

#9 lore raw
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Peter Zijlstra

Hi all,

So recently I've had two separate issues that touched upon smp_mb__before_spinlock().

Since its inception, our understanding of ACQUIRE, esp. as applied to spinlocks, has changed somewhat. Also, I wonder if, with a simple change, we cannot make it provide more.

The problem with the comment is that the STORE done by spin_lock isn't itself ordered by the ACQUIRE, and therefore a later LOAD can pass over it and cross with any prior STORE, rendering the default WMB insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, the former of which already defined smp_mb__before_spinlock() as a smp_mb(), the latter does not, Will?

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

That would make all schedule() calls fully transitive against one another.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

That is, would something like the below make sense?

Looks to me like you have reinvented smp_mb__after_unlock_lock()...

Will said the same, but that one doesn't in fact do the first bit, as ARM64 also needs a full barrier for that, while it doesn't need that to upgrade to RCsc.

Fair enough!

Thanx, Paul

#10 lore raw
In reply to Peter Zijlstra
url: https://github.com/0day-ci/linux/commits/Peter-Zijlstra/Question-on-smp_mb__before_spinlock/20160905-174026 config: i386-randconfig-s1-201636 (attached as .config) compiler: gcc-6 (Debian…

Hi Peter,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.8-rc5 next-20160825]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
[Suggest to use git(>=2.9.0) format-patch --base=<commit> (or --base=auto for convenience) to record what (public, well-known) commit your patch series was built on] [Check https://git-scm.com/docs/git-format-patch for more information]

url: https://github.com/0day-ci/linux/commits/Peter-Zijlstra/Question-on-smp_mb__before_spinlock/20160905-174026
config: i386-randconfig-s1-201636 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-9) 6.1.1 20160705 reproduce:

        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

fs/overlayfs/readdir.c: In function 'ovl_dir_fsync':

>> fs/overlayfs/readdir.c:449:4: error: implicit declaration of function 'smp_mb__before_spinlock' [-Werror=implicit-function-declaration]
       smp_mb__before_spinlock();
       ^~~~~~~~~~~~~~~~~~~~~~~

cc1: some warnings being treated as errors

vim +/smp_mb__before_spinlock +449 fs/overlayfs/readdir.c

d45f00ae Al Viro        2014-10-28  443  		realfile = lockless_dereference(od->upperfile);
e9be9d5e Miklos Szeredi 2014-10-24  444  		if (!realfile) {
e9be9d5e Miklos Szeredi 2014-10-24  445  			struct path upperpath;
e9be9d5e Miklos Szeredi 2014-10-24  446  
e9be9d5e Miklos Szeredi 2014-10-24  447  			ovl_path_upper(dentry, &upperpath);
e9be9d5e Miklos Szeredi 2014-10-24  448  			realfile = ovl_path_open(&upperpath, O_RDONLY);
d45f00ae Al Viro        2014-10-28 @449  			smp_mb__before_spinlock();
5955102c Al Viro        2016-01-22  450  			inode_lock(inode);
3d268c9b Al Viro        2014-10-23  451  			if (!od->upperfile) {
e9be9d5e Miklos Szeredi 2014-10-24  452  				if (IS_ERR(realfile)) {

:::::: The code at line 449 was first introduced by commit
:::::: d45f00ae43e63eff1b3d79df20610ae1ef645ebd overlayfs: barriers for opening upper-layer directory

:::::: TO: Al Viro <[email protected]>
:::::: CC: Al Viro <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#11 lore raw
In reply to Peter Zijlstra
Quoted 107 lines · On Mon, 5 Sep 2016 11:37:53 +0200 Peter Zijlstra <[email protected]> wrote:

Hi all,

So recently I've had two separate issues that touched upon smp_mb__before_spinlock().

Since its inception, our understanding of ACQUIRE, esp. as applied to spinlocks, has changed somewhat. Also, I wonder if, with a simple change, we cannot make it provide more.

The problem with the comment is that the STORE done by spin_lock isn't itself ordered by the ACQUIRE, and therefore a later LOAD can pass over it and cross with any prior STORE, rendering the default WMB insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, the former of which already defined smp_mb__before_spinlock() as a smp_mb(), the latter does not, Will?

The second issue I wondered about is spinlock transitivity. All except powerpc have RCsc locks, and since Power already does a full mb, would it not make sense to put it _after_ the spin_lock(), which would provide the same guarantee, but also upgrades the section to RCsc.

That would make all schedule() calls fully transitive against one another.

That is, would something like the below make sense?

(does not deal with mm_types.h and overlayfs using smp_mb__before_spnlock).

---
 arch/arm64/include/asm/barrier.h   |  2 ++
 arch/powerpc/include/asm/barrier.h |  2 +-
 include/linux/spinlock.h           | 41 +++++++++++++++++++++++++++++---------
 kernel/sched/core.c                |  5 +++--
 4 files changed, 38 insertions(+), 12 deletions(-)
arch/arm64/include/asm/barrier.h +2 -0
index 4eea7f618dce..d5cc8b58f942 100644
@@ -104,6 +104,8 @@ do {									\
 	VAL;								\
 })
 
+#define smp_mb__after_spinlock()	smp_mb()
+
 #include <asm-generic/barrier.h>
 
 #endif	/* __ASSEMBLY__ */
arch/powerpc/include/asm/barrier.h +1 -1
index c0deafc212b8..23d64d7196b7 100644
@@ -74,7 +74,7 @@ do {									\
 	___p1;								\
 })
 
-#define smp_mb__before_spinlock()   smp_mb()
+#define smp_mb__after_spinlock()   smp_mb()
 
 #include <asm-generic/barrier.h>
 
include/linux/spinlock.h +26 -7
index 47dd0cebd204..284616dad607 100644
@@ -118,16 +118,39 @@ do {								\
 #endif
 
 /*
- * Despite its name it doesn't necessarily has to be a full barrier.
- * It should only guarantee that a STORE before the critical section
- * can not be reordered with LOADs and STOREs inside this section.
- * spin_lock() is the one-way barrier, this LOAD can not escape out
- * of the region. So the default implementation simply ensures that
- * a STORE can not move into the critical section, smp_wmb() should
- * serialize it with another STORE done by spin_lock().
+ * This barrier must provide two things:
+ *
+ *   - it must guarantee a STORE before the spin_lock() is ordered against a
+ *     LOAD after it, see the comments at its two usage sites.
+ *
+ *   - it must ensure the critical section is RCsc.
+ *
+ * The latter is important for cases where we observe values written by other
+ * CPUs in spin-loops, without barriers, while being subject to scheduling.
+ *
+ * CPU0			CPU1			CPU2
+ * 
+ * 			for (;;) {
+ * 			  if (READ_ONCE(X))
+ * 			  	break;
+ * 			}
+ * X=1
+ * 			<sched-out>
+ * 						<sched-in>
+ * 						r = X;
+ *
+ * without transitivity it could be that CPU1 observes X!=0 breaks the loop,
+ * we get migrated and CPU2 sees X==0.
+ *
+ * Since most load-store architectures implement ACQUIRE with an smp_mb() after
+ * the LL/SC loop, they need no further barriers. Similarly all our TSO

include/linux/spinlock.h do { \ · line 154

Quoted 10 lines
include/linux/spinlock.h +6 -2
+ * architectures imlpy an smp_mb() for each atomic instruction and equally don't
+ * need more.
+ *
+ * Architectures that can implement ACQUIRE better need to take care.
  */
-#ifndef smp_mb__before_spinlock
-#define smp_mb__before_spinlock()	smp_wmb()
+#ifndef smp_mb__after_spinlock
+#define smp_mb__after_spinlock()	do { } while (0)
 #endif

It seems okay, but why not make it a special sched-only function name to prevent it being used in generic code?

I would not mind seeing responsibility for the switch barrier moved to generic context switch code like this (alternative for powerpc reducing number of hwsync instructions was to add documentation and warnings about the barriers in arch dependent and independent code). And pairing it with a spinlock is reasonable.

It may not strictly be an "smp_" style of barrier if MMIO accesses are to be ordered here too, despite critical section may only be providing acquire/release for cacheable memory, so maybe it's slightly more complicated than just cacheable RCsc?

This would end up flushing the store queue while holding the spinlock on POWER, as opposed to before acquiring the lock. I doubt that's ever going to be noticable, but if you already add a special new primitive here, then an arch wrapper for raw_spin_lock() can let archs do their own thing here.

Thanks,
Nick

author maintainer
#12 lore raw
Quoted 45 lines · Nicholas Piggin
Quoted 32 lines

/*
+ * This barrier must provide two things:
+ *
+ * - it must guarantee a STORE before the spin_lock() is ordered against a
+ * LOAD after it, see the comments at its two usage sites.
+ *
+ * - it must ensure the critical section is RCsc.
+ *
+ * The latter is important for cases where we observe values written by other
+ * CPUs in spin-loops, without barriers, while being subject to scheduling.
+ *
+ * CPU0 CPU1 CPU2
+ *
+ * for (;;) {
+ * if (READ_ONCE(X))
+ * break;
+ * }
+ * X=1
+ * <sched-out>
+ * <sched-in>
+ * r = X;
+ *
+ * without transitivity it could be that CPU1 observes X!=0 breaks the loop,
+ * we get migrated and CPU2 sees X==0.
+ *
+ * Since most load-store architectures implement ACQUIRE with an smp_mb() after
+ * the LL/SC loop, they need no further barriers. Similarly all our TSO
+ * architectures imlpy an smp_mb() for each atomic instruction and equally don't
+ * need more.
+ *
+ * Architectures that can implement ACQUIRE better need to take care. */

include/linux/spinlock.h do { \ · line 154

Quoted 3 lines
include/linux/spinlock.h +2 -0
+#ifndef smp_mb__after_spinlock
+#define smp_mb__after_spinlock()	do { } while (0)
 #endif

It seems okay, but why not make it a special sched-only function name to prevent it being used in generic code?

I would not mind seeing responsibility for the switch barrier moved to generic context switch code like this (alternative for powerpc reducing number of hwsync instructions was to add documentation and warnings about the barriers in arch dependent and independent code). And pairing it with a spinlock is reasonable.

Quoted 4 lines

It may not strictly be an "smp_" style of barrier if MMIO accesses are to be ordered here too, despite critical section may only be providing acquire/release for cacheable memory, so maybe it's slightly more complicated than just cacheable RCsc?

Interesting idea..

So I'm not a fan of that raw_spin_lock wrapper, since that would end up with a lot more boiler-plate code than just the one extra barrier.

But moving MMIO/DMA/TLB etc.. barriers into this spinlock might not be a good idea, since those are typically fairly heavy barriers, and its quite common to call schedule() without ending up in switch_to().

For PowerPC it works out, since there's only SYNC, no other option afaik.

But ARM/ARM64 will have to do DSB(ISH) instead of DMB(ISH). IA64 would need to issue "sync.i" and mips-octeon "synciobdma".

Will, any idea of the extra cost involved in DSB vs DMB?

#13 lore raw
Quoted 24 lines · Peter Zijlstra
Quoted 9 lines · Nicholas Piggin

It seems okay, but why not make it a special sched-only function name to prevent it being used in generic code?

I would not mind seeing responsibility for the switch barrier moved to generic context switch code like this (alternative for powerpc reducing number of hwsync instructions was to add documentation and warnings about the barriers in arch dependent and independent code). And pairing it with a spinlock is reasonable.

Quoted 4 lines

It may not strictly be an "smp_" style of barrier if MMIO accesses are to be ordered here too, despite critical section may only be providing acquire/release for cacheable memory, so maybe it's slightly more complicated than just cacheable RCsc?

Interesting idea..

So I'm not a fan of that raw_spin_lock wrapper, since that would end up with a lot more boiler-plate code than just the one extra barrier.

But moving MMIO/DMA/TLB etc.. barriers into this spinlock might not be a good idea, since those are typically fairly heavy barriers, and its quite common to call schedule() without ending up in switch_to().

Quoted 7 lines

For PowerPC it works out, since there's only SYNC, no other option afaik.

But ARM/ARM64 will have to do DSB(ISH) instead of DMB(ISH). IA64 would need to issue "sync.i" and mips-octeon "synciobdma".

Will, any idea of the extra cost involved in DSB vs DMB?

DSB is *much* more expensive, since it completes out-of-band communication such as MMIO accesses and TLB invalidation, as well as plain old memory accesses.

The only reason we have DSB in our __switch_to code is to complete cache maintenance in case the task is going to migrate to another CPU; there's just no way to know that at the point we need to do the barrier :(

Will

#14 lore raw
Quoted 33 lines · On Wed, 7 Sep 2016 14:51:47 +0100 Will Deacon <[email protected]> wrote:
Quoted 24 lines · Peter Zijlstra
Quoted 9 lines · Nicholas Piggin

It seems okay, but why not make it a special sched-only function name to prevent it being used in generic code?

I would not mind seeing responsibility for the switch barrier moved to generic context switch code like this (alternative for powerpc reducing number of hwsync instructions was to add documentation and warnings about the barriers in arch dependent and independent code). And pairing it with a spinlock is reasonable.

Quoted 4 lines

It may not strictly be an "smp_" style of barrier if MMIO accesses are to be ordered here too, despite critical section may only be providing acquire/release for cacheable memory, so maybe it's slightly more complicated than just cacheable RCsc?

Interesting idea..

So I'm not a fan of that raw_spin_lock wrapper, since that would end up with a lot more boiler-plate code than just the one extra barrier.

But moving MMIO/DMA/TLB etc.. barriers into this spinlock might not be a good idea, since those are typically fairly heavy barriers, and its quite common to call schedule() without ending up in switch_to().

Quoted 7 lines

For PowerPC it works out, since there's only SYNC, no other option afaik.

But ARM/ARM64 will have to do DSB(ISH) instead of DMB(ISH). IA64 would need to issue "sync.i" and mips-octeon "synciobdma".

Will, any idea of the extra cost involved in DSB vs DMB?

Quoted 7 lines

DSB is *much* more expensive, since it completes out-of-band communication such as MMIO accesses and TLB invalidation, as well as plain old memory accesses.

The only reason we have DSB in our __switch_to code is to complete cache maintenance in case the task is going to migrate to another CPU; there's just no way to know that at the point we need to do the barrier :(

Unfortunately it's not trivial to move such barriers to migrate-time, because the source CPU may not be involved after the task is switched out.

This won't prevent ARM32/64 from continuing to do what it does today, if we note that the arch must provide such barriers *either* in the context switch lock / barrier, or in its own switch code.

Thanks,
Nick

#15 lore raw
In reply to Peter Zijlstra
Quoted 51 lines · On Wed, 7 Sep 2016 15:23:54 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 45 lines · Nicholas Piggin
Quoted 32 lines

/*
+ * This barrier must provide two things:
+ *
+ * - it must guarantee a STORE before the spin_lock() is ordered against a
+ * LOAD after it, see the comments at its two usage sites.
+ *
+ * - it must ensure the critical section is RCsc.
+ *
+ * The latter is important for cases where we observe values written by other
+ * CPUs in spin-loops, without barriers, while being subject to scheduling.
+ *
+ * CPU0 CPU1 CPU2
+ *
+ * for (;;) {
+ * if (READ_ONCE(X))
+ * break;
+ * }
+ * X=1
+ * <sched-out>
+ * <sched-in>
+ * r = X;
+ *
+ * without transitivity it could be that CPU1 observes X!=0 breaks the loop,
+ * we get migrated and CPU2 sees X==0.
+ *
+ * Since most load-store architectures implement ACQUIRE with an smp_mb() after
+ * the LL/SC loop, they need no further barriers. Similarly all our TSO
+ * architectures imlpy an smp_mb() for each atomic instruction and equally don't
+ * need more.
+ *
+ * Architectures that can implement ACQUIRE better need to take care. */

include/linux/spinlock.h do { \ · line 154

Quoted 3 lines
include/linux/spinlock.h +2 -0
+#ifndef smp_mb__after_spinlock
+#define smp_mb__after_spinlock()	do { } while (0)
 #endif  

It seems okay, but why not make it a special sched-only function name to prevent it being used in generic code?

I would not mind seeing responsibility for the switch barrier moved to generic context switch code like this (alternative for powerpc reducing number of hwsync instructions was to add documentation and warnings about the barriers in arch dependent and independent code). And pairing it with a spinlock is reasonable.

Quoted 4 lines

It may not strictly be an "smp_" style of barrier if MMIO accesses are to be ordered here too, despite critical section may only be providing acquire/release for cacheable memory, so maybe it's slightly more complicated than just cacheable RCsc?

Quoted 4 lines

Interesting idea..

So I'm not a fan of that raw_spin_lock wrapper, since that would end up with a lot more boiler-plate code than just the one extra barrier.

#ifndef sched_ctxsw_raw_spin_lock
#define sched_ctxsw_raw_spin_lock(lock) raw_spin_lock(lock) #endif

#define sched_ctxsw_raw_spin_lock(lock) do { smp_mb() ; raw_spin_lock(lock); } while (0)

?

Quoted 3 lines

But moving MMIO/DMA/TLB etc.. barriers into this spinlock might not be a good idea, since those are typically fairly heavy barriers, and its quite common to call schedule() without ending up in switch_to().

That's true I guess, but if we already have the arch specific smp_mb__ specifically for this context switch code, and you are asking for them to implement *cacheable* memory barrier vs migration, then I see no reason not to allow them to implement uncacheable as well.

You make a good point about schedule() without switch_to(), but architectures will still have no less flexibility than they do now.

Thanks,
Nick

author maintainer
#16 lore raw
Quoted 5 lines · Nicholas Piggin
Quoted 2 lines · On Wed, 7 Sep 2016 15:23:54 +0200 Peter Zijlstra <[email protected]> wrote:

Interesting idea..

Quoted 8 lines
Quoted 2 lines

So I'm not a fan of that raw_spin_lock wrapper, since that would end up with a lot more boiler-plate code than just the one extra barrier.

#ifndef sched_ctxsw_raw_spin_lock
#define sched_ctxsw_raw_spin_lock(lock) raw_spin_lock(lock) #endif

#define sched_ctxsw_raw_spin_lock(lock) do { smp_mb() ; raw_spin_lock(lock); } while (0)

I was thinking you wanted to avoid the lwsync in arch_spin_lock() entirely, at which point you'll grow more layers. Because then you get an arch_spin_lock_mb() or something and then you'll have to do the raw_spin_lock wrappery for that.

Or am I missing the point of having the raw_spin_lock wrapper, as opposed to the extra barrier after it?

Afaict the benefit of having that wrapper is so you can avoid issuing multiple barriers.

Quoted 11 lines
Quoted 3 lines

But moving MMIO/DMA/TLB etc.. barriers into this spinlock might not be a good idea, since those are typically fairly heavy barriers, and its quite common to call schedule() without ending up in switch_to().

That's true I guess, but if we already have the arch specific smp_mb__ specifically for this context switch code, and you are asking for them to implement *cacheable* memory barrier vs migration, then I see no reason not to allow them to implement uncacheable as well.

You make a good point about schedule() without switch_to(), but architectures will still have no less flexibility than they do now.

Ah, so you're saying make it optional where they put it? I was initially thinking you wanted to add it to the list of requirements. Sure, optional works.

#17 lore raw
Quoted 20 lines · On Mon, 12 Sep 2016 14:54:03 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 5 lines · Nicholas Piggin
Quoted 2 lines · On Wed, 7 Sep 2016 15:23:54 +0200 Peter Zijlstra <[email protected]> wrote:

Interesting idea..

Quoted 8 lines
Quoted 2 lines

So I'm not a fan of that raw_spin_lock wrapper, since that would end up with a lot more boiler-plate code than just the one extra barrier.

#ifndef sched_ctxsw_raw_spin_lock
#define sched_ctxsw_raw_spin_lock(lock) raw_spin_lock(lock) #endif

#define sched_ctxsw_raw_spin_lock(lock) do { smp_mb() ; raw_spin_lock(lock); } while (0)

I was thinking you wanted to avoid the lwsync in arch_spin_lock() entirely, at which point you'll grow more layers. Because then you get an arch_spin_lock_mb() or something and then you'll have to do the raw_spin_lock wrappery for that.

Quoted 5 lines

Or am I missing the point of having the raw_spin_lock wrapper, as opposed to the extra barrier after it?

Afaict the benefit of having that wrapper is so you can avoid issuing multiple barriers.

Oh you could do that too yes. But it's all going to be in arch/powerpc, so I don't know if layers would be much problem.

I was thinking to avoid the hwsync inside the critical section.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

But moving MMIO/DMA/TLB etc.. barriers into this spinlock might not be a good idea, since those are typically fairly heavy barriers, and its quite common to call schedule() without ending up in switch_to().

That's true I guess, but if we already have the arch specific smp_mb__ specifically for this context switch code, and you are asking for them to implement *cacheable* memory barrier vs migration, then I see no reason not to allow them to implement uncacheable as well.

Quoted 6 lines
Quoted 2 lines

You make a good point about schedule() without switch_to(), but architectures will still have no less flexibility than they do now.

Ah, so you're saying make it optional where they put it? I was initially thinking you wanted to add it to the list of requirements. Sure, optional works.

Yes i.e., this primitive must provide minimally X, and optionally Y. If Y is not provided, then switch_to or other arch hook must provide it.

Thanks,
Nick