Skip to contentSkip to conversation

riscv: Support qspinlock with generic headers

PATCH In review [email protected] 3 participants 7 messages Jun 21, 2022 – Jun 23, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v6 cover riscv: Support qspinlock with generic headers

From: Guo Ren <[email protected]>

Enable qspinlock and meet the requirements mentioned in a8ad07e5240c9 ("asm-generic: qspinlock: Indicate the use of mixed-size atomics").

RISC-V LR/SC pairs could provide a strong/weak forward guarantee that depends on micro-architecture. And RISC-V ISA spec has given out several limitations to let hardware support strict forward guarantee (RISC-V User ISA - 8.3 Eventual Success of Store-Conditional Instructions):
We restricted the length of LR/SC loops to fit within 64 contiguous instruction bytes in the base ISA to avoid undue restrictions on instruction cache and TLB size and associativity. Similarly, we disallowed other loads and stores within the loops to avoid restrictions on data-cache associativity in simple implementations that track the reservation within a private cache. The restrictions on branches and jumps limit the time that can be spent in the sequence. Floating-point operations and integer multiply/divide were disallowed to simplify the operating system’s emulation of these instructions on implementations lacking appropriate hardware support.
Software is not forbidden from using unconstrained LR/SC sequences, but portable software must detect the case that the sequence repeatedly fails, then fall back to an alternate code sequence that does not rely on an unconstrained LR/SC sequence. Implementations are permitted to unconditionally fail any unconstrained LR/SC sequence.

eg:
Some riscv hardware such as BOOMv3 & XiangShan could provide strict & strong forward guarantee (The cache line would be kept in an exclusive state for Backoff cycles, and only this core's interrupt could break the LR/SC pair).
Qemu riscv give a weak forward guarantee by wrong implementation currently [1].

The first version of patch was made in 2019.1 [2].

[1] https://github.com/qemu/qemu/blob/master/target/riscv/insn_trans/trans_rva.c.inc
[2] https://lore.kernel.org/linux-riscv/[email protected]/#r

Change V6:
- Fixup Clang compile problem Reported-by: kernel test robot <[email protected]>
- Cleanup asm-generic/spinlock.h
- Remove changelog in patch main comment part, suggested by [email protected]
- Remove "default y if NUMA" in Kconfig

Change V5:
- Update comment with RISC-V forward guarantee feature.
- Back to V3 direction and optimize asm code.

Change V4:
- Remove custom sub-word xchg implementation
- Add ARCH_USE_QUEUED_SPINLOCKS_XCHG32 in locking/qspinlock

Change V3:
- Coding convention by Peter Zijlstra's advices

Change V2:
- Coding convention in cmpxchg.h
- Re-implement short xchg
- Remove char & cmpxchg implementations

Guo Ren (2):
asm-generic: spinlock: Move qspinlock & ticket-lock into generic
spinlock.h
riscv: Add qspinlock support

 arch/riscv/Kconfig                    |  8 +++
 arch/riscv/include/asm/Kbuild         |  2 +
 arch/riscv/include/asm/cmpxchg.h      | 17 +++++
 include/asm-generic/spinlock.h        | 90 ++------------------------
 include/asm-generic/spinlock_types.h  | 14 ++--
 include/asm-generic/tspinlock.h       | 92 +++++++++++++++++++++++++++
 include/asm-generic/tspinlock_types.h | 17 +++++
 7 files changed, 146 insertions(+), 94 deletions(-)
 create mode 100644 include/asm-generic/tspinlock.h
 create mode 100644 include/asm-generic/tspinlock_types.h
Signature · 7 lines
2.36.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#1 lore raw

PATCH v6 1/2 asm-generic: spinlock: Move qspinlock & ticket-lock into generic spinlock.h

From: Guo Ren <[email protected]>

Separate ticket-lock into tspinlock.h and let generic spinlock support qspinlock or ticket-lock selected by CONFIG_ARCH_USE_QUEUED_SPINLOCKS config.

Cc Peter Zijlstra (Intel) <[email protected]>
Cc Arnd Bergmann <[email protected]>
---
 include/asm-generic/spinlock.h        | 90 ++------------------------
 include/asm-generic/spinlock_types.h  | 14 ++--
 include/asm-generic/tspinlock.h       | 92 +++++++++++++++++++++++++++
 include/asm-generic/tspinlock_types.h | 17 +++++
 4 files changed, 119 insertions(+), 94 deletions(-)
 create mode 100644 include/asm-generic/tspinlock.h
 create mode 100644 include/asm-generic/tspinlock_types.h
include/asm-generic/spinlock.h +5 -85
index fdfebcb050f4..4eca2488af38 100644
@@ -1,92 +1,12 @@
 /* SPDX-License-Identifier: GPL-2.0 */
-
-/*
- * 'Generic' ticket-lock implementation.
- *
- * It relies on atomic_fetch_add() having well defined forward progress
- * guarantees under contention. If your architecture cannot provide this, stick
- * to a test-and-set lock.
- *
- * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a
- * sub-word of the value. This is generally true for anything LL/SC although
- * you'd be hard pressed to find anything useful in architecture specifications
- * about this. If your architecture cannot do this you might be better off with
- * a test-and-set.
- *
- * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence
- * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with
- * a full fence after the spin to upgrade the otherwise-RCpc
- * atomic_cond_read_acquire().
- *
- * The implementation uses smp_cond_load_acquire() to spin, so if the
- * architecture has WFE like instructions to sleep instead of poll for word
- * modifications be sure to implement that (see ARM64 for example).
- *
- */
-
 #ifndef __ASM_GENERIC_SPINLOCK_H
 #define __ASM_GENERIC_SPINLOCK_H
 
-#include <linux/atomic.h>
-#include <asm-generic/spinlock_types.h>
-
-static __always_inline void arch_spin_lock(arch_spinlock_t *lock)
-{
-	u32 val = atomic_fetch_add(1<<16, lock);
-	u16 ticket = val >> 16;
-
-	if (ticket == (u16)val)
-		return;
-
-	/*
-	 * atomic_cond_read_acquire() is RCpc, but rather than defining a
-	 * custom cond_read_rcsc() here we just emit a full fence.  We only
-	 * need the prior reads before subsequent writes ordering from
-	 * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we
-	 * have no outstanding writes due to the atomic_fetch_add() the extra
-	 * orderings are free.
-	 */
-	atomic_cond_read_acquire(lock, ticket == (u16)VAL);
-	smp_mb();
-}
-
-static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock)
-{
-	u32 old = atomic_read(lock);
-
-	if ((old >> 16) != (old & 0xffff))
-		return false;
-
-	return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */
-}
-
-static __always_inline void arch_spin_unlock(arch_spinlock_t *lock)
-{
-	u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN);
-	u32 val = atomic_read(lock);
-
-	smp_store_release(ptr, (u16)val + 1);
-}
-
-static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock)
-{
-	u32 val = atomic_read(lock);
-
-	return ((val >> 16) != (val & 0xffff));
-}
-
-static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock)
-{
-	u32 val = atomic_read(lock);
-
-	return (s16)((val >> 16) - (val & 0xffff)) > 1;
-}
-
-static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock)
-{
-	return !arch_spin_is_locked(&lock);
-}
-
+#ifdef CONFIG_ARCH_USE_QUEUED_SPINLOCKS
+#include <asm/qspinlock.h>
 #include <asm/qrwlock.h>
+#else
+#include <asm-generic/tspinlock.h>
+#endif
 
 #endif /* __ASM_GENERIC_SPINLOCK_H */
include/asm-generic/spinlock_types.h +5 -9
index 8962bb730945..9875c1d058b3 100644
@@ -3,15 +3,11 @@
 #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H
 #define __ASM_GENERIC_SPINLOCK_TYPES_H
 
-#include <linux/types.h>
-typedef atomic_t arch_spinlock_t;
-
-/*
- * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the
- * include.
- */
+#ifdef CONFIG_ARCH_USE_QUEUED_SPINLOCKS
+#include <asm-generic/qspinlock_types.h>
 #include <asm/qrwlock_types.h>
-
-#define __ARCH_SPIN_LOCK_UNLOCKED	ATOMIC_INIT(0)
+#else
+#include <asm-generic/tspinlock_types.h>
+#endif
 
 #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */
include/asm-generic/tspinlock.h new file +92 -0

92 lines not on this page · raw message

include/asm-generic/tspinlock_types.h new file +17 -0
new file mode 100644
index 000000000000..ca3ea5acd172
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __ASM_GENERIC_TSPINLOCK_TYPES_H
+#define __ASM_GENERIC_TSPINLOCK_TYPES_H
+
+#include <linux/types.h>
+typedef atomic_t arch_spinlock_t;
+
+/*
+ * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the
+ * include.
+ */
+#include <asm/qrwlock_types.h>
+
+#define __ARCH_SPIN_LOCK_UNLOCKED	ATOMIC_INIT(0)
+
+#endif /* __ASM_GENERIC_TSPINLOCK_TYPES_H */
Signature · 1 line
2.36.1
maintainer
#2 lore raw

Re: [PATCH V6 1/2] asm-generic: spinlock: Move qspinlock & ticket-lock into generic spinlock.h

Quoted 11 lines · On Tue, Jun 21, 2022 at 4:49 PM <[email protected]> wrote:

From: Guo Ren <[email protected]>

Separate ticket-lock into tspinlock.h and let generic spinlock support qspinlock or ticket-lock selected by CONFIG_ARCH_USE_QUEUED_SPINLOCKS config.

Cc Peter Zijlstra (Intel) <[email protected]>
Cc Arnd Bergmann <[email protected]>
Quoted 5 lines
---
 include/asm-generic/spinlock.h        | 90 ++------------------------
 include/asm-generic/spinlock_types.h  | 14 ++--
 include/asm-generic/tspinlock.h       | 92 +++++++++++++++++++++++++++
 include/asm-generic/tspinlock_types.h | 17 +++++

Unless someone has a very good argument for the "tspinlock" name, I would prefer naming the new file ticket_spinlock.h. While the 'qspinlock' name has an established meaning already, this is not the case for 'tspinlock', and the longer name would be less confusing in my opinion.

include/asm-generic/spinlock.h line 10

Quoted 6 lines
include/asm-generic/spinlock.h +5 -0
+#ifdef CONFIG_ARCH_USE_QUEUED_SPINLOCKS
+#include <asm/qspinlock.h>
 #include <asm/qrwlock.h>
+#else
+#include <asm-generic/tspinlock.h>
+#endif

As Huacai Chen suggested in the other thread, the asm/qrwlock.h include should be outside of the #ifdef here.

Quoted 13 lines
include/asm-generic/spinlock_types.h +0 -5
index 8962bb730945..9875c1d058b3 100644
@@ -3,15 +3,11 @@
 #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H
 #define __ASM_GENERIC_SPINLOCK_TYPES_H
 
-#include <linux/types.h>
-typedef atomic_t arch_spinlock_t;
-
-/*
- * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the

include/asm-generic/spinlock_types.h line 11

Quoted 10 lines
include/asm-generic/spinlock_types.h +5 -4
- * include.
- */
+#ifdef CONFIG_ARCH_USE_QUEUED_SPINLOCKS
+#include <asm-generic/qspinlock_types.h>
 #include <asm/qrwlock_types.h>
-
-#define __ARCH_SPIN_LOCK_UNLOCKED      ATOMIC_INIT(0)
+#else
+#include <asm-generic/tspinlock_types.h>
+#endif

I don't think this file warrants the extra indirection, since both versions have only a
few lines. Just put it all into one file, and change the files that include asm-generic/qspinlock_types.h to use asm-generic/spinlock_types.h instead.

Arnd

author
#3 lore raw

Re: [PATCH V6 1/2] asm-generic: spinlock: Move qspinlock & ticket-lock into generic spinlock.h

Quoted 19 lines · Arnd Bergmann
Quoted 11 lines · On Tue, Jun 21, 2022 at 4:49 PM <[email protected]> wrote:

From: Guo Ren <[email protected]>

Separate ticket-lock into tspinlock.h and let generic spinlock support qspinlock or ticket-lock selected by CONFIG_ARCH_USE_QUEUED_SPINLOCKS config.

Cc Peter Zijlstra (Intel) <[email protected]>
Cc Arnd Bergmann <[email protected]>
Quoted 5 lines
---
 include/asm-generic/spinlock.h        | 90 ++------------------------
 include/asm-generic/spinlock_types.h  | 14 ++--
 include/asm-generic/tspinlock.h       | 92 +++++++++++++++++++++++++++
 include/asm-generic/tspinlock_types.h | 17 +++++
Quoted 4 lines

Unless someone has a very good argument for the "tspinlock" name, I would prefer naming the new file ticket_spinlock.h. While the 'qspinlock' name has an established meaning already, this is not the case for 'tspinlock', and the longer name would be less confusing in my opinion.

Okay. ticket_spinlock is also good to me.

Quoted 10 lines

include/asm-generic/spinlock.h line 10

Quoted 7 lines
include/asm-generic/spinlock.h +5 -0
+#ifdef CONFIG_ARCH_USE_QUEUED_SPINLOCKS
+#include <asm/qspinlock.h>
 #include <asm/qrwlock.h>
+#else
+#include <asm-generic/tspinlock.h>
+#endif

As Huacai Chen suggested in the other thread, the asm/qrwlock.h include should be outside of the #ifdef here.

Okay

Quoted 21 lines
Quoted 21 lines
include/asm-generic/spinlock_types.h +2 -9
index 8962bb730945..9875c1d058b3 100644
@@ -3,15 +3,11 @@
 #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H
 #define __ASM_GENERIC_SPINLOCK_TYPES_H
 
-#include <linux/types.h>
-typedef atomic_t arch_spinlock_t;
-
-/*
- * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the
- * include.
- */
+#ifdef CONFIG_ARCH_USE_QUEUED_SPINLOCKS
+#include <asm-generic/qspinlock_types.h>
 #include <asm/qrwlock_types.h>
-
-#define __ARCH_SPIN_LOCK_UNLOCKED      ATOMIC_INIT(0)
Quoted 8 lines

include/asm-generic/spinlock_types.h line 11

Quoted 3 lines
include/asm-generic/spinlock_types.h +3 -0
+#else
+#include <asm-generic/tspinlock_types.h>
+#endif

I don't think this file warrants the extra indirection, since both versions have only a
few lines. Just put it all into one file, and change the files that include asm-generic/qspinlock_types.h to use asm-generic/spinlock_types.h instead.

Okay, I'll try that.

Quoted 2 lines

Arnd

--
Best Regards
Guo Ren

ML: https://lore.kernel.org/linux-csky/

#4 lore raw

PATCH v6 2/2 riscv: Add qspinlock support

In reply to [email protected]

From: Guo Ren <[email protected]>

Enable qspinlock by the requirements mentioned in a8ad07e5240c9 ("asm-generic: qspinlock: Indicate the use of mixed-size atomics").

- RISC-V atomic_*_release()/atomic_*_acquire() are implemented with own relaxed version plus acquire/release_fence for RCsc synchronization.

- RISC-V LR/SC pairs could provide a strong/weak forward guarantee that depends on micro-architecture. And RISC-V ISA spec has given out several limitations to let hardware support strict forward guarantee (RISC-V User ISA - 8.3 Eventual Success of Store-Conditional Instructions). Some riscv cores such as BOOMv3 & XiangShan could provide strict & strong forward guarantee (The cache line would be kept in an exclusive state for Backoff cycles, and only this core's interrupt could break the LR/SC pair).

- RISC-V provides cheap atomic_fetch_or_acquire() with RCsc.

- RISC-V only provides relaxed xchg16 to support qspinlock.

Cc Peter Zijlstra (Intel) <[email protected]>
Cc Waiman Long <[email protected]>
Cc Arnd Bergmann <[email protected]>
Cc Palmer Dabbelt <[email protected]>
---
 arch/riscv/Kconfig               |  8 ++++++++
 arch/riscv/include/asm/Kbuild    |  2 ++
 arch/riscv/include/asm/cmpxchg.h | 17 +++++++++++++++++
 3 files changed, 27 insertions(+)
arch/riscv/Kconfig +8 -0
index 32ffef9f6e5b..e1b57cb89189 100644
@@ -333,6 +333,14 @@ config NODES_SHIFT
 	  Specify the maximum number of NUMA Nodes available on the target
 	  system.  Increases memory reserved to accommodate various tables.
 
+config RISCV_USE_QUEUED_SPINLOCKS
+	bool "Using queued spinlock instead of ticket-lock"
+	depends on SMP && MMU
+	select ARCH_USE_QUEUED_SPINLOCKS
+	help
+	  Make sure your micro arch LL/SC has a strong forward progress guarantee.
+	  Otherwise, stay at ticket-lock.
+
 config RISCV_ALTERNATIVE
 	bool
 	depends on !XIP_KERNEL
arch/riscv/include/asm/Kbuild +2 -0
index 504f8b7e72d4..2cce98c7b653 100644
@@ -2,7 +2,9 @@
 generic-y += early_ioremap.h
 generic-y += flat.h
 generic-y += kvm_para.h
+generic-y += mcs_spinlock.h
 generic-y += parport.h
+generic-y += qspinlock.h
 generic-y += spinlock.h
 generic-y += spinlock_types.h
 generic-y += qrwlock.h
arch/riscv/include/asm/cmpxchg.h +17 -0
index 12debce235e5..492104d45a23 100644
@@ -17,6 +17,23 @@
 	__typeof__(new) __new = (new);					\
 	__typeof__(*(ptr)) __ret;					\
 	switch (size) {							\
+	case 2: { 							\
+		u32 temp;						\
+		u32 shif = ((ulong)__ptr & 2) ? 16 : 0;			\
+		u32 mask = 0xffff << shif;				\
+		__ptr = (__typeof__(ptr))((ulong)__ptr & ~(ulong)2);	\
+		__asm__ __volatile__ (					\
+			"0:	lr.w %0, %2\n"				\
+			"	and  %1, %0, %z3\n"			\
+			"	or   %1, %1, %z4\n"			\
+			"	sc.w %1, %1, %2\n"			\
+			"	bnez %1, 0b\n"				\
+			: "=&r" (__ret), "=&r" (temp), "+A" (*__ptr)	\
+			: "rJ" (~mask), "rJ" (__new << shif)		\
+			: "memory");					\
+		__ret = (__ret & mask) >> shif;				\
+		break;							\
+	}								\
 	case 4:								\
 		__asm__ __volatile__ (					\
 			"	amoswap.w %0, %2, %1\n"			\
Signature · 7 lines
2.36.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#5 lore raw
In reply to [email protected]
Quoted 5 lines · On 6/21/22 10:49, [email protected] wrote:

From: Guo Ren <[email protected]>

Enable qspinlock and meet the requirements mentioned in a8ad07e5240c9 ("asm-generic: qspinlock: Indicate the use of mixed-size atomics").

Quoted 8 lines

RISC-V LR/SC pairs could provide a strong/weak forward guarantee that depends on micro-architecture. And RISC-V ISA spec has given out several limitations to let hardware support strict forward guarantee (RISC-V User ISA - 8.3 Eventual Success of Store-Conditional Instructions):
We restricted the length of LR/SC loops to fit within 64 contiguous instruction bytes in the base ISA to avoid undue restrictions on instruction cache and TLB size and associativity. Similarly, we

Does the 64 contiguous bytes need to be cacheline aligned?

Regards,
Longman

author
#6 lore raw
Quoted 16 lines · Waiman Long
Quoted 5 lines · On 6/21/22 10:49, [email protected] wrote:

From: Guo Ren <[email protected]>

Enable qspinlock and meet the requirements mentioned in a8ad07e5240c9 ("asm-generic: qspinlock: Indicate the use of mixed-size atomics").

Quoted 8 lines

RISC-V LR/SC pairs could provide a strong/weak forward guarantee that depends on micro-architecture. And RISC-V ISA spec has given out several limitations to let hardware support strict forward guarantee (RISC-V User ISA - 8.3 Eventual Success of Store-Conditional Instructions):
We restricted the length of LR/SC loops to fit within 64 contiguous instruction bytes in the base ISA to avoid undue restrictions on instruction cache and TLB size and associativity. Similarly, we

Quoted 1 line

Does the 64 contiguous bytes need to be cacheline aligned?

No, they are instructions, and the IFU & issue units would guarantee that. The programmer needn't worry about that.

Quoted 4 lines

Regards,
Longman

Signature · 4 lines
Best Regards
 Guo Ren

ML: https://lore.kernel.org/linux-csky/