LoongArch: Add 128-bit atomic cmpxchg support (v4)
v4 cover LoongArch: Add 128-bit atomic cmpxchg support (v4)
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_central scx_qmap) where kmalloc_nolock_noprof returns NULL due to missing 128-bit atomics, leading to -ENOMEM errors during scheduler initialization
2. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support
3. "LoongArch: Add SCQ support detection"
- Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
4. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
The issue was identified through BPF scheduler test failures where scx_central and scx_qmap schedulers would fail to initialize. Testing was performed using the scx_qmap scheduler from tools/sched_ext/, confirming that the patches resolve the initialization failures.
---
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- dbar 0 -> __WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply __cmpxchg128_asm("ll.d", "sc.q", ptr, o, n) to __cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]
---
George Guo (3):
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg support
george (1):
LoongArch: Add SCQ support detection
arch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 4 ++ 6 files changed, 76 insertions(+)
---
base-commit: 2061f18ad76ecaddf8ed17df81b8611ea88dbddd
change-id: 20251120-2-d03862b2cf6d
Best regards,
Signature · 1 line
George Guo <[email protected]>
PATCH v4 1/4 LoongArch: Add 128-bit atomic cmpxchg support
From: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (scx_central scx_qmap) caused by kmalloc_nolock_noprof returning NULL due to missing 128-bit atomics. The NULL returns led to -ENOMEM errors during scheduler initialization, causing test cases to fail.
Verified by testing with the scx_qmap scheduler (located in tools/sched_ext/). Building with make and running ./tools/sched_ext/build/bin/scx_qmap.
--- arch/loongarch/include/asm/cmpxchg.h | 47 ++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
arch/loongarch/include/asm/cmpxchg.h +47 -0
@@ -111,6 +111,44 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -198,6 +236,15 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + __cmpxchg128_asm(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \
Signature · 1 line
2.49.0
PATCH v4 2/4 LoongArch: Add SCQ support detection
In reply to George GuoFrom: george <[email protected]>
Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 4 ++++ 4 files changed, 8 insertions(+)
arch/loongarch/include/asm/cpu-features.h +1 -0
@@ -70,5 +70,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */
arch/loongarch/include/asm/cpu.h +2 -0
@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */
arch/loongarch/include/asm/loongarch.h +1 -0
@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)
arch/loongarch/kernel/cpu-probe.c +4 -0
@@ -201,6 +201,10 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ; +else +pr_warn_once ("CPU does NOT support SCQ\n" );if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;
Signature · 1 line
2.49.0
PATCH v4 3/4 LoongArch: Use spinlock to emulate 128-bit cmpxchg
In reply to George GuoFrom: George Guo <[email protected]>
For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
--- arch/loongarch/include/asm/cmpxchg.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/cmpxchg.h +20 -1
@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_asm (amswap_db, m, val) \ ({ \
@@ -149,6 +150,23 @@ union __u128_halves { __ret.full; \ }) +#define __cmpxchg128_locked (ptr, old, new) \ +({ \ + u128 __ret; \ + static DEFINE_SPINLOCK(lock); \ + unsigned long flags; \ + \ + spin_lock_irqsave(&lock, flags); \ + \ + __ret = *(volatile u128 *)(ptr); \ + if (__ret == (old)) \ + *(volatile u128 *)(ptr) = (new); \ + \ + spin_unlock_irqrestore(&lock, flags); \ + \ + __ret; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -242,7 +260,8 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int#define arch_cmpxchg128 (ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ - __cmpxchg128_asm(ptr, o, n); \ + cpu_has_scq ? __cmpxchg128_asm(ptr, o, n) : \ + __cmpxchg128_locked(ptr, o, n); \ })#ifdef CONFIG_64BIT
Signature · 1 line
2.49.0
PATCH v4 4/4 LoongArch: Enable 128-bit atomics cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Add select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support on LoongArch.
--- arch/loongarch/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/loongarch/Kconfig +2 -0
@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE
@@ -140,6 +141,7 @@ config LOONGARCH select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT + select HAVE_CMPXCHG_DOUBLE select HAVE_EFFICIENT_UNALIGNED_ACCESS if !ARCH_STRICT_ALIGN select HAVE_EXIT_THREAD select HAVE_GENERIC_TIF_BITS
Signature · 1 line
2.49.0
+ Hev and Ruoyao
Please cc them in next respin.
Quoted 83 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization2. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg support
3. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.
4. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of__cmpxchg128using a spinlock to emulate the atomic operation.The issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.Signed-off-by George Guo <[email protected]>---
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]---
George Guo (3):
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg supportgeorge (1):
LoongArch: Add SCQ support detectionarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 4 ++ 6 files changed, 76 insertions(+)---
base-commit:2061f18ad76ecaddf8ed17df81b8611ea88dbddd
change-id: 20251120-2-d03862b2cf6dBest regards,
--
George Guo <[email protected]>
Quoted 14 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization
Quoted 6 lines
2. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg support
3. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.
The scq detection should be the first patch.
Quoted 63 lines
4. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of__cmpxchg128using a spinlock to emulate the atomic operation.The issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.Signed-off-by George Guo <[email protected]>---
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]---
George Guo (3):
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg supportgeorge (1):
LoongArch: Add SCQ support detectionarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 4 ++ 6 files changed, 76 insertions(+)---
base-commit:2061f18ad76ecaddf8ed17df81b8611ea88dbddd
change-id: 20251120-2-d03862b2cf6dBest regards,
--
George Guo <[email protected]>