LoongArch: Add 128-bit atomic cmpxchg support (v5)
v6 cover LoongArch: Add 128-bit atomic cmpxchg support (v5)
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add SCQ support detection"
- Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_central scx_qmap) where kmalloc_nolock_noprof returns NULL due to missing 128-bit atomics, leading to -ENOMEM errors during scheduler initialization
3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support
The issue was identified through BPF scheduler test failures where scx_central and scx_qmap schedulers would fail to initialize. Testing was performed using the scx_qmap scheduler from tools/sched_ext/, confirming that the patches resolve the initialization failures.
---
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- dbar 0 -> __WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply __cmpxchg128_asm("ll.d", "sc.q", ptr, o, n) to __cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]
---
George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg support
arch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)
---
base-commit: 612df905d7404450696e979c806ba4cdef8684f4
change-id: 20251120-2-d03862b2cf6d
Best regards,
Signature · 1 line
George Guo <[email protected]>
PATCH v6 1/4 LoongArch: Add SCQ support detection
From: George Guo <[email protected]>
Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)
arch/loongarch/include/asm/cpu-features.h +1 -0
@@ -70,5 +70,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */
arch/loongarch/include/asm/cpu.h +2 -0
@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */
arch/loongarch/include/asm/loongarch.h +1 -0
@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)
arch/loongarch/kernel/cpu-probe.c +2 -0
@@ -201,6 +201,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;
arch/loongarch/kernel/proc.c +1 -0
@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scp" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch));
Signature · 1 line
2.49.0
PATCH v6 2/4 LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (scx_central scx_qmap) caused by kmalloc_nolock_noprof returning NULL due to missing 128-bit atomics. The NULL returns led to -ENOMEM errors during scheduler initialization, causing test cases to fail.
Verified by testing with the scx_qmap scheduler (located in tools/sched_ext/). Building with make and running ./tools/sched_ext/build/bin/scx_qmap.
--- arch/loongarch/include/asm/cmpxchg.h | 47 ++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
arch/loongarch/include/asm/cmpxchg.h +47 -0
@@ -111,6 +111,44 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -198,6 +236,15 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + __cmpxchg128_asm(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \
Signature · 1 line
2.49.0
PATCH v6 3/4 LoongArch: Use spinlock to emulate 128-bit cmpxchg
In reply to George GuoFrom: George Guo <[email protected]>
For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
--- arch/loongarch/include/asm/cmpxchg.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/cmpxchg.h +20 -1
@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_asm (amswap_db, m, val) \ ({ \
@@ -149,6 +150,23 @@ union __u128_halves { __ret.full; \ }) +#define __cmpxchg128_locked (ptr, old, new) \ +({ \ + u128 __ret; \ + static DEFINE_SPINLOCK(lock); \ + unsigned long flags; \ + \ + spin_lock_irqsave(&lock, flags); \ + \ + __ret = *(volatile u128 *)(ptr); \ + if (__ret == (old)) \ + *(volatile u128 *)(ptr) = (new); \ + \ + spin_unlock_irqrestore(&lock, flags); \ + \ + __ret; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -242,7 +260,8 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int#define arch_cmpxchg128 (ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ - __cmpxchg128_asm(ptr, o, n); \ + cpu_has_scq ? __cmpxchg128_asm(ptr, o, n) : \ + __cmpxchg128_locked(ptr, o, n); \ })#ifdef CONFIG_64BIT
Signature · 1 line
2.49.0
Quoted 7 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
This series can not apply cleanly on top of loongarch-next branch, so I haven't tested it.
Quoted 84 lines
1. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of__cmpxchg128using a spinlock to emulate the atomic operation.4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg supportThe issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.Signed-off-by George Guo <[email protected]>---
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]---
George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg supportarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)---
base-commit:612df905d7404450696e979c806ba4cdef8684f4
change-id: 20251120-2-d03862b2cf6dBest regards,
--
George Guo <[email protected]>
cover LoongArch: Add 128-bit atomic cmpxchg support
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add SCQ support detection"
- Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_central scx_qmap) where kmalloc_nolock_noprof returns NULL due to missing 128-bit atomics, leading to -ENOMEM errors during scheduler initialization
3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support
The issue was identified through BPF scheduler test failures where scx_central and scx_qmap schedulers would fail to initialize. Testing was performed using the scx_qmap scheduler from tools/sched_ext/, confirming that the patches resolve the initialization failures.
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- dbar 0 -> __WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply __cmpxchg128_asm("ll.d", "sc.q", ptr, o, n) to __cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]
George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg support
arch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)
Signature · 1 line
2.49.0
PATCH v6 1/4 LoongArch: Add SCQ support detection
From: George Guo <[email protected]>
Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)
arch/loongarch/include/asm/cpu-features.h +1 -0
@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */
arch/loongarch/include/asm/cpu.h +2 -0
@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */
arch/loongarch/include/asm/loongarch.h +1 -0
@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)
arch/loongarch/kernel/cpu-probe.c +2 -0
@@ -205,6 +205,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;
arch/loongarch/kernel/proc.c +1 -0
@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scq" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch));
Signature · 1 line
2.49.0
PATCH v6 2/4 LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (scx_central scx_qmap) caused by kmalloc_nolock_noprof returning NULL due to missing 128-bit atomics. The NULL returns led to -ENOMEM errors during scheduler initialization, causing test cases to fail.
Verified by testing with the scx_qmap scheduler (located in tools/sched_ext/). Building with make and running ./tools/sched_ext/build/bin/scx_qmap.
--- arch/loongarch/include/asm/cmpxchg.h | 47 ++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
arch/loongarch/include/asm/cmpxchg.h +47 -0
@@ -137,6 +137,44 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -224,6 +262,15 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + __cmpxchg128_asm(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \
Signature · 1 line
2.49.0
PATCH v6 3/4 LoongArch: Use spinlock to emulate 128-bit cmpxchg
In reply to George GuoFrom: George Guo <[email protected]>
For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
--- arch/loongarch/include/asm/cmpxchg.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/cmpxchg.h +20 -1
@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_amo_asm (amswap_db, m, val) \ ({ \
@@ -175,6 +176,23 @@ union __u128_halves { __ret.full; \ }) +#define __cmpxchg128_locked (ptr, old, new) \ +({ \ + u128 __ret; \ + static DEFINE_SPINLOCK(lock); \ + unsigned long flags; \ + \ + spin_lock_irqsave(&lock, flags); \ + \ + __ret = *(volatile u128 *)(ptr); \ + if (__ret == (old)) \ + *(volatile u128 *)(ptr) = (new); \ + \ + spin_unlock_irqrestore(&lock, flags); \ + \ + __ret; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -268,7 +286,8 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int#define arch_cmpxchg128 (ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ - __cmpxchg128_asm(ptr, o, n); \ + cpu_has_scq ? __cmpxchg128_asm(ptr, o, n) : \ + __cmpxchg128_locked(ptr, o, n); \ })#ifdef CONFIG_64BIT
Signature · 1 line
2.49.0
PATCH v6 4/4 LoongArch: Enable 128-bit atomics cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Add select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support on LoongArch.
--- arch/loongarch/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/loongarch/Kconfig +2 -0
@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL
@@ -141,6 +142,7 @@ config LOONGARCH select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT + select HAVE_CMPXCHG_DOUBLE select HAVE_EFFICIENT_UNALIGNED_ACCESS if !ARCH_STRICT_ALIGN select HAVE_EXIT_THREAD select HAVE_GENERIC_TIF_BITS
Signature · 1 line
2.49.0
Re: [PATCH loongarch-next 0/4] LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoQuoted 32 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of__cmpxchg128using a spinlock to emulate the atomic operation.4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg supportThe issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.
Quoted 5 lines
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]
Please tag the subject line with v7 and resend, otherwise this confuses b4. Thanks.
Quoted 53 lines
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg supportarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)--
2.49.0
v7 cover LoongArch: Add 128-bit atomic cmpxchg support
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add SCQ support detection"
- Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_central scx_qmap) where kmalloc_nolock_noprof returns NULL due to missing 128-bit atomics, leading to -ENOMEM errors during scheduler initialization
3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support
The issue was identified through BPF scheduler test failures where scx_central and scx_qmap schedulers would fail to initialize. Testing was performed using the scx_qmap scheduler from tools/sched_ext/, confirming that the patches resolve the initialization failures.
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- dbar 0 -> __WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply __cmpxchg128_asm("ll.d", "sc.q", ptr, o, n) to __cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]
George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg support
arch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)
Signature · 1 line
2.49.0
PATCH v7 1/4 LoongArch: Add SCQ support detection
From: George Guo <[email protected]>
Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)
arch/loongarch/include/asm/cpu-features.h +1 -0
@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */
arch/loongarch/include/asm/cpu.h +2 -0
@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */
arch/loongarch/include/asm/loongarch.h +1 -0
@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)
arch/loongarch/kernel/cpu-probe.c +2 -0
@@ -205,6 +205,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;
arch/loongarch/kernel/proc.c +1 -0
@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scq" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch));
Signature · 1 line
2.49.0
Re: [PATCH v7 loongarch-next 1/4] LoongArch: Add SCQ support detection
Quoted 9 lines · George Guo
From: George Guo <[email protected]>
Check
CPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.Co-developed-by Yangyang Lian <[email protected]>Signed-off-by Yangyang Lian <[email protected]>Signed-off-by George Guo <[email protected]>
Quoted 8 lines
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)
Quoted 70 lines
arch/loongarch/include/asm/cpu-features.h +1
-0@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */ arch/loongarch/include/asm/cpu.h +2
-0@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */ arch/loongarch/include/asm/loongarch.h +1
-0@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)arch/loongarch/kernel/cpu-probe.c +2
-0@@ -205,6 +205,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;arch/loongarch/kernel/proc.c +1
-1@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scq" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch)); --2.49.0
Re: [PATCH v7 loongarch-next 1/4] LoongArch: Add SCQ support detection
In reply to George GuoQuoted 6 lines · George Guo
From: George Guo <[email protected]>
Check
CPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.
nit:
determin -> determine
instruction -> instruction
Quoted 81 lines
Co-developed-by Yangyang Lian <[email protected]>Signed-off-by Yangyang Lian <[email protected]>Signed-off-by George Guo <[email protected]>--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)arch/loongarch/include/asm/cpu-features.h +1
-0@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */ arch/loongarch/include/asm/cpu.h +2
-0@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */ arch/loongarch/include/asm/loongarch.h +1
-0@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)arch/loongarch/kernel/cpu-probe.c +2
-0@@ -205,6 +205,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;arch/loongarch/kernel/proc.c +1
-1@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scq" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch)); --2.49.0
PATCH v7 2/4 LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (scx_central scx_qmap) caused by kmalloc_nolock_noprof returning NULL due to missing 128-bit atomics. The NULL returns led to -ENOMEM errors during scheduler initialization, causing test cases to fail.
Verified by testing with the scx_qmap scheduler (located in tools/sched_ext/). Building with make and running ./tools/sched_ext/build/bin/scx_qmap.
--- arch/loongarch/include/asm/cmpxchg.h | 47 ++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
arch/loongarch/include/asm/cmpxchg.h +47 -0
@@ -137,6 +137,44 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -224,6 +262,15 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + __cmpxchg128_asm(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \
Signature · 1 line
2.49.0
Re: [PATCH v7 loongarch-next 2/4] LoongArch: Add 128-bit atomic cmpxchg support
Quoted 6 lines · George Guo
From: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
Quoted 9 lines
At the same time, fix BPF scheduler test failures (
scx_centralscx_qmap) caused bykmalloc_nolock_noprofreturning NULL due to missing 128-bit atomics. The NULL returns led to-ENOMEMerrors during scheduler initialization, causing test cases to fail.Verified by testing with the
scx_qmapscheduler (located intools/sched_ext/). Building withmakeand running./tools/sched_ext/build/bin/scx_qmap.
As I mentioned in last cycle, patch 2 and patch 3 can be merged into one. Please also add a link ([1]) to upstream commit that breaks these tests.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=5fb750e8a9ae
Quoted 73 lines
Signed-off-by George Guo <[email protected]>--- arch/loongarch/include/asm/cmpxchg.h | 47 ++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)arch/loongarch/include/asm/cmpxchg.h +47
-1@@ -137,6 +137,44 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {@@ -224,6 +262,15 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + __cmpxchg128_asm(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \ --2.49.0
PATCH v7 3/4 LoongArch: Use spinlock to emulate 128-bit cmpxchg
In reply to George GuoFrom: George Guo <[email protected]>
For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of __cmpxchg128 using a spinlock to emulate the atomic operation.
--- arch/loongarch/include/asm/cmpxchg.h | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
arch/loongarch/include/asm/cmpxchg.h +20 -1
@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_amo_asm (amswap_db, m, val) \ ({ \
@@ -175,6 +176,23 @@ union __u128_halves { __ret.full; \ }) +#define __cmpxchg128_locked (ptr, old, new) \ +({ \ + u128 __ret; \ + static DEFINE_SPINLOCK(lock); \ + unsigned long flags; \ + \ + spin_lock_irqsave(&lock, flags); \ + \ + __ret = *(volatile u128 *)(ptr); \ + if (__ret == (old)) \ + *(volatile u128 *)(ptr) = (new); \ + \ + spin_unlock_irqrestore(&lock, flags); \ + \ + __ret; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -268,7 +286,8 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int#define arch_cmpxchg128 (ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ - __cmpxchg128_asm(ptr, o, n); \ + cpu_has_scq ? __cmpxchg128_asm(ptr, o, n) : \ + __cmpxchg128_locked(ptr, o, n); \ })#ifdef CONFIG_64BIT
Signature · 1 line
2.49.0
PATCH v7 4/4 LoongArch: Enable 128-bit atomics cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Add select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support on LoongArch.
--- arch/loongarch/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/loongarch/Kconfig +2 -0
@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL
@@ -141,6 +142,7 @@ config LOONGARCH select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT + select HAVE_CMPXCHG_DOUBLE select HAVE_EFFICIENT_UNALIGNED_ACCESS if !ARCH_STRICT_ALIGN select HAVE_EXIT_THREAD select HAVE_GENERIC_TIF_BITS
Signature · 1 line
2.49.0
Re: [PATCH v7 loongarch-next 4/4] LoongArch: Enable 128-bit atomics cmpxchg support
Quoted 24 lines · George Guo
From: George Guo <[email protected]>
Add select
HAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg support on LoongArch.Signed-off-by George Guo <[email protected]>--- arch/loongarch/Kconfig | 2 ++ 1 file changed, 2 insertions(+)arch/loongarch/Kconfig +1
-0@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL
arch/loongarch/Kconfig config LOONGARCH
Quoted 8 lines
arch/loongarch/Kconfig +1
-1@@ -141,6 +142,7 @@ config LOONGARCH select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select HAVE_DYNAMIC_FTRACE_WITH_REGS select HAVE_EBPF_JIT + select HAVE_CMPXCHG_DOUBLE select HAVE_EFFICIENT_UNALIGNED_ACCESS if !ARCH_STRICT_ALIGN select HAVE_EXIT_THREAD select HAVE_GENERIC_TIF_BITS --
Keep the list sorted ?
Quoted 2 lines
2.49.0
Re: [PATCH v7 loongarch-next 0/4] LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoQuoted 23 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of__cmpxchg128using a spinlock to emulate the atomic operation.
Quoted 9 lines
4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg supportThe issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.
Testing good, this series fixes the BPF timer issues. For the series:
Quoted 58 lines
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg supportarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)--
2.49.0
v8 cover LoongArch: Add 128-bit atomic cmpxchg support
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of three patches:
1. "LoongArch: Add SCQ support detection"
- Check CPUCFG2_SCQ bit to determin if the CPU supports SCQ instrction.
2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), use a spinlock to emulate the atomic operation.
- Fixes BPF scheduler test failures (scx_central scx_qmap) where kmalloc_nolock_noprof returns NULL due to missing 128-bit atomics, leading to -ENOMEM errors during scheduler initialization
3. LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support
The issue was identified through BPF scheduler test failures where scx_central and scx_qmap schedulers would fail to initialize. Testing was performed using the scx_qmap scheduler from tools/sched_ext/, confirming that the patches resolve the initialization failures.
---
Changes in v8:
- Merge patch 2 and patch 3 into one patch
- Put HAVE_CMPXCHG_DOUBLE in order
- Link to v7: https://lore.kernel.org/all/[email protected]/
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- dbar 0 -> __WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply __cmpxchg128_asm("ll.d", "sc.q", ptr, o, n) to __cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]
George Guo (3):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Enable 128-bit atomics cmpxchg support
arch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)
Signature · 1 line
2.49.0
PATCH v8 1/3 LoongArch: Add SCQ support detection
From: George Guo <[email protected]>
Check CPUCFG2_SCQ bit to determine if the CPU supports SCQ instruction.
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)
arch/loongarch/include/asm/cpu-features.h +1 -0
@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */
arch/loongarch/include/asm/cpu.h +2 -0
@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */
arch/loongarch/include/asm/loongarch.h +1 -0
@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)
arch/loongarch/kernel/cpu-probe.c +2 -0
@@ -205,6 +205,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;
arch/loongarch/kernel/proc.c +1 -0
@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scq" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch));
Signature · 1 line
2.49.0
Re: [PATCH v8 loongarch-next 1/3] LoongArch: Add SCQ support detection
Quoted 10 lines · George Guo
From: George Guo <[email protected]>
Check
CPUCFG2_SCQbit to determine if the CPU supports SCQ instruction.Co-developed-by Yangyang Lian <[email protected]>Signed-off-by Yangyang Lian <[email protected]>Signed-off-by George Guo <[email protected]>---
There is a conflict with latest loongarch-next branch. Other than that
Quoted 77 lines
arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 ++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 7 insertions(+)arch/loongarch/include/asm/cpu-features.h +1
-0@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */ arch/loongarch/include/asm/cpu.h +2
-0@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */ arch/loongarch/include/asm/loongarch.h +1
-0@@ -94,6 +94,7 @@#define CPUCFG2_LSPW BIT(21)#define CPUCFG2_LAM BIT(22)#define CPUCFG2_PTW BIT(24) +#define CPUCFG2_SCQ BIT(30)#define LOONGARCH_CPUCFG3 0x3#define CPUCFG3_CCDMA BIT(0)arch/loongarch/kernel/cpu-probe.c +2
-0@@ -205,6 +205,8 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_PTW ; elf_hwcap |=HWCAP_LOONGARCH_PTW ; } +if (config &CPUCFG2_SCQ ) + c->options |=LOONGARCH_CPU_SCQ ;if (config &CPUCFG2_LSPW ) { c->options |=LOONGARCH_CPU_LSPW ; elf_hwcap |=HWCAP_LOONGARCH_LSPW ;arch/loongarch/kernel/proc.c +1
-1@@ -75,6 +75,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" );if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" );if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); +if (cpu_has_scq)seq_printf (m," scq" );seq_printf (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch)); --2.49.0
PATCH v8 2/3 LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
For LoongArch CPUs lacking 128-bit atomic instruction(e.g., the SCQ instruction on 3A5000), use a spinlock to emulate the atomic operation.
At the same time, fix BPF scheduler test failures (scx_central scx_qmap) caused by kmalloc_nolock_noprof returning NULL due to missing 128-bit atomics. The NULL returns led to -ENOMEM errors during scheduler initialization, causing test cases to fail.
Verified by testing with the scx_qmap scheduler (located in tools/sched_ext/). Building with make and running ./tools/sched_ext/build/bin/scx_qmap.
--- arch/loongarch/include/asm/cmpxchg.h | 66 ++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
arch/loongarch/include/asm/cmpxchg.h +66 -0
@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_amo_asm (amswap_db, m, val) \ ({ \
@@ -137,6 +138,61 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) + +#define __cmpxchg128_locked (ptr, old, new) \ +({ \ + u128 __ret; \ + static DEFINE_SPINLOCK(lock); \ + unsigned long flags; \ + \ + spin_lock_irqsave(&lock, flags); \ + \ + __ret = *(volatile u128 *)(ptr); \ + if (__ret == (old)) \ + *(volatile u128 *)(ptr) = (new); \ + \ + spin_unlock_irqrestore(&lock, flags); \ + \ + __ret; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {
@@ -224,6 +280,16 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + cpu_has_scq ? __cmpxchg128_asm(ptr, o, n) : \ + __cmpxchg128_locked(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \
Signature · 1 line
2.49.0
Re: [PATCH v8 loongarch-next 2/3] LoongArch: Add 128-bit atomic cmpxchg support
Quoted 15 lines · George Guo
From: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
For LoongArch CPUs lacking 128-bit atomic
instruction(e.g., the SCQ instruction on 3A5000), use a spinlock to emulate the atomic operation.At the same time, fix BPF scheduler test failures (
scx_centralscx_qmap) caused bykmalloc_nolock_noprofreturning NULL due to missing 128-bit atomics. The NULL returns led to-ENOMEMerrors during scheduler initialization, causing test cases to fail.
Quoted 7 lines
Verified by testing with the
scx_qmapscheduler (located intools/sched_ext/). Building withmakeand running./tools/sched_ext/build/bin/scx_qmap.Signed-off-by George Guo <[email protected]>---
Quoted 97 lines
arch/loongarch/include/asm/cmpxchg.h | 66 ++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)arch/loongarch/include/asm/cmpxchg.h +66
-1@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_amo_asm (amswap_db, m, val) \ ({ \@@ -137,6 +138,61 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __cmpxchg128_asm (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) + +#define __cmpxchg128_locked (ptr, old, new) \ +({ \ + u128 __ret; \ + static DEFINE_SPINLOCK(lock); \ + unsigned long flags; \ + \ + spin_lock_irqsave(&lock, flags); \ + \ + __ret = *(volatile u128 *)(ptr); \ + if (__ret == (old)) \ + *(volatile u128 *)(ptr) = (new); \ + \ + spin_unlock_irqrestore(&lock, flags); \ + \ + __ret; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {@@ -224,6 +280,16 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () 1 + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + cpu_has_scq ? __cmpxchg128_asm(ptr, o, n) : \ + __cmpxchg128_locked(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \ --2.49.0
PATCH v8 3/3 LoongArch: Enable 128-bit atomics cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Add select HAVE_CMPXCHG_DOUBLE and select HAVE_ALIGNED_STRUCT_PAGE in Kconfig to enable 128-bit atomic cmpxchg support on LoongArch.
--- arch/loongarch/Kconfig | 2 ++ 1 file changed, 2 insertions(+)
arch/loongarch/Kconfig +2 -0
@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL
@@ -130,6 +131,7 @@ config LOONGARCH select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_USERFAULTFD_MINOR if USERFAULTFD select HAVE_ASM_MODVERSIONS + select HAVE_CMPXCHG_DOUBLE select HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK
Signature · 1 line
2.49.0
Re: [PATCH v8 loongarch-next 3/3] LoongArch: Enable 128-bit atomics cmpxchg support
Quoted 6 lines · George Guo
From: George Guo <[email protected]>
Add select
HAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg support on LoongArch.
Quoted 28 lines
Signed-off-by George Guo <[email protected]>--- arch/loongarch/Kconfig | 2 ++ 1 file changed, 2 insertions(+)arch/loongarch/Kconfig +2
-1@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL@@ -130,6 +131,7 @@ config LOONGARCH select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_USERFAULTFD_MINOR if USERFAULTFD select HAVE_ASM_MODVERSIONS + select HAVE_CMPXCHG_DOUBLE select HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK --2.49.0
Huacai Chen
maintainerRe: [PATCH v8 loongarch-next 0/3] LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoHi, George,
Quoted 30 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of three patches:
1. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), use a spinlock to emulate the atomic operation.
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization3. LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg supportThe issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.
Quoted 5 lines
---
Changes in v8:
- Merge patch 2 and patch 3 into one patch
- PutHAVE_CMPXCHG_DOUBLEin order
- Link to v7: https://lore.kernel.org/all/[email protected]/
I don't know why you make all versions in a single thread, and the version numbers of cover letters are always wrong.
For the code itself:
1. You said you have set hwcaps, but you completely ignore arch/loongarch/include/uapi/asm/hwcap.h, I don't know why.
2. You can simply do
#define system_has_cmpxchg128() (cpu_has_scq)
and don't need to define __cmpxchg128_locked(), which is the same as X86 and RISC-V.
Huacai
Quoted 57 lines
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]George Guo (3):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Enable 128-bit atomics cmpxchg supportarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)--
2.49.0
Quoted 18 lines · George Guo
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
The series consists of four patches:
1. "LoongArch: Add SCQ support detection"
- CheckCPUCFG2_SCQbit to determin if the CPU supports SCQ instrction.2. "LoongArch: Add 128-bit atomic cmpxchg support"
- Implements 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions
- Fixes BPF scheduler test failures (scx_centralscx_qmap) wherekmalloc_nolock_noprofreturns NULL due to missing 128-bit atomics, leading to-ENOMEMerrors during scheduler initialization
Quoted 5 lines
3. "LoongArch: Use spinlock to emulate 128-bit cmpxchg"
- For LoongArch CPUs lacking 128-bit atomicinstruction(e.g., the SCQ instruction on 3A5000), provide a fallback implementation of__cmpxchg128using a spinlock to emulate the atomic operation.
Probably, you can combine patch 2 and patch 3 into a single patch.
Quoted 68 lines
4. "LoongArch: Enable 128-bit atomics cmpxchg support"
- Adds selectHAVE_CMPXCHG_DOUBLEand selectHAVE_ALIGNED_STRUCT_PAGEinKconfigto enable 128-bit atomic cmpxchg supportThe issue was identified through BPF scheduler test failures where
scx_centralandscx_qmapschedulers would fail to initialize. Testing was performed using thescx_qmapscheduler fromtools/sched_ext/, confirming that the patches resolve the initialization failures.Signed-off-by George Guo <[email protected]>---
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]Changes in v3:
- dbar 0 ->__WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply__cmpxchg128_asm("ll.d", "sc.q", ptr, o, n)to__cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]---
George Guo (4):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Use spinlock to emulate 128-bit cmpxchg
LoongArch: Enable 128-bit atomics cmpxchg supportarch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 66 +++++++++++++++++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/asm/loongarch.h | 1 + arch/loongarch/kernel/cpu-probe.c | 2 + arch/loongarch/kernel/proc.c | 1 + 7 files changed, 75 insertions(+)---
base-commit:612df905d7404450696e979c806ba4cdef8684f4
change-id: 20251120-2-d03862b2cf6dBest regards,
--
George Guo <[email protected]>