LoongArch: Add 128-bit atomic cmpxchg support
v10 cover LoongArch: Add 128-bit atomic cmpxchg support
This patch series adds 128-bit atomic compare-and-exchange support for LoongArch architecture, which fixes BPF scheduler test failures caused by missing 128-bit atomics support.
---
Changes in v10:
- move "scq" to be placed after "lam"
- squash last patch
- move patch: "LoongArch: Replace seq_printf with seq_puts for simple strings" to the end
- Link to v9: https://lore.kernel.org/all/[email protected]/
---
Changes in v9:
- Add patch: "LoongArch: Replace seq_printf with seq_puts for simple strings"
- #define system_has_cmpxchg128() (cpu_has_scq)
- Delete __cmpxchg128_locked
- #define HWCAP_LOONGARCH_CPU_SCQ (1 << 15) in hwcap.h
- Link to v8: https://lore.kernel.org/all/[email protected]/
---
Changes in v8:
- Merge patch 2 and patch 3 into one patch
- Put HAVE_CMPXCHG_DOUBLE in order
- Link to v7: https://lore.kernel.org/all/[email protected]/
---
Changes in v7:
- Create patches based on loongarch-next branch(previously used master)
- Link to v6: https://lore.kernel.org/r/[email protected]
Changes in v6:
- Put SCQ information in hwcap
- Link to v5: https://lore.kernel.org/r/[email protected]
Changes in v5:
- Reordered the patches
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- Add SCQ support detection
- Add spinlock to emulate 128-bit cmpxchg
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- dbar 0 -> __WEAK_LLSC_MB
- =ZB" (__ptr[0]) -> "r" (__ptr)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Use a normal ld.d for the high word instead of ll.d to avoid race condition
- Insert a dbar between ll.d and ld.d to prevent reordering
- Simply __cmpxchg128_asm("ll.d", "sc.q", ptr, o, n) to __cmpxchg128_asm(ptr, o, n)
- Fix address operand constraints after testing different approaches:
* ld.d with "m"
* ll.d with "ZC",
* sc.q with "ZB"(alternative constraints caused issues:
- "r" caused system hang
- "ZC" caused compiler error:
{standard input}: Assembler messages:
{standard input}:10037: Fatal error: Immediate overflow.
format: u0:0 )
- Link to v1: https://lore.kernel.org/r/[email protected]
George Guo (3):
LoongArch: Add SCQ support detection
LoongArch: Add 128-bit atomic cmpxchg support
LoongArch: Replace seq_printf with seq_puts for simple strings
arch/loongarch/Kconfig | 2 + arch/loongarch/include/asm/cmpxchg.h | 48 +++++++++++++++++ arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 + arch/loongarch/include/uapi/asm/hwcap.h | 1 + arch/loongarch/kernel/cpu-probe.c | 4 ++ arch/loongarch/kernel/proc.c | 63 ++++++++++++++--------- 7 files changed, 98 insertions(+), 23 deletions(-)
Signature · 1 line
2.49.0
PATCH v10 1/3 LoongArch: Add SCQ support detection
Check CPUCFG2_SCQ bit to determine if the CPU supports SCQ instruction.
--- arch/loongarch/include/asm/cpu-features.h | 1 + arch/loongarch/include/asm/cpu.h | 2 ++ arch/loongarch/include/uapi/asm/hwcap.h | 1 + arch/loongarch/kernel/cpu-probe.c | 4 ++++ arch/loongarch/kernel/proc.c | 1 + 5 files changed, 9 insertions(+)
arch/loongarch/include/asm/cpu-features.h +1 -0
@@ -67,5 +67,6 @@#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT) +#define cpu_has_scq cpu_opt(LOONGARCH_CPU_SCQ)#endif /* __ASM_CPU_FEATURES_H */
arch/loongarch/include/asm/cpu.h +2 -0
@@ -125,6 +125,7 @@ static inline char *id_to_core_name(unsigned int id)#define CPU_FEATURE_MSGINT 29/* CPU has MSG interrupt */ #define CPU_FEATURE_AVECINT 30/* CPU has AVEC interrupt */ #define CPU_FEATURE_REDIRECTINT 31/* CPU has interrupt remapping */ +#define CPU_FEATURE_SCQ 32/* CPU has SC.Q instruction */ #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -158,5 +159,6 @@ static inline char *id_to_core_name(unsigned int id)#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT) +#define LOONGARCH_CPU_SCQ BIT_ULL(CPU_FEATURE_SCQ)#endif /* _ASM_CPU_H */
arch/loongarch/include/uapi/asm/hwcap.h +1 -0
@@ -18,5 +18,6 @@#define HWCAP_LOONGARCH_LBT_MIPS (1 << 12)#define HWCAP_LOONGARCH_PTW (1 << 13)#define HWCAP_LOONGARCH_LSPW (1 << 14) +#define HWCAP_LOONGARCH_CPU_SCQ (1 << 15)#endif /* _UAPI_ASM_HWCAP_H */
arch/loongarch/kernel/cpu-probe.c +4 -0
@@ -177,6 +177,10 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c) c->options |=LOONGARCH_CPU_LAM ; elf_hwcap |=HWCAP_LOONGARCH_LAM ; } +if (config &CPUCFG2_SCQ ) { + c->options |=LOONGARCH_CPU_SCQ ; + elf_hwcap |=HWCAP_LOONGARCH_CPU_SCQ ; + }if (config &CPUCFG2_FP ) { c->options |=LOONGARCH_CPU_FPU ; elf_hwcap |=HWCAP_LOONGARCH_FPU ;
arch/loongarch/kernel/proc.c +1 -0
@@ -62,6 +62,7 @@ static int show_cpuinfo(struct seq_file *m, void *v)seq_printf (m,"Features\t\t:" );if (cpu_has_cpucfg)seq_printf (m," cpucfg" );if (cpu_has_lam)seq_printf (m," lam" ); +if (cpu_has_scq)seq_printf (m," scq" );if (cpu_has_ual)seq_printf (m," ual" );if (cpu_has_fpu)seq_printf (m," fpu" );if (cpu_has_lsx)seq_printf (m," lsx" );
Signature · 1 line
2.49.0
PATCH v10 2/3 LoongArch: Add 128-bit atomic cmpxchg support
In reply to George GuoFrom: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (scx_central scx_qmap) caused by kmalloc_nolock_noprof returning NULL due to missing 128-bit atomics. The NULL returns led to -ENOMEM errors during scheduler initialization, causing test cases to fail.
Verified by testing with the scx_qmap scheduler (located in tools/sched_ext/). Building with make and running ./tools/sched_ext/build/bin/scx_qmap.
--- arch/loongarch/Kconfig | 2 ++ arch/loongarch/include/asm/cmpxchg.h | 48 ++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+)
arch/loongarch/Kconfig +2 -0
@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL
@@ -130,6 +131,7 @@ config LOONGARCH select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_USERFAULTFD_MINOR if USERFAULTFD select HAVE_ASM_MODVERSIONS + select HAVE_CMPXCHG_DOUBLE select HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK
arch/loongarch/include/asm/cmpxchg.h +48 -0
66 lines not on this page · raw message
Signature · 1 line
2.49.0
Huacai Chen
maintainerApplied and add arch_cmpxchg128_local(), you can check whether everthing is OK.
https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.git/commit/?h=loongarch-next&id=09a3171203974a30c6d0ab66f8ef0a6d9e7e780e
Huacai
Quoted 121 lines · George Guo
From: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (
scx_centralscx_qmap) caused bykmalloc_nolock_noprofreturning NULL due to missing 128-bit atomics. The NULL returns led to-ENOMEMerrors during scheduler initialization, causing test cases to fail.Verified by testing with the
scx_qmapscheduler (located intools/sched_ext/). Building withmakeand running./tools/sched_ext/build/bin/scx_qmap.Acked-by Hengqi Chen <[email protected]>Tested-by Hengqi Chen <[email protected]>Signed-off-by George Guo <[email protected]>--- arch/loongarch/Kconfig | 2 ++ arch/loongarch/include/asm/cmpxchg.h | 48 ++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+)arch/loongarch/Kconfig +2
-0@@ -114,6 +114,7 @@ config LOONGARCH select GENERIC_TIME_VSYSCALL select GPIOLIB select HAS_IOPORT + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_BITREVERSE select HAVE_ARCH_JUMP_LABEL@@ -130,6 +131,7 @@ config LOONGARCH select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_USERFAULTFD_MINOR if USERFAULTFD select HAVE_ASM_MODVERSIONS + select HAVE_CMPXCHG_DOUBLE select HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAKarch/loongarch/include/asm/cmpxchg.h +48
-1@@ -8,6 +8,7 @@#include <linux/bits.h> #include <linux/build_bug.h> #include <asm/barrier.h> +#include <asm/cpu-features.h> #define __xchg_amo_asm (amswap_db, m, val) \ ({ \@@ -137,6 +138,44 @@ __arch_xchg(volatile void *ptr, unsigned long x, int size) __ret; \ }) +union __u128_halves { +u128 full; +struct { +u64 low; +u64 high; + }; +}; + +#define __arch_cmpxchg128 (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \ + __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +}) +static inline unsigned int __cmpxchg_small (volatile void *ptr,unsigned int old,unsigned int new,unsigned int size) {@@ -224,6 +263,15 @@ __cmpxchg(volatile void *ptr, unsigned long old, unsigned long new, unsigned int __res; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128 () (cpu_has_scq) + +#define arch_cmpxchg128 (ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 16); \ + __arch_cmpxchg128(ptr, o, n); \ +}) +#ifdef CONFIG_64BIT #define arch_cmpxchg64_local (ptr, o, n) \ ({ \ --2.49.0
Hi!
Quoted 18 lines · George Guo
From: George Guo <[email protected]>
Implement 128-bit atomic compare-and-exchange using LoongArch's LL.D/SC.Q instructions.
At the same time, fix BPF scheduler test failures (
scx_centralscx_qmap) caused bykmalloc_nolock_noprofreturning NULL due to missing 128-bit atomics. The NULL returns led to-ENOMEMerrors during scheduler initialization, causing test cases to fail.Verified by testing with the
scx_qmapscheduler (located intools/sched_ext/). Building withmakeand running./tools/sched_ext/build/bin/scx_qmap.
(...)
Quoted 19 lines
arch/loongarch/include/asm/cmpxchg.h +19
-0+#define __arch_cmpxchg128 (ptr, old, new) \ +({ \ + union __u128_halves __old, __new, __ret; \ + volatile u64 *__ptr = (volatile u64 *)(ptr); \ + \ + __old.full = (old); \ + __new.full = (new); \ + \ + __asm__ __volatile__( \ + "1: ll.d %0, %3 # 128-bit cmpxchg low \n" \ + __WEAK_LLSC_MB \ + " ld.d %1, %4 # 128-bit cmpxchg high \n" \ + " bne %0, %z5, 2f \n" \ + " bne %1, %z6, 2f \n" \ + " move $t0, %z7 \n" \ + " move $t1, %z8 \n" \ + " sc.q $t0, $t1, %2 \n" \ + " beqz $t0, 1b \n" \ + "2: \n" \
arch/loongarch/include/asm/cmpxchg.h __arch_xchg(volatile void *ptr, unsigned long x, int size) · line 177
Quoted 10 lines
arch/loongarch/include/asm/cmpxchg.h +10
-0+ __WEAK_LLSC_MB \ + : "=&r" (__ret.low), "=&r" (__ret.high) \ + : "r" (__ptr), \ + "ZC" (__ptr[0]), "m" (__ptr[1]), \ + "Jr" (__old.low), "Jr" (__old.high), \ + "Jr" (__new.low), "Jr" (__new.high) \ + : "t0", "t1", "memory"); \ + \ + __ret.full; \ +})
Older versions of binutils[0] seem to lack support for the sc.q instruction. So there should be some validation of the toolchain capabilities.
ERROR:root:{standard input}: Assembler messages:
{standard input}:4831: Error: no match insn: sc.q $t0,$t1,$r14
{standard input}:6407: Error: no match insn: sc.q $t0,$t1,$r23
{standard input}:10856: Error: no match insn: sc.q $t0,$t1,$r14
make[4]: *** [../scripts/Makefile.build:289: mm/slub.o] Error 1
[0] Tested with binutils 2.41 from the GCC 13.2 kernel.org toolchain.
Thomas
PATCH v10 3/3 LoongArch: Replace seq_printf with seq_puts for simple strings
In reply to George GuoFix warnings like: "Prefer seq_puts to seq_printf" by checkpatch.pl.
Replace seq_printf() calls with seq_puts() in show_cpuinfo() when outputting simple constant strings without format specifiers.
This improves performance slightly as seq_puts() avoids parsing the format string.
--- arch/loongarch/kernel/proc.c | 64 ++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 24 deletions(-)
arch/loongarch/kernel/proc.c +40 -24
@@ -50,33 +50,49 @@ static int show_cpuinfo(struct seq_file *m, void *v)seq_printf (m,"Address Sizes\t\t: %d bits physical, %d bits virtual\n" , cpu_pabits +1 , cpu_vabits +1 ); -seq_printf (m,"ISA\t\t\t:" ); +seq_puts (m,"ISA\t\t\t:" );if (isa &LOONGARCH_CPU_ISA_LA32R ) -seq_printf (m," loongarch32r" ); +seq_puts (m," loongarch32r" );if (isa &LOONGARCH_CPU_ISA_LA32S ) -seq_printf (m," loongarch32s" ); +seq_puts (m," loongarch32s" );if (isa &LOONGARCH_CPU_ISA_LA64 ) -seq_printf (m," loongarch64" ); -seq_printf (m,"\n" ); +seq_puts (m," loongarch64" ); +seq_puts (m,"\n" ); -seq_printf (m,"Features\t\t:" ); -if (cpu_has_cpucfg)seq_printf (m," cpucfg" ); -if (cpu_has_lam)seq_printf (m," lam" ); -if (cpu_has_scq)seq_printf (m," scq" ); -if (cpu_has_ual)seq_printf (m," ual" ); -if (cpu_has_fpu)seq_printf (m," fpu" ); -if (cpu_has_lsx)seq_printf (m," lsx" ); -if (cpu_has_lasx)seq_printf (m," lasx" ); -if (cpu_has_crc32)seq_printf (m," crc32" ); -if (cpu_has_complex)seq_printf (m," complex" ); -if (cpu_has_crypto)seq_printf (m," crypto" ); -if (cpu_has_ptw)seq_printf (m," ptw" ); -if (cpu_has_lspw)seq_printf (m," lspw" ); -if (cpu_has_lvz)seq_printf (m," lvz" ); -if (cpu_has_lbt_x86)seq_printf (m," lbt_x86" ); -if (cpu_has_lbt_arm)seq_printf (m," lbt_arm" ); -if (cpu_has_lbt_mips)seq_printf (m," lbt_mips" ); -seq_printf (m,"\n" ); +seq_puts (m,"Features\t\t:" ); +if (cpu_has_cpucfg) +seq_puts (m," cpucfg" ); +if (cpu_has_lam) +seq_puts (m," lam" ); +if (cpu_has_scq) +seq_puts (m," scq" ); +if (cpu_has_ual) +seq_puts (m," ual" ); +if (cpu_has_fpu) +seq_puts (m," fpu" ); +if (cpu_has_lsx) +seq_puts (m," lsx" ); +if (cpu_has_lasx) +seq_puts (m," lasx" ); +if (cpu_has_crc32) +seq_puts (m," crc32" ); +if (cpu_has_complex) +seq_puts (m," complex" ); +if (cpu_has_crypto) +seq_puts (m," crypto" ); +if (cpu_has_ptw) +seq_puts (m," ptw" ); +if (cpu_has_lspw) +seq_puts (m," lspw" ); +if (cpu_has_lvz) +seq_puts (m," lvz" ); +if (cpu_has_lbt_x86) +seq_puts (m," lbt_x86" ); +if (cpu_has_lbt_arm) +seq_puts (m," lbt_arm" ); +if (cpu_has_lbt_mips) +seq_puts (m," lbt_mips" ); +seq_puts (m,"\n" );seq_printf (m,"Hardware Watchpoint\t: %s" ,str_yes_no (cpu_has_watch));if (cpu_has_watch) {
@@ -84,7 +100,7 @@ static int show_cpuinfo(struct seq_file *m, void *v) cpu_data[n].watch_ireg_count, cpu_data[n].watch_dreg_count); } -seq_printf (m,"\n\n" ); +seq_puts (m,"\n\n" );return 0 ; }
Signature · 1 line
2.49.0
Re: [PATCH v10 loongarch-next 3/3] LoongArch: Replace seq_printf with seq_puts for simple strings
Quoted 7 lines · George Guo
Fix warnings like: "Prefer
seq_putstoseq_printf" by checkpatch.pl.Replace
seq_printf()calls withseq_puts()inshow_cpuinfo()when outputting simple constant strings without format specifiers.This improves performance slightly as
seq_puts()avoids parsing the format string.
[]
arch/loongarch/kernel/proc.c
Quoted 1 line
arch/loongarch/kernel/proc.c +0
-0
[]
arch/loongarch/kernel/proc.c static int show_cpuinfo(struct seq_file *m, void *v)
Quoted 1 line
arch/loongarch/kernel/proc.c +0
-0@@ -50,33 +50,49 @@ static int show_cpuinfo(struct seq_file *m, void *v)
[]
arch/loongarch/kernel/proc.c static int show_cpuinfo(struct seq_file *m, void *v) · line 64
Quoted 3 lines
arch/loongarch/kernel/proc.c +0
-3-seq_printf (m,"Features\t\t:" ); -if (cpu_has_cpucfg)seq_printf (m," cpucfg" ); -if (cpu_has_lam)seq_printf (m," lam" );
[etc]
arch/loongarch/kernel/proc.c static int show_cpuinfo(struct seq_file *m, void *v) · line 66
Quoted 5 lines
arch/loongarch/kernel/proc.c +5
-0+seq_puts (m,"Features\t\t:" ); +if (cpu_has_cpucfg) +seq_puts (m," cpucfg" ); +if (cpu_has_lam) +seq_puts (m," lam" );
trivia:
Not sure this is better style as it's fairly difficult to read.
Maybe a macro might help, something like:
#define seq_cpu_feature(m, feature) \ if (cpu_has_##feature) seq_puts(m, " " #feature) seq_cpu_feature(m, cpucfg); seq_cpu_feature(m, lam);
etc.
Huacai Chen
maintainerRe: [PATCH v10 loongarch-next 3/3] LoongArch: Replace seq_printf with seq_puts for simple strings
Hi, Joe,
Quoted 28 lines · Joe Perches
Quoted 7 lines · George Guo
Fix warnings like: "Prefer
seq_putstoseq_printf" by checkpatch.pl.Replace
seq_printf()calls withseq_puts()inshow_cpuinfo()when outputting simple constant strings without format specifiers.This improves performance slightly as
seq_puts()avoids parsing the format string.[]
arch/loongarch/kernel/proc.c
Quoted 1 line
arch/loongarch/kernel/proc.c +0
-0[]
arch/loongarch/kernel/proc.c static int show_cpuinfo(struct seq_file *m, void *v)
Quoted 1 line
arch/loongarch/kernel/proc.c +0
-0@@ -50,33 +50,49 @@ static int show_cpuinfo(struct seq_file *m, void *v)[]
arch/loongarch/kernel/proc.c static int show_cpuinfo(struct seq_file *m, void *v) · line 64
Quoted 3 lines
arch/loongarch/kernel/proc.c +0
-3-seq_printf (m,"Features\t\t:" ); -if (cpu_has_cpucfg)seq_printf (m," cpucfg" ); -if (cpu_has_lam)seq_printf (m," lam" );[etc]
arch/loongarch/kernel/proc.c static int show_cpuinfo(struct seq_file *m, void *v) · line 66
Quoted 5 lines
arch/loongarch/kernel/proc.c +5
-0+seq_puts (m,"Features\t\t:" ); +if (cpu_has_cpucfg) +seq_puts (m," cpucfg" ); +if (cpu_has_lam) +seq_puts (m," lam" );trivia:
Not sure this is better style as it's fairly difficult to read.
Quoted 7 lines
Maybe a macro might help, something like:
#define seq_cpu_feature(m, feature) \ if (cpu_has_##feature) seq_puts(m, " " #feature) seq_cpu_feature(m, cpucfg); seq_cpu_feature(m, lam);
Seems just some bikesheedings, the current style is OK at least for me. Since George has worked on this series for a very long time, let's stop changing and get it merged.
Huacai
Quoted 3 lines
etc.