riscv: Add Zalasr ISA extension support
Subsystems (7)
v3 cover riscv: Add Zalasr ISA extension support
This patch adds support for the Zalasr ISA extension, which supplies the real load acquire/store release instructions.
The specification can be found here:
https://github.com/riscv/riscv-zalasr/blob/main/chapter2.adoc
This patch seires has been tested with ltp on Qemu with Brensan's zalasr support patch[1].
Some false positive spacing error happens during patch checking. Thus I CCed maintainers of checkpatch.pl as well.
[1] https://lore.kernel.org/all/CAGPSXwJEdtqW=nx71oufZp64nK6tK=0rytVEcz4F-gfvCOXk2w@mail.gmail.com/
v3:
- Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations so as to ensure FENCE.TSO ordering between operations which precede the UNLOCK+LOCK sequence and operations which follow the sequence. Thanks to Andrea.
- Support hwprobe of Zalasr.
- Allow Zalasr extensions for Guest/VM.
v2:
- Adjust the order of Zalasr and Zalrsc in dt-bindings. Thanks to Conor.
Xu Lu (8):
riscv: add ISA extension parsing for Zalasr
dt-bindings: riscv: Add Zalasr ISA extension description
riscv: hwprobe: Export Zalasr extension
riscv: Introduce Zalasr instructions
riscv: Use Zalasr for smp_load_acquire/smp_store_release
riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg
operations
RISC-V: KVM: Allow Zalasr extensions for Guest/VM
KVM: riscv: selftests: Add Zalasr extensions to get-reg-list test
Documentation/arch/riscv/hwprobe.rst | 5 +- .../devicetree/bindings/riscv/extensions.yaml | 5 + arch/riscv/include/asm/atomic.h | 6 - arch/riscv/include/asm/barrier.h | 91 ++++++++++-- arch/riscv/include/asm/cmpxchg.h | 136 ++++++++---------- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 79 ++++++++++ arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + arch/riscv/kvm/vcpu_onereg.c | 2 + .../selftests/kvm/riscv/get-reg-list.c | 4 + 13 files changed, 242 insertions(+), 91 deletions(-)
Signature · 1 line
2.20.1
PATCH v3 1/8 riscv: add ISA extension parsing for Zalasr
Add parsing for Zalasr ISA extension.
--- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+)
arch/riscv/include/asm/hwcap.h +1 -0
@@ -106,6 +106,7 @@#define RISCV_ISA_EXT_ZAAMO 97#define RISCV_ISA_EXT_ZALRSC 98#define RISCV_ISA_EXT_ZICBOP 99 +#define RISCV_ISA_EXT_ZALASR 100#define RISCV_ISA_EXT_XLINUXENVCFG 127
arch/riscv/kernel/cpufeature.c +1 -0
@@ -472,6 +472,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = {__RISCV_ISA_EXT_DATA (zaamo,RISCV_ISA_EXT_ZAAMO ),__RISCV_ISA_EXT_DATA (zabha,RISCV_ISA_EXT_ZABHA ),__RISCV_ISA_EXT_DATA (zacas,RISCV_ISA_EXT_ZACAS ), +__RISCV_ISA_EXT_DATA (zalasr,RISCV_ISA_EXT_ZALASR ),__RISCV_ISA_EXT_DATA (zalrsc,RISCV_ISA_EXT_ZALRSC ),__RISCV_ISA_EXT_DATA (zawrs,RISCV_ISA_EXT_ZAWRS ), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA),
Signature · 1 line
2.20.1
PATCH v3 2/8 dt-bindings: riscv: Add Zalasr ISA extension description
In reply to Xu LuAdd description for the Zalasr ISA extension
--- Documentation/devicetree/bindings/riscv/extensions.yaml | 5 +++++ 1 file changed, 5 insertions(+)
Documentation/devicetree/bindings/riscv/extensions.yaml +5 -0
@@ -242,6 +242,11 @@ properties: is supported as ratified at commit 5059e0ca641c ("update to ratified") of the riscv-zacas. + - const: zalasr + description: | + The standard Zalasr extension for load-acquire/store-release as frozen + at commit 194f0094 ("Version 0.9 for freeze") of riscv-zalasr. + - const: zalrsc description: | The standard Zalrsc extension for load-reserved/store-conditional as
Signature · 1 line
2.20.1
PATCH v3 3/8 riscv: hwprobe: Export Zalasr extension
In reply to Xu LuExport the Zalasr extension to userspace using hwprobe.
--- Documentation/arch/riscv/hwprobe.rst | 5 ++++- arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + 3 files changed, 6 insertions(+), 1 deletion(-)
Documentation/arch/riscv/hwprobe.rst +4 -1
@@ -249,6 +249,9 @@ The following keys are defined: defined in the in the RISC-V ISA manual starting from commit e87412e621f1 ("integrate Zaamo and Zalrsc text (#1304)"). + * :c:macro:`RISCV_HWPROBE_EXT_ZALASR`: The Zalasr extension is supported as + frozen at commit 194f0094 ("Version 0.9 for freeze") of riscv-zalasr. + * :c:macro:`RISCV_HWPROBE_EXT_ZALRSC`: The Zalrsc extension is supported as defined in the in the RISC-V ISA manual starting from commit e87412e621f1 ("integrate Zaamo and Zalrsc text (#1304)").
@@ -360,4 +363,4 @@ The following keys are defined: * :c:macro:`RISCV_HWPROBE_VENDOR_EXT_XSFVFWMACCQQQ`: The Xsfvfwmaccqqq vendor extension is supported in version 1.0 of Matrix Multiply Accumulate - Instruction Extensions Specification. + Instruction Extensions Specification.
arch/riscv/include/uapi/asm/hwprobe.h +1 -0
@@ -82,6 +82,7 @@ struct riscv_hwprobe {#define RISCV_HWPROBE_EXT_ZAAMO (1ULL << 56)#define RISCV_HWPROBE_EXT_ZALRSC (1ULL << 57)#define RISCV_HWPROBE_EXT_ZABHA (1ULL << 58) +#define RISCV_HWPROBE_EXT_ZALASR (1ULL << 59)#define RISCV_HWPROBE_KEY_CPUPERF_0 5#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0)#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0)
arch/riscv/kernel/sys_hwprobe.c +1 -0
@@ -99,6 +99,7 @@ static void hwprobe_isa_ext0(struct riscv_hwprobe *pair,EXT_KEY (ZAAMO );EXT_KEY (ZABHA );EXT_KEY (ZACAS ); +EXT_KEY (ZALASR );EXT_KEY (ZALRSC );EXT_KEY (ZAWRS );EXT_KEY (ZBA );
Signature · 1 line
2.20.1
PATCH v3 4/8 riscv: Introduce Zalasr instructions
In reply to Xu LuIntroduce l{b|h|w|d}.{aq|aqrl} and s{b|h|w|d}.{rl|aqrl} instruction encodings.
--- arch/riscv/include/asm/insn-def.h | 79 +++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+)
arch/riscv/include/asm/insn-def.h +79 -0
@@ -179,6 +179,7 @@#define RV___RS1 (v) __RV_REG(v)#define RV___RS2 (v) __RV_REG(v) +#define RV_OPCODE_AMO RV_OPCODE(47)#define RV_OPCODE_MISC_MEM RV_OPCODE(15)#define RV_OPCODE_OP_IMM RV_OPCODE(19)#define RV_OPCODE_SYSTEM RV_OPCODE(115)
@@ -208,6 +209,84 @@ __ASM_STR(.error "hlv.d requires 64-bit support")#endif +#define LB_AQ (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(0), FUNC7(26), \ + RD(dest), RS1(addr), __RS2(0)) + +#define LB_AQRL (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(0), FUNC7(27), \ + RD(dest), RS1(addr), __RS2(0)) + +#define LH_AQ (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(1), FUNC7(26), \ + RD(dest), RS1(addr), __RS2(0)) + +#define LH_AQRL (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(1), FUNC7(27), \ + RD(dest), RS1(addr), __RS2(0)) + +#define LW_AQ (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(2), FUNC7(26), \ + RD(dest), RS1(addr), __RS2(0)) + +#define LW_AQRL (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(2), FUNC7(27), \ + RD(dest), RS1(addr), __RS2(0)) + +#define SB_RL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(0), FUNC7(29), \ + __RD(0), RS1(addr), RS2(src)) + +#define SB_AQRL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(0), FUNC7(31), \ + __RD(0), RS1(addr), RS2(src)) + +#define SH_RL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(1), FUNC7(29), \ + __RD(0), RS1(addr), RS2(src)) + +#define SH_AQRL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(1), FUNC7(31), \ + __RD(0), RS1(addr), RS2(src)) + +#define SW_RL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(2), FUNC7(29), \ + __RD(0), RS1(addr), RS2(src)) + +#define SW_AQRL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(2), FUNC7(31), \ + __RD(0), RS1(addr), RS2(src)) + +#ifdef CONFIG_64BIT +#define LD_AQ (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(3), FUNC7(26), \ + RD(dest), RS1(addr), __RS2(0)) + +#define LD_AQRL (dest, addr) \ + INSN_R(OPCODE_AMO, FUNC3(3), FUNC7(27), \ + RD(dest), RS1(addr), __RS2(0)) + +#define SD_RL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(3), FUNC7(29), \ + __RD(0), RS1(addr), RS2(src)) + +#define SD_AQRL (src, addr) \ + INSN_R(OPCODE_AMO, FUNC3(3), FUNC7(31), \ + __RD(0), RS1(addr), RS2(src)) +#else +#define LD_AQ (dest, addr) \ + __ASM_STR(.error "ld.aq requires 64-bit support") + +#define LD_AQRL (dest, addr) \ + __ASM_STR(.error "ld.aqrl requires 64-bit support") + +#define SD_RL (dest, addr) \ + __ASM_STR(.error "sd.rl requires 64-bit support") + +#define SD_AQRL (dest, addr) \ + __ASM_STR(.error "sd.aqrl requires 64-bit support") +#endif +#define SINVAL_VMA (vaddr, asid) \ INSN_R(OPCODE_SYSTEM, FUNC3(0), FUNC7(11), \ __RD(0), RS1(vaddr), RS2(asid))
Signature · 1 line
2.20.1
PATCH v3 5/8 riscv: Use Zalasr for smp_load_acquire/smp_store_release
In reply to Xu LuReplace fence instructions with Zalasr instructions during smp_load_acquire() and smp_store_release() operations.
|----------------------------------|
| | __smp_store_release |
| |-----------------------------|
| | zalasr | !zalasr |
| rl |-----------------------------|
| | s{b|h|w|d}.rl | fence rw, w |
| | | s{b|h|w|d} |
|----------------------------------|
| | __smp_load_acquire |
| |-----------------------------|
| | zalasr | !zalasr |
| aq |-----------------------------|
| | l{b|h|w|d}.rl | l{b|h|w|d} |
| | | fence r, rw |
|----------------------------------|--- arch/riscv/include/asm/barrier.h | 91 ++++++++++++++++++++++++++++---- 1 file changed, 80 insertions(+), 11 deletions(-)
arch/riscv/include/asm/barrier.h +80 -11
@@ -51,19 +51,88 @@*/ #define smp_mb__after_spinlock () RISCV_FENCE(iorw, iorw) -#define __smp_store_release (p, v) \ -do { \ - compiletime_assert_atomic_type(*p); \ - RISCV_FENCE(rw, w); \ - WRITE_ONCE(*p, v); \ +extern void __bad_size_call_parameter (void ); + +#define __smp_store_release (p, v) \ +do { \ + typeof(p) __p = (p); \ + union { typeof(*p) __val; char __c[1]; } __u = \ + { .__val = (__force typeof(*p)) (v) }; \ + compiletime_assert_atomic_type(*p); \ + switch (sizeof(*p)) { \ + case 1: \ + asm volatile(ALTERNATIVE("fence rw, w;\t\nsb %0, 0(%1)\t\n", \ + SB_RL(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : : "r" (*(__u8 *)__u.__c), "r" (__p) \ + : "memory"); \ + break; \ + case 2: \ + asm volatile(ALTERNATIVE("fence rw, w;\t\nsh %0, 0(%1)\t\n", \ + SH_RL(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : : "r" (*(__u16 *)__u.__c), "r" (__p) \ + : "memory"); \ + break; \ + case 4: \ + asm volatile(ALTERNATIVE("fence rw, w;\t\nsw %0, 0(%1)\t\n", \ + SW_RL(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : : "r" (*(__u32 *)__u.__c), "r" (__p) \ + : "memory"); \ + break; \ + case 8: \ + asm volatile(ALTERNATIVE("fence rw, w;\t\nsd %0, 0(%1)\t\n", \ + SD_RL(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : : "r" (*(__u64 *)__u.__c), "r" (__p) \ + : "memory"); \ + break; \ + default: \ + __bad_size_call_parameter(); \ + break; \ + } \ } while (0) -#define __smp_load_acquire (p) \ -({ \ - typeof(*p) ___p1 = READ_ONCE(*p); \ - compiletime_assert_atomic_type(*p); \ - RISCV_FENCE(r, rw); \ - ___p1; \ +#define __smp_load_acquire (p) \ +({ \ + union { typeof(*p) __val; char __c[1]; } __u; \ + typeof(p) __p = (p); \ + compiletime_assert_atomic_type(*p); \ + switch (sizeof(*p)) { \ + case 1: \ + asm volatile(ALTERNATIVE("lb %0, 0(%1)\t\nfence r, rw\t\n", \ + LB_AQ(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : "=r" (*(__u8 *)__u.__c) : "r" (__p) \ + : "memory"); \ + break; \ + case 2: \ + asm volatile(ALTERNATIVE("lh %0, 0(%1)\t\nfence r, rw\t\n", \ + LH_AQ(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : "=r" (*(__u16 *)__u.__c) : "r" (__p) \ + : "memory"); \ + break; \ + case 4: \ + asm volatile(ALTERNATIVE("lw %0, 0(%1)\t\nfence r, rw\t\n", \ + LW_AQ(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : "=r" (*(__u32 *)__u.__c) : "r" (__p) \ + : "memory"); \ + break; \ + case 8: \ + asm volatile(ALTERNATIVE("ld %0, 0(%1)\t\nfence r, rw\t\n", \ + LD_AQ(%0, %1) "\t\nnop\t\n", \ + 0, RISCV_ISA_EXT_ZALASR, 1) \ + : "=r" (*(__u64 *)__u.__c) : "r" (__p) \ + : "memory"); \ + break; \ + default: \ + __bad_size_call_parameter(); \ + break; \ + } \ + __u.__val; \ })#ifdef CONFIG_RISCV_ISA_ZAWRS
Signature · 1 line
2.20.1
PATCH v3 6/8 riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations
In reply to Xu LuThe existing arch_xchg/arch_cmpxchg operations are implemented by inserting fence instructions before or after atomic instructions. This commit replaces them with real acquire/release semantics.
|----------------------------------------------------------------| | | arch_xchg_release | arch_cmpxchg_release | | |-----------------------------------------------------------| | | zabha | !zabha | zabha+zacas | !(zabha+zacas) | | rl |-----------------------------------------------------------| | | | (fence rw, w) | | (fence rw, w) | | | amoswap.rl | lr.w | amocas.rl | lr.w | | | | sc.w.rl | | sc.w.rl | |----------------------------------------------------------------| | | arch_xchg_acquire | arch_cmpxchg_acquire | | |-----------------------------------------------------------| | | zabha | !zabha | zabha+zacas | !(zabha+zacas) | | aq |-----------------------------------------------------------| | | | lr.w.aq | | lr.w.aq | | | amoswap.aq | sc.w | amocas.aq | sc.w | | | | (fence r, rw) | | (fence r, rw) | |----------------------------------------------------------------|
(fence rw, w), (fence r, rw) here means such instructions will only be inserted when zalasr is not implemented.
--- arch/riscv/include/asm/atomic.h | 6 -- arch/riscv/include/asm/cmpxchg.h | 136 ++++++++++++++----------------- 2 files changed, 63 insertions(+), 79 deletions(-)
arch/riscv/include/asm/atomic.h +0 -6
@@ -18,12 +18,6 @@#include <asm/cmpxchg.h> -#define __atomic_acquire_fence () \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") - -#define __atomic_release_fence () \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); - static __always_inline int arch_atomic_read(const atomic_t *v) {return READ_ONCE (v->counter);
arch/riscv/include/asm/cmpxchg.h +63 -73
289 lines not on this page · raw message
Signature · 1 line
2.20.1
kernel test robot
build botRe: [PATCH v3 6/8] riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations
kernel test robot noticed the following build errors:
Hi Xu,
kernel test robot noticed the following build errors:
[auto build test ERROR on robh/for-next]
[also build test ERROR on kvm/queue kvm/next linus/master v6.17-rc6] [cannot apply to kvm/linux-next]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Xu-Lu/riscv-add-ISA-extension-parsing-for-Zalasr/20250919-154304 base: https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git for-next patch link: https://lore.kernel.org/r/20250919073714.83063-7-luxu.kernel%40bytedance.com patch subject: [PATCH v3 6/8] riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations config: riscv-randconfig-002-20250920 (https://download.01.org/0day-ci/archive/20250920/[email protected]/config) compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 7c861bcedf61607b6c087380ac711eb7ff918ca6) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250920/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from lib/objpool.c:3:
>> include/linux/objpool.h:156:7: error: invalid .org offset '1528' (at offset '1532')
156 | if (try_cmpxchg_release(&slot->head, &head, head + 1))
| ^
include/linux/atomic/atomic-instrumented.h:4899:2: note: expanded from macro 'try_cmpxchg_release'
4899 | raw_try_cmpxchg_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:228:9: note: expanded from macro 'raw_try_cmpxchg_release'
228 | ___r = raw_cmpxchg_release((_ptr), ___o, (_new)); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
In file included from lib/objpool.c:3:
>> include/linux/objpool.h:156:7: error: invalid .org offset '1528' (at offset '1532')
156 | if (try_cmpxchg_release(&slot->head, &head, head + 1))
| ^
include/linux/atomic/atomic-instrumented.h:4899:2: note: expanded from macro 'try_cmpxchg_release'
4899 | raw_try_cmpxchg_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:228:9: note: expanded from macro 'raw_try_cmpxchg_release'
228 | ___r = raw_cmpxchg_release((_ptr), ___o, (_new)); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
note: (skipping 4 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
2 errors generated.
--
>> lib/generic-radix-tree.c:53:12: error: invalid .org offset '1850' (at offset '1854')
53 | if ((v = cmpxchg_release(&radix->root, r, new_root)) == r) {
| ^
include/linux/atomic/atomic-instrumented.h:4803:2: note: expanded from macro 'cmpxchg_release'
4803 | raw_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
lib/generic-radix-tree.c:74:14: error: invalid .org offset '1862' (at offset '1866')
74 | if (!(n = cmpxchg_release(p, NULL, new_node)))
| ^
include/linux/atomic/atomic-instrumented.h:4803:2: note: expanded from macro 'cmpxchg_release'
4803 | raw_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
>> lib/generic-radix-tree.c:53:12: error: invalid .org offset '1850' (at offset '1854')
53 | if ((v = cmpxchg_release(&radix->root, r, new_root)) == r) {
| ^
include/linux/atomic/atomic-instrumented.h:4803:2: note: expanded from macro 'cmpxchg_release'
4803 | raw_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
lib/generic-radix-tree.c:74:14: error: invalid .org offset '1862' (at offset '1866')
74 | if (!(n = cmpxchg_release(p, NULL, new_node)))
| ^
include/linux/atomic/atomic-instrumented.h:4803:2: note: expanded from macro 'cmpxchg_release'
4803 | raw_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^ 4 errors generated.
--
In file included from lib/refcount.c:6:
In file included from include/linux/mutex.h:17:
In file included from include/linux/lockdep.h:14:
In file included from include/linux/smp.h:13:
In file included from include/linux/cpumask.h:14:
In file included from include/linux/atomic.h:80:
>> include/linux/atomic/atomic-arch-fallback.h:2083:9: error: invalid .org offset '528' (at offset '532')
2083 | return raw_cmpxchg_release(&v->counter, old, new);
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
arch/riscv/include/asm/cmpxchg.h:238:3: note: expanded from macro '_arch_cmpxchg'
238 | __arch_cmpxchg(".w" lr_sfx, ".w" sc_sfx, ".w" cas_sfx, \
| ^
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
In file included from lib/refcount.c:6:
In file included from include/linux/mutex.h:17:
In file included from include/linux/lockdep.h:14:
In file included from include/linux/smp.h:13:
In file included from include/linux/cpumask.h:14:
In file included from include/linux/atomic.h:80:
include/linux/atomic/atomic-arch-fallback.h:2083:9: error: invalid .org offset '540' (at offset '544')
2083 | return raw_cmpxchg_release(&v->counter, old, new);
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
arch/riscv/include/asm/cmpxchg.h:238:3: note: expanded from macro '_arch_cmpxchg'
238 | __arch_cmpxchg(".w" lr_sfx, ".w" sc_sfx, ".w" cas_sfx, \
| ^
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
In file included from lib/refcount.c:6:
In file included from include/linux/mutex.h:17:
In file included from include/linux/lockdep.h:14:
In file included from include/linux/smp.h:13:
In file included from include/linux/cpumask.h:14:
In file included from include/linux/atomic.h:80:
>> include/linux/atomic/atomic-arch-fallback.h:2083:9: error: invalid .org offset '528' (at offset '532')
2083 | return raw_cmpxchg_release(&v->counter, old, new);
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
arch/riscv/include/asm/cmpxchg.h:238:3: note: expanded from macro '_arch_cmpxchg'
238 | __arch_cmpxchg(".w" lr_sfx, ".w" sc_sfx, ".w" cas_sfx, \
| ^
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
In file included from lib/refcount.c:6:
In file included from include/linux/mutex.h:17:
In file included from include/linux/lockdep.h:14:
In file included from include/linux/smp.h:13:
In file included from include/linux/cpumask.h:14:
In file included from include/linux/atomic.h:80:
include/linux/atomic/atomic-arch-fallback.h:2083:9: error: invalid .org offset '540' (at offset '544')
2083 | return raw_cmpxchg_release(&v->counter, old, new);
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
arch/riscv/include/asm/cmpxchg.h:238:3: note: expanded from macro '_arch_cmpxchg'
238 | __arch_cmpxchg(".w" lr_sfx, ".w" sc_sfx, ".w" cas_sfx, \
| ^
note: (skipping 2 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
4 errors generated.
--
>> fs/overlayfs/file.c:147:10: error: invalid .org offset '3640' (at offset '3644')
147 | old = cmpxchg_release(&of->upperfile, NULL, upperfile);
| ^
include/linux/atomic/atomic-instrumented.h:4803:2: note: expanded from macro 'cmpxchg_release'
4803 | raw_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^
>> fs/overlayfs/file.c:147:10: error: invalid .org offset '3640' (at offset '3644')
147 | old = cmpxchg_release(&of->upperfile, NULL, upperfile);
| ^
include/linux/atomic/atomic-instrumented.h:4803:2: note: expanded from macro 'cmpxchg_release'
4803 | raw_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
| ^
include/linux/atomic/atomic-arch-fallback.h:77:29: note: expanded from macro 'raw_cmpxchg_release'
77 | #define raw_cmpxchg_release arch_cmpxchg_release
| ^
arch/riscv/include/asm/cmpxchg.h:265:2: note: expanded from macro 'arch_cmpxchg_release'
265 | _arch_cmpxchg((ptr), (o), (n), \
| ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
arch/riscv/include/asm/alternative-macros.h:104:2: note: expanded from macro '_ALTERNATIVE_CFG'
104 | __ALTERNATIVE_CFG(old_c, new_c, vendor_id, patch_id, IS_ENABLED(CONFIG_k))
| ^
arch/riscv/include/asm/alternative-macros.h:94:2: note: expanded from macro '__ALTERNATIVE_CFG'
94 | ALT_NEW_CONTENT(vendor_id, patch_id, enable, new_c)
| ^
arch/riscv/include/asm/alternative-macros.h:81:3: note: expanded from macro 'ALT_NEW_CONTENT'
81 | ".org . - (887b - 886b) + (889b - 888b)\n" \
| ^
<inline asm>:27:6: note: instantiated into assembly here
27 | .org . - (887b - 886b) + (889b - 888b)
| ^ 2 errors generated.
..
vim +156 include/linux/objpool.h
b4edb8d2d4647a wuqiang.matt 2023-10-17 100
b4edb8d2d4647a wuqiang.matt 2023-10-17 101 /**
b4edb8d2d4647a wuqiang.matt 2023-10-17 102 * objpool_init() - initialize objpool and pre-allocated objects
b4edb8d2d4647a wuqiang.matt 2023-10-17 103 * @pool: the object pool to be initialized, declared by caller
b4edb8d2d4647a wuqiang.matt 2023-10-17 104 * @nr_objs: total objects to be pre-allocated by this object pool
b4edb8d2d4647a wuqiang.matt 2023-10-17 105 * @object_size: size of an object (should be > 0)
b4edb8d2d4647a wuqiang.matt 2023-10-17 106 * @gfp: flags for memory allocation (via kmalloc or vmalloc)
b4edb8d2d4647a wuqiang.matt 2023-10-17 107 * @context: user context for object initialization callback
b4edb8d2d4647a wuqiang.matt 2023-10-17 108 * @objinit: object initialization callback for extra setup
b4edb8d2d4647a wuqiang.matt 2023-10-17 109 * @release: cleanup callback for extra cleanup task
b4edb8d2d4647a wuqiang.matt 2023-10-17 110 *
b4edb8d2d4647a wuqiang.matt 2023-10-17 111 * return value: 0 for success, otherwise error code
b4edb8d2d4647a wuqiang.matt 2023-10-17 112 *
b4edb8d2d4647a wuqiang.matt 2023-10-17 113 * All pre-allocated objects are to be zeroed after memory allocation.
b4edb8d2d4647a wuqiang.matt 2023-10-17 114 * Caller could do extra initialization in objinit callback. objinit()
b4edb8d2d4647a wuqiang.matt 2023-10-17 115 * will be called just after slot allocation and called only once for
b4edb8d2d4647a wuqiang.matt 2023-10-17 116 * each object. After that the objpool won't touch any content of the
b4edb8d2d4647a wuqiang.matt 2023-10-17 117 * objects. It's caller's duty to perform reinitialization after each
b4edb8d2d4647a wuqiang.matt 2023-10-17 118 * pop (object allocation) or do clearance before each push (object
b4edb8d2d4647a wuqiang.matt 2023-10-17 119 * reclamation).
b4edb8d2d4647a wuqiang.matt 2023-10-17 120 */
b4edb8d2d4647a wuqiang.matt 2023-10-17 121 int objpool_init(struct objpool_head *pool, int nr_objs, int object_size,
b4edb8d2d4647a wuqiang.matt 2023-10-17 122 gfp_t gfp, void *context, objpool_init_obj_cb objinit,
b4edb8d2d4647a wuqiang.matt 2023-10-17 123 objpool_fini_cb release);
b4edb8d2d4647a wuqiang.matt 2023-10-17 124
a3b00f10da808b Andrii Nakryiko 2024-04-24 125 /* try to retrieve object from slot */
a3b00f10da808b Andrii Nakryiko 2024-04-24 126 static inline void *__objpool_try_get_slot(struct objpool_head *pool, int cpu)
a3b00f10da808b Andrii Nakryiko 2024-04-24 127 {
a3b00f10da808b Andrii Nakryiko 2024-04-24 128 struct objpool_slot *slot = pool->cpu_slots[cpu];
a3b00f10da808b Andrii Nakryiko 2024-04-24 129 /* load head snapshot, other cpus may change it */
a3b00f10da808b Andrii Nakryiko 2024-04-24 130 uint32_t head = smp_load_acquire(&slot->head);
a3b00f10da808b Andrii Nakryiko 2024-04-24 131
a3b00f10da808b Andrii Nakryiko 2024-04-24 132 while (head != READ_ONCE(slot->last)) {
a3b00f10da808b Andrii Nakryiko 2024-04-24 133 void *obj;
a3b00f10da808b Andrii Nakryiko 2024-04-24 134
a3b00f10da808b Andrii Nakryiko 2024-04-24 135 /*
a3b00f10da808b Andrii Nakryiko 2024-04-24 136 * data visibility of 'last' and 'head' could be out of
a3b00f10da808b Andrii Nakryiko 2024-04-24 137 * order since memory updating of 'last' and 'head' are
a3b00f10da808b Andrii Nakryiko 2024-04-24 138 * performed in push() and pop() independently
a3b00f10da808b Andrii Nakryiko 2024-04-24 139 *
a3b00f10da808b Andrii Nakryiko 2024-04-24 140 * before any retrieving attempts, pop() must guarantee
a3b00f10da808b Andrii Nakryiko 2024-04-24 141 * 'last' is behind 'head', that is to say, there must
a3b00f10da808b Andrii Nakryiko 2024-04-24 142 * be available objects in slot, which could be ensured
a3b00f10da808b Andrii Nakryiko 2024-04-24 143 * by condition 'last != head && last - head <= nr_objs'
a3b00f10da808b Andrii Nakryiko 2024-04-24 144 * that is equivalent to 'last - head - 1 < nr_objs' as
a3b00f10da808b Andrii Nakryiko 2024-04-24 145 * 'last' and 'head' are both unsigned int32
a3b00f10da808b Andrii Nakryiko 2024-04-24 146 */
a3b00f10da808b Andrii Nakryiko 2024-04-24 147 if (READ_ONCE(slot->last) - head - 1 >= pool->nr_objs) {
a3b00f10da808b Andrii Nakryiko 2024-04-24 148 head = READ_ONCE(slot->head);
a3b00f10da808b Andrii Nakryiko 2024-04-24 149 continue;
a3b00f10da808b Andrii Nakryiko 2024-04-24 150 }
a3b00f10da808b Andrii Nakryiko 2024-04-24 151
a3b00f10da808b Andrii Nakryiko 2024-04-24 152 /* obj must be retrieved before moving forward head */
a3b00f10da808b Andrii Nakryiko 2024-04-24 153 obj = READ_ONCE(slot->entries[head & slot->mask]);
a3b00f10da808b Andrii Nakryiko 2024-04-24 154
a3b00f10da808b Andrii Nakryiko 2024-04-24 155 /* move head forward to mark it's consumption */
a3b00f10da808b Andrii Nakryiko 2024-04-24 @156 if (try_cmpxchg_release(&slot->head, &head, head + 1))
a3b00f10da808b Andrii Nakryiko 2024-04-24 157 return obj;
a3b00f10da808b Andrii Nakryiko 2024-04-24 158 }
a3b00f10da808b Andrii Nakryiko 2024-04-24 159
a3b00f10da808b Andrii Nakryiko 2024-04-24 160 return NULL;
a3b00f10da808b Andrii Nakryiko 2024-04-24 161 }
a3b00f10da808b Andrii Nakryiko 2024-04-24 162 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v3 7/8 RISC-V: KVM: Allow Zalasr extensions for Guest/VM
In reply to Xu LuExtend the KVM ISA extension ONE_REG interface to allow KVM user space to detect and enable Zalasr extensions for Guest/VM.
--- arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kvm/vcpu_onereg.c | 2 ++ 2 files changed, 3 insertions(+)
arch/riscv/include/uapi/asm/kvm.h +1 -0
@@ -185,6 +185,7 @@ enum KVM_RISCV_ISA_EXT_ID {KVM_RISCV_ISA_EXT_ZICCRSE ,KVM_RISCV_ISA_EXT_ZAAMO ,KVM_RISCV_ISA_EXT_ZALRSC , +KVM_RISCV_ISA_EXT_ZALASR , KVM_RISCV_ISA_EXT_MAX, };
arch/riscv/kvm/vcpu_onereg.c +2 -0
@@ -50,6 +50,7 @@ static const unsigned long kvm_isa_ext_arr[] = {KVM_ISA_EXT_ARR (ZAAMO ),KVM_ISA_EXT_ARR (ZABHA ),KVM_ISA_EXT_ARR (ZACAS ), +KVM_ISA_EXT_ARR (ZALASR ),KVM_ISA_EXT_ARR (ZALRSC ),KVM_ISA_EXT_ARR (ZAWRS ), KVM_ISA_EXT_ARR(ZBA),
@@ -184,6 +185,7 @@ static bool kvm_riscv_vcpu_isa_disable_allowed(unsigned long ext)case KVM_RISCV_ISA_EXT_ZAAMO :case KVM_RISCV_ISA_EXT_ZABHA :case KVM_RISCV_ISA_EXT_ZACAS : +case KVM_RISCV_ISA_EXT_ZALASR :case KVM_RISCV_ISA_EXT_ZALRSC :case KVM_RISCV_ISA_EXT_ZAWRS :case KVM_RISCV_ISA_EXT_ZBA :
Signature · 1 line
2.20.1
PATCH v3 8/8 KVM: riscv: selftests: Add Zalasr extensions to get-reg-list test
In reply to Xu LuThe KVM RISC-V allows Zalasr extensions for Guest/VM so add these extensions to get-reg-list test.
--- tools/testing/selftests/kvm/riscv/get-reg-list.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/testing/selftests/kvm/riscv/get-reg-list.c +4 -0
@@ -65,6 +65,7 @@ bool filter_reg(__u64 reg)case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZAAMO :case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZABHA :case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZACAS : +case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZALASR :case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZALRSC :case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZAWRS :case KVM_REG_RISCV_ISA_EXT |KVM_REG_RISCV_ISA_SINGLE |KVM_RISCV_ISA_EXT_ZBA :
@@ -517,6 +518,7 @@ static const char *isa_ext_single_id_to_str(__u64 reg_off)KVM_ISA_EXT_ARR (ZAAMO ),KVM_ISA_EXT_ARR (ZABHA ),KVM_ISA_EXT_ARR (ZACAS ), +KVM_ISA_EXT_ARR (ZALASR ),KVM_ISA_EXT_ARR (ZALRSC ),KVM_ISA_EXT_ARR (ZAWRS ), KVM_ISA_EXT_ARR(ZBA),
@@ -1112,6 +1114,7 @@ KVM_ISA_EXT_SIMPLE_CONFIG(svvptc, SVVPTC);KVM_ISA_EXT_SIMPLE_CONFIG (zaamo,ZAAMO );KVM_ISA_EXT_SIMPLE_CONFIG (zabha,ZABHA );KVM_ISA_EXT_SIMPLE_CONFIG (zacas,ZACAS ); +KVM_ISA_EXT_SIMPLE_CONFIG (zalasr,ZALASR );KVM_ISA_EXT_SIMPLE_CONFIG (zalrsc,ZALRSC );KVM_ISA_EXT_SIMPLE_CONFIG (zawrs,ZAWRS );KVM_ISA_EXT_SIMPLE_CONFIG (zba,ZBA );
@@ -1187,6 +1190,7 @@ struct vcpu_reg_list *vcpu_configs[] = { &config_zabha, &config_zacas, &config_zalrsc, + &config_zalasr, &config_zawrs, &config_zba, &config_zbb,
Signature · 1 line
2.20.1
Quoted 37 lines · Xu Lu
This patch adds support for the Zalasr ISA extension, which supplies the real load acquire/store release instructions.
The specification can be found here:
https://github.com/riscv/riscv-zalasr/blob/main/chapter2.adocThis patch seires has been tested with ltp on Qemu with Brensan's zalasr support patch[1].
Some false positive spacing error happens during patch checking. Thus I CCed maintainers of checkpatch.pl as well.
[1] https://lore.kernel.org/all/CAGPSXwJEdtqW=nx71oufZp64nK6tK=0rytVEcz4F-gfvCOXk2w@mail.gmail.com/
v3:
- Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations so as to ensure FENCE.TSO ordering between operations which precede the UNLOCK+LOCK sequence and operations which follow the sequence. Thanks to Andrea.
- Support hwprobe of Zalasr.
- Allow Zalasr extensions for Guest/VM.v2:
- Adjust the order of Zalasr and Zalrsc in dt-bindings. Thanks to Conor.Xu Lu (8):
riscv: add ISA extension parsing for Zalasr
dt-bindings: riscv: Add Zalasr ISA extension description
riscv: hwprobe: Export Zalasr extension
riscv: Introduce Zalasr instructions
riscv: Use Zalasr for smp_load_acquire/smp_store_release
riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg
operations
RISC-V: KVM: Allow Zalasr extensions for Guest/VM
KVM: riscv: selftests: Add Zalasr extensions to get-reg-list test
Quoted 14 lines
Documentation/arch/riscv/hwprobe.rst | 5 +- .../devicetree/bindings/riscv/extensions.yaml | 5 + arch/riscv/include/asm/atomic.h | 6 - arch/riscv/include/asm/barrier.h | 91 ++++++++++-- arch/riscv/include/asm/cmpxchg.h | 136 ++++++++---------- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 79 ++++++++++ arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + arch/riscv/kvm/vcpu_onereg.c | 2 + .../selftests/kvm/riscv/get-reg-list.c | 4 + 13 files changed, 242 insertions(+), 91 deletions(-)
I wouldn't have rushed this submission while the discussion on v2 seems so much alive; IAC, to add and link to that discussion, this version (not a review, just looking at this diff stat) is changing the fastpath
read_unlock()
read_lock()
from something like
fence rw,w
amodadd.w
amoadd.w
fence r,rw
to
fence rw,rw
amoadd.w
amoadd.w
fence rw,rw
no matter Zalasr or !Zalasr. Similarly for other atomic operations with release or acquire semantics. I guess the change was not intentional? If it was intentional, it should be properly mentioned in the changelog.
Andrea
Hi Andrea,
Quoted 54 lines · Andrea Parri
Quoted 37 lines · Xu Lu
This patch adds support for the Zalasr ISA extension, which supplies the real load acquire/store release instructions.
The specification can be found here:
https://github.com/riscv/riscv-zalasr/blob/main/chapter2.adocThis patch seires has been tested with ltp on Qemu with Brensan's zalasr support patch[1].
Some false positive spacing error happens during patch checking. Thus I CCed maintainers of checkpatch.pl as well.
[1] https://lore.kernel.org/all/CAGPSXwJEdtqW=nx71oufZp64nK6tK=0rytVEcz4F-gfvCOXk2w@mail.gmail.com/
v3:
- Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations so as to ensure FENCE.TSO ordering between operations which precede the UNLOCK+LOCK sequence and operations which follow the sequence. Thanks to Andrea.
- Support hwprobe of Zalasr.
- Allow Zalasr extensions for Guest/VM.v2:
- Adjust the order of Zalasr and Zalrsc in dt-bindings. Thanks to Conor.Xu Lu (8):
riscv: add ISA extension parsing for Zalasr
dt-bindings: riscv: Add Zalasr ISA extension description
riscv: hwprobe: Export Zalasr extension
riscv: Introduce Zalasr instructions
riscv: Use Zalasr for smp_load_acquire/smp_store_release
riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg
operations
RISC-V: KVM: Allow Zalasr extensions for Guest/VM
KVM: riscv: selftests: Add Zalasr extensions to get-reg-list testQuoted 14 lines
Documentation/arch/riscv/hwprobe.rst | 5 +- .../devicetree/bindings/riscv/extensions.yaml | 5 + arch/riscv/include/asm/atomic.h | 6 - arch/riscv/include/asm/barrier.h | 91 ++++++++++-- arch/riscv/include/asm/cmpxchg.h | 136 ++++++++---------- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 79 ++++++++++ arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + arch/riscv/kvm/vcpu_onereg.c | 2 + .../selftests/kvm/riscv/get-reg-list.c | 4 + 13 files changed, 242 insertions(+), 91 deletions(-)
Quoted 2 lines
I wouldn't have rushed this submission while the discussion on v2 seems so much alive; IAC, to add and link to that discussion, this version
Thanks. This version is sent out to show my solution to the FENCE.TSO problem you pointed out before. I will continue to improve it. Look forward to more suggestions from you.
Quoted 14 lines
(not a review, just looking at this diff stat) is changing the fastpath
read_unlock()
read_lock()from something like
fence rw,w
amodadd.w
amoadd.w
fence r,rwto
Quoted 8 lines
fence rw,rw
amoadd.w
amoadd.w
fence rw,rwno matter Zalasr or !Zalasr. Similarly for other atomic operations with release or acquire semantics. I guess the change was not intentional? If it was intentional, it should be properly mentioned in the changelog.
Sorry about that. It is intended. The atomic operation before __atomic_acquire_fence or operation after __atomic_release_fence can be just a single ld or sd instruction instead of amocas or amoswap. In such cases, when the store release operation becomes 'sd.rl', the __atomic_acquire_fence via 'fence r, rw' can not ensure FENCE.TSO anymore. Thus I replace it with 'fence rw, rw'.
I will make it a separate commit and provide more messages in the changelog. Maybe alternative mechanism can be applied to accelerate it.
Best Regards,
Xu Lu
Quoted 2 lines
Andrea
Quoted 88 lines · Xu Lu
Hi Andrea,
Quoted 54 lines · Andrea Parri
Quoted 37 lines · Xu Lu
This patch adds support for the Zalasr ISA extension, which supplies the real load acquire/store release instructions.
The specification can be found here:
https://github.com/riscv/riscv-zalasr/blob/main/chapter2.adocThis patch seires has been tested with ltp on Qemu with Brensan's zalasr support patch[1].
Some false positive spacing error happens during patch checking. Thus I CCed maintainers of checkpatch.pl as well.
[1] https://lore.kernel.org/all/CAGPSXwJEdtqW=nx71oufZp64nK6tK=0rytVEcz4F-gfvCOXk2w@mail.gmail.com/
v3:
- Apply acquire/release semantics to arch_xchg/arch_cmpxchg operations so as to ensure FENCE.TSO ordering between operations which precede the UNLOCK+LOCK sequence and operations which follow the sequence. Thanks to Andrea.
- Support hwprobe of Zalasr.
- Allow Zalasr extensions for Guest/VM.v2:
- Adjust the order of Zalasr and Zalrsc in dt-bindings. Thanks to Conor.Xu Lu (8):
riscv: add ISA extension parsing for Zalasr
dt-bindings: riscv: Add Zalasr ISA extension description
riscv: hwprobe: Export Zalasr extension
riscv: Introduce Zalasr instructions
riscv: Use Zalasr for smp_load_acquire/smp_store_release
riscv: Apply acquire/release semantics to arch_xchg/arch_cmpxchg
operations
RISC-V: KVM: Allow Zalasr extensions for Guest/VM
KVM: riscv: selftests: Add Zalasr extensions to get-reg-list testQuoted 14 lines
Documentation/arch/riscv/hwprobe.rst | 5 +- .../devicetree/bindings/riscv/extensions.yaml | 5 + arch/riscv/include/asm/atomic.h | 6 - arch/riscv/include/asm/barrier.h | 91 ++++++++++-- arch/riscv/include/asm/cmpxchg.h | 136 ++++++++---------- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 79 ++++++++++ arch/riscv/include/uapi/asm/hwprobe.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/sys_hwprobe.c | 1 + arch/riscv/kvm/vcpu_onereg.c | 2 + .../selftests/kvm/riscv/get-reg-list.c | 4 + 13 files changed, 242 insertions(+), 91 deletions(-)Quoted 2 lines
I wouldn't have rushed this submission while the discussion on v2 seems so much alive; IAC, to add and link to that discussion, this version
Thanks. This version is sent out to show my solution to the FENCE.TSO problem you pointed out before. I will continue to improve it. Look forward to more suggestions from you.
Quoted 14 lines
(not a review, just looking at this diff stat) is changing the fastpath
read_unlock()
read_lock()from something like
fence rw,w
amodadd.w
amoadd.w
fence r,rwto
Quoted 8 lines
fence rw,rw
amoadd.w
amoadd.w
fence rw,rwno matter Zalasr or !Zalasr. Similarly for other atomic operations with release or acquire semantics. I guess the change was not intentional? If it was intentional, it should be properly mentioned in the changelog.
Quoted 6 lines
Sorry about that. It is intended. The atomic operation before
__atomic_acquire_fenceor operation after__atomic_release_fencecan be just a single ld or sd instruction instead of amocas or amoswap. In such cases, when the store release operation becomes 'sd.rl', the__atomic_acquire_fencevia 'fence r, rw' can not ensure FENCE.TSO anymore. Thus I replace it with 'fence rw, rw'.
This is also the common implementation on other architectures who use aq/rl instructions like ARM. And you certainly already knew it~
Quoted 10 lines
I will make it a separate commit and provide more messages in the changelog. Maybe alternative mechanism can be applied to accelerate it.
Best Regards,
Xu LuQuoted 2 lines
Andrea
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
(not a review, just looking at this diff stat) is changing the fastpath
read_unlock()
read_lock()from something like
fence rw,w
amodadd.w
amoadd.w
fence r,rwto
fence rw,rw
amoadd.w
amoadd.w
fence rw,rwno matter Zalasr or !Zalasr. Similarly for other atomic operations with release or acquire semantics. I guess the change was not intentional? If it was intentional, it should be properly mentioned in the changelog.
Quoted 6 lines
Quoted 6 lines
Sorry about that. It is intended. The atomic operation before
__atomic_acquire_fenceor operation after__atomic_release_fencecan be just a single ld or sd instruction instead of amocas or amoswap. In such cases, when the store release operation becomes 'sd.rl', the__atomic_acquire_fencevia 'fence r, rw' can not ensure FENCE.TSO anymore. Thus I replace it with 'fence rw, rw'.
But you could apply similar changes you performed for xchg & cmpxchg: use .AQ and .RL for other atomic RMW operations as well, no? AFAICS, that is what arm64 actually does in arch/arm64/include/asm/atomic_{ll_sc,lse}.h .
Andrea
Quoted 2 lines
This is also the common implementation on other architectures who use aq/rl instructions like ARM. And you certainly already knew it~
Quoted 31 lines · Andrea Parri
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
(not a review, just looking at this diff stat) is changing the fastpath
read_unlock()
read_lock()from something like
fence rw,w
amodadd.w
amoadd.w
fence r,rwto
fence rw,rw
amoadd.w
amoadd.w
fence rw,rwno matter Zalasr or !Zalasr. Similarly for other atomic operations with release or acquire semantics. I guess the change was not intentional? If it was intentional, it should be properly mentioned in the changelog.
Quoted 6 lines
Quoted 6 lines
Sorry about that. It is intended. The atomic operation before
__atomic_acquire_fenceor operation after__atomic_release_fencecan be just a single ld or sd instruction instead of amocas or amoswap. In such cases, when the store release operation becomes 'sd.rl', the__atomic_acquire_fencevia 'fence r, rw' can not ensure FENCE.TSO anymore. Thus I replace it with 'fence rw, rw'.
Quoted 3 lines
But you could apply similar changes you performed for xchg & cmpxchg: use .AQ and .RL for other atomic RMW operations as well, no? AFAICS, that is what arm64 actually does in
arch/arm64/include/asm/atomic_{ll_sc,lse}.h.
I see. I will study the implementation of ARM and refine my patch. Thanks a lot.
Best regards,
Xu Lu
Quoted 6 lines
Andrea
Quoted 2 lines
This is also the common implementation on other architectures who use aq/rl instructions like ARM. And you certainly already knew it~