riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO
Subsystems (9)
v7 cover riscv: restrict the ISA used for atomics to avoid unsupported or slow AMO
The A extension is Zaamo (AMO*.w/d) plus Zalrsc (LR/SC). The kernel requires all of it: -march has "a", atomic_add, bitops, xchg and the like are AMO instructions, cmpxchg is an LR/SC loop.
This series lets the kernel restrict the ISA it uses for atomics to Zalrsc, when the toolchain can express that, for CPUs where AMO is missing or slow. MIPS P8700 [1] traps every AMO instruction into M-mode and emulates it there. The extension is formally present, but a typical uncontended atomic increment costs:
native AMO 2..5 cycles
LR/SC, private L1 reservation monitor 6..12 cycles
LR/SC, PoC (L2) reservation monitor about 40 cycles
AMO emulated by an M-mode trap about 600 cycles
The last two were measured on Mobileye's eyeq7h, which is built on this CPU for ADAS and autonomous driving. Kernel atomics are on its scheduling, memory-management and driver paths, so the emulated AMO cost is not acceptable there. perf bench on eyeq7h, same tree built with RISCV_ISA_ZAAMO=y and =n, same userspace, built with LR/SC atomics in both cases so that only the kernel changes:
ZAAMO=y ZAAMO=n
sched messaging, 400 processes 0.653 s 0.536 s -18% futex hash, 24 threads 64131 op/s 79142 op/s +23% futex wake, 24 threads 0.221 ms 0.168 ms -24% futex requeue, 24 threads 0.064 ms 0.036 ms -44% futex lock-pi, 24 threads 435 op/s 478 op/s +10%
futex wake-parallel swings by an order of magnitude between runs on both kernels and is left out. With the ZAAMO=y kernel the firmware counter of emulated instructions, fw_illegal_insn, runs at about 4000 per second per CPU at idle, and at about 85000 per second on CPU0, where network Rx runs, under iperf3 Rx traffic.
Patch 1 adds RISCV_ISA_ZAAMO, default y, with no change to the default build. With it off, -march carries _zalrsc instead of "a" and every kernel atomic is an LR/SC loop, selected with the preprocessor like the other ISA options. The loops use the ordering convention of the existing LR/SC code: lr, sc.rl and a full fence when fully ordered, lr.aq for acquire, sc.rl for release. The choice is made at compile time rather than with alternatives, so the many kernels for CPUs with native AMO do not pay for nop padding at every atomic site. Such a kernel runs on any hart with A, so there is no NONPORTABLE dependency. It needs a compiler that accepts _zaamo and _zalrsc in -march, GCC 15.1 or later; TOOLCHAIN_HAS_ZALRSC_ZAAMO keeps the option fixed at y otherwise. Zabha, Zacas, userspace CFI and Rust emit AMO instructions and depend on RISCV_ISA_ZAAMO. Zalrsc stays mandatory: cmpxchg, the conditional atomics and futex CAS are LR/SC sequences. Zaamo plus Zacas, with CAS loops as on x86, would also provide every atomic and is left for whoever has such hardware.
Patch 2 makes the boot-time check accept a hart that lists "zalrsc" alone, or "zaamo" and "zalrsc" instead of "a", when the kernel was built for that. Userspace sees whatever the hart advertises, and hwprobe reports the IMA base behaviour only when every hart has A.
Question: shall I add something like RISCV_HWPROBE_BASE_BEHAVIOR_IM (without A) and report it if CPU haz no A?
Patch 3 sets "a" when a hart lists "zaamo" and "zalrsc" instead of "a", so ELF hwcap and /proc/cpuinfo report the same ISA either way.
Patch 4 makes the BPF JIT emit LR/SC loops when Zaamo is off, so BPF programs do not trap either.
Patch 5 is a fix on its own: since the revert of the lazy vDSO fill in v7.3-rc1, the vDSO copy of the hwprobe data is made before the misaligned access speed probes run, so the vDSO fast path reports the two MISALIGNED_*_PERF keys as unknown. Refresh the vDSO data once the probes are done. Patch 7 needs the same for its key.
Patch 6 measures at boot whether an uncontended AMO is faster than the equivalent LR/SC sequence on each CPU and reports the answer to userspace through a new hwprobe key, RISCV_HWPROBE_KEY_AMO_PERF, the way the misaligned access speed is reported. On P8700 it says "slow"; a program or a runtime that can choose then knows to use LR/SC. This patch is optional. eyeq7h does not need it: it is an embedded platform whose userspace is built for the hardware, not a generic distribution that has to find out at run time. It is here for whoever has to run generic userspace on such a CPU.
How the pieces fit together, with "..." for the rest of -march:
Hardware AMO DT, cpuinfo RISCV_ISA_ZAAMO Userspace -march
------------ ----------- --------------- ---------------------------
native a y (default) ...a, unchanged
native a n ...a; only the kernel
switches to LR/SC
emulated a y ...a or ..._zalrsc; the
kernel traps on every
AMO. This is today.
emulated a n ...a runs but traps on
every AMO; ..._zalrsc
does not
emulated zalrsc n (y: hart ..._zalrsc; ...a still
rejected) runs, slowly
absent zalrsc n (y: hart ..._zalrsc; ...a dies
rejected) with SIGILLThe kernel expands "a" to zaamo and zalrsc, and with patch 4 folds the two back into "a", so userspace sees the same ISA in hwcap, hwprobe and /proc/cpuinfo whether the DT lists "a" or both sub-extensions. For an emulating CPU the firmware chooses between "a", which is formally true and keeps every existing binary running, and "zalrsc" alone, which lets userspace that checks hwcap or hwprobe know not to use AMO; with patch 7, hwprobe says so in either case. Nothing in the series depends on the last column:
a kernel built without Zaamo runs userspace built either way. Building userspace without "a" needs the same GCC 15.1 as the kernel and an -march that spells out the base, e.g. rv64imfdc_zicsr_zifencei_zalrsc, since "g" implies "a".
Both "AMO not supported" and "AMO is slow" configurations are tested on the eyeq7h hardware (24 harts) running locktorture for a several hours.
This series grew out of the January patches for Zalrsc-only P8700 cores [4], which (as Jisheng Zhang <[email protected]> noticed) Paul asked to hold until common userspace could run on such a configuration. It is no longer that series. The hart keeps advertising "a" unless openSBI disable AMO emulation, the default kernel build is unchanged, userspace built for "a" keeps running on the resulting kernel, and no change in glibc or in a distribution is required. What is left is a compile-time choice of the atomic sequence the kernel itself uses, plus the probes fix and the hwprobe hints, which stand on their own.
[1] https://mips.com/products/hardware/p8700/
[2] https://lore.kernel.org/linux-riscv/20261106-smock-headband-d6a7d568c228@spud/
[3] https://lore.kernel.org/linux-riscv/20261110-revivable-saint-9bcfaa2fd0e7@spud/
[4] https://lore.kernel.org/linux-riscv/[email protected]/
Changes in v7:
- remove patch rejecting probes inside LR/SC block (was patch 1), submitted on its own
- Drop Conor Dooley's Nacked-by from patch 1, as agreed on the v6
thread. NAKed were the v5 Kconfig option [2]. The points raised
were addressed in v6 and Conor confirmed on the v6 thread [3] they
were resolved, so the trailer is dropped.
- hwprobe reports RISCV_HWPROBE_BASE_BEHAVIOR_IMA only when every
selected hart has A, so a Zalrsc-only hart is not described as IMA (patch 2).
- BPF JIT: the fetch and xchg LR/SC loops are fully ordered the way emit_cmpxchg() is, plain lr, sc.rl and a trailing fence rw,rw,
instead of lr.aqrl/sc.aqrl (patch 4).
- New patch 5: refresh the vDSO hwprobe data after the boot-time
speed probes; fixes the stale MISALIGNED_*_PERF values in the vDSO
since v7.3-rc1.
- AMO speed probe: refresh the vDSO data when done, report a CPU
without Zaamo as unsupported even with amo_speed= on the command
line, and return 0 from the __setup() handler on a bad value
(patch 6).
- Link to v6: https://patch.msgid.link/[email protected]
Changes in v6:
- Fully ordered LR/SC loops in atomic_fetch_*, test_and_*_bit, xchg
and futex are now lr, sc.rl and a trailing fence rw,rw, following
commit 5ce6c1f3535f ("riscv/atomic: Strengthen implementations with fences"), instead of lr.aqrl/sc.aqrl.
- test_and_set_bit_lock uses lr.aq with a plain sc, and
clear_bit_unlock and xor_unlock_is_negative_byte use a plain lr
with sc.rl, instead of putting the ordering bit on both.
- Bound the futex LR/SC loop to 128 retries and return -EAGAIN, as
arm64 does; futex_wake_op() retries.
- RISCV_ISA_ZABHA, RISCV_ISA_ZACAS, RISCV_USER_CFI and RUSTC_SUPPORTS_RISCV depend on RISCV_ISA_ZAAMO, since Zabha, Zacas, ssamoswap and the Rust target emit AMO instructions.
- Drop RISCV_ISA_ZALRSC and RISCV_ISA_A. The Makefile and the boot
check key on RISCV_ISA_ZAAMO alone.
- kprobes scan forward with the same constrained-sequence rules as uprobes, so an lr without an sc, as in __cmpwait(), no longer
rejects every later probe in the function.
- Move the probes patch first.
- Bring back the boot-time extension check as patch 3, keyed on RISCV_ISA_ZAAMO. A hart may list "zalrsc" alone, or "zaamo" and
"zalrsc" instead of "a".
- Add patch 4: a hart that lists "zaamo" and "zalrsc" gets A set,
so hwcap and /proc/cpuinfo match a hart that lists "a".
- Add patch 6: probe AMO speed against LR/SC at boot and report it
through hwprobe, RISCV_HWPROBE_KEY_AMO_PERF.
- Record Conor Dooley's NAK on patch 2.
- Rewrite the cover letter; add perf bench results and a table of hardware, DT, Kconfig and userspace combinations.
- Link to v5: https://patch.msgid.link/[email protected]
Changes in v5:
- Treat Zaamo as supported but slow rather than absent. The hart keeps advertising "a" and the kernel keeps accepting it, so the boot-time extension check patch is dropped.
- Drop the NONPORTABLE dependency. A kernel built without Zaamo runs
on any hart that implements A.
- Rename TOOLCHAIN_HAS_ZALRSC to TOOLCHAIN_HAS_ZALRSC_ZAAMO and test
both _zaamo and _zalrsc in -march.
- Explain why RISCV_ISA_ZALRSC and RISCV_ISA_A exist, and why the
choice is made at compile time rather than with alternatives.
- Add the BPF JIT patch. Without Zaamo the JIT emits LR/SC loops for
the BPF read-modify-write atomics on RV32 and RV64, and reserves a second exception-table entry for the sc of an arena atomic.
- Reject kprobes and uprobes on lr and sc, and on any instruction
between them.
- Link to v4: https://patch.msgid.link/[email protected]
Changes in v4:
- Require a compiler that accepts _zalrsc in -march before Zaamo can
be disabled. GCC 15.1 introduces that support. Older compilers keep Zaamo enabled and -march keeps "a".
- Link to v3: https://patch.msgid.link/[email protected]
Changes in v3:
- Make Zalrsc mandatory. cmpxchg and the conditional atomics already use LR/SC, so RISCV_ISA_ZALRSC can no longer be turned off.
- Check AMO and LR/SC support independently. A kernel that emits both requires "a", or both "zaamo" and "zalrsc".
- Put the LR/SC bitop sequence in its own block so two inlined set_bit or clear_bit calls compile.
- Remove the LR/SC loop that ran after the hart lottery and accessed address 1. Build the lottery increment with add so RV32 assembles.
- Add an exception-table entry for sc as well as lr in the futex LR/SC helper.
Changes in v2:
- Switch from a runtime choice of atomic implementation to a compile-time one, gated by NONPORTABLE.
---
Vladimir Kondratiev (6):
riscv: build kernel atomics from LR/SC when Zaamo is disabled
riscv: accept the atomic extensions this kernel was built with
riscv: report the A extension when Zaamo and Zalrsc are both present riscv, bpf: emit LR/SC loops when Zaamo is disabled
riscv: hwprobe: update the vDSO data after the boot-time speed probes
riscv: hwprobe: report whether AMO is faster than LR/SC
Documentation/admin-guide/kernel-parameters.txt | 7 + Documentation/arch/riscv/hwprobe.rst | 22 ++- arch/riscv/Kconfig | 49 +++++++ arch/riscv/Makefile | 15 +- arch/riscv/include/asm/atomic.h | 53 ++++++- arch/riscv/include/asm/bitops.h | 76 +++++++++- arch/riscv/include/asm/cmpxchg.h | 39 ++++- arch/riscv/include/asm/cpufeature.h | 4 + arch/riscv/include/asm/futex.h | 58 ++++++++ arch/riscv/include/asm/hwprobe.h | 8 +- arch/riscv/include/asm/insn-def.h | 5 + arch/riscv/include/uapi/asm/hwprobe.h | 5 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/amo_speed.c | 183 ++++++++++++++++++++++++ arch/riscv/kernel/cpu.c | 23 ++- arch/riscv/kernel/cpufeature.c | 9 ++ arch/riscv/kernel/entry.S | 7 + arch/riscv/kernel/head.S | 12 ++ arch/riscv/kernel/sys_hwprobe.c | 76 ++++++++-- arch/riscv/kernel/unaligned_access_speed.c | 8 +- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp32.c | 84 ++++++++++- arch/riscv/net/bpf_jit_comp64.c | 109 +++++++++++++- arch/riscv/net/bpf_jit_core.c | 29 ++++ 24 files changed, 844 insertions(+), 39 deletions(-)
---
base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
change-id: 20260120-lrsc-only-cc06422f5bb1
Best regards,
Signature · 1 line
Vladimir Kondratiev <[email protected]>
PATCH v7 1/6 riscv: build kernel atomics from LR/SC when Zaamo is disabled
The A extension is Zaamo plus Zalrsc. On some CPUs Zaamo is present only formally. MIPS P8700 traps every AMO instruction into M-mode and emulates it there, so the extension is supported but very slow. A typical uncontended atomic increment costs:
native AMO 2..5 cycles
LR/SC, private L1 reservation monitor 6..12 cycles
LR/SC, PoC (L2) reservation monitor about 40 cycles
AMO emulated by an M-mode trap about 600 cycles
The last two were measured on the eyeq7h hardware. On such a CPU the kernel must use LR/SC for every atomic.
Add RISCV_ISA_ZAAMO, default y. The default keeps "a" in -march and the generated code unchanged. When it is disabled, -march carries _zalrsc instead of "a", and atomic_*, bitops, xchg, futex and the early boot code use LR/SC loops. The sequence is chosen with the preprocessor, on CONFIG_RISCV_ISA_ZAAMO like the other ISA options in the headers, because the assembler rejects AMO mnemonics once -march lacks Zaamo. A kernel built this way runs on any hart that implements A; it just does not use AMO. It is therefore a plain option, not a NONPORTABLE one, and the hart keeps advertising "a" to userspace, which is true.
The LR/SC loops follow the ordering convention of the existing ones in cmpxchg.h, from commit 5ce6c1f3535f ("riscv/atomic: Strengthen implementations with fences") and commit 1d84afaf0252 ("riscv: Fix fully ordered LR/SC xchg[8|16]() implementations"): a fully ordered operation is lr, sc.rl and a trailing fence rw,rw; acquire is lr.aq with a plain sc; release is a plain lr with sc.rl. The futex loop is bounded to 128 retries and returns -EAGAIN after that, as arm64 does, because userspace can keep the word changing from another thread and futex_wake_op() retries on -EAGAIN anyway.
Zabha, Zacas and the ssamoswap instruction of Zicfiss are AMO instructions, so RISCV_ISA_ZABHA, RISCV_ISA_ZACAS and RISCV_USER_CFI depend on RISCV_ISA_ZAAMO. The riscv64imac Rust target emits AMO too, so RUSTC_SUPPORTS_RISCV depends on it as well.
The choice is made at compile time rather than with alternatives. Nearly every CPU has a native A extension, and an alternative would inflate all those kernels with the nops needed to pad one AMO to the size of an LR/SC loop, at every atomic site, for the benefit of a few.
Turning Zaamo off needs a compiler that accepts _zaamo and _zalrsc in -march. GCC 15.1 is the first release that does. TOOLCHAIN_HAS_ZALRSC_ZAAMO checks for it; with an older compiler the option stays at y.
Three sets of extensions provide every atomic Linux needs. "a" is what the kernel supports today. Zalrsc plus Ziccrse is what this patch adds:
every atomic is a constrained LR/SC loop, and Ziccrse is the forward-progress guarantee for such loops. Zaamo plus Zacas would do as well, with CAS loops in place of LR/SC as on x86, and is left for whoever has such hardware.
--- arch/riscv/Kconfig | 36 +++++++++++++++++++ arch/riscv/Makefile | 15 +++++--- arch/riscv/include/asm/atomic.h | 53 +++++++++++++++++++++++++++- arch/riscv/include/asm/bitops.h | 76 ++++++++++++++++++++++++++++++++++++++-- arch/riscv/include/asm/cmpxchg.h | 39 +++++++++++++++++---- arch/riscv/include/asm/futex.h | 58 ++++++++++++++++++++++++++++++ arch/riscv/kernel/entry.S | 7 ++++ arch/riscv/kernel/head.S | 12 +++++++ 8 files changed, 281 insertions(+), 15 deletions(-)
arch/riscv/Kconfig +36 -0
@@ -237,6 +237,8 @@ config RISCV config RUSTC_SUPPORTS_RISCV def_bool y depends on 64BIT + # The riscv64imac target emits AMO instructions + depends on RISCV_ISA_ZAAMO config CLANG_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_CLANG
@@ -706,6 +708,7 @@ config RISCV_ISA_ZABHA bool "Zabha extension support for atomic byte/halfword operations" depends on TOOLCHAIN_HAS_ZABHA depends on RISCV_ALTERNATIVE + depends on RISCV_ISA_ZAAMO default y help Enable the use of the Zabha ISA-extension to implement kernel
@@ -723,6 +726,7 @@ config TOOLCHAIN_HAS_ZACAS config RISCV_ISA_ZACAS bool "Zacas extension support for atomic CAS" depends on RISCV_ALTERNATIVE + depends on RISCV_ISA_ZAAMO default y help Enable the use of the Zacas ISA-extension to implement kernel atomic
@@ -730,6 +734,36 @@ config RISCV_ISA_ZACAS If you don't know what to do here, say Y. +config TOOLCHAIN_HAS_ZALRSC_ZAAMO + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64im_zaamo_zalrsc) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32im_zaamo_zalrsc) + +config RISCV_ISA_ZAAMO + bool "Zaamo extension support for AMO atomics" if TOOLCHAIN_HAS_ZALRSC_ZAAMO + default y + help + Use AMO instructions for kernel atomic memory operations. When + disabled, those operations are built from LR/SC and the kernel + is compiled with zalrsc instead of the A extension. + + Disable this for a CPU where AMO is much slower than LR/SC. That + is the case when AMO instructions are emulated by a trap into + M-mode, as on the MIPS P8700: an emulated atomic costs hundreds + of cycles where LR/SC costs tens. + + A kernel built without Zaamo runs on any CPU that implements the + A extension. Zabha, Zacas, userspace control flow integrity and + Rust need Zaamo and are unavailable without it. + + The compiler must accept _zaamo and _zalrsc in -march to make + this choice. GCC 15.1 is the first release that does. With an + older compiler the option stays enabled and -march keeps the A + extension. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZBB bool default y
@@ -1206,6 +1240,8 @@ config RISCV_USER_CFI depends on 64BIT && MMU && \ $(cc-option,-mabi=lp64 -march=rv64ima_zicfiss_zicfilp -fcf-protection=full) depends on RISCV_ALTERNATIVE + # ssamoswap is an AMO instruction + depends on RISCV_ISA_ZAAMO select RISCV_SBI select ARCH_HAS_USER_SHADOW_STACK select ARCH_USES_HIGH_VMA_FLAGS
arch/riscv/Makefile +11 -4
@@ -58,8 +58,9 @@ ifeq ($(CONFIG_SHADOW_CALL_STACK),y) endif # ISA string setting -riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima -riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima +riscv-march-$(CONFIG_ARCH_RV32I) := rv32im +riscv-march-$(CONFIG_ARCH_RV64I) := rv64im +riscv-march-$(CONFIG_RISCV_ISA_ZAAMO) := $(riscv-march-y)a riscv-march-$(CONFIG_FPU) := $(riscv-march-y)fd riscv-march-$(CONFIG_RISCV_ISA_C) := $(riscv-march-y)c riscv-march-$(CONFIG_RISCV_ISA_V) := $(riscv-march-y)v
@@ -75,9 +76,15 @@ else riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei endif -KBUILD_BASE_ISA = -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') +KBUILD_BASE_ISA = -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima?|rv64ima?)fd([^v_]*)v?/\1\2/') export KBUILD_BASE_ISA +# Without Zaamo only the LR/SC half of "a" is used. The compiler accepts +# _zalrsc here, see TOOLCHAIN_HAS_ZALRSC_ZAAMO. +ifneq ($(CONFIG_RISCV_ISA_ZAAMO),y) +riscv-march-y := $(riscv-march-y)_zalrsc +endif + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += $(KBUILD_BASE_ISA)
@@ -85,7 +92,7 @@ KBUILD_CFLAGS += $(KBUILD_BASE_ISA) KBUILD_AFLAGS += -march=$(riscv-march-y) # For C code built with floating-point support, exclude V but keep F and D. -CC_FLAGS_FPU := -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)([^v_]*)v?/\1\2/') +CC_FLAGS_FPU := -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima?|rv64ima?)([^v_]*)v?/\1\2/') KBUILD_CFLAGS += -mno-save-restore
arch/riscv/include/asm/atomic.h +52 -1
77 lines not on this page · raw message
arch/riscv/include/asm/bitops.h +73 -3
129 lines not on this page · raw message
arch/riscv/include/asm/cmpxchg.h +32 -7
@@ -57,16 +57,39 @@ } \ }) -#define __arch_xchg (sfx, prepend, append, r, p, n) \ +/* +* Word and double-word xchg is one amoswap, or an LR/SC loop without +* Zaamo. Each form uses its own pair of ordering arguments, as in +* __arch_xchg_masked() above. +*/ +#ifdef CONFIG_RISCV_ISA_ZAAMO +#define __arch_xchg (w, sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ ({ \ __asm__ __volatile__ ( \ prepend \ - " amoswap" sfx " %0, %2, %1\n" \ - append \ + " amoswap" w swap_sfx " %0, %2, %1\n" \ + swap_append \ : "=r" (r), "+A" (*(p)) \ : "r" (n) \ : "memory"); \ }) +#else +#define __arch_xchg (w, sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ +({ \ + __typeof__(*(p)) __temp; \ + __asm__ __volatile__ ( \ + prepend \ + "1: lr" w " %0, %1\n" \ + " sc" w sc_sfx " %2, %3, %1\n" \ + " bnez %2, 1b\n" \ + sc_append \ + : "=&r" (r), "+A" (*(p)), "=&r" (__temp) \ + : "r" (n) \ + : "memory"); \ +}) +#endif #define _arch_xchg (ptr, new, sc_sfx, swap_sfx, prepend, \ sc_append, swap_append) \
@@ -87,12 +110,14 @@ __ret, __ptr, __new); \break ; \case 4 : \ -__arch_xchg (".w" swap_sfx, prepend, swap_append, \ - __ret, __ptr, __new); \ +__arch_xchg (".w" , sc_sfx, swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \break ; \case 8 : \ -__arch_xchg (".d" swap_sfx, prepend, swap_append, \ - __ret, __ptr, __new); \ +__arch_xchg (".d" , sc_sfx, swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \break ; \default : \BUILD_BUG (); \
arch/riscv/include/asm/futex.h +58 -0
88 lines not on this page · raw message
arch/riscv/kernel/entry.S +7 -0
@@ -73,7 +73,14 @@ beq a2, zero, .Lnew_valid_map_cpus_restore_context /* Atomically reset the current cpu bit in new_valid_map_cpus */ +#ifdef CONFIG_RISCV_ISA_ZAAMO amoxor.d a0, a1, (a0) +#else +1: lr.d a2, (a0) + xor a2, a1, a2 + sc.d a2, a2, (a0) + bnez a2, 1b +#endif /* * A sfence.vma is required here. Even if we had Svvptc, there's no
arch/riscv/kernel/head.S +12 -0
@@ -265,7 +265,19 @@ SYM_CODE_START(_start_kernel) /* Pick one hart to run the main boot sequence */ la a3, hart_lottery li a2, 1 +#ifdef CONFIG_RISCV_ISA_ZAAMO amoadd.w a3, a2, (a3) +#else + /* + * hart_lottery is a small 32-bit counter. add is valid on RV32 + * and RV64; sc.w stores the low 32 bits. + */ +1: lr.w t0, (a3) + add t1, t0, a2 + sc.w t1, t1, (a3) + bnez t1, 1b + mv a3, t0 +#endif bnez a3, .Lsecondary_start #endif /* CONFIG_RISCV_BOOT_SPINWAIT */
Signature · 1 line
2.43.0
PATCH v7 2/6 riscv: accept the atomic extensions this kernel was built with
In reply to Vladimir Kondratievriscv_early_of_processor_hartid() rejects a hart unless its riscv,isa-extensions property contains "a". A hart that lists "zaamo" and "zalrsc" instead of "a" is rejected, and so is one that lists only "zalrsc", even when the kernel was built to use only LR/SC.
Accept "a" as before. Without it, require "zalrsc", which every kernel uses for cmpxchg, and "zaamo" when RISCV_ISA_ZAAMO is enabled. The default build therefore needs "a", or "zaamo" together with "zalrsc"; a kernel built without Zaamo needs only "zalrsc".
The legacy "riscv,isa" fallback keeps its rv{32,64}ima check.
Once such a hart can boot, hwprobe must stop telling userspace that the base behaviour is IMA regardless of the hardware: report RISCV_HWPROBE_BASE_BEHAVIOR_IMA only when every selected hart has A. A hart with Zalrsc alone still shows what it has through RISCV_HWPROBE_EXT_ZALRSC in RISCV_HWPROBE_KEY_IMA_EXT_0.
--- Documentation/arch/riscv/hwprobe.rst | 4 +++- arch/riscv/kernel/cpu.c | 23 ++++++++++++++++++++--- arch/riscv/kernel/sys_hwprobe.c | 23 ++++++++++++++++++----- 3 files changed, 41 insertions(+), 9 deletions(-)
Documentation/arch/riscv/hwprobe.rst +3 -1
@@ -61,7 +61,9 @@ The following keys are defined: * :c:macro:`RISCV_HWPROBE_BASE_BEHAVIOR_IMA`: Support for rv32ima or rv64ima, as defined by version 2.2 of the user ISA and version 1.10 of the privileged ISA, with the following known exceptions (more exceptions may be - added, but only if it can be demonstrated that the user ABI is not broken): + added, but only if it can be demonstrated that the user ABI is not broken). + This bit is clear on a hart that implements Zalrsc without Zaamo; such a + hart lists its atomic support in :c:macro:`RISCV_HWPROBE_KEY_IMA_EXT_0`. * The ``fence.i`` instruction cannot be directly executed by userspace programs (it may still be executed in userspace via a
arch/riscv/kernel/cpu.c +20 -3
@@ -82,12 +82,29 @@ int __init riscv_early_of_processor_hartid(struct device_node *node, unsigned loreturn -ENODEV ;if (of_property_match_string (node,"riscv,isa-extensions" ,"i" ) <0 || -of_property_match_string (node,"riscv,isa-extensions" ,"m" ) <0 || -of_property_match_string (node,"riscv,isa-extensions" ,"a" ) <0 ) { -pr_warn ("CPU with hartid=%lu does not support ima" , *hart); +of_property_match_string (node,"riscv,isa-extensions" ,"m" ) <0 ) { +pr_warn ("CPU with hartid=%lu does not support im" , *hart);return -ENODEV ; } +/* +* Require the atomic extensions this kernel was built with. "a" +* covers both; otherwise LR/SC is always needed and AMO is needed +* when the kernel uses it. +*/ +if (of_property_match_string (node,"riscv,isa-extensions" ,"a" ) <0 ) { +if (of_property_match_string (node,"riscv,isa-extensions" ,"zalrsc" ) <0 ) { +pr_warn ("CPU with hartid=%lu does not support zalrsc" , *hart); +return -ENODEV ; + } + +if (IS_ENABLED (CONFIG_RISCV_ISA_ZAAMO ) && +of_property_match_string (node,"riscv,isa-extensions" ,"zaamo" ) <0 ) { +pr_warn ("CPU with hartid=%lu does not support zaamo" , *hart); +return -ENODEV ; + } + } +return 0 ; old_interface:
arch/riscv/kernel/sys_hwprobe.c +18 -5
@@ -213,6 +213,18 @@ static void hwprobe_isa_ext1(struct riscv_hwprobe *pair, pair->value &= ~missing; } +static bool hwprobe_all_cpus_have (const struct cpumask *cpus,unsigned long ext) +{ +int cpu; + +for_each_cpu (cpu, cpus) { +if (!__riscv_isa_extension_available (hart_isa[cpu].isa, ext)) +return false; + } + +return true; +} +static bool hwprobe_ext0_has (const struct cpumask *cpus,u64 ext) {struct riscv_hwprobe pair;
@@ -306,13 +318,14 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pair,hwprobe_arch_id (pair, cpus);break ;/* -* The kernel already assumes that the base single-letter ISA -* extensions are supported on all harts, and only supports the -* IMA base, so just cheat a bit here and tell that to -* userspace. +* The kernel requires I and M on every hart, so IMA holds when +* every selected hart has A as well. A hart that lists only +* Zalrsc boots on a kernel built without Zaamo, but it has no +* AMO instructions and must not be reported as IMA. */ case RISCV_HWPROBE_KEY_BASE_BEHAVIOR : - pair->value =RISCV_HWPROBE_BASE_BEHAVIOR_IMA ; +if (hwprobe_all_cpus_have (cpus,RISCV_ISA_EXT_A )) + pair->value =RISCV_HWPROBE_BASE_BEHAVIOR_IMA ;break ;case RISCV_HWPROBE_KEY_IMA_EXT_0 :
Signature · 1 line
2.43.0
PATCH v7 3/6 riscv: report the A extension when Zaamo and Zalrsc are both present
In reply to Vladimir KondratievA hart may describe its atomics as "a", or as "zaamo" and "zalrsc". The two are the same thing: the A extension is defined as Zaamo plus Zalrsc. The kernel already expands "a" into both sub-extensions, but not the other way round, so a hart that lists the sub-extensions ends up without RISCV_ISA_EXT_A. /proc/cpuinfo then shows rv64imfdc... and the A bit is missing from ELF hwcap, although the hardware is the same as one that lists "a".
Set A in the source ISA bitmap when both Zaamo and Zalrsc are there, before the bitmap is resolved, so that hwcap and cpuinfo do not depend on how the firmware spelled it. This covers the "riscv,isa" string and ACPI as well as "riscv,isa-extensions".
--- arch/riscv/kernel/cpufeature.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/riscv/kernel/cpufeature.c +9 -0
@@ -682,6 +682,15 @@ static void __init riscv_resolve_isa(unsigned long *source_isa,int max_loop_count = riscv_isa_ext_count, ret;unsigned int bit; +/* +* "a" is exactly Zaamo plus Zalrsc. A hart that lists both has A, +* however the firmware spelled it, so report it the same way in +* ELF hwcap and /proc/cpuinfo. +*/ +if (test_bit (RISCV_ISA_EXT_ZAAMO , source_isa) && +test_bit (RISCV_ISA_EXT_ZALRSC , source_isa)) +set_bit (RISCV_ISA_EXT_A , source_isa); + do { loop = false; if (max_loop_count-- < 0) {
Signature · 1 line
2.43.0
PATCH v7 4/6 riscv, bpf: emit LR/SC loops when Zaamo is disabled
In reply to Vladimir KondratievA kernel built with CONFIG_RISCV_ISA_ZAAMO disabled is meant for a hart where AMO instructions are emulated by an M-mode trap. The BPF JIT still encodes amo* instructions for the BPF atomic read-modify-write operations, so every such operation takes that trap.
Emit an LR/SC loop for BPF_ADD, BPF_AND, BPF_OR, BPF_XOR, their BPF_FETCH forms and BPF_XCHG when Zaamo is disabled, on RV32 and RV64. The sequences follow the Zalrsc variants in arch/riscv/include/asm/atomic.h and the existing LR/SC loop of emit_cmpxchg(): a relaxed op discards the old value; a fetch op is fully ordered with a plain lr, sc.rl and a trailing fence rw,rw, and returns the old value. BPF_CMPXCHG is unchanged; it already uses amocas with Zacas and an LR/SC loop otherwise.
The JIT encodes instructions as integers, so no assembler constraint applies and both paths stay compiled. The choice is a plain IS_ENABLED() test, the same compile-time decision the rest of the kernel makes. The hart advertises "a" either way, so a runtime extension check would not tell the two cases apart.
An arena (BPF_PROBE_ATOMIC) access needs an exception table entry for each instruction that may fault. The verifier reserves one entry per atomic instruction, but an LR/SC loop can fault on either the lr or the sc. Count the extra entries once the instruction offsets converge, before the image is allocated, and add an entry for the sc that shares the fixup of the lr, so a fault skips the retry and the result move.
--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp32.c | 84 +++++++++++++++++++++++++++++-- arch/riscv/net/bpf_jit_comp64.c | 109 +++++++++++++++++++++++++++++++++++++++- arch/riscv/net/bpf_jit_core.c | 29 +++++++++++ 4 files changed, 217 insertions(+), 6 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
@@ -79,6 +79,7 @@ struct rv_jit_context {int nexentries;int ex_insn_off;int ex_jmp_off; +int ex_sc_off;/* sc in an lr/sc pair; 0 if unused */ unsigned long flags;int stack_size;int tcc_offset;
arch/riscv/net/bpf_jit_comp32.c +80 -4
97 lines not on this page · raw message
arch/riscv/net/bpf_jit_comp64.c +107 -2
133 lines not on this page · raw message
arch/riscv/net/bpf_jit_core.c +29 -0
@@ -16,6 +16,34 @@/* Number of iterations to try until offsets converge. */ #define NR_JIT_ITERATIONS 32 +/* +* The verifier reserves one extable slot per BPF_PROBE_ATOMIC. Without +* Zaamo a read-modify-write atomic is an lr/sc loop that can fault on +* either instruction, so reserve a second slot for the sc. +*/ +static void rv_jit_add_lrsc_exentries (struct bpf_prog *prog) +{ +const struct bpf_insn *insn = prog->insnsi; +int i; + +if (IS_ENABLED (CONFIG_RISCV_ISA_ZAAMO ) || __riscv_xlen !=64 ) +return ; + +for (i =0 ; i < prog->len; i++, insn++) { +if (BPF_CLASS (insn->code) !=BPF_STX || +BPF_MODE (insn->code) !=BPF_PROBE_ATOMIC || +bpf_atomic_is_load_store (insn)) +continue ; +if (BPF_SIZE (insn->code) !=BPF_W && +BPF_SIZE (insn->code) !=BPF_DW ) +continue ; +/* amocas is a single faulting instruction. */ +if (insn->imm ==BPF_CMPXCHG &&rv_ext_enabled (ZACAS )) +continue ; + prog->aux->num_exentries++; + } +} +static int build_body (struct rv_jit_context *ctx,bool extra_pass,int *offset) {const struct bpf_prog *prog = ctx->prog;
@@ -100,6 +128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prif (ctx->ninsns == prev_ninsns) {if (jit_data->header)break ; +rv_jit_add_lrsc_exentries (prog);/* obtain the actual image size */ extable_size = prog->aux->num_exentries *sizeof (struct exception_table_entry );
Signature · 1 line
2.43.0
PATCH v7 5/6 riscv: hwprobe: update the vDSO data after the boot-time speed probes
In reply to Vladimir Kondratievinit_hwprobe_vdso_data() fills the vDSO copy of the hwprobe answers at arch_initcall_sync. The misaligned access speed probes run later, at fs_initcall_sync, so the vDSO keeps RISCV_HWPROBE_MISALIGNED_*_UNKNOWN for RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF and _VECTOR_PERF, and every hwprobe() call that the vDSO answers itself, which is the common "all CPUs" case, returns UNKNOWN where the syscall would return the measured value.
This was handled by filling the vDSO data lazily on the first syscall, which commit 7e14e42edfaa ("Revert "riscv: hwprobe: Fix stale vDSO data for late-initialized keys at boot"") removed once the probes became synchronous; the ordering between the two initcalls was left as it was.
Split the filling out of the initcall into riscv_hwprobe_update_vdso_data() and call it again at the end of check_unaligned_access_all_cpus(). That still runs before rootfs_initcall() enables usermode helpers, so no userspace can read the vDSO data before it is complete.
--- arch/riscv/include/asm/hwprobe.h | 6 ++++++ arch/riscv/kernel/sys_hwprobe.c | 16 +++++++++++----- arch/riscv/kernel/unaligned_access_speed.c | 8 +++++--- 3 files changed, 22 insertions(+), 8 deletions(-)
arch/riscv/include/asm/hwprobe.h +6 -0
@@ -43,4 +43,10 @@ static inline bool riscv_hwprobe_pair_cmp(struct riscv_hwprobe *pair, return pair->value == other_pair->value; } +#ifdef CONFIG_MMU +void riscv_hwprobe_update_vdso_data (void ); +#else +static inline void riscv_hwprobe_update_vdso_data (void ) {} +#endif +#endif
arch/riscv/kernel/sys_hwprobe.c +11 -5
@@ -534,17 +534,18 @@ static int do_riscv_hwprobe(struct riscv_hwprobe __user *pairs,#ifdef CONFIG_MMU -static int __init init_hwprobe_vdso_data (void ) +/* +* Fill the vDSO data with the answers for the "all CPUs" case, to save a +* syscall in the common case. Called once the ISA is known, and again by +* the boot-time probes whose results feed some of the keys. +*/ +void __init riscv_hwprobe_update_vdso_data (void ) {struct vdso_arch_data *avd = vdso_k_arch_data;u64 id_bitsmash =0 ;struct riscv_hwprobe pair;int key; -/* -* Initialize vDSO data with the answers for the "all CPUs" case, to -* save a syscall in the common case. -*/ for (key =0 ; key <=RISCV_HWPROBE_MAX_KEY ; key++) { pair.key = key;hwprobe_one_pair (&pair, cpu_online_mask);
@@ -568,6 +569,11 @@ static int __init init_hwprobe_vdso_data(void)* vDSO should defer to the kernel for exotic cpu masks. */ avd->homogeneous_cpus = id_bitsmash !=0 && id_bitsmash !=-1 ; +} + +static int __init init_hwprobe_vdso_data (void ) +{ +riscv_hwprobe_update_vdso_data ();return 0 ; }
arch/riscv/kernel/unaligned_access_speed.c +5 -3
@@ -393,13 +393,15 @@ static int __init check_unaligned_access_all_cpus(void)modify_unaligned_access_branches (cpu_online_mask);cpus_read_unlock (); +/* The vDSO was filled before the probes ran; give it the results. */ +riscv_hwprobe_update_vdso_data (); +return 0 ; }/* * Run after clocksource_done_booting() so measure_cycles() uses a stable -* clocksource, but before rootfs_initcall() enables usermode helpers. Those -* helpers can reach hwprobe and populate the vDSO cache, so async hwprobe -* probes must be registered first. +* clocksource, but before rootfs_initcall() enables usermode helpers, which +* can reach hwprobe and must see the results in the vDSO data. */ fs_initcall_sync (check_unaligned_access_all_cpus);
Signature · 1 line
2.43.0
PATCH v7 6/6 riscv: hwprobe: report whether AMO is faster than LR/SC
In reply to Vladimir KondratievOn a CPU that traps AMO instructions into firmware and emulates them there, such as MIPS P8700, an uncontended AMO costs about 600 cycles where the equivalent LR/SC sequence costs about 40. Userspace built for the A extension uses AMO for every atomic and pays that on every one. It cannot tell from the ISA string: such a CPU still has Zaamo, and correctly so.
Give userspace the same kind of hint the kernel gives for misaligned access. At boot, time AMO_SPEED_ITERS amoadd.w against the same number of lr.w/addi/sc.w loops on each CPU, take the best sample from an 8 ms window of each, and report the result through a new hwprobe key, RISCV_HWPROBE_KEY_AMO_PERF:
RISCV_HWPROBE_AMO_FAST AMO is at least as fast as LR/SC RISCV_HWPROBE_AMO_SLOW LR/SC is faster; use it where possible RISCV_HWPROBE_AMO_UNSUPPORTED the CPU has no Zaamo RISCV_HWPROBE_AMO_UNKNOWN not measured, or differs across CPUs
Only the direction of the difference is reported, not its size: the question a program can act on is which sequence to pick. Each CPU logs its ratio, and a kernel whose own atomics are AMO warns once when the CPU turns out to be slow, pointing at RISCV_ISA_ZAAMO.
The amoadd.w is spelled with .insn through a new AMOADD_W() in insn-def.h, so the file assembles when -march has no Zaamo, and the probe is skipped on a CPU whose ISA lacks the extension. CPUs that come online later are measured by a hotplug callback. amo_speed={slow|fast} on the command line skips the measurement, as unaligned_scalar_speed= does; a CPU without Zaamo is reported unsupported either way. The probe runs at the same initcall level as the misaligned access probe, for the same reasons, and refreshes the vDSO hwprobe data when done.
Tested on the eyeq7h hardware, with the DT advertising "a":
[ 0.163714] cpu10: AMO speed is 0.07x LR/SC speed (slow) [ 0.163727] cpu11: AMO speed is 0.07x LR/SC speed (slow) [ 0.163724] cpu6: AMO speed is 0.07x LR/SC speed (slow)
and, with CONFIG_RISCV_ISA_ZAAMO set:
[ 0.163729] AMO is slow but kernel atomics use it, consider disabling RISCV_ISA_ZAAMO
--- Documentation/admin-guide/kernel-parameters.txt | 7 + Documentation/arch/riscv/hwprobe.rst | 18 +++ arch/riscv/Kconfig | 13 ++ arch/riscv/include/asm/cpufeature.h | 4 + arch/riscv/include/asm/hwprobe.h | 2 +- arch/riscv/include/asm/insn-def.h | 5 + arch/riscv/include/uapi/asm/hwprobe.h | 5 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/amo_speed.c | 183 ++++++++++++++++++++++++ arch/riscv/kernel/sys_hwprobe.c | 37 +++++ 10 files changed, 274 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -513,6 +513,13 @@ Kernel parameters Format: <a>,<b> See also Documentation/input/joydev/joystick.rst + amo_speed= [RISCV] + Format: {slow | fast} + Allow skipping the AMO instruction speed test. This is + useful for testing alternative code paths and to skip + the test in environments where it runs too slowly. All + CPUs must have the same AMO speed. + analog.map= [HW,JOY] Analog joystick and gamepad support Specifies type or capabilities of an analog joystick connected to one of 16 gameports
Documentation/arch/riscv/hwprobe.rst +18 -0
@@ -425,3 +425,21 @@ The following keys are defined: * :c:macro:`RISCV_HWPROBE_EXT_ZA64RS`: The Za64rs extension is supported, as defined in the RISC-V Profiles specification starting from commit b1d80660 ("Updated to ratified state.") + +* :c:macro:`RISCV_HWPROBE_KEY_AMO_PERF`: An enum value describing the + performance of the AMO instructions of the Zaamo extension, relative to + the equivalent LR/SC sequence, on the selected set of processors. + + * :c:macro:`RISCV_HWPROBE_AMO_UNKNOWN`: The performance of AMO instructions + is unknown. + + * :c:macro:`RISCV_HWPROBE_AMO_SLOW`: An uncontended AMO instruction takes + longer than the equivalent LR/SC sequence. This is the case when AMO + instructions are trapped and emulated by firmware. Code that can choose + should use LR/SC. + + * :c:macro:`RISCV_HWPROBE_AMO_FAST`: An uncontended AMO instruction is at + least as fast as the equivalent LR/SC sequence. + + * :c:macro:`RISCV_HWPROBE_AMO_UNSUPPORTED`: AMO instructions are not + supported, the Zaamo extension is absent.
arch/riscv/Kconfig +13 -0
@@ -764,6 +764,19 @@ config RISCV_ISA_ZAAMO If you don't know what to do here, say Y. +config RISCV_PROBE_AMO_SPEED + bool "Probe AMO instruction speed at boot" + default y + help + During boot, time an uncontended AMO instruction against the + equivalent LR/SC sequence on every CPU, and report which one is + faster to userspace through the hwprobe syscall, as + RISCV_HWPROBE_KEY_AMO_PERF. On a CPU that traps AMO instructions + into firmware and emulates them there, LR/SC is much faster, and + code that can choose should use it. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZBB bool default y
arch/riscv/include/asm/cpufeature.h +4 -0
@@ -83,6 +83,10 @@ static inline bool unaligned_ctl_available(void) }#endif +#ifdef CONFIG_RISCV_PROBE_AMO_SPEED +DECLARE_PER_CPU (long, amo_speed); +#endif +#if defined(CONFIG_RISCV_MISALIGNED )DECLARE_PER_CPU (long, misaligned_access_speed);bool misaligned_traps_can_delegate (void );
arch/riscv/include/asm/hwprobe.h +1 -1
@@ -8,7 +8,7 @@#include <uapi/asm/hwprobe.h> -#define RISCV_HWPROBE_MAX_KEY 16 +#define RISCV_HWPROBE_MAX_KEY 17static inline bool riscv_hwprobe_key_is_valid (__s64 key) {
arch/riscv/include/asm/insn-def.h +5 -0
@@ -219,6 +219,11 @@ ALTERNATIVE("nop", "fence rw, rw", SIFIVE_VENDOR_ID, \ __ASM_STR(.error "hlv.d requires 64-bit support")#endif +/* Spelled out so that it assembles when -march lacks Zaamo */ +#define AMOADD_W (dest, addr, val) \ + INSN_R(OPCODE_AMO, FUNC3(2), FUNC7(0), \ + RD(dest), RS1(addr), RS2(val)) +#define LB_AQ (dest, addr) \ INSN_R(OPCODE_AMO, FUNC3(0), FUNC7(26), \ RD(dest), RS1(addr), __RS2(0))
arch/riscv/include/uapi/asm/hwprobe.h +5 -0
@@ -122,6 +122,11 @@ struct riscv_hwprobe {#define RISCV_HWPROBE_EXT_ZICCIF _BITULL(3)#define RISCV_HWPROBE_EXT_ZICCRSE _BITULL(4)#define RISCV_HWPROBE_EXT_ZA64RS _BITULL(5) +#define RISCV_HWPROBE_KEY_AMO_PERF 17 +#define RISCV_HWPROBE_AMO_UNKNOWN 0 +#define RISCV_HWPROBE_AMO_SLOW 1 +#define RISCV_HWPROBE_AMO_FAST 2 +#define RISCV_HWPROBE_AMO_UNSUPPORTED 3/* Increase RISCV_HWPROBE_MAX_KEY when adding items. */
arch/riscv/kernel/Makefile +1 -0
@@ -79,6 +79,7 @@ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o obj-$(CONFIG_RISCV_MISALIGNED) += unaligned_access_speed.o obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) += copy-unaligned.o obj-$(CONFIG_RISCV_PROBE_VECTOR_UNALIGNED_ACCESS) += vec-copy-unaligned.o +obj-$(CONFIG_RISCV_PROBE_AMO_SPEED) += amo_speed.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_FPU) += kernel_mode_fpu.o
arch/riscv/kernel/amo_speed.c new file +183 -0
183 lines not on this page · raw message
arch/riscv/kernel/sys_hwprobe.c +37 -0
@@ -269,6 +269,39 @@ static u64 hwprobe_misaligned(const struct cpumask *cpus) }#endif +#ifdef CONFIG_RISCV_PROBE_AMO_SPEED +static u64 hwprobe_amo (const struct cpumask *cpus) +{ +int cpu; +u64 perf =-1ULL ; + +for_each_cpu (cpu, cpus) { +int this_perf =per_cpu (amo_speed, cpu); + +if (perf ==-1ULL ) + perf = this_perf; + +if (perf != this_perf) { + perf =RISCV_HWPROBE_AMO_UNKNOWN ; +break ; + } + } + +if (perf ==-1ULL ) +return RISCV_HWPROBE_AMO_UNKNOWN ; + +return perf; +} +#else +static u64 hwprobe_amo (const struct cpumask *cpus) +{ +if (!hwprobe_ext0_has (cpus,RISCV_HWPROBE_EXT_ZAAMO )) +return RISCV_HWPROBE_AMO_UNSUPPORTED ; + +return RISCV_HWPROBE_AMO_UNKNOWN ; +} +#endif +#ifdef CONFIG_RISCV_VECTOR_MISALIGNED static u64 hwprobe_vec_misaligned (const struct cpumask *cpus) {
@@ -345,6 +378,10 @@ static void hwprobe_one_pair(struct riscv_hwprobe *pair, pair->value =hwprobe_vec_misaligned (cpus);break ; +case RISCV_HWPROBE_KEY_AMO_PERF : + pair->value =hwprobe_amo (cpus); +break ; +case RISCV_HWPROBE_KEY_ZICBOZ_BLOCK_SIZE :if (hwprobe_ext0_has (cpus,RISCV_HWPROBE_EXT_ZICBOZ )) pair->value = riscv_cboz_block_size;
Signature · 1 line
2.43.0