membarrier: riscv: Core serializing command
Subsystems (4)
v4 cover membarrier: riscv: Core serializing command
Changes since v3 ([1]):
- amend documentation
Changes since v2 ([2]):
- amend inline comments
- drop ARCH_HAS_MEMBARRIER, create membarrrier.rst
Changes since v1 ([3]):
- add smp_mb() in switch_mm()
- introduce ARCH_HAS_MEMBARRIER, amend documentation
Changes since RFC ([4]):
- introduce prepare_sync_core_cmd()
- fix nosmp builds
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/[email protected]/
Andrea Parri (4):
membarrier: riscv: Add full memory barrier in switch_mm()
membarrier: Create Documentation/scheduler/membarrier.rst
locking: Introduce prepare_sync_core_cmd()
membarrier: riscv: Provide core serializing command
.../membarrier-sync-core/arch-support.txt | 18 ++++++- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/membarrier.rst | 39 +++++++++++++++ MAINTAINERS | 4 +- arch/riscv/Kconfig | 4 ++ arch/riscv/include/asm/membarrier.h | 50 +++++++++++++++++++ arch/riscv/include/asm/sync_core.h | 29 +++++++++++ arch/riscv/mm/context.c | 2 + include/linux/sync_core.h | 16 +++++- init/Kconfig | 3 ++ kernel/sched/core.c | 16 ++++-- kernel/sched/membarrier.c | 13 +++-- 12 files changed, 185 insertions(+), 10 deletions(-) create mode 100644 Documentation/scheduler/membarrier.rst create mode 100644 arch/riscv/include/asm/membarrier.h create mode 100644 arch/riscv/include/asm/sync_core.h
Signature · 1 line
2.34.1
PATCH v4 1/4 membarrier: riscv: Add full memory barrier in switch_mm()
The membarrier system call requires a full memory barrier after storing to rq->curr, before going back to user-space. The barrier is only needed when switching between processes: the barrier is implied by mmdrop() when switching from kernel to userspace, and it's not needed when switching from userspace to kernel.
Rely on the feature/mechanism ARCH_HAS_MEMBARRIER_CALLBACKS and on the primitive membarrier_arch_switch_mm(), already adopted by the PowerPC architecture, to insert the required barrier.
--- MAINTAINERS | 2 +- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/membarrier.h | 31 +++++++++++++++++++++++++++++ arch/riscv/mm/context.c | 2 ++ kernel/sched/core.c | 5 +++-- 5 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 arch/riscv/include/asm/membarrier.h
MAINTAINERS +1 -1
@@ -14040,7 +14040,7 @@ M: Mathieu Desnoyers <[email protected]> M: "Paul E. McKenney" <[email protected]> L: [email protected] S: Supported -F: arch/powerpc/include/asm/membarrier.h +F: arch/*/include/asm/membarrier.h F: include/uapi/linux/membarrier.h F: kernel/sched/membarrier.c
arch/riscv/Kconfig +1 -0
@@ -27,6 +27,7 @@ config RISCV select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV + select ARCH_HAS_MEMBARRIER_CALLBACKS select ARCH_HAS_MMIOWB select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PMEM_API
arch/riscv/include/asm/membarrier.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _ASM_RISCV_MEMBARRIER_H +#define _ASM_RISCV_MEMBARRIER_H + +static inline void membarrier_arch_switch_mm (struct mm_struct *prev, +struct mm_struct *next, +struct task_struct *tsk) +{ +/* +* Only need the full barrier when switching between processes. +* Barrier when switching from kernel to userspace is not +* required here, given that it is implied by mmdrop(). Barrier +* when switching from userspace to kernel is not needed after +* store to rq->curr. +*/ +if (IS_ENABLED (CONFIG_SMP ) && +likely (!(atomic_read (&next->membarrier_state) & + (MEMBARRIER_STATE_PRIVATE_EXPEDITED | +MEMBARRIER_STATE_GLOBAL_EXPEDITED )) || !prev)) +return ; + +/* +* The membarrier system call requires a full memory barrier +* after storing to rq->curr, before going back to user-space. +* Matches a full barrier in the proximity of the membarrier +* system call entry. +*/ +smp_mb (); +} + +#endif /* _ASM_RISCV_MEMBARRIER_H */
arch/riscv/mm/context.c +2 -0
@@ -323,6 +323,8 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,if (unlikely (prev == next))return ; +membarrier_arch_switch_mm (prev, next, task); +/* * Mark the current MM context as inactive, and the next as * active. This is at least used by the icache flushing
kernel/sched/core.c +3 -2
@@ -6709,8 +6709,9 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * Here are the schemes providing that barrier on the * various architectures: -* - mm ? switch_mm() : mmdrop() for x86, s390, sparc, PowerPC. -* switch_mm() rely on membarrier_arch_switch_mm() on PowerPC. +* - mm ? switch_mm() : mmdrop() for x86, s390, sparc, PowerPC, +* RISC-V. switch_mm() relies on membarrier_arch_switch_mm() +* on PowerPC and on RISC-V. * - finish_lock_switch() for weakly-ordered * architectures where spin_unlock is a full barrier, * - switch_to() for arm64 (weakly-ordered, spin_unlock
Signature · 1 line
2.34.1
PATCH v4 2/4 membarrier: Create Documentation/scheduler/membarrier.rst
In reply to Andrea ParriTo gather the architecture requirements of the "private/global expedited" membarrier commands. The file will be expanded to integrate further information about the membarrier syscall (as needed/desired in the future). While at it, amend some related inline comments in the membarrier codebase.
--- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/membarrier.rst | 39 ++++++++++++++++++++++++++ MAINTAINERS | 1 + kernel/sched/core.c | 7 ++++- kernel/sched/membarrier.c | 8 +++--- 5 files changed, 51 insertions(+), 5 deletions(-) create mode 100644 Documentation/scheduler/membarrier.rst
Documentation/scheduler/index.rst +1 -0
@@ -7,6 +7,7 @@ Scheduler completion + membarrier sched-arch sched-bwc sched-deadline
Documentation/scheduler/membarrier.rst new file +39 -0
@@ -0,0 +1,39 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================== +membarrier() System Call +======================== + +MEMBARRIER_CMD_{PRIVATE,GLOBAL}_EXPEDITED - Architecture requirements +===================================================================== + +Memory barriers before updating rq->curr +---------------------------------------- + +The commands MEMBARRIER_CMD_PRIVATE_EXPEDITED and MEMBARRIER_CMD_GLOBAL_EXPEDITED +require each architecture to have a full memory barrier after coming from +user-space, before updating rq->curr. This barrier is implied by the sequence +rq_lock(); smp_mb__after_spinlock() in __schedule(). The barrier matches a full +barrier in the proximity of the membarrier system call exit, cf. +membarrier_{private,global}_expedited(). + +Memory barriers after updating rq->curr +--------------------------------------- + +The commands MEMBARRIER_CMD_PRIVATE_EXPEDITED and MEMBARRIER_CMD_GLOBAL_EXPEDITED +require each architecture to have a full memory barrier after updating rq->curr, +before returning to user-space. The schemes providing this barrier on the various +architectures are as follows. + + - alpha, arc, arm, hexagon, mips rely on the full barrier implied by + spin_unlock() in finish_lock_switch(). + + - arm64 relies on the full barrier implied by switch_to(). + + - powerpc, riscv, s390, sparc, x86 rely on the full barrier implied by + switch_mm(), if mm is not NULL; they rely on the full barrier implied + by mmdrop(), otherwise. On powerpc and riscv, switch_mm() relies on + membarrier_arch_switch_mm(). + +The barrier matches a full barrier in the proximity of the membarrier system call +entry, cf. membarrier_{private,global}_expedited().
MAINTAINERS +1 -0
@@ -14040,6 +14040,7 @@ M: Mathieu Desnoyers <[email protected]> M: "Paul E. McKenney" <[email protected]> L: [email protected] S: Supported +F: Documentation/scheduler/membarrier.rst F: arch/*/include/asm/membarrier.h F: include/uapi/linux/membarrier.h F: kernel/sched/membarrier.c
kernel/sched/core.c +6 -1
@@ -6638,7 +6638,9 @@ static void __sched notrace __schedule(unsigned int sched_mode)* if (signal_pending_state()) if (p->state & @state) * * Also, the membarrier system call requires a full memory barrier -* after coming from user-space, before storing to rq->curr. +* after coming from user-space, before storing to rq->curr; this +* barrier matches a full barrier in the proximity of the membarrier +* system call exit. */ rq_lock (rq, &rf);smp_mb__after_spinlock ();
@@ -6716,6 +6718,9 @@ static void __sched notrace __schedule(unsigned int sched_mode)* architectures where spin_unlock is a full barrier, * - switch_to() for arm64 (weakly-ordered, spin_unlock * is a RELEASE barrier), +* +* The barrier matches a full barrier in the proximity of +* the membarrier system call entry. */ ++*switch_count;
kernel/sched/membarrier.c +4 -4
@@ -251,7 +251,7 @@ static int membarrier_global_expedited(void)return 0 ;/* -* Matches memory barriers around rq->curr modification in +* Matches memory barriers after rq->curr modification in * scheduler. */ smp_mb ();/* system call entry is not a mb. */
@@ -300,7 +300,7 @@ static int membarrier_global_expedited(void)/* * Memory barrier on the caller thread _after_ we finished -* waiting for the last IPI. Matches memory barriers around +* waiting for the last IPI. Matches memory barriers before * rq->curr modification in scheduler. */ smp_mb ();/* exit from system call is not a mb */
@@ -339,7 +339,7 @@ static int membarrier_private_expedited(int flags, int cpu_id)return 0 ;/* -* Matches memory barriers around rq->curr modification in +* Matches memory barriers after rq->curr modification in * scheduler. */ smp_mb ();/* system call entry is not a mb. */
@@ -415,7 +415,7 @@ static int membarrier_private_expedited(int flags, int cpu_id)/* * Memory barrier on the caller thread _after_ we finished -* waiting for the last IPI. Matches memory barriers around +* waiting for the last IPI. Matches memory barriers before * rq->curr modification in scheduler. */ smp_mb ();/* exit from system call is not a mb */
Signature · 1 line
2.34.1
PATCH v4 3/4 locking: Introduce prepare_sync_core_cmd()
In reply to Andrea ParriIntroduce an architecture function that architectures can use to set up ("prepare") SYNC_CORE commands.
The function will be used by RISC-V to update its "deferred icache- flush" data structures (icache_stale_mask).
Architectures defining prepare_sync_core_cmd() static inline need to select ARCH_HAS_PREPARE_SYNC_CORE_CMD.
--- include/linux/sync_core.h | 16 +++++++++++++++- init/Kconfig | 3 +++ kernel/sched/membarrier.c | 1 + 3 files changed, 19 insertions(+), 1 deletion(-)
include/linux/sync_core.h +15 -1
@@ -17,5 +17,19 @@ static inline void sync_core_before_usermode(void) }#endif -#endif /* _LINUX_SYNC_CORE_H */ +#ifdef CONFIG_ARCH_HAS_PREPARE_SYNC_CORE_CMD +#include <asm/sync_core.h> +#else +/* +* This is a dummy prepare_sync_core_cmd() implementation that can be used on +* all architectures which provide unconditional core serializing instructions +* in switch_mm(). +* If your architecture doesn't provide such core serializing instructions in +* switch_mm(), you may need to write your own functions. +*/ +static inline void prepare_sync_core_cmd (struct mm_struct *mm) +{ +} +#endif +#endif /* _LINUX_SYNC_CORE_H */
init/Kconfig +3 -0
@@ -1982,6 +1982,9 @@ source "kernel/Kconfig.locks" config ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE bool +config ARCH_HAS_PREPARE_SYNC_CORE_CMD + bool + config ARCH_HAS_SYNC_CORE_BEFORE_USERMODE bool
kernel/sched/membarrier.c +1 -0
@@ -320,6 +320,7 @@ static int membarrier_private_expedited(int flags, int cpu_id) MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY)) return -EPERM; ipi_func = ipi_sync_core; +prepare_sync_core_cmd (mm); }else if (flags ==MEMBARRIER_FLAG_RSEQ ) {if (!IS_ENABLED (CONFIG_RSEQ ))return -EINVAL ;
Signature · 1 line
2.34.1
PATCH v4 4/4 membarrier: riscv: Provide core serializing command
In reply to Andrea ParriRISC-V uses xRET instructions on return from interrupt and to go back to user-space; the xRET instruction is not core serializing.
Use FENCE.I for providing core serialization as follows:
- by calling sync_core_before_usermode() on return from interrupt (cf. ipi_sync_core()),
- via switch_mm() and sync_core_before_usermode() (respectively, for uthread->uthread and kthread->uthread transitions) before returning to user-space.
On RISC-V, the serialization in switch_mm() is activated by resetting the icache_stale_mask of the mm at prepare_sync_core_cmd().
--- .../membarrier-sync-core/arch-support.txt | 18 +++++++++++- MAINTAINERS | 1 + arch/riscv/Kconfig | 3 ++ arch/riscv/include/asm/membarrier.h | 19 ++++++++++++ arch/riscv/include/asm/sync_core.h | 29 +++++++++++++++++++ kernel/sched/core.c | 4 +++ kernel/sched/membarrier.c | 4 +++ 7 files changed, 77 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/sync_core.h
Documentation/features/sched/membarrier-sync-core/arch-support.txt +17 -1
@@ -10,6 +10,22 @@ # Rely on implicit context synchronization as a result of exception return # when returning from IPI handler, and when returning to user-space. # +# * riscv +# +# riscv uses xRET as return from interrupt and to return to user-space. +# +# Given that xRET is not core serializing, we rely on FENCE.I for providing +# core serialization: +# +# - by calling sync_core_before_usermode() on return from interrupt (cf. +# ipi_sync_core()), +# +# - via switch_mm() and sync_core_before_usermode() (respectively, for +# uthread->uthread and kthread->uthread transitions) before returning +# to user-space. +# +# The serialization in switch_mm() is activated by prepare_sync_core_cmd(). +# # * x86 # # x86-32 uses IRET as return from interrupt, which takes care of the IPI.
@@ -43,7 +59,7 @@ | openrisc: | TODO | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | ok | | sh: | TODO | | sparc: | TODO |
MAINTAINERS +1 -0
@@ -14042,6 +14042,7 @@ L: [email protected] S: Supported F: Documentation/scheduler/membarrier.rst F: arch/*/include/asm/membarrier.h +F: arch/*/include/asm/sync_core.h F: include/uapi/linux/membarrier.h F: kernel/sched/membarrier.c
arch/riscv/Kconfig +3 -0
@@ -28,14 +28,17 @@ config RISCV select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV select ARCH_HAS_MEMBARRIER_CALLBACKS + select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_MMIOWB select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PMEM_API + select ARCH_HAS_PREPARE_SYNC_CORE_CMD select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_DIRECT_MAP if MMU select ARCH_HAS_SET_MEMORY if MMU select ARCH_HAS_STRICT_KERNEL_RWX if MMU && !XIP_KERNEL select ARCH_HAS_STRICT_MODULE_RWX if MMU && !XIP_KERNEL + select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE select ARCH_HAS_SYSCALL_WRAPPER select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_HAS_UBSAN_SANITIZE_ALL
arch/riscv/include/asm/membarrier.h +19 -0
@@ -22,6 +22,25 @@ static inline void membarrier_arch_switch_mm(struct mm_struct *prev,/* * The membarrier system call requires a full memory barrier * after storing to rq->curr, before going back to user-space. +* +* This barrier is also needed for the SYNC_CORE command when +* switching between processes; in particular, on a transition +* from a thread belonging to another mm to a thread belonging +* to the mm for which a membarrier SYNC_CORE is done on CPU0: +* +* - [CPU0] sets all bits in the mm icache_stale_mask (in +* prepare_sync_core_cmd()); +* +* - [CPU1] stores to rq->curr (by the scheduler); +* +* - [CPU0] loads rq->curr within membarrier and observes +* cpu_rq(1)->curr->mm != mm, so the IPI is skipped on +* CPU1; this means membarrier relies on switch_mm() to +* issue the sync-core; +* +* - [CPU1] switch_mm() loads icache_stale_mask; if the bit +* is zero, switch_mm() may incorrectly skip the sync-core. +* * Matches a full barrier in the proximity of the membarrier * system call entry. */
arch/riscv/include/asm/sync_core.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RISCV_SYNC_CORE_H +#define _ASM_RISCV_SYNC_CORE_H + +/* +* RISC-V implements return to user-space through an xRET instruction, +* which is not core serializing. +*/ +static inline void sync_core_before_usermode (void ) +{ + asmvolatile ("fence.i" :::"memory" ); +} + +#ifdef CONFIG_SMP +/* +* Ensure the next switch_mm() on every CPU issues a core serializing +* instruction for the given @mm. +*/ +static inline void prepare_sync_core_cmd (struct mm_struct *mm) +{ +cpumask_setall (&mm->context.icache_stale_mask); +} +#else +static inline void prepare_sync_core_cmd (struct mm_struct *mm) +{ +} +#endif /* CONFIG_SMP */ + +#endif /* _ASM_RISCV_SYNC_CORE_H */
kernel/sched/core.c +4 -0
@@ -6721,6 +6721,10 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * The barrier matches a full barrier in the proximity of * the membarrier system call entry. +* +* On RISC-V, this barrier pairing is also needed for the +* SYNC_CORE command when switching between processes, cf. +* the inline comments in membarrier_arch_switch_mm(). */ ++*switch_count;
kernel/sched/membarrier.c +4 -0
@@ -342,6 +342,10 @@ static int membarrier_private_expedited(int flags, int cpu_id)/* * Matches memory barriers after rq->curr modification in * scheduler. +* +* On RISC-V, this barrier pairing is also needed for the +* SYNC_CORE command when switching between processes, cf. +* the inline comments in membarrier_arch_switch_mm(). */ smp_mb ();/* system call entry is not a mb. */
Signature · 1 line
2.34.1
Re: [PATCH v4 4/4] membarrier: riscv: Provide core serializing command
Quoted 9 lines · Andrea Parri
RISC-V uses xRET instructions on return from interrupt and to go back to user-space; the xRET instruction is not core serializing.
Use FENCE.I for providing core serialization as follows:
- by calling
sync_core_before_usermode()on return from interrupt (cf.ipi_sync_core()),- via
switch_mm()andsync_core_before_usermode()(respectively, for
Quoted 8 lines
uthread->uthread and kthread->uthread transitions) before returning to user-space.On RISC-V, the serialization in
switch_mm()is activated by resetting theicache_stale_maskof the mm atprepare_sync_core_cmd().Suggested-by Palmer Dabbelt <[email protected]>Signed-off-by Andrea Parri <[email protected]>
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Palmer Dabbelt
maintainerQuoted 26 lines · On Wed, 31 Jan 2024 06:49:32 PST (-0800), [email protected] wrote:
Changes since v3 ([1]):
- amend documentationChanges since v2 ([2]):
- amend inline comments
- dropARCH_HAS_MEMBARRIER, createmembarrrier.rstChanges since v1 ([3]):
- addsmp_mb()inswitch_mm()
- introduceARCH_HAS_MEMBARRIER, amend documentationChanges since RFC ([4]):
- introduceprepare_sync_core_cmd()
- fix nosmp builds[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/[email protected]/Andrea Parri (4):
membarrier: riscv: Add full memory barrier inswitch_mm()
membarrier: CreateDocumentation/scheduler/membarrier.rst
locking: Introduceprepare_sync_core_cmd()
membarrier: riscv: Provide core serializing command
Quoted 16 lines
.../membarrier-sync-core/arch-support.txt | 18 ++++++- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/membarrier.rst | 39 +++++++++++++++ MAINTAINERS | 4 +- arch/riscv/Kconfig | 4 ++ arch/riscv/include/asm/membarrier.h | 50 +++++++++++++++++++ arch/riscv/include/asm/sync_core.h | 29 +++++++++++ arch/riscv/mm/context.c | 2 + include/linux/sync_core.h | 16 +++++- init/Kconfig | 3 ++ kernel/sched/core.c | 16 ++++-- kernel/sched/membarrier.c | 13 +++-- 12 files changed, 185 insertions(+), 10 deletions(-) create mode 100644 Documentation/scheduler/membarrier.rst create mode 100644 arch/riscv/include/asm/membarrier.h create mode 100644 arch/riscv/include/asm/sync_core.h
I picked these up into my staging tree for the tester this morning, there's a few hiccups from my latest master merge but they don't seem super exciting. So it might take a bit for these to show up in my for-next, but they're headed in that direction.
LMK if you guys were aiming for some other tree, or want to do some sort of shared tag for them. Otherwise they'll show up in linux-next soon.
Thanks!
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:
Hello:
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:
Quoted 8 lines · On Wed, 31 Jan 2024 15:49:32 +0100 you wrote:
Changes since v3 ([1]):
- amend documentationChanges since v2 ([2]):
- amend inline comments
- dropARCH_HAS_MEMBARRIER, createmembarrrier.rst[...]
Here is the summary with links:
- [v4,1/4] membarrier: riscv: Add full memory barrier in switch_mm()
https://git.kernel.org/riscv/c/d6cfd1770f20 - [v4,2/4] membarrier: Create Documentation/scheduler/membarrier.rst https://git.kernel.org/riscv/c/a14d11a0f5f4 - [v4,3/4] locking: Introduce prepare_sync_core_cmd() https://git.kernel.org/riscv/c/4ff4c745a16c - [v4,4/4] membarrier: riscv: Provide core serializing command https://git.kernel.org/riscv/c/cd9b29014dc6
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html