membarrier fixes
Subsystems (5)
Andy Lutomirski
maintainerv3 cover membarrier fixes
Various membarrier fixes.
Changes from v2:
- Added reviewed-bys
- Don't rseq_preempt the caller (Mathieu)
- Fix single-thread short circuit (Mathieu)
Changes from v1:
- patch 1: comment fixes from Mathieu
- patch 2: improved comments
- patch 3: split out as a separate patch
- patch 4: now has a proper explanation
Mathieu, I think we have to make sync_core sync the caller. See patch 4.
Andy Lutomirski (4):
x86/membarrier: Get rid of a dubious optimization
membarrier: Add an actual barrier before rseq_preempt()
membarrier: Explicitly sync remote cores when SYNC_CORE is requested
membarrier: Execute SYNC_CORE on the calling thread
arch/x86/include/asm/sync_core.h | 9 ++-- arch/x86/mm/tlb.c | 10 ++++- kernel/sched/membarrier.c | 77 ++++++++++++++++++++++++-------- 3 files changed, 72 insertions(+), 24 deletions(-)
Signature · 1 line
2.28.0
Andy Lutomirski
maintainerPATCH v3 1/4 x86/membarrier: Get rid of a dubious optimization
sync_core_before_usermode() had an incorrect optimization. If we're in an IRQ, we can get to usermode without IRET -- we just have to schedule to a different task in the same mm and do SYSRET. Fortunately, there were no callers of sync_core_before_usermode() that could have had in_irq() or in_nmi() equal to true, because it's only ever called from the scheduler.
While we're at it, clarify a related comment.
--- arch/x86/include/asm/sync_core.h | 9 +++++---- arch/x86/mm/tlb.c | 10 ++++++++-- 2 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/include/asm/sync_core.h +5 -4
@@ -98,12 +98,13 @@ static inline void sync_core_before_usermode(void)/* With PTI, we unconditionally serialize before running user code. */ if (static_cpu_has (X86_FEATURE_PTI ))return ; +/* -* Return from interrupt and NMI is done through iret, which is core -* serializing. +* Even if we're in an interrupt, we might reschedule before returning, +* in which case we could switch to a different thread in the same mm +* and return using SYSRET or SYSEXIT. Instead of trying to keep +* track of our need to sync the core, just sync right away. */ -if (in_irq () ||in_nmi ()) -return ;sync_core (); }
arch/x86/mm/tlb.c +8 -2
@@ -474,8 +474,14 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* * The membarrier system call requires a full memory barrier and * core serialization before returning to user-space, after -* storing to rq->curr. Writing to CR3 provides that full -* memory barrier and core serializing instruction. +* storing to rq->curr, when changing mm. This is because +* membarrier() sends IPIs to all CPUs that are in the target mm +* to make them issue memory barriers. However, if another CPU +* switches to/from the target mm concurrently with +* membarrier(), it can cause that CPU not to receive an IPI +* when it really should issue a memory barrier. Writing to CR3 +* provides that full memory barrier and core serializing +* instruction. */ if (real_prev == next) { VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=
Signature · 1 line
2.28.0
Andy Lutomirski
maintainerRe: [PATCH v3 1/4] x86/membarrier: Get rid of a dubious optimization
Quoted 10 lines · Andy Lutomirski
sync_core_before_usermode()had an incorrect optimization. If we're in an IRQ, we can get to usermode without IRET -- we just have to schedule to a different task in the same mm and do SYSRET. Fortunately, there were no callers ofsync_core_before_usermode()that could have hadin_irq()orin_nmi()equal to true, because it's only ever called from the scheduler.While we're at it, clarify a related comment.
Quoted 3 lines
Reviewed-by Mathieu Desnoyers <[email protected]>Signed-off-by Andy Lutomirski <[email protected]>
[tip: x86/urgent] x86/membarrier: Get rid of a dubious optimization
In reply to Andy LutomirskiThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: a493d1ca1a03b532871f1da27f8dbda2b28b04c4 Gitweb: https://git.kernel.org/tip/a493d1ca1a03b532871f1da27f8dbda2b28b04c4 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 03 Dec 2020 21:07:03 -08:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 09 Dec 2020 09:37:42 +01:00
x86/membarrier: Get rid of a dubious optimization
sync_core_before_usermode() had an incorrect optimization. If the kernel returns from an interrupt, it can get to usermode without IRET. It just has to schedule to a different task in the same mm and do SYSRET. Fortunately, there were no callers of sync_core_before_usermode() that could have had in_irq() or in_nmi() equal to true, because it's only ever called from the scheduler.
While at it, clarify a related comment.
--- arch/x86/include/asm/sync_core.h | 9 +++++---- arch/x86/mm/tlb.c | 10 ++++++++-- 2 files changed, 13 insertions(+), 6 deletions(-)
arch/x86/include/asm/sync_core.h +5 -4
@@ -98,12 +98,13 @@ static inline void sync_core_before_usermode(void)/* With PTI, we unconditionally serialize before running user code. */ if (static_cpu_has (X86_FEATURE_PTI ))return ; +/* -* Return from interrupt and NMI is done through iret, which is core -* serializing. +* Even if we're in an interrupt, we might reschedule before returning, +* in which case we could switch to a different thread in the same mm +* and return using SYSRET or SYSEXIT. Instead of trying to keep +* track of our need to sync the core, just sync right away. */ -if (in_irq () ||in_nmi ()) -return ;sync_core (); }
arch/x86/mm/tlb.c +8 -2
@@ -474,8 +474,14 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* * The membarrier system call requires a full memory barrier and * core serialization before returning to user-space, after -* storing to rq->curr. Writing to CR3 provides that full -* memory barrier and core serializing instruction. +* storing to rq->curr, when changing mm. This is because +* membarrier() sends IPIs to all CPUs that are in the target mm +* to make them issue memory barriers. However, if another CPU +* switches to/from the target mm concurrently with +* membarrier(), it can cause that CPU not to receive an IPI +* when it really should issue a memory barrier. Writing to CR3 +* provides that full memory barrier and core serializing +* instruction. */ if (real_prev == next) { VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) !=
Andy Lutomirski
maintainerPATCH v3 2/4 membarrier: Add an actual barrier before rseq_preempt()
In reply to Andy LutomirskiIt seems to me that most RSEQ membarrier users will expect any stores done before the membarrier() syscall to be visible to the target task(s). While this is extremely likely to be true in practice, nothing actually guarantees it by a strict reading of the x86 manuals. Rather than providing this guarantee by accident and potentially causing a problem down the road, just add an explicit barrier.
--- kernel/sched/membarrier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/membarrier.c +8 -0
@@ -168,6 +168,14 @@ static void ipi_mb(void *info)static void ipi_rseq (void *info) { +/* +* Ensure that all stores done by the calling thread are visible +* to the current task before the current task resumes. We could +* probably optimize this away on most architectures, but by the +* time we've already sent an IPI, the cost of the extra smp_mb() +* is negligible. +*/ +smp_mb ();rseq_preempt (current); }
Signature · 1 line
2.28.0
Andy Lutomirski
maintainerRe: [PATCH v3 2/4] membarrier: Add an actual barrier before rseq_preempt()
Quoted 10 lines · Andy Lutomirski
It seems to me that most RSEQ membarrier users will expect any stores done before the
membarrier()syscall to be visible to the target task(s). While this is extremely likely to be true in practice, nothing actually guarantees it by a strict reading of the x86 manuals. Rather than providing this guarantee by accident and potentially causing a problem down the road, just add an explicit barrier.
MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ")
which is new in 5.10, so it doesn't need cc:stable if it makes 5.10.
[tip: x86/urgent] membarrier: Add an actual barrier before rseq_preempt()
In reply to Andy LutomirskiThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 2ecedd7569080fd05c1a457e8af2165afecfa29f Gitweb: https://git.kernel.org/tip/2ecedd7569080fd05c1a457e8af2165afecfa29f Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 03 Dec 2020 21:07:04 -08:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 09 Dec 2020 09:37:43 +01:00
membarrier: Add an actual barrier before rseq_preempt()
It seems that most RSEQ membarrier users will expect any stores done before the membarrier() syscall to be visible to the target task(s). While this is extremely likely to be true in practice, nothing actually guarantees it by a strict reading of the x86 manuals. Rather than providing this guarantee by accident and potentially causing a problem down the road, just add an explicit barrier.
--- kernel/sched/membarrier.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/sched/membarrier.c +8 -0
@@ -40,6 +40,14 @@ static void ipi_mb(void *info)static void ipi_rseq (void *info) { +/* +* Ensure that all stores done by the calling thread are visible +* to the current task before the current task resumes. We could +* probably optimize this away on most architectures, but by the +* time we've already sent an IPI, the cost of the extra smp_mb() +* is negligible. +*/ +smp_mb ();rseq_preempt (current); }
Andy Lutomirski
maintainerRe: [tip: x86/urgent] membarrier: Add an actual barrier before rseq_preempt()
Quoted 12 lines · On Wed, Dec 9, 2020 at 12:42 AM tip-bot2 for Andy Lutomirski…
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 2ecedd7569080fd05c1a457e8af2165afecfa29f Gitweb: https://git.kernel.org/tip/2ecedd7569080fd05c1a457e8af2165afecfa29f Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 03 Dec 2020 21:07:04 -08:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 09 Dec 2020 09:37:43 +01:00membarrier: Add an actual barrier before
rseq_preempt()
Quoted 8 lines
It seems that most RSEQ membarrier users will expect any stores done before the
membarrier()syscall to be visible to the target task(s). While this is extremely likely to be true in practice, nothing actually guarantees it by a strict reading of the x86 manuals. Rather than providing this guarantee by accident and potentially causing a problem down the road, just add an explicit barrier.Fixes 70216e18e519 ("membarrier: Provide core serializing command, *_SYNC_CORE")
Whoops, this got mangled on its way to tip. This should be:
MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ")
and this patch does not need to be backported.
Andy Lutomirski
maintainerPATCH v3 3/4 membarrier: Explicitly sync remote cores when SYNC_CORE is requested
In reply to Andy Lutomirskimembarrier() does not explicitly sync_core() remote CPUs; instead, it relies on the assumption that an IPI will result in a core sync. On x86, I think this may be true in practice, but it's not architecturally reliable. In particular, the SDM and APM do not appear to guarantee that interrupt delivery is serializing. While IRET does serialize, IPI return can schedule, thereby switching to another task in the same mm that was sleeping in a syscall. The new task could then SYSRET back to usermode without ever executing IRET.
Make this more robust by explicitly calling sync_core_before_usermode() on remote cores. (This also helps people who search the kernel tree for instances of sync_core() and sync_core_before_usermode() -- one might be surprised that the core membarrier code doesn't currently show up in a such a search.)
--- kernel/sched/membarrier.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/sched/membarrier.c +18 -0
@@ -166,6 +166,23 @@ static void ipi_mb(void *info)smp_mb ();/* IPIs should be serializing but paranoid. */ } +static void ipi_sync_core (void *info) +{ +/* +* The smp_mb() in membarrier after all the IPIs is supposed to +* ensure that memory on remote CPUs that occur before the IPI +* become visible to membarrier()'s caller -- see scenario B in +* the big comment at the top of this file. +* +* A sync_core() would provide this guarantee, but +* sync_core_before_usermode() might end up being deferred until +* after membarrier()'s smp_mb(). +*/ +smp_mb ();/* IPIs should be serializing but paranoid. */ + +sync_core_before_usermode (); +} +static void ipi_rseq (void *info) {/*
@@ -301,6 +318,7 @@ static int membarrier_private_expedited(int flags, int cpu_id)if (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY ))return -EPERM ; + ipi_func = ipi_sync_core; }else if (flags ==MEMBARRIER_FLAG_RSEQ ) {if (!IS_ENABLED (CONFIG_RSEQ ))return -EINVAL ;
Signature · 1 line
2.28.0
Andy Lutomirski
maintainerRe: [PATCH v3 3/4] membarrier: Explicitly sync remote cores when SYNC_CORE is requested
Quoted 10 lines · Andy Lutomirski
membarrier()does not explicitlysync_core()remote CPUs; instead, it relies on the assumption that an IPI will result in a core sync. On x86, I think this may be true in practice, but it's not architecturally reliable. In particular, the SDM and APM do not appear to guarantee that interrupt delivery is serializing. While IRET does serialize, IPI return can schedule, thereby switching to another task in the same mm that was sleeping in a syscall. The new task could then SYSRET back to usermode without ever executing IRET.
Quoted 6 lines
Make this more robust by explicitly calling
sync_core_before_usermode()on remote cores. (This also helps people who search the kernel tree for instances ofsync_core()andsync_core_before_usermode()-- one might be surprised that the core membarrier code doesn't currently show up in a such a search.)
*_SYNC_CORE")
Quoted 3 lines
Reviewed-by Mathieu Desnoyers <[email protected]>Signed-off-by Andy Lutomirski <[email protected]>
[tip: x86/urgent] membarrier: Explicitly sync remote cores when SYNC_CORE is requested
In reply to Andy LutomirskiThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 758c9373d84168dc7d039cf85a0e920046b17b41 Gitweb: https://git.kernel.org/tip/758c9373d84168dc7d039cf85a0e920046b17b41 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 03 Dec 2020 21:07:05 -08:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 09 Dec 2020 09:37:43 +01:00
membarrier: Explicitly sync remote cores when SYNC_CORE is requested
membarrier() does not explicitly sync_core() remote CPUs; instead, it relies on the assumption that an IPI will result in a core sync. On x86, this may be true in practice, but it's not architecturally reliable. In particular, the SDM and APM do not appear to guarantee that interrupt delivery is serializing. While IRET does serialize, IPI return can schedule, thereby switching to another task in the same mm that was sleeping in a syscall. The new task could then SYSRET back to usermode without ever executing IRET.
Make this more robust by explicitly calling sync_core_before_usermode() on remote cores. (This also helps people who search the kernel tree for instances of sync_core() and sync_core_before_usermode() -- one might be surprised that the core membarrier code doesn't currently show up in a such a search.)
--- kernel/sched/membarrier.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/sched/membarrier.c +18 -0
@@ -38,6 +38,23 @@ static void ipi_mb(void *info)smp_mb ();/* IPIs should be serializing but paranoid. */ } +static void ipi_sync_core (void *info) +{ +/* +* The smp_mb() in membarrier after all the IPIs is supposed to +* ensure that memory on remote CPUs that occur before the IPI +* become visible to membarrier()'s caller -- see scenario B in +* the big comment at the top of this file. +* +* A sync_core() would provide this guarantee, but +* sync_core_before_usermode() might end up being deferred until +* after membarrier()'s smp_mb(). +*/ +smp_mb ();/* IPIs should be serializing but paranoid. */ + +sync_core_before_usermode (); +} +static void ipi_rseq (void *info) {/*
@@ -162,6 +179,7 @@ static int membarrier_private_expedited(int flags, int cpu_id)if (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY ))return -EPERM ; + ipi_func = ipi_sync_core; }else if (flags ==MEMBARRIER_FLAG_RSEQ ) {if (!IS_ENABLED (CONFIG_RSEQ ))return -EINVAL ;
Andy Lutomirski
maintainerPATCH v3 4/4 membarrier: Execute SYNC_CORE on the calling thread
In reply to Andy Lutomirskimembarrier()'s MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE is documented as syncing the core on all sibling threads but not necessarily the calling thread. This behavior is fundamentally buggy and cannot be used safely. Suppose a user program has two threads. Thread A is on CPU 0 and thread B is on CPU 1. Thread A modifies some text and calls membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE). Then thread B executes the modified code. If, at any point after membarrier() decides which CPUs to target, thread A could be preempted and replaced by thread B on CPU 0. This could even happen on exit from the membarrier() syscall. If this happens, thread B will end up running on CPU 0 without having synced.
In principle, this could be fixed by arranging for the scheduler to sync_core_before_usermode() whenever switching between two threads in the same mm if there is any possibility of a concurrent membarrier() call, but this would have considerable overhead. Instead, make membarrier() sync the calling CPU as well.
As an optimization, this avoids an extra smp_mb() in the default barrier-only mode.
--- kernel/sched/membarrier.c | 51 +++++++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 18 deletions(-)
kernel/sched/membarrier.c +33 -18
@@ -333,7 +333,8 @@ static int membarrier_private_expedited(int flags, int cpu_id)return -EPERM ; } -if (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 ) +if (flags !=MEMBARRIER_FLAG_SYNC_CORE && + (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 ))return 0 ;/*
@@ -352,8 +353,6 @@ static int membarrier_private_expedited(int flags, int cpu_id)if (cpu_id >= nr_cpu_ids || !cpu_online (cpu_id)) goto out; -if (cpu_id ==raw_smp_processor_id ()) - goto out;rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu_id)->curr);if (!p || p->mm != mm) {
@@ -368,16 +367,6 @@ static int membarrier_private_expedited(int flags, int cpu_id)for_each_online_cpu (cpu) {struct task_struct *p; -/* -* Skipping the current CPU is OK even through we can be -* migrated at any point. The current CPU, at the point -* where we read raw_smp_processor_id(), is ensured to -* be in program order with respect to the caller -* thread. Therefore, we can skip this CPU from the -* iteration. -*/ -if (cpu ==raw_smp_processor_id ()) -continue ; p =rcu_dereference (cpu_rq (cpu)->curr);if (p && p->mm == mm)__cpumask_set_cpu (cpu, tmpmask);
@@ -385,12 +374,38 @@ static int membarrier_private_expedited(int flags, int cpu_id)rcu_read_unlock (); } -preempt_disable (); -if (cpu_id >=0 ) +if (cpu_id >=0 ) { +/* +* smp_call_function_single() will call ipi_func() if cpu_id +* is the calling CPU. +*/ smp_call_function_single (cpu_id, ipi_func,NULL ,1 ); -else -smp_call_function_many (tmpmask, ipi_func,NULL ,1 ); -preempt_enable (); + }else { +/* +* For regular membarrier, we can save a few cycles by +* skipping the current cpu -- we're about to do smp_mb() +* below, and if we migrate to a different cpu, this cpu +* and the new cpu will execute a full barrier in the +* scheduler. +* +* For CORE_SYNC, we do need a barrier on the current cpu -- +* otherwise, if we are migrated and replaced by a different +* task in the same mm just before, during, or after +* membarrier, we will end up with some thread in the mm +* running without a core sync. +* +* For RSEQ, don't rseq_preempt() the caller. User code +* is not supposed to issue syscalls at all from inside an +* rseq critical section. +*/ +if (flags !=MEMBARRIER_FLAG_SYNC_CORE ) { +preempt_disable (); +smp_call_function_many (tmpmask, ipi_func,NULL , true); +preempt_enable (); + }else { +on_each_cpu_mask (tmpmask, ipi_func,NULL , true); + } + } out:if (cpu_id <0 )
Signature · 1 line
2.28.0
Re: [PATCH v3 4/4] membarrier: Execute SYNC_CORE on the calling thread
Quoted 12 lines · ----- On Dec 4, 2020, at 12:07 AM, Andy Lutomirski [email protected] wrote:
membarrier()'s
MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_COREis documented as syncing the core on all sibling threads but not necessarily the calling thread. This behavior is fundamentally buggy and cannot be used safely. Suppose a user program has two threads. Thread A is on CPU 0 and thread B is on CPU 1. Thread A modifies some text and callsmembarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE). Then thread B executes the modified code. If, at any point aftermembarrier()decides which CPUs to target, thread A could be preempted and replaced by thread B on CPU 0. This could even happen on exit from themembarrier()syscall. If this happens, thread B will end up running on CPU 0 without having synced.
Quoted 8 lines
In principle, this could be fixed by arranging for the scheduler to
sync_core_before_usermode()whenever switching between two threads in the same mm if there is any possibility of a concurrentmembarrier()call, but this would have considerable overhead. Instead, makemembarrier()sync the calling CPU as well.As an optimization, this avoids an extra
smp_mb()in the default barrier-only mode.
^ we could also add to the commit message that it avoids doing rseq preempt on the caller as well.
Other than that:
Thanks!
Mathieu
Quoted 97 lines
Signed-off-by Andy Lutomirski <[email protected]>---
kernel/sched/membarrier.c| 51 +++++++++++++++++++++++++--------------1 file changed, 33 insertions(+), 18 deletions(-)kernel/sched/membarrier.c +2
-1@@ -333,7 +333,8 @@ static int membarrier_private_expedited(int flags, int cpu_id)return -EPERM ; } -if (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 ) +if (flags !=MEMBARRIER_FLAG_SYNC_CORE && + (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 ))return 0 ;/*kernel/sched/membarrier.c +0
-2@@ -352,8 +353,6 @@ static int membarrier_private_expedited(int flags, int cpu_id)if (cpu_id >= nr_cpu_ids || !cpu_online (cpu_id)) goto out; -if (cpu_id ==raw_smp_processor_id ()) - goto out;rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu_id)->curr);if (!p || p->mm != mm) {kernel/sched/membarrier.c +0
-10@@ -368,16 +367,6 @@ static int membarrier_private_expedited(int flags, int cpu_id)for_each_online_cpu (cpu) {struct task_struct *p; -/* -* Skipping the current CPU is OK even through we can be -* migrated at any point. The current CPU, at the point -* where we read raw_smp_processor_id(), is ensured to -* be in program order with respect to the caller -* thread. Therefore, we can skip this CPU from the -* iteration. -*/ -if (cpu ==raw_smp_processor_id ()) -continue ; p =rcu_dereference (cpu_rq (cpu)->curr);if (p && p->mm == mm)__cpumask_set_cpu(cpu, tmpmask);kernel/sched/membarrier.c +31
-5@@ -385,12 +374,38 @@ static int membarrier_private_expedited(int flags, int cpu_id)rcu_read_unlock (); } -preempt_disable (); -if (cpu_id >=0 ) +if (cpu_id >=0 ) { +/* +* smp_call_function_single() will call ipi_func() if cpu_id +* is the calling CPU. +*/ smp_call_function_single (cpu_id, ipi_func,NULL ,1 ); -else -smp_call_function_many (tmpmask, ipi_func,NULL ,1 ); -preempt_enable (); + }else { +/* +* For regular membarrier, we can save a few cycles by +* skipping the current cpu -- we're about to do smp_mb() +* below, and if we migrate to a different cpu, this cpu +* and the new cpu will execute a full barrier in the +* scheduler. +* +* For CORE_SYNC, we do need a barrier on the current cpu -- +* otherwise, if we are migrated and replaced by a different +* task in the same mm just before, during, or after +* membarrier, we will end up with some thread in the mm +* running without a core sync. +* +* For RSEQ, don't rseq_preempt() the caller. User code +* is not supposed to issue syscalls at all from inside an +* rseq critical section. +*/ +if (flags !=MEMBARRIER_FLAG_SYNC_CORE ) { +preempt_disable (); +smp_call_function_many (tmpmask, ipi_func,NULL , true); +preempt_enable (); + }else { +on_each_cpu_mask (tmpmask, ipi_func,NULL , true); + } + }out:
if (cpu_id < 0)--
2.28.0
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Andy Lutomirski
maintainerRe: [PATCH v3 4/4] membarrier: Execute SYNC_CORE on the calling thread
In reply to Andy LutomirskiQuoted 13 lines · Andy Lutomirski
membarrier()'s
MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_COREis documented as syncing the core on all sibling threads but not necessarily the calling thread. This behavior is fundamentally buggy and cannot be used safely. Suppose a user program has two threads. Thread A is on CPU 0 and thread B is on CPU 1. Thread A modifies some text and callsmembarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE). Then thread B executes the modified code. If, at any point aftermembarrier()decides which CPUs to target, thread A could be preempted and replaced by thread B on CPU 0. This could even happen on exit from themembarrier()syscall. If this happens, thread B will end up running on CPU 0 without having synced.
Quoted 8 lines
In principle, this could be fixed by arranging for the scheduler to
sync_core_before_usermode()whenever switching between two threads in the same mm if there is any possibility of a concurrentmembarrier()call, but this would have considerable overhead. Instead, makemembarrier()sync the calling CPU as well.As an optimization, this avoids an extra
smp_mb()in the default barrier-only mode.
*_SYNC_CORE")
also:
kernel/sched/membarrier.c static int membarrier_private_expedited(int flags, int cpu_id) · line 391
Quoted 8 lines
kernel/sched/membarrier.c +8
-0+/* +* For regular membarrier, we can save a few cycles by +* skipping the current cpu -- we're about to do smp_mb() +* below, and if we migrate to a different cpu, this cpu +* and the new cpu will execute a full barrier in the +* scheduler. +* +* For CORE_SYNC, we do need a barrier on the current cpu --
s/CORE_SYNC/SYNC_CORE/
--Andy
[tip: x86/urgent] membarrier: Execute SYNC_CORE on the calling thread
In reply to Andy LutomirskiThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: e45cdc71d1fa5ac3a57b23acc31eb959e4f60135 Gitweb: https://git.kernel.org/tip/e45cdc71d1fa5ac3a57b23acc31eb959e4f60135 Author: Andy Lutomirski <[email protected]> AuthorDate: Thu, 03 Dec 2020 21:07:06 -08:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Wed, 09 Dec 2020 09:37:43 +01:00
membarrier: Execute SYNC_CORE on the calling thread
membarrier()'s MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE is documented as syncing the core on all sibling threads but not necessarily the calling thread. This behavior is fundamentally buggy and cannot be used safely.
Suppose a user program has two threads. Thread A is on CPU 0 and thread B is on CPU 1. Thread A modifies some text and calls membarrier(MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE).
Then thread B executes the modified code. If, at any point after membarrier() decides which CPUs to target, thread A could be preempted and replaced by thread B on CPU 0. This could even happen on exit from the membarrier() syscall. If this happens, thread B will end up running on CPU 0 without having synced.
In principle, this could be fixed by arranging for the scheduler to issue sync_core_before_usermode() whenever switching between two threads in the same mm if there is any possibility of a concurrent membarrier() call, but this would have considerable overhead. Instead, make membarrier() sync the calling CPU as well.
As an optimization, this avoids an extra smp_mb() in the default barrier-only mode and an extra rseq preempt on the caller.
--- kernel/sched/membarrier.c | 51 ++++++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 18 deletions(-)
kernel/sched/membarrier.c +33 -18
@@ -194,7 +194,8 @@ static int membarrier_private_expedited(int flags, int cpu_id)return -EPERM ; } -if (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 ) +if (flags !=MEMBARRIER_FLAG_SYNC_CORE && + (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 ))return 0 ;/*
@@ -213,8 +214,6 @@ static int membarrier_private_expedited(int flags, int cpu_id)if (cpu_id >= nr_cpu_ids || !cpu_online (cpu_id)) goto out; -if (cpu_id ==raw_smp_processor_id ()) - goto out;rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu_id)->curr);if (!p || p->mm != mm) {
@@ -229,16 +228,6 @@ static int membarrier_private_expedited(int flags, int cpu_id)for_each_online_cpu (cpu) {struct task_struct *p; -/* -* Skipping the current CPU is OK even through we can be -* migrated at any point. The current CPU, at the point -* where we read raw_smp_processor_id(), is ensured to -* be in program order with respect to the caller -* thread. Therefore, we can skip this CPU from the -* iteration. -*/ -if (cpu ==raw_smp_processor_id ()) -continue ; p =rcu_dereference (cpu_rq (cpu)->curr);if (p && p->mm == mm)__cpumask_set_cpu (cpu, tmpmask);
@@ -246,12 +235,38 @@ static int membarrier_private_expedited(int flags, int cpu_id)rcu_read_unlock (); } -preempt_disable (); -if (cpu_id >=0 ) +if (cpu_id >=0 ) { +/* +* smp_call_function_single() will call ipi_func() if cpu_id +* is the calling CPU. +*/ smp_call_function_single (cpu_id, ipi_func,NULL ,1 ); -else -smp_call_function_many (tmpmask, ipi_func,NULL ,1 ); -preempt_enable (); + }else { +/* +* For regular membarrier, we can save a few cycles by +* skipping the current cpu -- we're about to do smp_mb() +* below, and if we migrate to a different cpu, this cpu +* and the new cpu will execute a full barrier in the +* scheduler. +* +* For SYNC_CORE, we do need a barrier on the current cpu -- +* otherwise, if we are migrated and replaced by a different +* task in the same mm just before, during, or after +* membarrier, we will end up with some thread in the mm +* running without a core sync. +* +* For RSEQ, don't rseq_preempt() the caller. User code +* is not supposed to issue syscalls at all from inside an +* rseq critical section. +*/ +if (flags !=MEMBARRIER_FLAG_SYNC_CORE ) { +preempt_disable (); +smp_call_function_many (tmpmask, ipi_func,NULL , true); +preempt_enable (); + }else { +on_each_cpu_mask (tmpmask, ipi_func,NULL , true); + } + } out:if (cpu_id <0 )