Membarrier updates
Subsystems (3)
Mathieu Desnoyers
maintainercover Membarrier updates
Please find the following membarrier updates series posted for inclusion upstream.
Thanks,
Mathieu
Mathieu Desnoyers (3):
sched: fix exit_mm vs membarrier (v4)
sched: membarrier: cover kthread_use_mm (v4)
sched: membarrier: document memory ordering scenarios
include/linux/sched/mm.h | 5 ++ kernel/exit.c | 16 ++++- kernel/kthread.c | 21 ++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 147 ++++++++++++++++++++++++++++++++++++-- 5 files changed, 185 insertions(+), 5 deletions(-)
Signature · 1 line
2.17.1
Mathieu Desnoyers
maintainerPATCH 1/3 sched: fix exit_mm vs membarrier (v4)
exit_mm should issue memory barriers after user-space memory accesses, before clearing current->mm, to order user-space memory accesses performed prior to exit_mm before clearing tsk->mm, which has the effect of skipping the membarrier private expedited IPIs.
exit_mm should also update the runqueue's membarrier_state so membarrier global expedited IPIs are not sent when they are not needed.
The membarrier system call can be issued concurrently with do_exit if we have thread groups created with CLONE_VM but not CLONE_THREAD.
Here is the scenario I have in mind:
Two thread groups are created, A and B. Thread group B is created by issuing clone from group A with flag CLONE_VM set, but not CLONE_THREAD. Let's assume we have a single thread within each thread group (Thread A and Thread B).
The AFAIU we can have:
Userspace variables:
int x = 0, y = 0;
CPU 0 CPU 1 Thread A Thread B (in thread group A) (in thread group B)
x = 1barrier()
y = 1exit()exit_mm()current->mm = NULL;
r1 = load y
membarrier()
skips CPU 0 (no IPI) because its current mm is NULL
r2 = load x
BUG_ON(r1 == 1 && r2 == 0)
---
Changes since v1:
- Use smp_mb__after_spinlock rather than smp_mb.
- Document race scenario in commit message.
Changes since v2:
- Introduce membarrier_update_current_mm,
- Use membarrier_update_current_mm to update rq's membarrier_state from exit_mm.
Changes since v3:
- Disable interrupts around call to membarrier_update_current_mm, which is required to access the runqueue's fields.
--- include/linux/sched/mm.h | 5 +++++ kernel/exit.c | 16 +++++++++++++++- kernel/sched/membarrier.c | 12 ++++++++++++ 3 files changed, 32 insertions(+), 1 deletion(-)
include/linux/sched/mm.h +5 -0
@@ -370,6 +370,8 @@ static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm)extern void membarrier_exec_mmap (struct mm_struct *mm); +extern void membarrier_update_current_mm (struct mm_struct *next_mm); +#else #ifdef CONFIG_ARCH_HAS_MEMBARRIER_CALLBACKS static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
@@ -384,6 +386,9 @@ static inline void membarrier_exec_mmap(struct mm_struct *mm)static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) { } +static inline void membarrier_update_current_mm (struct mm_struct *next_mm) +{ +}#endif #endif /* _LINUX_SCHED_MM_H */
kernel/exit.c +15 -1
@@ -475,10 +475,24 @@ static void exit_mm(void)BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); +/* +* When a thread stops operating on an address space, the loop +* in membarrier_private_expedited() may not observe that +* tsk->mm, and the loop in membarrier_global_expedited() may +* not observe a MEMBARRIER_STATE_GLOBAL_EXPEDITED +* rq->membarrier_state, so those would not issue an IPI. +* Membarrier requires a memory barrier after accessing +* user-space memory, before clearing tsk->mm or the +* rq->membarrier_state. +*/ +smp_mb__after_spinlock (); +local_irq_disable (); current->mm =NULL ; -mmap_read_unlock (mm); +membarrier_update_current_mm (NULL );enter_lazy_tlb (mm, current); +local_irq_enable ();task_unlock (current); +mmap_read_unlock (mm);mm_update_next_owner (mm);mmput (mm);if (test_thread_flag (TIF_MEMDIE ))
kernel/sched/membarrier.c +12 -0
@@ -63,6 +63,18 @@ void membarrier_exec_mmap(struct mm_struct *mm)this_cpu_write (runqueues.membarrier_state,0 ); } +void membarrier_update_current_mm (struct mm_struct *next_mm) +{ +struct rq *rq =this_rq (); +int membarrier_state =0 ; + +if (next_mm) + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state) +return ; +WRITE_ONCE (rq->membarrier_state, membarrier_state); +} +static int membarrier_global_expedited (void ) {int cpu;
Signature · 1 line
2.17.1
Peter Zijlstra
maintainerRe: [PATCH 1/3] sched: fix exit_mm vs membarrier (v4)
Review
Quoted 11 lines · Mathieu Desnoyers
(quoted diff) +11
-0+void membarrier_update_current_mm(struct mm_struct *next_mm) +{ + struct rq *rq = this_rq(); + int membarrier_state = 0; + + if (next_mm) + membarrier_state = atomic_read(&next_mm->membarrier_state); + if (READ_ONCE(rq->membarrier_state) == membarrier_state) + return; + WRITE_ONCE(rq->membarrier_state, membarrier_state); +}
This is suspisioucly similar to membarrier_switch_mm().
Would something like so make sense?
---
kernel/sched/membarrier.c +1 -8
@@ -206,14 +206,7 @@ void membarrier_exec_mmap(struct mm_struvoid membarrier_update_current_mm (struct mm_struct *next_mm) { -struct rq *rq =this_rq (); -int membarrier_state =0 ; - -if (next_mm) - membarrier_state =atomic_read (&next_mm->membarrier_state); -if (READ_ONCE (rq->membarrier_state) == membarrier_state) -return ; -WRITE_ONCE (rq->membarrier_state, membarrier_state); +membarrier_switch_mm (this_rq (),NULL , next_mm); }static int membarrier_global_expedited (void )
kernel/sched/sched.h +4 -2
@@ -2645,12 +2645,14 @@ static inline void membarrier_switch_mm(struct rq *rq,struct mm_struct *prev_mm, struct mm_struct *next_mm) { - int membarrier_state; + int membarrier_state = 0;if (prev_mm == next_mm)return ; - membarrier_state =atomic_read (&next_mm->membarrier_state); +if (next_mm) + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state)return ;
Mathieu Desnoyers
maintainerRe: [PATCH 1/3] sched: fix exit_mm vs membarrier (v4)
Quoted 15 lines · ----- On Oct 20, 2020, at 10:36 AM, Peter Zijlstra [email protected] wrote:
Review
Quoted 11 lines · Mathieu Desnoyers
(quoted diff) +11
-0+void membarrier_update_current_mm(struct mm_struct *next_mm) +{ + struct rq *rq = this_rq(); + int membarrier_state = 0; + + if (next_mm) + membarrier_state = atomic_read(&next_mm->membarrier_state); + if (READ_ONCE(rq->membarrier_state) == membarrier_state) + return; + WRITE_ONCE(rq->membarrier_state, membarrier_state); +}This is suspisioucly similar to
membarrier_switch_mm().
Quoted 1 line
Would something like so make sense?
Very much yes. Do you want me to re-send the series, or you want to fold this in as you merge it ?
Thanks,
Mathieu
Quoted 40 lines
---
kernel/sched/membarrier.c +1
-8@@ -206,14 +206,7 @@ void membarrier_exec_mmap(struct mm_struvoid membarrier_update_current_mm (struct mm_struct *next_mm) { -struct rq *rq =this_rq (); -int membarrier_state =0 ; - -if (next_mm) - membarrier_state =atomic_read (&next_mm->membarrier_state); -if (READ_ONCE (rq->membarrier_state) == membarrier_state) -return ; -WRITE_ONCE (rq->membarrier_state, membarrier_state); +membarrier_switch_mm (this_rq (),NULL , next_mm); }static int membarrier_global_expedited (void )kernel/sched/sched.h +4
-2@@ -2645,12 +2645,14 @@ static inline void membarrier_switch_mm(struct rq *rq,struct mm_struct *prev_mm, struct mm_struct *next_mm) { - int membarrier_state; + int membarrier_state = 0;if (prev_mm == next_mm)return ; - membarrier_state =atomic_read (&next_mm->membarrier_state); +if (next_mm) + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state)return ;
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [PATCH 1/3] sched: fix exit_mm vs membarrier (v4)
Hi,
Quoted 57 lines · Mathieu Desnoyers
Quoted 15 lines · ----- On Oct 20, 2020, at 10:36 AM, Peter Zijlstra [email protected] wrote:
Review
Quoted 11 lines · Mathieu Desnoyers
(quoted diff) +11
-0+void membarrier_update_current_mm(struct mm_struct *next_mm) +{ + struct rq *rq = this_rq(); + int membarrier_state = 0; + + if (next_mm) + membarrier_state = atomic_read(&next_mm->membarrier_state); + if (READ_ONCE(rq->membarrier_state) == membarrier_state) + return; + WRITE_ONCE(rq->membarrier_state, membarrier_state); +}This is suspisioucly similar to
membarrier_switch_mm().Quoted 1 line
Would something like so make sense?
Very much yes. Do you want me to re-send the series, or you want to fold this in as you merge it ?
Thanks,
Mathieu
Quoted 31 lines
---
kernel/sched/membarrier.c +1
-8@@ -206,14 +206,7 @@ void membarrier_exec_mmap(struct mm_struvoid membarrier_update_current_mm (struct mm_struct *next_mm) { -struct rq *rq =this_rq (); -int membarrier_state =0 ; - -if (next_mm) - membarrier_state =atomic_read (&next_mm->membarrier_state); -if (READ_ONCE (rq->membarrier_state) == membarrier_state) -return ; -WRITE_ONCE (rq->membarrier_state, membarrier_state); +membarrier_switch_mm (this_rq (),NULL , next_mm); }static int membarrier_global_expedited (void )kernel/sched/sched.h +1
-1@@ -2645,12 +2645,14 @@ static inline void membarrier_switch_mm(struct rq *rq,struct mm_struct *prev_mm, struct mm_struct *next_mm) { - int membarrier_state; + int membarrier_state = 0;
Quoted 1 line
kernel/sched/sched.h static inline void membarrier_switch_mm(struct rq *rq, · line 2650
Quoted 1 line
kernel/sched/sched.h +0
-0if (prev_mm == next_mm)
Unless I'm missing something subtle, in exit_mm(), membarrier_update_current_mm() is called with @next_mm == NULL, and inside membarrier_update_current_mm(), membarrier_switch_mm() is called wiht @prev_mm == NULL. As a result, the branch above is taken, so membarrier_update_current_mm() becomes a nop. I think we should use the previous value of current->mm as the @prev_mm, something like below maybe?
void update_current_mm(struct mm_struct *next_mm)
{
struct mm_struct *prev_mm;
unsigned long flags;
local_irq_save(flags);
prev_mm = current->mm;
current->mm = next_mm;
membarrier_switch_mm(this_rq(), prev_mm, next_mm);
local_irq_restore(flags);
}, and replace all settings for "current->mm" in kernel with update_current_mm().
Thoughts?
Regards,
Boqun
Quoted 13 lines
Quoted 8 lines
kernel/sched/sched.h +3
-1return ; - membarrier_state =atomic_read (&next_mm->membarrier_state); +if (next_mm) + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state)return ;Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
[tip: sched/core] sched: fix exit_mm vs membarrier (v4)
In reply to Mathieu DesnoyersThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5bc78502322a5e4eef3f1b2a2813751dc6434143 Gitweb: https://git.kernel.org/tip/5bc78502322a5e4eef3f1b2a2813751dc6434143 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 20 Oct 2020 09:47:13 -04:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 29 Oct 2020 11:00:30 +01:00
sched: fix exit_mm vs membarrier (v4)
exit_mm should issue memory barriers after user-space memory accesses, before clearing current->mm, to order user-space memory accesses performed prior to exit_mm before clearing tsk->mm, which has the effect of skipping the membarrier private expedited IPIs.
exit_mm should also update the runqueue's membarrier_state so membarrier global expedited IPIs are not sent when they are not needed.
The membarrier system call can be issued concurrently with do_exit if we have thread groups created with CLONE_VM but not CLONE_THREAD.
Here is the scenario I have in mind:
Two thread groups are created, A and B. Thread group B is created by issuing clone from group A with flag CLONE_VM set, but not CLONE_THREAD. Let's assume we have a single thread within each thread group (Thread A and Thread B).
The AFAIU we can have:
Userspace variables:
int x = 0, y = 0;
CPU 0 CPU 1 Thread A Thread B (in thread group A) (in thread group B)
x = 1barrier()
y = 1exit()exit_mm()current->mm = NULL;
r1 = load y
membarrier()
skips CPU 0 (no IPI) because its current mm is NULL
r2 = load x
BUG_ON(r1 == 1 && r2 == 0)
--- include/linux/sched/mm.h | 5 +++++ kernel/exit.c | 16 +++++++++++++++- kernel/sched/membarrier.c | 12 ++++++++++++ 3 files changed, 32 insertions(+), 1 deletion(-)
include/linux/sched/mm.h +5 -0
@@ -347,6 +347,8 @@ static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm)extern void membarrier_exec_mmap (struct mm_struct *mm); +extern void membarrier_update_current_mm (struct mm_struct *next_mm); +#else #ifdef CONFIG_ARCH_HAS_MEMBARRIER_CALLBACKS static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
@@ -361,6 +363,9 @@ static inline void membarrier_exec_mmap(struct mm_struct *mm)static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) { } +static inline void membarrier_update_current_mm (struct mm_struct *next_mm) +{ +}#endif #endif /* _LINUX_SCHED_MM_H */
kernel/exit.c +15 -1
@@ -475,10 +475,24 @@ static void exit_mm(void)BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); +/* +* When a thread stops operating on an address space, the loop +* in membarrier_private_expedited() may not observe that +* tsk->mm, and the loop in membarrier_global_expedited() may +* not observe a MEMBARRIER_STATE_GLOBAL_EXPEDITED +* rq->membarrier_state, so those would not issue an IPI. +* Membarrier requires a memory barrier after accessing +* user-space memory, before clearing tsk->mm or the +* rq->membarrier_state. +*/ +smp_mb__after_spinlock (); +local_irq_disable (); current->mm =NULL ; -mmap_read_unlock (mm); +membarrier_update_current_mm (NULL );enter_lazy_tlb (mm, current); +local_irq_enable ();task_unlock (current); +mmap_read_unlock (mm);mm_update_next_owner (mm);mmput (mm);if (test_thread_flag (TIF_MEMDIE ))
kernel/sched/membarrier.c +12 -0
@@ -76,6 +76,18 @@ void membarrier_exec_mmap(struct mm_struct *mm)this_cpu_write (runqueues.membarrier_state,0 ); } +void membarrier_update_current_mm (struct mm_struct *next_mm) +{ +struct rq *rq =this_rq (); +int membarrier_state =0 ; + +if (next_mm) + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state) +return ; +WRITE_ONCE (rq->membarrier_state, membarrier_state); +} +static int membarrier_global_expedited (void ) {int cpu;
Mathieu Desnoyers
maintainerPATCH 2/3 sched: membarrier: cover kthread_use_mm (v4)
In reply to Mathieu DesnoyersAdd comments and memory barrier to kthread_use_mm and kthread_unuse_mm to allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.
Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in membarrier_global_expedited to exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.
Now that membarrier_global_expedited can IPI kthreads, the scheduler also needs to update the runqueue's membarrier_state when entering lazy TLB state.
---
Changes since v1:
- Add WARN_ON_ONCE(current->mm) in play_idle_precise (PeterZ),
- Use smp_mb__after_spinlock rather than smp_mb after task_lock.
Changes since v2:
- Update the rq's membarrier state on kthread use/unuse mm,
- The scheduler must use membarrier_switch_mm for lazy TLB case as well, now that global expedited membarrier IPIs kthreads.
Changes since v3:
- Revert back to not using membarrier_switch_mm for lazy TLB case. This is made OK by ensuring that the global expedited membarrier only skips kthreads which have a NULL mm, so it only skips kthreads which are in lazy TLB mode.
--- kernel/kthread.c | 21 +++++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 7 +++---- 3 files changed, 25 insertions(+), 4 deletions(-)
kernel/kthread.c +21 -0
@@ -1248,6 +1248,7 @@ void kthread_use_mm(struct mm_struct *mm) tsk->active_mm = mm; } tsk->mm = mm; +membarrier_update_current_mm (mm);switch_mm_irqs_off (active_mm, mm, tsk);local_irq_enable ();task_unlock (tsk);
@@ -1255,8 +1256,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();to_kthread (tsk)->oldfs =force_uaccess_begin (); }
@@ -1276,9 +1288,18 @@ void kthread_unuse_mm(struct mm_struct *mm)force_uaccess_end (to_kthread (tsk)->oldfs);task_lock (tsk); +/* +* When a kthread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock ();sync_mm_rss (mm);local_irq_disable (); tsk->mm =NULL ; +membarrier_update_current_mm (NULL );/* active_mm is still 'mm' */ enter_lazy_tlb (mm, tsk);local_irq_enable ();
kernel/sched/idle.c +1 -0
@@ -338,6 +338,7 @@ void play_idle_precise(u64 duration_ns, u64 latency_ns)WARN_ON_ONCE (!(current->flags &PF_KTHREAD ));WARN_ON_ONCE (!(current->flags &PF_NO_SETAFFINITY ));WARN_ON_ONCE (!duration_ns); +WARN_ON_ONCE (current->mm);rcu_sleep_check ();preempt_disable ();
kernel/sched/membarrier.c +3 -4
@@ -113,12 +113,11 @@ static int membarrier_global_expedited(void)continue ;/* -* Skip the CPU if it runs a kernel thread. The scheduler -* leaves the prior task mm in place as an optimization when -* scheduling a kthread. +* Skip the CPU if it runs a kernel thread which is not using +* a task mm. */ p =rcu_dereference (cpu_rq (cpu)->curr); -if (p->flags &PF_KTHREAD ) +if (!p->mm)continue ;__cpumask_set_cpu (cpu, tmpmask);
Signature · 1 line
2.17.1
[tip: sched/core] sched: membarrier: cover kthread_use_mm (v4)
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 618758ed3a4f7d790414d020b362111748ebbf9f Gitweb: https://git.kernel.org/tip/618758ed3a4f7d790414d020b362111748ebbf9f Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 20 Oct 2020 09:47:14 -04:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 29 Oct 2020 11:00:31 +01:00
sched: membarrier: cover kthread_use_mm (v4)
Add comments and memory barrier to kthread_use_mm and kthread_unuse_mm to allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.
Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in membarrier_global_expedited to exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.
Now that membarrier_global_expedited can IPI kthreads, the scheduler also needs to update the runqueue's membarrier_state when entering lazy TLB state.
--- kernel/kthread.c | 21 +++++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 7 +++---- 3 files changed, 25 insertions(+), 4 deletions(-)
kernel/kthread.c +21 -0
@@ -1248,6 +1248,7 @@ void kthread_use_mm(struct mm_struct *mm) tsk->active_mm = mm; } tsk->mm = mm; +membarrier_update_current_mm (mm);switch_mm_irqs_off (active_mm, mm, tsk);local_irq_enable ();task_unlock (tsk);
@@ -1255,8 +1256,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();to_kthread (tsk)->oldfs =force_uaccess_begin (); }
@@ -1276,9 +1288,18 @@ void kthread_unuse_mm(struct mm_struct *mm)force_uaccess_end (to_kthread (tsk)->oldfs);task_lock (tsk); +/* +* When a kthread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock ();sync_mm_rss (mm);local_irq_disable (); tsk->mm =NULL ; +membarrier_update_current_mm (NULL );/* active_mm is still 'mm' */ enter_lazy_tlb (mm, tsk);local_irq_enable ();
kernel/sched/idle.c +1 -0
@@ -338,6 +338,7 @@ void play_idle_precise(u64 duration_ns, u64 latency_ns)WARN_ON_ONCE (!(current->flags &PF_KTHREAD ));WARN_ON_ONCE (!(current->flags &PF_NO_SETAFFINITY ));WARN_ON_ONCE (!duration_ns); +WARN_ON_ONCE (current->mm);rcu_sleep_check ();preempt_disable ();
kernel/sched/membarrier.c +3 -4
@@ -126,12 +126,11 @@ static int membarrier_global_expedited(void)continue ;/* -* Skip the CPU if it runs a kernel thread. The scheduler -* leaves the prior task mm in place as an optimization when -* scheduling a kthread. +* Skip the CPU if it runs a kernel thread which is not using +* a task mm. */ p =rcu_dereference (cpu_rq (cpu)->curr); -if (p->flags &PF_KTHREAD ) +if (!p->mm)continue ;__cpumask_set_cpu (cpu, tmpmask);
Mathieu Desnoyers
maintainerPATCH 3/3 sched: membarrier: document memory ordering scenarios
In reply to Mathieu DesnoyersDocument membarrier ordering scenarios in membarrier.c. Thanks to Alan Stern for refreshing my memory. Now that I have those in mind, it seems appropriate to serialize them to comments for posterity.
--- kernel/sched/membarrier.c | 128 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+)
kernel/sched/membarrier.c +128 -0
@@ -6,6 +6,134 @@*/ #include "sched.h" +/* +* For documentation purposes, here are some membarrier ordering +* scenarios to keep in mind: +* +* A) Userspace thread execution after IPI vs membarrier's memory +* barrier before sending the IPI +* +* Userspace variables: +* +* int x = 0, y = 0; +* +* The memory barrier at the start of membarrier() on CPU0 is necessary in +* order to enforce the guarantee that any writes occurring on CPU0 before +* the membarrier() is executed will be visible to any code executing on +* CPU1 after the IPI-induced memory barrier: +* +* CPU0 CPU1 +* +* x = 1 +* membarrier(): +* a: smp_mb() +* b: send IPI IPI-induced mb +* c: smp_mb() +* r2 = y +* y = 1 +* barrier() +* r1 = x +* +* BUG_ON(r1 == 0 && r2 == 0) +* +* The write to y and load from x by CPU1 are unordered by the hardware, +* so it's possible to have "r1 = x" reordered before "y = 1" at any +* point after (b). If the memory barrier at (a) is omitted, then "x = 1" +* can be reordered after (a) (although not after (c)), so we get r1 == 0 +* and r2 == 0. This violates the guarantee that membarrier() is +* supposed by provide. +* +* The timing of the memory barrier at (a) has to ensure that it executes +* before the IPI-induced memory barrier on CPU1. +* +* B) Userspace thread execution before IPI vs membarrier's memory +* barrier after completing the IPI +* +* Userspace variables: +* +* int x = 0, y = 0; +* +* The memory barrier at the end of membarrier() on CPU0 is necessary in +* order to enforce the guarantee that any writes occurring on CPU1 before +* the membarrier() is executed will be visible to any code executing on +* CPU0 after the membarrier(): +* +* CPU0 CPU1 +* +* x = 1 +* barrier() +* y = 1 +* r2 = y +* membarrier(): +* a: smp_mb() +* b: send IPI IPI-induced mb +* c: smp_mb() +* r1 = x +* BUG_ON(r1 == 0 && r2 == 1) +* +* The writes to x and y are unordered by the hardware, so it's possible to +* have "r2 = 1" even though the write to x doesn't execute until (b). If +* the memory barrier at (c) is omitted then "r1 = x" can be reordered +* before (b) (although not before (a)), so we get "r1 = 0". This violates +* the guarantee that membarrier() is supposed to provide. +* +* The timing of the memory barrier at (c) has to ensure that it executes +* after the IPI-induced memory barrier on CPU1. +* +* C) Scheduling userspace thread -> kthread -> userspace thread vs membarrier +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* d: switch to kthread (includes mb) +* b: read rq->curr->mm == NULL +* e: switch to user (includes mb) +* c: smp_mb() +* +* Using the scenario from (A), we can show that (a) needs to be paired +* with (e). Using the scenario from (B), we can show that (c) needs to +* be paired with (d). +* +* D) exit_mm vs membarrier +* +* Two thread groups are created, A and B. Thread group B is created by +* issuing clone from group A with flag CLONE_VM set, but not CLONE_THREAD. +* Let's assume we have a single thread within each thread group (Thread A +* and Thread B). Thread A runs on CPU0, Thread B runs on CPU1. +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* exit_mm(): +* d: smp_mb() +* e: current->mm = NULL +* b: read rq->curr->mm == NULL +* c: smp_mb() +* +* Using scenario (B), we can show that (c) needs to be paired with (d). +* +* E) kthread_{use,unuse}_mm vs membarrier +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* kthread_unuse_mm() +* d: smp_mb() +* e: current->mm = NULL +* b: read rq->curr->mm == NULL +* kthread_use_mm() +* f: current->mm = mm +* g: smp_mb() +* c: smp_mb() +* +* Using the scenario from (A), we can show that (a) needs to be paired +* with (g). Using the scenario from (B), we can show that (c) needs to +* be paired with (d). +*/ +/* * Bitmask made from a "or" of all commands within enum membarrier_cmd, * except MEMBARRIER_CMD_QUERY.
Signature · 1 line
2.17.1
[tip: sched/core] sched: membarrier: document memory ordering scenarios
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 25595eb6aaa9fbb31330f1e0b400642694bc6574 Gitweb: https://git.kernel.org/tip/25595eb6aaa9fbb31330f1e0b400642694bc6574 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Tue, 20 Oct 2020 09:47:15 -04:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 29 Oct 2020 11:00:31 +01:00
sched: membarrier: document memory ordering scenarios
Document membarrier ordering scenarios in membarrier.c. Thanks to Alan Stern for refreshing my memory. Now that I have those in mind, it seems appropriate to serialize them to comments for posterity.
--- kernel/sched/membarrier.c | 128 +++++++++++++++++++++++++++++++++++++- 1 file changed, 128 insertions(+)
kernel/sched/membarrier.c +128 -0
@@ -7,6 +7,134 @@#include "sched.h" /* +* For documentation purposes, here are some membarrier ordering +* scenarios to keep in mind: +* +* A) Userspace thread execution after IPI vs membarrier's memory +* barrier before sending the IPI +* +* Userspace variables: +* +* int x = 0, y = 0; +* +* The memory barrier at the start of membarrier() on CPU0 is necessary in +* order to enforce the guarantee that any writes occurring on CPU0 before +* the membarrier() is executed will be visible to any code executing on +* CPU1 after the IPI-induced memory barrier: +* +* CPU0 CPU1 +* +* x = 1 +* membarrier(): +* a: smp_mb() +* b: send IPI IPI-induced mb +* c: smp_mb() +* r2 = y +* y = 1 +* barrier() +* r1 = x +* +* BUG_ON(r1 == 0 && r2 == 0) +* +* The write to y and load from x by CPU1 are unordered by the hardware, +* so it's possible to have "r1 = x" reordered before "y = 1" at any +* point after (b). If the memory barrier at (a) is omitted, then "x = 1" +* can be reordered after (a) (although not after (c)), so we get r1 == 0 +* and r2 == 0. This violates the guarantee that membarrier() is +* supposed by provide. +* +* The timing of the memory barrier at (a) has to ensure that it executes +* before the IPI-induced memory barrier on CPU1. +* +* B) Userspace thread execution before IPI vs membarrier's memory +* barrier after completing the IPI +* +* Userspace variables: +* +* int x = 0, y = 0; +* +* The memory barrier at the end of membarrier() on CPU0 is necessary in +* order to enforce the guarantee that any writes occurring on CPU1 before +* the membarrier() is executed will be visible to any code executing on +* CPU0 after the membarrier(): +* +* CPU0 CPU1 +* +* x = 1 +* barrier() +* y = 1 +* r2 = y +* membarrier(): +* a: smp_mb() +* b: send IPI IPI-induced mb +* c: smp_mb() +* r1 = x +* BUG_ON(r1 == 0 && r2 == 1) +* +* The writes to x and y are unordered by the hardware, so it's possible to +* have "r2 = 1" even though the write to x doesn't execute until (b). If +* the memory barrier at (c) is omitted then "r1 = x" can be reordered +* before (b) (although not before (a)), so we get "r1 = 0". This violates +* the guarantee that membarrier() is supposed to provide. +* +* The timing of the memory barrier at (c) has to ensure that it executes +* after the IPI-induced memory barrier on CPU1. +* +* C) Scheduling userspace thread -> kthread -> userspace thread vs membarrier +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* d: switch to kthread (includes mb) +* b: read rq->curr->mm == NULL +* e: switch to user (includes mb) +* c: smp_mb() +* +* Using the scenario from (A), we can show that (a) needs to be paired +* with (e). Using the scenario from (B), we can show that (c) needs to +* be paired with (d). +* +* D) exit_mm vs membarrier +* +* Two thread groups are created, A and B. Thread group B is created by +* issuing clone from group A with flag CLONE_VM set, but not CLONE_THREAD. +* Let's assume we have a single thread within each thread group (Thread A +* and Thread B). Thread A runs on CPU0, Thread B runs on CPU1. +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* exit_mm(): +* d: smp_mb() +* e: current->mm = NULL +* b: read rq->curr->mm == NULL +* c: smp_mb() +* +* Using scenario (B), we can show that (c) needs to be paired with (d). +* +* E) kthread_{use,unuse}_mm vs membarrier +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* kthread_unuse_mm() +* d: smp_mb() +* e: current->mm = NULL +* b: read rq->curr->mm == NULL +* kthread_use_mm() +* f: current->mm = mm +* g: smp_mb() +* c: smp_mb() +* +* Using the scenario from (A), we can show that (a) needs to be paired +* with (g). Using the scenario from (B), we can show that (c) needs to +* be paired with (d). +*/ + +/* * Bitmask made from a "or" of all commands within enum membarrier_cmd, * except MEMBARRIER_CMD_QUERY. */