sched: membarrier updates
Subsystems (3)
Mathieu Desnoyers
maintainercover sched: membarrier updates
Hi,
Please find those three changes based on our recent discussions about interaction between membarrier and exit_mm and kthread_use_mm. I added documentation of memory ordering scenarios to membarrier.c as well.
Thanks,
Mathieu
Mathieu Desnoyers (3):
sched: fix exit_mm vs membarrier (v2)
sched: membarrier: cover kthread_use_mm (v2)
sched: membarrier: document memory ordering scenarios
kernel/exit.c | 8 +++ kernel/kthread.c | 19 +++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 136 ++++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 158 insertions(+), 6 deletions(-)
Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRFC PATCH 1/3 sched: fix exit_mm vs membarrier (v2)
exit_mm should issue memory barriers after user-space memory accesses, before clearing current->mm, to order user-space memory accesses performed prior to exit_mm before clearing tsk->mm, which has the effect of skipping the membarrier private expedited IPIs.
The membarrier system call can be issued concurrently with do_exit if we have thread groups created with CLONE_VM but not CLONE_THREAD.
Here is the scenario I have in mind:
Two thread groups are created, A and B. Thread group B is created by issuing clone from group A with flag CLONE_VM set, but not CLONE_THREAD. Let's assume we have a single thread within each thread group (Thread A and Thread B).
The AFAIU we can have:
Userspace variables:
int x = 0, y = 0;
CPU 0 CPU 1 Thread A Thread B (in thread group A) (in thread group B)
x = 1barrier()
y = 1exit()exit_mm()current->mm = NULL;
r1 = load y
membarrier()
skips CPU 0 (no IPI) because its current mm is NULL
r2 = load x
BUG_ON(r1 == 1 && r2 == 0)
---
Changes since v1:
- Use smp_mb__after_spinlock rather than smp_mb.
- Document race scenario in commit message.
--- kernel/exit.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/exit.c +8 -0
@@ -475,6 +475,14 @@ static void exit_mm(void)BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); +/* +* When a thread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock (); current->mm =NULL ;mmap_read_unlock (mm);enter_lazy_tlb (mm, current);
Signature · 1 line
2.11.0
Re: [RFC PATCH 1/3] sched: fix exit_mm vs membarrier (v2)
Hi Mathieu,
Quoted 60 lines · Mathieu Desnoyers
exit_mmshould issue memory barriers after user-space memory accesses, before clearingcurrent->mm, to order user-space memory accesses performed prior toexit_mmbefore clearingtsk->mm, which has the effect of skipping the membarrier private expedited IPIs.The membarrier system call can be issued concurrently with
do_exitif we have thread groups created withCLONE_VMbut notCLONE_THREAD.Here is the scenario I have in mind:
Two thread groups are created, A and B. Thread group B is created by issuing clone from group A with flag
CLONE_VMset, but notCLONE_THREAD. Let's assume we have a single thread within each thread group (Thread A and Thread B).The AFAIU we can have:
Userspace variables:
int x = 0, y = 0;
CPU 0 CPU 1 Thread A Thread B (in thread group A) (in thread group B)x = 1
barrier()
y = 1exit()exit_mm()current->mm= NULL;
r1 = load y
membarrier()
skips CPU 0 (no IPI) because its current mm is NULL
r2 = load x
BUG_ON(r1 == 1 && r2 == 0)Signed-off-by Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Will Deacon <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Linus Torvalds <[email protected]>Cc Alan Stern <[email protected]>---
Changes since v1:
- Usesmp_mb__after_spinlockrather thansmp_mb.
- Document race scenario in commit message.--- kernel/exit.c | 8 ++++++++ 1 file changed, 8 insertions(+)kernel/exit.c +0
-0
kernel/exit.c static void exit_mm(void) · line 480
Quoted 6 lines
kernel/exit.c +3
-0@@ -475,6 +475,14 @@ static void exit_mm(void)BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); +/* +* When a thread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe
Is it accurate to say that the correctness of membarrier_global_expedited() relies on the observation of ->mm? Because IIUC membarrier_global_expedited() loop doesn't check ->mm.
Regards,
Boqun
Quoted 11 lines
kernel/exit.c +5
-0+* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock (); current->mm =NULL ;mmap_read_unlock (mm);enter_lazy_tlb (mm, current);Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRe: [RFC PATCH 1/3] sched: fix exit_mm vs membarrier (v2)
Quoted 58 lines · ----- On Aug 16, 2020, at 11:23 AM, Boqun Feng [email protected] wrote:
Hi Mathieu,
Quoted 55 lines · Mathieu Desnoyers
exit_mmshould issue memory barriers after user-space memory accesses, before clearingcurrent->mm, to order user-space memory accesses performed prior toexit_mmbefore clearingtsk->mm, which has the effect of skipping the membarrier private expedited IPIs.The membarrier system call can be issued concurrently with
do_exitif we have thread groups created withCLONE_VMbut notCLONE_THREAD.Here is the scenario I have in mind:
Two thread groups are created, A and B. Thread group B is created by issuing clone from group A with flag
CLONE_VMset, but notCLONE_THREAD. Let's assume we have a single thread within each thread group (Thread A and Thread B).The AFAIU we can have:
Userspace variables:
int x = 0, y = 0;
CPU 0 CPU 1 Thread A Thread B (in thread group A) (in thread group B)x = 1
barrier()
y = 1exit()exit_mm()current->mm= NULL;
r1 = load y
membarrier()
skips CPU 0 (no IPI) because its current mm is NULL
r2 = load x
BUG_ON(r1 == 1 && r2 == 0)Signed-off-by Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Will Deacon <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Linus Torvalds <[email protected]>Cc Alan Stern <[email protected]>---
Changes since v1:
- Usesmp_mb__after_spinlockrather thansmp_mb.
- Document race scenario in commit message.--- kernel/exit.c | 8 ++++++++ 1 file changed, 8 insertions(+)
Quoted 15 lines
kernel/exit.c static void exit_mm(void) · line 480
Quoted 11 lines
kernel/exit.c +3
-0@@ -475,6 +475,14 @@ static void exit_mm(void)BUG_ON (mm != current->active_mm);/* more a memory barrier than a real lock */ task_lock (current); +/* +* When a thread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe Is it accurate to say that the correctness of
membarrier_global_expedited()relies on the observation of ->mm? Because IIUCmembarrier_global_expedited()loop doesn't check ->mm.
Good point, I was wrong. Will instead reword as:
/*
* When a thread stops operating on an address space, the loop
* in membarrier_private_expedited() may not observe that
* tsk->mm, and the loop in membarrier_global_expedited() may
* not observe a MEMBARRIER_STATE_GLOBAL_EXPEDITED
* rq->membarrier_state, so those would not issue an IPI.
* Membarrier requires a memory barrier after accessing
* user-space memory, before clearing tsk->mm or the
* rq->membarrier_state.
*/
And I'll make sure exit_mm clears this_rq()->membarrier_state as well.
Thanks,
Mathieu
Quoted 14 lines
Regards,
BoqunQuoted 10 lines
kernel/exit.c +5
-1+* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock (); current->mm =NULL ;mmap_read_unlock (mm);enter_lazy_tlb (mm, current); --2.11.0
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRFC PATCH 2/3 sched: membarrier: cover kthread_use_mm (v2)
In reply to Mathieu DesnoyersAdd comments and memory barrier to kthread_use_mm and kthread_unuse_mm to allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.
Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in membarrier_global_expedited to exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.
---
Changes since v1:
- Add WARN_ON_ONCE(current->mm) in play_idle_precise (PeterZ),
- Use smp_mb__after_spinlock rather than smp_mb after task_lock.
--- kernel/kthread.c | 19 +++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 8 ++------ 3 files changed, 22 insertions(+), 6 deletions(-)
kernel/kthread.c +19 -0
@@ -1255,8 +1255,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();to_kthread (tsk)->oldfs =force_uaccess_begin (); }
@@ -1276,6 +1287,14 @@ void kthread_unuse_mm(struct mm_struct *mm)force_uaccess_end (to_kthread (tsk)->oldfs);task_lock (tsk); +/* +* When a kthread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock ();sync_mm_rss (mm);local_irq_disable (); tsk->mm =NULL ;
kernel/sched/idle.c +1 -0
@@ -341,6 +341,7 @@ void play_idle_precise(u64 duration_ns, u64 latency_ns)WARN_ON_ONCE (!(current->flags &PF_KTHREAD ));WARN_ON_ONCE (!(current->flags &PF_NO_SETAFFINITY ));WARN_ON_ONCE (!duration_ns); +WARN_ON_ONCE (current->mm);rcu_sleep_check ();preempt_disable ();
kernel/sched/membarrier.c +2 -6
@@ -100,13 +100,9 @@ static int membarrier_global_expedited(void) MEMBARRIER_STATE_GLOBAL_EXPEDITED))continue ; -/* -* Skip the CPU if it runs a kernel thread. The scheduler -* leaves the prior task mm in place as an optimization when -* scheduling a kthread. -*/ +/* Skip the CPU if it runs the idle thread. */ p =rcu_dereference (cpu_rq (cpu)->curr); -if (p->flags &PF_KTHREAD ) +if (is_idle_task (p))continue ;__cpumask_set_cpu (cpu, tmpmask);
Signature · 1 line
2.11.0
Re: [RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v2)
Quoted 38 lines · Mathieu Desnoyers
Add comments and memory barrier to
kthread_use_mmandkthread_unuse_mmto allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in
membarrier_global_expeditedto exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.Signed-off-by Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Will Deacon <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>---
Changes since v1:
- AddWARN_ON_ONCE(current->mm)inplay_idle_precise(PeterZ),
- Usesmp_mb__after_spinlockrather thansmp_mbaftertask_lock.--- kernel/kthread.c | 19 +++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 8 ++------ 3 files changed, 22 insertions(+), 6 deletions(-)kernel/kthread.c +3
-0@@ -1255,8 +1255,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe
kernel/kthread.c void kthread_use_mm(struct mm_struct *mm) · line 1270
Quoted 10 lines
kernel/kthread.c +8
-0+* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();
Similar question here: could smp_mb() guarantee the correctness of GLOBAL_EXPEDITED? Don't you need membarrier_switch_mm() here and in kthread_unuse_mm(), too?
Am I miss something here?
Regards,
Boqun
Quoted 53 lines
kernel/kthread.c +8
-0to_kthread (tsk)->oldfs =force_uaccess_begin (); }@@ -1276,6 +1287,14 @@ void kthread_unuse_mm(struct mm_struct *mm)force_uaccess_end (to_kthread (tsk)->oldfs);task_lock (tsk); +/* +* When a kthread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock ();sync_mm_rss (mm);local_irq_disable (); tsk->mm =NULL ;kernel/sched/idle.c +1
-0@@ -341,6 +341,7 @@ void play_idle_precise(u64 duration_ns, u64 latency_ns)WARN_ON_ONCE (!(current->flags &PF_KTHREAD ));WARN_ON_ONCE (!(current->flags &PF_NO_SETAFFINITY ));WARN_ON_ONCE (!duration_ns); +WARN_ON_ONCE (current->mm);rcu_sleep_check ();preempt_disable ();kernel/sched/membarrier.c +2
-6@@ -100,13 +100,9 @@ static int membarrier_global_expedited(void) MEMBARRIER_STATE_GLOBAL_EXPEDITED))continue ; -/* -* Skip the CPU if it runs a kernel thread. The scheduler -* leaves the prior task mm in place as an optimization when -* scheduling a kthread. -*/ +/* Skip the CPU if it runs the idle thread. */ p =rcu_dereference (cpu_rq (cpu)->curr); -if (p->flags &PF_KTHREAD ) +if (is_idle_task (p))continue ;__cpumask_set_cpu (cpu, tmpmask);Signature · 1 line
2.11.0
Mathieu Desnoyers
maintainerRe: [RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v2)
Quoted 54 lines · ----- On Aug 16, 2020, at 11:29 AM, Boqun Feng [email protected] wrote:
Quoted 38 lines · Mathieu Desnoyers
Add comments and memory barrier to
kthread_use_mmandkthread_unuse_mmto allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in
membarrier_global_expeditedto exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.Signed-off-by Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Will Deacon <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>---
Changes since v1:
- AddWARN_ON_ONCE(current->mm)inplay_idle_precise(PeterZ),
- Usesmp_mb__after_spinlockrather thansmp_mbaftertask_lock.--- kernel/kthread.c | 19 +++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 8 ++------ 3 files changed, 22 insertions(+), 6 deletions(-)kernel/kthread.c +3
-0@@ -1255,8 +1255,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe kernel/kthread.c void kthread_use_mm(struct mm_struct *mm) · line 1270
Quoted 10 lines
kernel/kthread.c +8
-0+* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();Similar question here: could
smp_mb()guarantee the correctness ofGLOBAL_EXPEDITED? Don't you needmembarrier_switch_mm()here and inkthread_unuse_mm(), too?
Quoted 1 line
Am I miss something here?
I think you have a good point there. Which brings me to wonder why we don't have membarrier_switch_mm() when entering/leaving lazy TLB mode. This means an IPI can be sent to a kthread even if it does not use an mm, just because the membarrier state in the runqueue is that of the active_mm.
Thoughts ?
Thanks,
Mathieu
Quoted 56 lines
Regards,
BoqunQuoted 52 lines
kernel/kthread.c +8
-0to_kthread (tsk)->oldfs =force_uaccess_begin (); }@@ -1276,6 +1287,14 @@ void kthread_unuse_mm(struct mm_struct *mm)force_uaccess_end (to_kthread (tsk)->oldfs);task_lock (tsk); +/* +* When a kthread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock ();sync_mm_rss (mm);local_irq_disable (); tsk->mm =NULL ;kernel/sched/idle.c +1
-0@@ -341,6 +341,7 @@ void play_idle_precise(u64 duration_ns, u64 latency_ns)WARN_ON_ONCE (!(current->flags &PF_KTHREAD ));WARN_ON_ONCE (!(current->flags &PF_NO_SETAFFINITY ));WARN_ON_ONCE (!duration_ns); +WARN_ON_ONCE (current->mm);rcu_sleep_check ();preempt_disable ();kernel/sched/membarrier.c +2
-7@@ -100,13 +100,9 @@ static int membarrier_global_expedited(void) MEMBARRIER_STATE_GLOBAL_EXPEDITED))continue ; -/* -* Skip the CPU if it runs a kernel thread. The scheduler -* leaves the prior task mm in place as an optimization when -* scheduling a kthread. -*/ +/* Skip the CPU if it runs the idle thread. */ p =rcu_dereference (cpu_rq (cpu)->curr); -if (p->flags &PF_KTHREAD ) +if (is_idle_task (p))continue ;__cpumask_set_cpu (cpu, tmpmask); --2.11.0
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v2)
Quoted 58 lines · Mathieu Desnoyers
Quoted 54 lines · ----- On Aug 16, 2020, at 11:29 AM, Boqun Feng [email protected] wrote:
Quoted 38 lines · Mathieu Desnoyers
Add comments and memory barrier to
kthread_use_mmandkthread_unuse_mmto allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in
membarrier_global_expeditedto exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.Signed-off-by Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Will Deacon <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>---
Changes since v1:
- AddWARN_ON_ONCE(current->mm)inplay_idle_precise(PeterZ),
- Usesmp_mb__after_spinlockrather thansmp_mbaftertask_lock.--- kernel/kthread.c | 19 +++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 8 ++------ 3 files changed, 22 insertions(+), 6 deletions(-)kernel/kthread.c +3
-0@@ -1255,8 +1255,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe kernel/kthread.c void kthread_use_mm(struct mm_struct *mm) · line 1270
Quoted 10 lines
kernel/kthread.c +8
-0+* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();Similar question here: could
smp_mb()guarantee the correctness ofGLOBAL_EXPEDITED? Don't you needmembarrier_switch_mm()here and inkthread_unuse_mm(), too?Quoted 1 line
Am I miss something here?
Quoted 8 lines
I think you have a good point there. Which brings me to wonder why we don't have
membarrier_switch_mm()when entering/leaving lazy TLB mode. This means an IPI can be sent to a kthread even if it does not use an mm, just because the membarrier state in the runqueue is that of theactive_mm.Thoughts ?
Right, I think we should also handle the percpu membarrier_state. The basic rule is whenever we change current->mm or current (i.e. rq->curr) itself, we need to update the percpu membarrier_state accordingly.
Regards,
Boqun
Quoted 65 lines
Thanks,
Mathieu
Quoted 56 lines
Regards,
BoqunQuoted 52 lines
kernel/kthread.c +8
-0to_kthread (tsk)->oldfs =force_uaccess_begin (); }@@ -1276,6 +1287,14 @@ void kthread_unuse_mm(struct mm_struct *mm)force_uaccess_end (to_kthread (tsk)->oldfs);task_lock (tsk); +/* +* When a kthread stops operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe +* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after accessing user-space memory, before +* clearing tsk->mm. +*/ +smp_mb__after_spinlock ();sync_mm_rss (mm);local_irq_disable (); tsk->mm =NULL ;kernel/sched/idle.c +1
-0@@ -341,6 +341,7 @@ void play_idle_precise(u64 duration_ns, u64 latency_ns)WARN_ON_ONCE (!(current->flags &PF_KTHREAD ));WARN_ON_ONCE (!(current->flags &PF_NO_SETAFFINITY ));WARN_ON_ONCE (!duration_ns); +WARN_ON_ONCE (current->mm);rcu_sleep_check ();preempt_disable ();kernel/sched/membarrier.c +2
-7@@ -100,13 +100,9 @@ static int membarrier_global_expedited(void) MEMBARRIER_STATE_GLOBAL_EXPEDITED))continue ; -/* -* Skip the CPU if it runs a kernel thread. The scheduler -* leaves the prior task mm in place as an optimization when -* scheduling a kthread. -*/ +/* Skip the CPU if it runs the idle thread. */ p =rcu_dereference (cpu_rq (cpu)->curr); -if (p->flags &PF_KTHREAD ) +if (is_idle_task (p))continue ;__cpumask_set_cpu (cpu, tmpmask); --2.11.0
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRe: [RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v2)
Quoted 65 lines · ----- On Aug 24, 2020, at 10:06 PM, Boqun Feng [email protected] wrote:
Quoted 64 lines · Mathieu Desnoyers
Quoted 54 lines · ----- On Aug 16, 2020, at 11:29 AM, Boqun Feng [email protected] wrote:
Quoted 38 lines · Mathieu Desnoyers
Add comments and memory barrier to
kthread_use_mmandkthread_unuse_mmto allow the effect of membarrier(2) to apply to kthreads accessing user-space memory as well.Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Refine the check in
membarrier_global_expeditedto exclude runqueues running the idle thread rather than all kthreads from the IPI cpumask.Signed-off-by Mathieu Desnoyers <[email protected]>Cc Peter Zijlstra (Intel) <[email protected]>Cc Will Deacon <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Andrew Morton <[email protected]>---
Changes since v1:
- AddWARN_ON_ONCE(current->mm)inplay_idle_precise(PeterZ),
- Usesmp_mb__after_spinlockrather thansmp_mbaftertask_lock.--- kernel/kthread.c | 19 +++++++++++++++++++ kernel/sched/idle.c | 1 + kernel/sched/membarrier.c | 8 ++------ 3 files changed, 22 insertions(+), 6 deletions(-)kernel/kthread.c +3
-0@@ -1255,8 +1255,19 @@ void kthread_use_mm(struct mm_struct *mm)finish_arch_post_lock_switch ();#endif +/* +* When a kthread starts operating on an address space, the loop +* in membarrier_{private,global}_expedited() may not observe kernel/kthread.c void kthread_use_mm(struct mm_struct *mm) · line 1270
Quoted 10 lines
kernel/kthread.c +8
-0+* that tsk->mm, and not issue an IPI. Membarrier requires a +* memory barrier after storing to tsk->mm, before accessing +* user-space memory. A full memory barrier for membarrier +* {PRIVATE,GLOBAL}_EXPEDITED is implicitly provided by +* mmdrop(), or explicitly with smp_mb(). +*/ if (active_mm != mm)mmdrop (active_mm); +else +smp_mb ();Similar question here: could
smp_mb()guarantee the correctness ofGLOBAL_EXPEDITED? Don't you needmembarrier_switch_mm()here and inkthread_unuse_mm(), too?Quoted 1 line
Am I miss something here?
I think you have a good point there. Which brings me to wonder why we don't have
membarrier_switch_mm()when entering/leaving lazy TLB mode. This means an IPI can be sent to a kthread even if it does not use an mm, just because the membarrier state in the runqueue is that of theactive_mm.
Quoted 6 lines
Quoted 2 lines
Thoughts ?
Right, I think we should also handle the percpu
membarrier_state. The basic rule is whenever we changecurrent->mmor current (i.e.rq->curr) itself, we need to update the percpumembarrier_stateaccordingly.
OK, so as we introduce IPIs to kthreads which are using kthread_use_mm, we need to reconsider how the scheduler deals with runqueues entering lazy TLB state. Currently, membarrier_switch_mm() is not issued when entering lazy TLB state. But as we start considering kthreads as candidates for sending IPIs, we need to update the rq->membarrier_state whenever we enter lazy TLB state as well.
So I plan to do this change to cover this:
kernel/sched/core.c +2 -1
@@ -3736,6 +3736,8 @@ context_switch(struct rq *rq, struct task_struct *prev,*/ arch_start_context_switch (prev); +membarrier_switch_mm (rq, prev->mm, next->mm); +/* * kernel -> kernel lazy + transfer active * user -> kernel lazy + mmgrab() active
@@ -3752,7 +3754,6 @@ context_switch(struct rq *rq, struct task_struct *prev,else prev->active_mm =NULL ; }else {// to user -membarrier_switch_mm (rq, prev->active_mm, next->mm);/* * sys_membarrier() requires an smp_mb() between setting * rq->curr / membarrier_switch_mm() and returning to userspace.
kernel/sched/sched.h +3 -2
@@ -2592,12 +2592,13 @@ static inline void membarrier_switch_mm(struct rq *rq,struct mm_struct *prev_mm, struct mm_struct *next_mm) { - int membarrier_state; + int membarrier_state = 0;if (prev_mm == next_mm)return ; - membarrier_state =atomic_read (&next_mm->membarrier_state); +if (next_mm) + membarrier_state =atomic_read (&next_mm->membarrier_state);if (READ_ONCE (rq->membarrier_state) == membarrier_state)return ;
Thoughts ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Mathieu Desnoyers
maintainerRFC PATCH 3/3 sched: membarrier: document memory ordering scenarios
In reply to Mathieu DesnoyersDocument membarrier ordering scenarios in membarrier.c. Thanks to Alan Stern for refreshing my memory. Now that I have those in mind, it seems appropriate to serialize them to comments for posterity.
--- kernel/sched/membarrier.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+)
kernel/sched/membarrier.c +128 -0
@@ -7,6 +7,134 @@#include "sched.h" /* +* For documentation purposes, here are some membarrier ordering +* scenarios to keep in mind: +* +* A) Userspace thread execution after IPI vs membarrier's memory +* barrier before sending the IPI +* +* Userspace variables: +* +* int x = 0, y = 0; +* +* The memory barrier at the start of membarrier() on CPU0 is necessary in +* order to enforce the guarantee that any writes occurring on CPU0 before +* the membarrier() is executed will be visible to any code executing on +* CPU1 after the IPI-induced memory barrier: +* +* CPU0 CPU1 +* +* x = 1 +* membarrier(): +* a: smp_mb() +* b: send IPI IPI-induced mb +* c: smp_mb() +* r2 = y +* y = 1 +* barrier() +* r1 = x +* +* BUG_ON(r1 == 0 && r2 == 0) +* +* The write to y and load from x by CPU1 are unordered by the hardware, +* so it's possible to have "r1 = x" reordered before "y = 1" at any +* point after (b). If the memory barrier at (a) is omitted, then "x = 1" +* can be reordered after (a) (although not after (c)), so we get r1 == 0 +* and r2 == 0. This violates the guarantee that membarrier() is +* supposed by provide. +* +* The timing of the memory barrier at (a) has to ensure that it executes +* before the IPI-induced memory barrier on CPU1. +* +* B) Userspace thread execution before IPI vs membarrier's memory +* barrier after completing the IPI +* +* Userspace variables: +* +* int x = 0, y = 0; +* +* The memory barrier at the end of membarrier() on CPU0 is necessary in +* order to enforce the guarantee that any writes occurring on CPU1 before +* the membarrier() is executed will be visible to any code executing on +* CPU0 after the membarrier(): +* +* CPU0 CPU1 +* +* x = 1 +* barrier() +* y = 1 +* r2 = y +* membarrier(): +* a: smp_mb() +* b: send IPI IPI-induced mb +* c: smp_mb() +* r1 = x +* BUG_ON(r1 == 0 && r2 == 1) +* +* The writes to x and y are unordered by the hardware, so it's possible to +* have "r2 = 1" even though the write to x doesn't execute until (b). If +* the memory barrier at (c) is omitted then "r1 = x" can be reordered +* before (b) (although not before (a)), so we get "r1 = 0". This violates +* the guarantee that membarrier() is supposed to provide. +* +* The timing of the memory barrier at (c) has to ensure that it executes +* after the IPI-induced memory barrier on CPU1. +* +* C) Scheduling userspace thread -> kthread -> userspace thread vs membarrier +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* d: switch to kthread (includes mb) +* b: read rq->curr->mm == NULL +* e: switch to user (includes mb) +* c: smp_mb() +* +* Using the scenario from (A), we can show that (a) needs to be paired +* with (e). Using the scenario from (B), we can show that (c) needs to +* be paired with (d). +* +* D) exit_mm vs membarrier +* +* Two thread groups are created, A and B. Thread group B is created by +* issuing clone from group A with flag CLONE_VM set, but not CLONE_THREAD. +* Let's assume we have a single thread within each thread group (Thread A +* and Thread B). Thread A runs on CPU0, Thread B runs on CPU1. +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* exit_mm(): +* d: smp_mb() +* e: current->mm = NULL +* b: read rq->curr->mm == NULL +* c: smp_mb() +* +* Using scenario (B), we can show that (c) needs to be paired with (d). +* +* E) kthread_{use,unuse}_mm vs membarrier +* +* CPU0 CPU1 +* +* membarrier(): +* a: smp_mb() +* kthread_unuse_mm() +* d: smp_mb() +* e: current->mm = NULL +* b: read rq->curr->mm == NULL +* kthread_use_mm() +* f: current->mm = mm +* g: smp_mb() +* c: smp_mb() +* +* Using the scenario from (A), we can show that (a) needs to be paired +* with (g). Using the scenario from (B), we can show that (c) needs to +* be paired with (d). +*/ + +/* * Bitmask made from a "or" of all commands within enum membarrier_cmd, * except MEMBARRIER_CMD_QUERY. */
Signature · 1 line
2.11.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Mathieu Desnoyers
maintainerRe: [RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v2)
Quoted 7 lines · ----- On Aug 16, 2020, at 3:09 AM, Hillf Danton [email protected] wrote:
Quoted 3 lines · On Fri, 14 Aug 2020 12:43:57 -0400 Mathieu Desnoyers wrote:
Given that no prior kthread use this guarantee and that it only affects kthreads, adding this guarantee does not affect user-space ABI.
Can you expand a bit on why kthreads like ksoftirqd have to ack the IPIs from Dave who's not
CAP_SYS_ADMIN.
Do ksoftirqd kthreads ever use kthread_use_mm() to access user processes' memory ? If not, then they won't be disturbed by any IPI.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com