Membarrier fixes and cleanups
Mathieu Desnoyers
maintainercover Membarrier fixes and cleanups
Hi,
Those series of fixes and cleanups are initially motivated by the report of race in membarrier, which can load p->mm->membarrier_state after mm has been freed (use-after-free).
Thanks,
Mathieu
Mathieu Desnoyers (7):
Fix: sched/membarrier: Private expedited registration check
Cleanup: sched/membarrier: Remove redundant check
Cleanup: sched/membarrier: Only sync_core before usermode for same mm
Fix: sched/membarrier: p->mm->membarrier_state racy load (v4)
selftests: sched/membarrier: Add multi-threaded test sched/membarrier: Skip IPIs when mm->mm_users == 1 sched/membarrier: Return -ENOMEM to userspace on memory allocation
failure
fs/exec.c | 2 +-
include/linux/mm_types.h | 14 +-
include/linux/sched/mm.h | 10 +-
kernel/sched/core.c | 4 +-
kernel/sched/membarrier.c | 236 +++++++++++-------
kernel/sched/sched.h | 34 +++
tools/testing/selftests/membarrier/.gitignore | 3 +-
tools/testing/selftests/membarrier/Makefile | 5 +-
...mbarrier_test.c => membarrier_test_impl.h} | 40 +--
.../membarrier/membarrier_test_multi_thread.c | 73 ++++++
.../membarrier_test_single_thread.c | 24 ++
11 files changed, 329 insertions(+), 116 deletions(-)
rename tools/testing/selftests/membarrier/{membarrier_test.c => membarrier_test_impl.h} (95%)
create mode 100644 tools/testing/selftests/membarrier/membarrier_test_multi_thread.c
create mode 100644 tools/testing/selftests/membarrier/membarrier_test_single_thread.cSignature · 1 line
2.17.1
Mathieu Desnoyers
maintainerRFC PATCH 1/7 Fix: sched/membarrier: Private expedited registration check
Fix a logic flaw in the way membarrier_register_private_expedited() handles ready state checks for private expedited sync core and private expedited registrations.
If a private expedited membarrier registration is first performed, and then a private expedited sync_core registration is performed, the ready state check will skip the second registration when it really should not.
--- kernel/sched/membarrier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/membarrier.c +1 -1
@@ -226,7 +226,7 @@ static int membarrier_register_private_expedited(int flags)* groups, which use the same mm. (CLONE_VM but not * CLONE_THREAD). */ -if (atomic_read (&mm->membarrier_state) & state) +if ((atomic_read (&mm->membarrier_state) & state) == state)return 0 ;atomic_or (MEMBARRIER_STATE_PRIVATE_EXPEDITED , &mm->membarrier_state);if (flags &MEMBARRIER_FLAG_SYNC_CORE )
Signature · 1 line
2.17.1
[tip: sched/urgent] sched/membarrier: Fix private expedited registration check
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: fc0d77387cb5ae883fd774fc559e056a8dde024c Gitweb: https://git.kernel.org/tip/fc0d77387cb5ae883fd774fc559e056a8dde024c Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:36:59 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:30 +02:00
sched/membarrier: Fix private expedited registration check
Fix a logic flaw in the way membarrier_register_private_expedited() handles ready state checks for private expedited sync core and private expedited registrations.
If a private expedited membarrier registration is first performed, and then a private expedited sync_core registration is performed, the ready state check will skip the second registration when it really should not.
--- kernel/sched/membarrier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/membarrier.c +1 -1
@@ -226,7 +226,7 @@ static int membarrier_register_private_expedited(int flags)* groups, which use the same mm. (CLONE_VM but not * CLONE_THREAD). */ -if (atomic_read (&mm->membarrier_state) & state) +if ((atomic_read (&mm->membarrier_state) & state) == state)return 0 ;atomic_or (MEMBARRIER_STATE_PRIVATE_EXPEDITED , &mm->membarrier_state);if (flags &MEMBARRIER_FLAG_SYNC_CORE )
Mathieu Desnoyers
maintainerRFC PATCH 2/7 Cleanup: sched/membarrier: Remove redundant check
In reply to Mathieu DesnoyersChecking that the number of threads is 1 is redundant with checking mm_users == 1.
--- kernel/sched/membarrier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/membarrier.c +2 -2
@@ -186,7 +186,7 @@ static int membarrier_register_global_expedited(void) MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY)return 0 ;atomic_or (MEMBARRIER_STATE_GLOBAL_EXPEDITED , &mm->membarrier_state); -if (atomic_read (&mm->mm_users) ==1 &&get_nr_threads (p) ==1 ) { +if (atomic_read (&mm->mm_users) ==1 ) {/* * For single mm user, single threaded process, we can * simply issue a memory barrier after setting
@@ -232,7 +232,7 @@ static int membarrier_register_private_expedited(int flags)if (flags &MEMBARRIER_FLAG_SYNC_CORE )atomic_or (MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE , &mm->membarrier_state); -if (!(atomic_read (&mm->mm_users) ==1 &&get_nr_threads (p) ==1 )) { +if (atomic_read (&mm->mm_users) !=1 ) {/* * Ensure all future scheduler executions will observe the * new thread flag state for this process.
Signature · 1 line
2.17.1
[tip: sched/urgent] sched/membarrier: Remove redundant check
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 09554009c0cad4cb2223dd943c813c9257c6883a Gitweb: https://git.kernel.org/tip/09554009c0cad4cb2223dd943c813c9257c6883a Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:00 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:30 +02:00
sched/membarrier: Remove redundant check
Checking that the number of threads is 1 is redundant with checking mm_users == 1.
No change in functionality intended.
--- kernel/sched/membarrier.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/sched/membarrier.c +2 -2
@@ -186,7 +186,7 @@ static int membarrier_register_global_expedited(void) MEMBARRIER_STATE_GLOBAL_EXPEDITED_READY)return 0 ;atomic_or (MEMBARRIER_STATE_GLOBAL_EXPEDITED , &mm->membarrier_state); -if (atomic_read (&mm->mm_users) ==1 &&get_nr_threads (p) ==1 ) { +if (atomic_read (&mm->mm_users) ==1 ) {/* * For single mm user, single threaded process, we can * simply issue a memory barrier after setting
@@ -232,7 +232,7 @@ static int membarrier_register_private_expedited(int flags)if (flags &MEMBARRIER_FLAG_SYNC_CORE )atomic_or (MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE , &mm->membarrier_state); -if (!(atomic_read (&mm->mm_users) ==1 &&get_nr_threads (p) ==1 )) { +if (atomic_read (&mm->mm_users) !=1 ) {/* * Ensure all future scheduler executions will observe the * new thread flag state for this process.
Mathieu Desnoyers
maintainerRFC PATCH 3/7 Cleanup: sched/membarrier: Only sync_core before usermode for same mm
In reply to Mathieu DesnoyersWhen the prev and next task's mm change, switch_mm() provides the core serializing guarantees before returning to usermode. The only case where an explicit core serialization is needed is when the scheduler keeps the same mm for prev and next.
--- include/linux/sched/mm.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/sched/mm.h +2 -0
@@ -362,6 +362,8 @@ enum {static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) { +if (current->mm != mm) +return ;if (likely (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE )))return ;
Signature · 1 line
2.17.1
[tip: sched/urgent] sched/membarrier: Call sync_core only before usermode for same mm
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 2840cf02fae627860156737e83326df354ee4ec6 Gitweb: https://git.kernel.org/tip/2840cf02fae627860156737e83326df354ee4ec6 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:01 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:30 +02:00
sched/membarrier: Call sync_core only before usermode for same mm
When the prev and next task's mm change, switch_mm() provides the core serializing guarantees before returning to usermode. The only case where an explicit core serialization is needed is when the scheduler keeps the same mm for prev and next.
--- include/linux/sched/mm.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/sched/mm.h +2 -0
@@ -362,6 +362,8 @@ enum {static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm) { +if (current->mm != mm) +return ;if (likely (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE )))return ;
Mathieu Desnoyers
maintainerRFC PATCH 4/7 Fix: sched/membarrier: p->mm->membarrier_state racy load (v4)
In reply to Mathieu DesnoyersThe membarrier_state field is located within the mm_struct, which is not guaranteed to exist when used from runqueue-lock-free iteration on runqueues by the membarrier system call.
Copy the membarrier_state from the mm_struct into the scheduler runqueue when the scheduler switches between mm.
When registering membarrier for mm, after setting the registration bit in the mm membarrier state, issue a synchronize_rcu() to ensure the scheduler observes the change. In order to take care of the case where a runqueue keeps executing the target mm without swapping to other mm, iterate over each runqueue and issue an IPI to copy the membarrier_state from the mm_struct into each runqueue which have the same mm which state has just been modified.
Move the mm membarrier_state field closer to pgd in mm_struct to use a cache line already touched by the scheduler switch_mm.
The membarrier_execve() (now membarrier_exec_mmap) hook now needs to clear the runqueue's membarrier state in addition to clear the mm membarrier state, so move its implementation into the scheduler membarrier code so it can access the runqueue structure.
Add memory barrier in membarrier_exec_mmap() prior to clearing the membarrier state, ensuring memory accesses executed prior to exec are not reordered with the stores clearing the membarrier state.
As suggested by Linus, move all membarrier.c RCU read-side locks outside of the for each cpu loops.
---
Changes since v1:
- Take care of Peter Zijlstra's feedback, moving callsites closer to switch_mm() (scheduler) and activate_mm() (execve).
- Add memory barrier in membarrier_exec_mmap() prior to clearing the membarrier state.
Changes since v2:
- Move the call to membarrier_exec_mmap() before setting tsk->mm to its new value, so the memory barrier is done before this store.
Changes since v3:
- Use GFP_KERNEL for registration cpumask allocation.
- Use this_cpu_write() to update the runqueue's membarrier_state.
- Return -ENOMEM to userspace if cpumask allocation fails. Remove fallback code.
--- fs/exec.c | 2 +- include/linux/mm_types.h | 14 ++- include/linux/sched/mm.h | 8 +- kernel/sched/core.c | 4 +- kernel/sched/membarrier.c | 176 ++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 34 ++++++++ 6 files changed, 184 insertions(+), 54 deletions(-)
fs/exec.c +1 -1
@@ -1033,6 +1033,7 @@ static int exec_mmap(struct mm_struct *mm) }task_lock (tsk); active_mm = tsk->active_mm; +membarrier_exec_mmap (mm); tsk->mm = mm; tsk->active_mm = mm;activate_mm (active_mm, mm);
@@ -1825,7 +1826,6 @@ static int __do_execve_file(int fd, struct filename *filename,/* execve succeeded */ current->fs->in_exec =0 ; current->in_execve =0 ; -membarrier_execve (current);rseq_execve (current);acct_update_integrals (current);task_numa_free (current, false);
include/linux/mm_types.h +11 -3
@@ -383,6 +383,16 @@ struct mm_struct {unsigned long highest_vm_end;/* highest vma end address */ pgd_t * pgd; +#ifdef CONFIG_MEMBARRIER +/** +* @membarrier_state: Flags controlling membarrier behavior. +* +* This field is close to @pgd to hopefully fit in the same +* cache-line, which needs to be touched by switch_mm(). +*/ +atomic_t membarrier_state; +#endif +/** * @mm_users: The number of users including userspace. *
@@ -452,9 +462,7 @@ struct mm_struct {unsigned long flags;/* Must use atomic bitops to access */ struct core_state *core_state;/* coredumping support */ -#ifdef CONFIG_MEMBARRIER -atomic_t membarrier_state; -#endif +#ifdef CONFIG_AIO spinlock_t ioctx_lock;struct kioctx_table __rcu *ioctx_table;
include/linux/sched/mm.h +3 -5
@@ -370,10 +370,8 @@ static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm)sync_core_before_usermode (); } -static inline void membarrier_execve (struct task_struct *t) -{ -atomic_set (&t->mm->membarrier_state,0 ); -} +extern void membarrier_exec_mmap (struct mm_struct *mm); +#else #ifdef CONFIG_ARCH_HAS_MEMBARRIER_CALLBACKS static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
@@ -382,7 +380,7 @@ static inline void membarrier_arch_switch_mm(struct mm_struct *prev, { }#endif -static inline void membarrier_execve (struct task_struct *t) +static inline void membarrier_exec_mmap (struct mm_struct *mm) { }static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm)
kernel/sched/core.c +3 -1
@@ -3238,8 +3238,10 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = oldmm;mmgrab (oldmm);enter_lazy_tlb (oldmm, next); - }else + }else { +membarrier_switch_mm (rq, oldmm, mm);switch_mm_irqs_off (oldmm, mm, next); + }if (!prev->mm) { prev->active_mm =NULL ;
kernel/sched/membarrier.c +132 -44
254 lines not on this page · raw message
kernel/sched/sched.h +34 -0
@@ -907,6 +907,10 @@ struct rq {atomic_t nr_iowait; +#ifdef CONFIG_MEMBARRIER +int membarrier_state; +#endif +#ifdef CONFIG_SMP struct root_domain *rd;struct sched_domain __rcu *sd;
@@ -2423,3 +2427,33 @@ static inline bool sched_energy_enabled(void)static inline bool sched_energy_enabled (void ) {return false; }#endif /* CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL */ + +#ifdef CONFIG_MEMBARRIER +/* +* The scheduler provides memory barriers required by membarrier between: +* - prior user-space memory accesses and store to rq->membarrier_state, +* - store to rq->membarrier_state and following user-space memory accesses. +* In the same way it provides those guarantees around store to rq->curr. +*/ +static inline void membarrier_switch_mm (struct rq *rq, +struct mm_struct *prev_mm, +struct mm_struct *next_mm) +{ +int membarrier_state; + +if (prev_mm == next_mm) +return ; + + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state) +return ; + +WRITE_ONCE (rq->membarrier_state, membarrier_state); +} +#else +static inline void membarrier_switch_mm (struct rq *rq, +struct mm_struct *prev_mm, +struct mm_struct *next_mm) +{ +} +#endif
Signature · 1 line
2.17.1
[tip: sched/urgent] sched/membarrier: Fix p->mm->membarrier_state racy load
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 227a4aadc75ba22fcb6c4e1c078817b8cbaae4ce Gitweb: https://git.kernel.org/tip/227a4aadc75ba22fcb6c4e1c078817b8cbaae4ce Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:02 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:30 +02:00
sched/membarrier: Fix p->mm->membarrier_state racy load
The membarrier_state field is located within the mm_struct, which is not guaranteed to exist when used from runqueue-lock-free iteration on runqueues by the membarrier system call.
Copy the membarrier_state from the mm_struct into the scheduler runqueue when the scheduler switches between mm.
When registering membarrier for mm, after setting the registration bit in the mm membarrier state, issue a synchronize_rcu() to ensure the scheduler observes the change. In order to take care of the case where a runqueue keeps executing the target mm without swapping to other mm, iterate over each runqueue and issue an IPI to copy the membarrier_state from the mm_struct into each runqueue which have the same mm which state has just been modified.
Move the mm membarrier_state field closer to pgd in mm_struct to use a cache line already touched by the scheduler switch_mm.
The membarrier_execve() (now membarrier_exec_mmap) hook now needs to clear the runqueue's membarrier state in addition to clear the mm membarrier state, so move its implementation into the scheduler membarrier code so it can access the runqueue structure.
Add memory barrier in membarrier_exec_mmap() prior to clearing the membarrier state, ensuring memory accesses executed prior to exec are not reordered with the stores clearing the membarrier state.
As suggested by Linus, move all membarrier.c RCU read-side locks outside of the for each cpu loops.
--- fs/exec.c | 2 +- include/linux/mm_types.h | 14 ++- include/linux/sched/mm.h | 8 +-- kernel/sched/core.c | 4 +- kernel/sched/membarrier.c | 175 +++++++++++++++++++++++++++---------- kernel/sched/sched.h | 34 +++++++- 6 files changed, 183 insertions(+), 54 deletions(-)
fs/exec.c +1 -1
@@ -1033,6 +1033,7 @@ static int exec_mmap(struct mm_struct *mm) }task_lock (tsk); active_mm = tsk->active_mm; +membarrier_exec_mmap (mm); tsk->mm = mm; tsk->active_mm = mm;activate_mm (active_mm, mm);
@@ -1825,7 +1826,6 @@ static int __do_execve_file(int fd, struct filename *filename,/* execve succeeded */ current->fs->in_exec =0 ; current->in_execve =0 ; -membarrier_execve (current);rseq_execve (current);acct_update_integrals (current);task_numa_free (current, false);
include/linux/mm_types.h +11 -3
@@ -383,6 +383,16 @@ struct mm_struct {unsigned long highest_vm_end;/* highest vma end address */ pgd_t * pgd; +#ifdef CONFIG_MEMBARRIER +/** +* @membarrier_state: Flags controlling membarrier behavior. +* +* This field is close to @pgd to hopefully fit in the same +* cache-line, which needs to be touched by switch_mm(). +*/ +atomic_t membarrier_state; +#endif +/** * @mm_users: The number of users including userspace. *
@@ -452,9 +462,7 @@ struct mm_struct {unsigned long flags;/* Must use atomic bitops to access */ struct core_state *core_state;/* coredumping support */ -#ifdef CONFIG_MEMBARRIER -atomic_t membarrier_state; -#endif +#ifdef CONFIG_AIO spinlock_t ioctx_lock;struct kioctx_table __rcu *ioctx_table;
include/linux/sched/mm.h +3 -5
@@ -370,10 +370,8 @@ static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm)sync_core_before_usermode (); } -static inline void membarrier_execve (struct task_struct *t) -{ -atomic_set (&t->mm->membarrier_state,0 ); -} +extern void membarrier_exec_mmap (struct mm_struct *mm); +#else #ifdef CONFIG_ARCH_HAS_MEMBARRIER_CALLBACKS static inline void membarrier_arch_switch_mm(struct mm_struct *prev,
@@ -382,7 +380,7 @@ static inline void membarrier_arch_switch_mm(struct mm_struct *prev, { }#endif -static inline void membarrier_execve (struct task_struct *t) +static inline void membarrier_exec_mmap (struct mm_struct *mm) { }static inline void membarrier_mm_sync_core_before_usermode (struct mm_struct *mm)
kernel/sched/core.c +2 -2
@@ -3358,15 +3358,15 @@ context_switch(struct rq *rq, struct task_struct *prev,else prev->active_mm =NULL ; }else {// to user +membarrier_switch_mm (rq, prev->active_mm, next->mm);/* * sys_membarrier() requires an smp_mb() between setting -* rq->curr and returning to userspace. +* rq->curr / membarrier_switch_mm() and returning to userspace. * * The below provides this either through switch_mm(), or in * case 'prev->active_mm == next->mm' through * finish_task_switch()'s mmdrop(). */ -switch_mm_irqs_off (prev->active_mm, next->mm, next);if (!prev->mm) {// from kernel
kernel/sched/membarrier.c +132 -43
247 lines not on this page · raw message
kernel/sched/sched.h +34 -0
@@ -911,6 +911,10 @@ struct rq {atomic_t nr_iowait; +#ifdef CONFIG_MEMBARRIER +int membarrier_state; +#endif +#ifdef CONFIG_SMP struct root_domain *rd;struct sched_domain __rcu *sd;
@@ -2438,3 +2442,33 @@ static inline bool sched_energy_enabled(void)static inline bool sched_energy_enabled (void ) {return false; }#endif /* CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL */ + +#ifdef CONFIG_MEMBARRIER +/* +* The scheduler provides memory barriers required by membarrier between: +* - prior user-space memory accesses and store to rq->membarrier_state, +* - store to rq->membarrier_state and following user-space memory accesses. +* In the same way it provides those guarantees around store to rq->curr. +*/ +static inline void membarrier_switch_mm (struct rq *rq, +struct mm_struct *prev_mm, +struct mm_struct *next_mm) +{ +int membarrier_state; + +if (prev_mm == next_mm) +return ; + + membarrier_state =atomic_read (&next_mm->membarrier_state); +if (READ_ONCE (rq->membarrier_state) == membarrier_state) +return ; + +WRITE_ONCE (rq->membarrier_state, membarrier_state); +} +#else +static inline void membarrier_switch_mm (struct rq *rq, +struct mm_struct *prev_mm, +struct mm_struct *next_mm) +{ +} +#endif
Re: [tip: sched/urgent] sched/membarrier: Fix p->mm->membarrier_state racy load
Quoted 24 lines · * tip-bot2 for Mathieu Desnoyers <[email protected]> wrote:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 227a4aadc75ba22fcb6c4e1c078817b8cbaae4ce Gitweb: https://git.kernel.org/tip/227a4aadc75ba22fcb6c4e1c078817b8cbaae4ce Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:02 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:30 +02:00sched/membarrier: Fix
p->mm->membarrier_stateracy loadkernel/sched/membarrier.c +2
-0+rcu_read_unlock ();if (!fallback) {preempt_disable ();smp_call_function_many (tmpmask, ipi_mb,NULL ,1 );@@ -136,6 +178,7 @@ static int membarrier_private_expedited(int flags) }cpus_read_lock (); +rcu_read_lock ();for_each_online_cpu (cpu) {struct task_struct *p;
kernel/sched/membarrier.c static int membarrier_private_expedited(int flags) · line 204
Quoted 6 lines
kernel/sched/membarrier.c +1
-1@@ -157,8 +200,8 @@ static int membarrier_private_expedited(int flags)else smp_call_function_single (cpu, ipi_mb,NULL ,1 ); } -rcu_read_unlock (); } +rcu_read_unlock ();
I noticed this too late, but the locking in this part is now bogus:
rcu_read_lock();
for_each_online_cpu(cpu) {
struct task_struct *p;
/*
* Skipping the current CPU is OK even through we can be
* migrated at any point. The current CPU, at the point
* where we read raw_smp_processor_id(), is ensured to
* be in program order with respect to the caller
* thread. Therefore, we can skip this CPU from the
* iteration.
*/
if (cpu == raw_smp_processor_id())
continue;
rcu_read_lock();
p = rcu_dereference(cpu_rq(cpu)->curr);
if (p && p->mm == mm)
__cpumask_set_cpu(cpu, tmpmask);
}
rcu_read_unlock();Note the double rcu_read_lock() ....
This bug is now upstream, so requires an urgent fix, as it should be trivial to trigger with pretty much any membarrier user.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [tip: sched/urgent] sched/membarrier: Fix p->mm->membarrier_state racy load
Quoted 36 lines · Ingo Molnar
Quoted 24 lines · * tip-bot2 for Mathieu Desnoyers <[email protected]> wrote:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 227a4aadc75ba22fcb6c4e1c078817b8cbaae4ce Gitweb: https://git.kernel.org/tip/227a4aadc75ba22fcb6c4e1c078817b8cbaae4ce Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:02 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:30 +02:00sched/membarrier: Fix
p->mm->membarrier_stateracy load(quoted diff) +2
-0+ rcu_read_unlock(); if (!fallback) { preempt_disable(); smp_call_function_many(tmpmask, ipi_mb, NULL, 1);@@ -136,6 +178,7 @@ static int membarrier_private_expedited(int flags) } cpus_read_lock(); + rcu_read_lock(); for_each_online_cpu(cpu) { struct task_struct *p;Review static int membarrier_private_expedited(int flags) · line 204
Quoted 6 lines
(quoted diff) +1
-1@@ -157,8 +200,8 @@ static int membarrier_private_expedited(int flags) else smp_call_function_single(cpu, ipi_mb, NULL, 1); } - rcu_read_unlock(); } + rcu_read_unlock();I noticed this too late, but the locking in this part is now bogus:
Quoted 15 lines
rcu_read_lock();
for_each_online_cpu(cpu) {
struct task_struct *p;
/*
* Skipping the current CPU is OK even through we can be
* migrated at any point. The current CPU, at the point
* where we read raw_smp_processor_id(), is ensured to
* be in program order with respect to the caller
* thread. Therefore, we can skip this CPU from the
* iteration.
*/
if (cpu == raw_smp_processor_id())
continue;
rcu_read_lock();Yeah, that one needs to go.
Quoted 10 lines
p = rcu_dereference(cpu_rq(cpu)->curr); if (p && p->mm == mm) __cpumask_set_cpu(cpu, tmpmask); } rcu_read_unlock();Note the double
rcu_read_lock()....This bug is now upstream, so requires an urgent fix, as it should be trivial to trigger with pretty much any membarrier user.
---
Subject: membarrier: Fix faulty merge
Commit 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load") got fat fingered by me when merging it with other patches. It meant to move the rcu section out of the for loop but ended up doing it partially, leaving a superfluous rcu_read_lock() inside, causing havok.
--- kernel/sched/membarrier.c | 1 - 1 file changed, 1 deletion(-)
kernel/sched/membarrier.c +0 -1
@@ -174,7 +174,6 @@ static int membarrier_private_expedited(int flags)*/ if (cpu ==raw_smp_processor_id ())continue ; -rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu)->curr);if (p && p->mm == mm)__cpumask_set_cpu (cpu, tmpmask);
[tip: sched/urgent] membarrier: Fix RCU locking bug caused by faulty merge
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 73956fc07dd7b25d4a33ab3fdd6247c60d0b237c Gitweb: https://git.kernel.org/tip/73956fc07dd7b25d4a33ab3fdd6247c60d0b237c Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Oct 2019 10:50:33 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 01 Oct 2019 21:27:50 +02:00
membarrier: Fix RCU locking bug caused by faulty merge
The following commit:
227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load")
got fat fingered by me when merging it with other patches. It meant to move the RCU section out of the for loop but ended up doing it partially, leaving a superfluous rcu_read_lock() inside, causing havok.
--- kernel/sched/membarrier.c | 1 - 1 file changed, 1 deletion(-)
kernel/sched/membarrier.c +0 -1
@@ -174,7 +174,6 @@ static int membarrier_private_expedited(int flags)*/ if (cpu ==raw_smp_processor_id ())continue ; -rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu)->curr);if (p && p->mm == mm)__cpumask_set_cpu (cpu, tmpmask);
Mathieu Desnoyers
maintainerRFC PATCH 5/7 selftests: sched/membarrier: Add multi-threaded test
In reply to Mathieu Desnoyersmembarrier commands cover very different code paths if they are in a single-threaded vs multi-threaded process. Therefore, exercise both scenarios in the kernel selftests to increase coverage of this selftest.
---
tools/testing/selftests/membarrier/.gitignore | 3 +-
tools/testing/selftests/membarrier/Makefile | 5 +-
...mbarrier_test.c => membarrier_test_impl.h} | 40 +++++-----
.../membarrier/membarrier_test_multi_thread.c | 73 +++++++++++++++++++
.../membarrier_test_single_thread.c | 24 ++++++
5 files changed, 124 insertions(+), 21 deletions(-)
rename tools/testing/selftests/membarrier/{membarrier_test.c => membarrier_test_impl.h} (95%)
create mode 100644 tools/testing/selftests/membarrier/membarrier_test_multi_thread.c
create mode 100644 tools/testing/selftests/membarrier/membarrier_test_single_thread.ctools/testing/selftests/membarrier/.gitignore +2 -1
@@ -1 +1,2 @@ -membarrier_test +membarrier_test_multi_thread +membarrier_test_single_thread
tools/testing/selftests/membarrier/Makefile +3 -2
@@ -1,7 +1,8 @@ # SPDX-License-Identifier: GPL-2.0-only CFLAGS += -g -I../../../../usr/include/ +LDLIBS += -lpthread -TEST_GEN_PROGS := membarrier_test +TEST_GEN_PROGS := membarrier_test_single_thread \ + membarrier_test_multi_thread include ../lib.mk -
tools/testing/selftests/membarrier/membarrier_test_impl.h renamed from tools/testing/selftests/membarrier/membarrier_test.c +22 -18
@@ -1,10 +1,11 @@ -// SPDX-License-Identifier: GPL-2.0 +/* SPDX-License-Identifier: GPL-2.0 */ #define _GNU_SOURCE#include <linux/membarrier.h> #include <syscall.h> #include <stdio.h> #include <errno.h> #include <string.h> +#include <pthread.h> #include "../kselftest.h"
@@ -223,7 +224,7 @@ static int test_membarrier_global_expedited_success(void)return 0 ; } -static int test_membarrier (void ) +static int test_membarrier_fail (void ) {int status;
@@ -233,10 +234,27 @@ static int test_membarrier(void) status =test_membarrier_flags_fail ();if (status)return status; - status =test_membarrier_global_success (); + status =test_membarrier_private_expedited_fail ();if (status)return status; - status =test_membarrier_private_expedited_fail (); + status =sys_membarrier (MEMBARRIER_CMD_QUERY ,0 ); +if (status <0 ) { +ksft_test_result_fail ("sys_membarrier() failed\n" ); +return status; + } +if (status &MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE ) { + status =test_membarrier_private_expedited_sync_core_fail (); +if (status) +return status; + } +return 0 ; +} + +static int test_membarrier_success (void ) +{ +int status; + + status =test_membarrier_global_success ();if (status)return status; status =test_membarrier_register_private_expedited_success ();
@@ -251,9 +269,6 @@ static int test_membarrier(void)return status; }if (status &MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE ) { - status =test_membarrier_private_expedited_sync_core_fail (); -if (status) -return status; status =test_membarrier_register_private_expedited_sync_core_success ();if (status)return status;
@@ -300,14 +315,3 @@ static int test_membarrier_query(void)ksft_test_result_pass ("sys_membarrier available\n" );return 0 ; } - -int main (int argc,char **argv) -{ -ksft_print_header (); -ksft_set_plan (13 ); - -test_membarrier_query (); -test_membarrier (); - -return ksft_exit_pass (); -}
tools/testing/selftests/membarrier/membarrier_test_multi_thread.c new file +73 -0
73 lines not on this page · raw message
tools/testing/selftests/membarrier/membarrier_test_single_thread.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <linux/membarrier.h> +#include <syscall.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <pthread.h> + +#include "membarrier_test_impl.h" + +int main (int argc,char **argv) +{ +ksft_print_header (); +ksft_set_plan (13 ); + +test_membarrier_query (); + +test_membarrier_fail (); + +test_membarrier_success (); + +return ksft_exit_pass (); +}
Signature · 1 line
2.17.1
[tip: sched/urgent] selftests, sched/membarrier: Add multi-threaded test
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 19a4ff534bb09686f53800564cb977bad2177c00 Gitweb: https://git.kernel.org/tip/19a4ff534bb09686f53800564cb977bad2177c00 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:03 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:31 +02:00
selftests, sched/membarrier: Add multi-threaded test
membarrier commands cover very different code paths if they are in a single-threaded vs multi-threaded process. Therefore, exercise both scenarios in the kernel selftests to increase coverage of this selftest.
--- tools/testing/selftests/membarrier/.gitignore | 3 +- tools/testing/selftests/membarrier/Makefile | 5 +- tools/testing/selftests/membarrier/membarrier_test.c | 313 +--------------------------------------------------------------------- tools/testing/selftests/membarrier/membarrier_test_impl.h | 317 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- tools/testing/selftests/membarrier/membarrier_test_multi_thread.c | 73 ++++++++++++++++- tools/testing/selftests/membarrier/membarrier_test_single_thread.c | 24 +++++- 6 files changed, 419 insertions(+), 316 deletions(-) delete mode 100644 tools/testing/selftests/membarrier/membarrier_test.c create mode 100644 tools/testing/selftests/membarrier/membarrier_test_impl.h create mode 100644 tools/testing/selftests/membarrier/membarrier_test_multi_thread.c create mode 100644 tools/testing/selftests/membarrier/membarrier_test_single_thread.c
tools/testing/selftests/membarrier/.gitignore +2 -1
@@ -1 +1,2 @@ -membarrier_test +membarrier_test_multi_thread +membarrier_test_single_thread
tools/testing/selftests/membarrier/Makefile +3 -2
@@ -1,7 +1,8 @@ # SPDX-License-Identifier: GPL-2.0-only CFLAGS += -g -I../../../../usr/include/ +LDLIBS += -lpthread -TEST_GEN_PROGS := membarrier_test +TEST_GEN_PROGS := membarrier_test_single_thread \ + membarrier_test_multi_thread include ../lib.mk -
tools/testing/selftests/membarrier/membarrier_test.c deleted +0 -313
313 lines not on this page · raw message
tools/testing/selftests/membarrier/membarrier_test_impl.h new file +317 -0
317 lines not on this page · raw message
tools/testing/selftests/membarrier/membarrier_test_multi_thread.c new file +73 -0
73 lines not on this page · raw message
tools/testing/selftests/membarrier/membarrier_test_single_thread.c new file +24 -0
@@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <linux/membarrier.h> +#include <syscall.h> +#include <stdio.h> +#include <errno.h> +#include <string.h> +#include <pthread.h> + +#include "membarrier_test_impl.h" + +int main (int argc,char **argv) +{ +ksft_print_header (); +ksft_set_plan (13 ); + +test_membarrier_query (); + +test_membarrier_fail (); + +test_membarrier_success (); + +return ksft_exit_pass (); +}
Mathieu Desnoyers
maintainerRFC PATCH 6/7 sched/membarrier: Skip IPIs when mm->mm_users == 1
In reply to Mathieu DesnoyersIf there is only a single mm_user for the mm, the private expedited membarrier command can skip the IPIs, because only a single thread is using the mm.
--- kernel/sched/membarrier.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/sched/membarrier.c +5 -4
@@ -145,20 +145,21 @@ static int membarrier_private_expedited(int flags)int cpu;bool fallback = false;cpumask_var_t tmpmask; +struct mm_struct *mm = current->mm;if (flags &MEMBARRIER_FLAG_SYNC_CORE ) {if (!IS_ENABLED (CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE ))return -EINVAL ; -if (!(atomic_read (¤t->mm->membarrier_state) & +if (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY ))return -EPERM ; }else { -if (!(atomic_read (¤t->mm->membarrier_state) & +if (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY ))return -EPERM ; } -if (num_online_cpus () ==1 ) +if (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 )return 0 ;/*
@@ -193,7 +194,7 @@ static int membarrier_private_expedited(int flags)if (cpu ==raw_smp_processor_id ())continue ; p =task_rcu_dereference (&cpu_rq (cpu)->curr); -if (p && p->mm == current->mm) { +if (p && p->mm == mm) {if (!fallback)__cpumask_set_cpu (cpu, tmpmask);else
Signature · 1 line
2.17.1
[tip: sched/urgent] sched/membarrier: Skip IPIs when mm->mm_users == 1
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: c6d68c1c4a4d6611fc0f8145d764226571d737ca Gitweb: https://git.kernel.org/tip/c6d68c1c4a4d6611fc0f8145d764226571d737ca Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:04 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:31 +02:00
sched/membarrier: Skip IPIs when mm->mm_users == 1
If there is only a single mm_user for the mm, the private expedited membarrier command can skip the IPIs, because only a single thread is using the mm.
--- kernel/sched/membarrier.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
kernel/sched/membarrier.c +5 -4
@@ -145,20 +145,21 @@ static int membarrier_private_expedited(int flags)int cpu;bool fallback = false;cpumask_var_t tmpmask; +struct mm_struct *mm = current->mm;if (flags &MEMBARRIER_FLAG_SYNC_CORE ) {if (!IS_ENABLED (CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE ))return -EINVAL ; -if (!(atomic_read (¤t->mm->membarrier_state) & +if (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE_READY ))return -EPERM ; }else { -if (!(atomic_read (¤t->mm->membarrier_state) & +if (!(atomic_read (&mm->membarrier_state) &MEMBARRIER_STATE_PRIVATE_EXPEDITED_READY ))return -EPERM ; } -if (num_online_cpus () ==1 ) +if (atomic_read (&mm->mm_users) ==1 ||num_online_cpus () ==1 )return 0 ;/*
@@ -194,7 +195,7 @@ static int membarrier_private_expedited(int flags)continue ;rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu)->curr); -if (p && p->mm == current->mm) { +if (p && p->mm == mm) {if (!fallback)__cpumask_set_cpu (cpu, tmpmask);else
Mathieu Desnoyers
maintainerRFC PATCH 7/7 sched/membarrier: Return -ENOMEM to userspace on memory allocation failure
In reply to Mathieu DesnoyersRemove the IPI fallback code from membarrier to deal with very infrequent cpumask memory allocation failure. Use GFP_KERNEL rather than GFP_NOWAIT, and relax the blocking guarantees for the expedited membarrier system call commands, allowing it to block if waiting for memory to be made available.
In addition, now -ENOMEM can be returned to user-space if the cpumask memory allocation fails.
--- kernel/sched/membarrier.c | 61 ++++++++++++--------------------------- 1 file changed, 19 insertions(+), 42 deletions(-)
kernel/sched/membarrier.c +19 -42
@@ -66,7 +66,6 @@ void membarrier_exec_mmap(struct mm_struct *mm)static int membarrier_global_expedited (void ) {int cpu; -bool fallback = false;cpumask_var_t tmpmask;if (num_online_cpus () ==1 )
@@ -78,15 +77,8 @@ static int membarrier_global_expedited(void)*/ smp_mb ();/* system call entry is not a mb. */ -/* -* Expedited membarrier commands guarantee that they won't -* block, hence the GFP_NOWAIT allocation flag and fallback -* implementation. -*/ -if (!zalloc_cpumask_var (&tmpmask,GFP_NOWAIT )) { -/* Fallback for OOM. */ - fallback = true; - } +if (!zalloc_cpumask_var (&tmpmask,GFP_KERNEL )) +return -ENOMEM ;cpus_read_lock ();rcu_read_lock ();
@@ -117,18 +109,15 @@ static int membarrier_global_expedited(void)if (p->flags &PF_KTHREAD )continue ; -if (!fallback) -__cpumask_set_cpu (cpu, tmpmask); -else -smp_call_function_single (cpu, ipi_mb,NULL ,1 ); +__cpumask_set_cpu (cpu, tmpmask); }rcu_read_unlock (); -if (!fallback) { -preempt_disable (); -smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); -preempt_enable (); -free_cpumask_var (tmpmask); - } + +preempt_disable (); +smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); +preempt_enable (); + +free_cpumask_var (tmpmask);cpus_read_unlock ();/*
@@ -143,7 +132,6 @@ static int membarrier_global_expedited(void)static int membarrier_private_expedited (int flags) {int cpu; -bool fallback = false;cpumask_var_t tmpmask;struct mm_struct *mm = current->mm;
@@ -168,15 +156,8 @@ static int membarrier_private_expedited(int flags)*/ smp_mb ();/* system call entry is not a mb. */ -/* -* Expedited membarrier commands guarantee that they won't -* block, hence the GFP_NOWAIT allocation flag and fallback -* implementation. -*/ -if (!zalloc_cpumask_var (&tmpmask,GFP_NOWAIT )) { -/* Fallback for OOM. */ - fallback = true; - } +if (!zalloc_cpumask_var (&tmpmask,GFP_KERNEL )) +return -ENOMEM ;cpus_read_lock ();rcu_read_lock ();
@@ -194,20 +175,16 @@ static int membarrier_private_expedited(int flags)if (cpu ==raw_smp_processor_id ())continue ; p =task_rcu_dereference (&cpu_rq (cpu)->curr); -if (p && p->mm == mm) { -if (!fallback) -__cpumask_set_cpu (cpu, tmpmask); -else -smp_call_function_single (cpu, ipi_mb,NULL ,1 ); - } +if (p && p->mm == mm) +__cpumask_set_cpu (cpu, tmpmask); }rcu_read_unlock (); -if (!fallback) { -preempt_disable (); -smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); -preempt_enable (); -free_cpumask_var (tmpmask); - } + +preempt_disable (); +smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); +preempt_enable (); + +free_cpumask_var (tmpmask);cpus_read_unlock ();/*
Signature · 1 line
2.17.1
[tip: sched/urgent] sched/membarrier: Return -ENOMEM to userspace on memory allocation failure
The following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: c172e0a3e8e65a4c6fffec5bc4d6de08d6f894f7 Gitweb: https://git.kernel.org/tip/c172e0a3e8e65a4c6fffec5bc4d6de08d6f894f7 Author: Mathieu Desnoyers <[email protected]> AuthorDate: Thu, 19 Sep 2019 13:37:05 -04:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 25 Sep 2019 17:42:31 +02:00
sched/membarrier: Return -ENOMEM to userspace on memory allocation failure
Remove the IPI fallback code from membarrier to deal with very infrequent cpumask memory allocation failure. Use GFP_KERNEL rather than GFP_NOWAIT, and relax the blocking guarantees for the expedited membarrier system call commands, allowing it to block if waiting for memory to be made available.
In addition, now -ENOMEM can be returned to user-space if the cpumask memory allocation fails.
--- kernel/sched/membarrier.c | 63 ++++++++++++-------------------------- 1 file changed, 20 insertions(+), 43 deletions(-)
kernel/sched/membarrier.c +20 -43
@@ -66,7 +66,6 @@ void membarrier_exec_mmap(struct mm_struct *mm)static int membarrier_global_expedited (void ) {int cpu; -bool fallback = false;cpumask_var_t tmpmask;if (num_online_cpus () ==1 )
@@ -78,15 +77,8 @@ static int membarrier_global_expedited(void)*/ smp_mb ();/* system call entry is not a mb. */ -/* -* Expedited membarrier commands guarantee that they won't -* block, hence the GFP_NOWAIT allocation flag and fallback -* implementation. -*/ -if (!zalloc_cpumask_var (&tmpmask,GFP_NOWAIT )) { -/* Fallback for OOM. */ - fallback = true; - } +if (!zalloc_cpumask_var (&tmpmask,GFP_KERNEL )) +return -ENOMEM ;cpus_read_lock ();rcu_read_lock ();
@@ -117,18 +109,15 @@ static int membarrier_global_expedited(void)if (p->flags &PF_KTHREAD )continue ; -if (!fallback) -__cpumask_set_cpu (cpu, tmpmask); -else -smp_call_function_single (cpu, ipi_mb,NULL ,1 ); +__cpumask_set_cpu (cpu, tmpmask); }rcu_read_unlock (); -if (!fallback) { -preempt_disable (); -smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); -preempt_enable (); -free_cpumask_var (tmpmask); - } + +preempt_disable (); +smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); +preempt_enable (); + +free_cpumask_var (tmpmask);cpus_read_unlock ();/*
@@ -143,7 +132,6 @@ static int membarrier_global_expedited(void)static int membarrier_private_expedited (int flags) {int cpu; -bool fallback = false;cpumask_var_t tmpmask;struct mm_struct *mm = current->mm;
@@ -168,15 +156,8 @@ static int membarrier_private_expedited(int flags)*/ smp_mb ();/* system call entry is not a mb. */ -/* -* Expedited membarrier commands guarantee that they won't -* block, hence the GFP_NOWAIT allocation flag and fallback -* implementation. -*/ -if (!zalloc_cpumask_var (&tmpmask,GFP_NOWAIT )) { -/* Fallback for OOM. */ - fallback = true; - } +if (!zalloc_cpumask_var (&tmpmask,GFP_KERNEL )) +return -ENOMEM ;cpus_read_lock ();rcu_read_lock ();
@@ -195,20 +176,16 @@ static int membarrier_private_expedited(int flags)continue ;rcu_read_lock (); p =rcu_dereference (cpu_rq (cpu)->curr); -if (p && p->mm == mm) { -if (!fallback) -__cpumask_set_cpu (cpu, tmpmask); -else -smp_call_function_single (cpu, ipi_mb,NULL ,1 ); - } +if (p && p->mm == mm) +__cpumask_set_cpu (cpu, tmpmask); }rcu_read_unlock (); -if (!fallback) { -preempt_disable (); -smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); -preempt_enable (); -free_cpumask_var (tmpmask); - } + +preempt_disable (); +smp_call_function_many (tmpmask, ipi_mb,NULL ,1 ); +preempt_enable (); + +free_cpumask_var (tmpmask);cpus_read_unlock ();/*
@@ -264,7 +241,7 @@ static int sync_runqueues_membarrier_state(struct mm_struct *mm)struct rq *rq =cpu_rq (cpu);struct task_struct *p; - p =rcu_dereference (&rq->curr); + p =rcu_dereference (rq->curr);if (p && p->mm == mm)__cpumask_set_cpu (cpu, tmpmask); }
Peter Zijlstra
maintainerQuoted 6 lines · Mathieu Desnoyers
Hi,
Those series of fixes and cleanups are initially motivated by the report of race in membarrier, which can load
p->mm->membarrier_stateafter mm has been freed (use-after-free).
The lot looks good to me; what do you want done with them (them being RFC and all) ?
Mathieu Desnoyers
maintainerQuoted 10 lines · ----- On Sep 23, 2019, at 5:06 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Those series of fixes and cleanups are initially motivated by the report of race in membarrier, which can load
p->mm->membarrier_stateafter mm has been freed (use-after-free).The lot looks good to me; what do you want done with them (them being RFC and all) ?
I can either re-send them without the RFC tag, or you can pick them directly through the scheduler tree.
As you prefer,
Thanks!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 10 lines · Mathieu Desnoyers
Quoted 8 lines · ----- On Sep 23, 2019, at 5:06 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Those series of fixes and cleanups are initially motivated by the report of race in membarrier, which can load
p->mm->membarrier_stateafter mm has been freed (use-after-free).
Quoted 5 lines
Quoted 2 lines
The lot looks good to me; what do you want done with them (them being RFC and all) ?
I can either re-send them without the RFC tag, or you can pick them directly through the scheduler tree.
I've picked them up (and fixed them up, they didn't apply to tip) and merge them with Eric's task_rcu_dereference() patches.
I'll push it out in a bit.
Mathieu Desnoyers
maintainerQuoted 14 lines · ----- On Sep 25, 2019, at 4:07 AM, Peter Zijlstra [email protected] wrote:
Quoted 13 lines · Mathieu Desnoyers
Quoted 10 lines · ----- On Sep 23, 2019, at 5:06 AM, Peter Zijlstra [email protected] wrote:
Quoted 6 lines · Mathieu Desnoyers
Hi,
Those series of fixes and cleanups are initially motivated by the report of race in membarrier, which can load
p->mm->membarrier_stateafter mm has been freed (use-after-free).The lot looks good to me; what do you want done with them (them being RFC and all) ?
Quoted 7 lines
Quoted 2 lines
I can either re-send them without the RFC tag, or you can pick them directly through the scheduler tree.
I've picked them up (and fixed them up, they didn't apply to tip) and merge them with Eric's
task_rcu_dereference()patches.I'll push it out in a bit.
Thanks Peter!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com