mm: Fix use_mm() vs TLB invalidate
PATCH v2 1/1 mm: Fix use_mm() vs TLB invalidate
For SMP systems using IPI based TLB invalidation, looking at current->active_mm is entirely reasonable. This then presents the following race condition:
CPU0 CPU1
flush_tlb_mm(mm) use_mm(mm)
<send-IPI> tsk->active_mm = mm; <IPI> if (tsk->active_mm == mm) // flush TLBs </IPI> switch_mm(old_mm,mm,tsk);
Where it is possible the IPI flushed the TLBs for @old_mm, not @mm, because the IPI lands before we actually switched.
Avoid this by disabling IRQs across changing ->active_mm and switch_mm().
[ There are all sorts of reasons this might be harmless for various architecture specific reasons, but best not leave the door open at all. ]
---
v2 now with WARN_ON_ONCE
mmu_context.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
mm/mmu_context.c +10 -1
@@ -24,14 +24,19 @@ void use_mm(struct mm_struct *mm)struct mm_struct *active_mm;struct task_struct *tsk = current; +WARN_ON (!(tsk->flags &PF_KTHREAD )); +WARN_ON (tsk->mm !=NULL ); +task_lock (tsk); +local_irq_disable (); active_mm = tsk->active_mm;if (active_mm != mm) {mmgrab (mm); tsk->active_mm = mm; } tsk->mm = mm; -switch_mm (active_mm, mm, tsk); +switch_mm_irqs_off (active_mm, mm, tsk); +local_irq_enable ();task_unlock (tsk);#ifdef finish_arch_post_lock_switchfinish_arch_post_lock_switch ();
@@ -54,11 +59,15 @@ void unuse_mm(struct mm_struct *mm) {struct task_struct *tsk = current; +WARN_ON (!(tsk->flags &PF_KTHREAD )); +task_lock (tsk);sync_mm_rss (mm); +local_irq_disable (); tsk->mm =NULL ;/* active_mm is still 'mm' */ enter_lazy_tlb (mm, tsk); +local_irq_enable ();task_unlock (tsk); }EXPORT_SYMBOL_GPL (unuse_mm);
Quoted 21 lines · Peter Zijlstra
For SMP systems using IPI based TLB invalidation, looking at
current->active_mmis entirely reasonable. This then presents the following race condition:CPU0 CPU1
flush_tlb_mm(mm)use_mm(mm)<send-IPI> tsk->active_mm = mm; <IPI> if (tsk->active_mm == mm) // flush TLBs </IPI> switch_mm(old_mm,mm,tsk);Where it is possible the IPI flushed the TLBs for @old_mm, not @mm, because the IPI lands before we actually switched.
Quoted 6 lines
Avoid this by disabling IRQs across changing ->active_mm and
switch_mm().[ There are all sorts of reasons this might be harmless for various architecture specific reasons, but best not leave the door open at all. ]
Not that I'm worried about it breaking anything, but ran it through the usual testing and might as well report:
Signature · 1 line
Jens Axboe
mm/mmu_context.c void use_mm(struct mm_struct *mm)
Quoted 6 lines · Peter Zijlstra
v2 now with
WARN_ON_ONCE
[...]mm/mmu_context.c +3
-0+WARN_ON (!(tsk->flags &PF_KTHREAD )); +WARN_ON (tsk->mm !=NULL ); +WARN_ON (!(tsk->flags &PF_KTHREAD ));
Version mismatch?
Signature · 1 line
Kees Cook
Quoted 9 lines · Kees Cook
mm/mmu_context.c void use_mm(struct mm_struct *mm)
Quoted 6 lines · Peter Zijlstra
v2 now with
WARN_ON_ONCE
[...]mm/mmu_context.c +3
-0+WARN_ON (!(tsk->flags &PF_KTHREAD )); +WARN_ON (tsk->mm !=NULL ); +WARN_ON (!(tsk->flags &PF_KTHREAD ));Version mismatch?
Dammit; no, that's just me being an idiot for doing too many things at once :/
I suppose I'll go send v3..