Skip to contentSkip to conversation

sparc64: remove mm_cpumask clearing to fix kthread_use_mm race

RFC PATCH New Nicholas Piggin 2 participants 3 messages Aug 18, 2020

Subsystems (4)
lore
Share & tools
lore

Nicholas Piggin

author maintainer
#0 lore raw

RFC PATCH 1/2 sparc64: remove mm_cpumask clearing to fix kthread_use_mm race

The de facto (and apparently uncommented) standard for using an mm had, thanks to this code in sparc if nothing else, been that you must have a reference on mm_users *and that reference must have been obtained with mmget()*, i.e., from a thread with a reference to mm_users that had used the mm.

The introduction of mmget_not_zero() in commit d2005e3f41d4 ("userfaultfd: don't pin the user memory in userfaultfd_file_create()") allowed mm_count holders to aoperate on user mappings asynchronously from the actual threads using the mm, but they were not to load those mappings into their TLB (i.e., walking vmas and page tables is okay, kthread_use_mm() is not).

io_uring 2b188cc1bb857 ("Add io_uring IO interface") added code which does a kthread_use_mm() from a mmget_not_zero() refcount.

The problem with this is code which previously assumed mm = current->mm and mm->mm_users = 1 implies the mm will remain single-threaded at least until this thread creates another mm_users reference, has now broken.

arch/sparc/kernel/smp_64.c:

    if (atomic_read(&mm->mm_users) = 1) {
        cpumask_copy(mm_cpumask(mm), cpumask_of(cpu));
        goto local_flush_and_out;
    }

vs fs/io_uring.c

    if (unlikely(!(ctx->flags & IORING_SETUP_SQPOLL) ||
                 !mmget_not_zero(ctx->sqo_mm)))
        return -EFAULT;
    kthread_use_mm(ctx->sqo_mm);

mmget_not_zero() could come in right after the mm_users = 1 test, then kthread_use_mm() which sets its CPU in the mm_cpumask. That update could be lost if cpumask_copy() occurs afterward.

I propose we fix this by allowing mmget_not_zero() to be a first-class reference, and not have this obscure undocumented and unchecked restriction.

The basic fix for sparc64 is to remove its mm_cpumask clearing code. The optimisation could be effectively restored by sending IPIs to mm_cpumask members and having them remove themselves from mm_cpumask. This is more tricky so I leave it as an exercise for someone with a sparc64 SMP. powerpc has a (currently similarly broken) example.

---
 arch/sparc/kernel/smp_64.c | 58 ++++++--------------------------------
 1 file changed, 9 insertions(+), 49 deletions(-)
arch/sparc/kernel/smp_64.c +9 -49
index e286e2badc8a..41423c6077d0 100644
@@ -1039,38 +1039,9 @@ void smp_fetch_global_pmu(void)
  * are flush_tlb_*() routines, and these run after flush_cache_*()
  * which performs the flushw.
  *
- * The SMP TLB coherency scheme we use works as follows:
- *
- * 1) mm->cpu_vm_mask is a bit mask of which cpus an address
- *    space has (potentially) executed on, this is the heuristic
- *    we use to avoid doing cross calls.
- *
- *    Also, for flushing from kswapd and also for clones, we
- *    use cpu_vm_mask as the list of cpus to make run the TLB.
- *
- * 2) TLB context numbers are shared globally across all processors
- *    in the system, this allows us to play several games to avoid
- *    cross calls.
- *
- *    One invariant is that when a cpu switches to a process, and
- *    that processes tsk->active_mm->cpu_vm_mask does not have the
- *    current cpu's bit set, that tlb context is flushed locally.
- *
- *    If the address space is non-shared (ie. mm->count = 1) we avoid
- *    cross calls when we want to flush the currently running process's
- *    tlb state.  This is done by clearing all cpu bits except the current
- *    processor's in current->mm->cpu_vm_mask and performing the
- *    flush locally only.  This will force any subsequent cpus which run
- *    this task to flush the context from the local tlb if the process
- *    migrates to another cpu (again).
- *
- * 3) For shared address spaces (threads) and swapping we bite the
- *    bullet for most cases and perform the cross call (but only to
- *    the cpus listed in cpu_vm_mask).
- *
- *    The performance gain from "optimizing" away the cross call for threads is
- *    questionable (in theory the big win for threads is the massive sharing of
- *    address space state across processors).
+ * mm->cpu_vm_mask is a bit mask of which cpus an address
+ * space has (potentially) executed on, this is the heuristic
+ * we use to limit cross calls.
  */
 
 /* This currently is only used by the hugetlb arch pre-fault
@@ -1082,16 +1053,10 @@ void smp_flush_tlb_mm(struct mm_struct *mm)
 	u32 ctx = CTX_HWBITS(mm->context);
 	int cpu = get_cpu();
 
-	if (atomic_read(&mm->mm_users) = 1) {
-		cpumask_copy(mm_cpumask(mm), cpumask_of(cpu));
-		goto local_flush_and_out;
-	}
-
 	smp_cross_call_masked(&xcall_flush_tlb_mm,
 			      ctx, 0, 0,
 			      mm_cpumask(mm));
 
-local_flush_and_out:
 	__flush_tlb_mm(ctx, SECONDARY_CONTEXT);
 
 	put_cpu();
@@ -1120,11 +1085,8 @@ void smp_flush_tlb_pending(struct mm_struct *mm, unsigned long nr, unsigned long
 	info.nr = nr;
 	info.vaddrs = vaddrs;
 
-	if (mm = current->mm && atomic_read(&mm->mm_users) = 1)
-		cpumask_copy(mm_cpumask(mm), cpumask_of(cpu));
-	else
-		smp_call_function_many(mm_cpumask(mm), tlb_pending_func,
-				       &info, 1);
+	smp_call_function_many(mm_cpumask(mm), tlb_pending_func,
+			       &info, 1);
 
 	__flush_tlb_pending(ctx, nr, vaddrs);
 
@@ -1136,12 +1098,10 @@ void smp_flush_tlb_page(struct mm_struct *mm, unsigned long vaddr)
 	unsigned long context = CTX_HWBITS(mm->context);
 	int cpu = get_cpu();
 
-	if (mm = current->mm && atomic_read(&mm->mm_users) = 1)
-		cpumask_copy(mm_cpumask(mm), cpumask_of(cpu));
-	else
-		smp_cross_call_masked(&xcall_flush_tlb_page,
-				      context, vaddr, 0,
-				      mm_cpumask(mm));
+	smp_cross_call_masked(&xcall_flush_tlb_page,
+			      context, vaddr, 0,
+			      mm_cpumask(mm));
+
 	__flush_tlb_page(context, vaddr);
 
 	put_cpu();
Signature · 1 line
2.23.0
author maintainer
#1 lore raw

RFC PATCH 2/2 powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm

Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference.

That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up.

Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask if it's okay. This relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") to disable irqs over the mm switch.

Fixes 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask")
---
 arch/powerpc/include/asm/tlb.h       | 13 -------------
 arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++-------
 2 files changed, 16 insertions(+), 20 deletions(-)
arch/powerpc/include/asm/tlb.h +0 -13
index fbc6f3002f23..d97f061fecac 100644
@@ -66,19 +66,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm)
 		return false;
 	return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm));
 }
-static inline void mm_reset_thread_local(struct mm_struct *mm)
-{
-	WARN_ON(atomic_read(&mm->context.copros) > 0);
-	/*
-	 * It's possible for mm_access to take a reference on mm_users to
-	 * access the remote mm from another thread, but it's not allowed
-	 * to set mm_cpumask, so mm_users may be > 1 here.
-	 */
-	WARN_ON(current->mm != mm);
-	atomic_set(&mm->context.active_cpus, 1);
-	cpumask_clear(mm_cpumask(mm));
-	cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm));
-}
 #else /* CONFIG_PPC_BOOK3S_64 */
 static inline int mm_is_thread_local(struct mm_struct *mm)
 {
arch/powerpc/mm/book3s64/radix_tlb.c +16 -7
index 0d233763441f..a421a0e3f930 100644
@@ -645,19 +645,29 @@ static void do_exit_flush_lazy_tlb(void *arg)
 	struct mm_struct *mm = arg;
 	unsigned long pid = mm->context.id;
 
+	/*
+	 * A kthread could have done a mmget_not_zero() after the flushing CPU
+	 * checked mm_users = 1, and be in the process of kthread_use_mm when
+	 * interrupted here. In that case, current->mm will be set to mm,
+	 * because kthread_use_mm() setting ->mm and switching to the mm is
+	 * done with interrupts off.
+	 */
 	if (current->mm = mm)
-		return; /* Local CPU */
+		goto out_flush;
 
 	if (current->active_mm = mm) {
-		/*
-		 * Must be a kernel thread because sender is single-threaded.
-		 */
-		BUG_ON(current->mm);
+		WARN_ON_ONCE(current->mm != NULL);
+		/* Is a kernel thread and is using mm as the lazy tlb */
 		mmgrab(&init_mm);
-		switch_mm(mm, &init_mm, current);
 		current->active_mm = &init_mm;
+		switch_mm_irqs_off(mm, &init_mm, current);
 		mmdrop(mm);
 	}
+
+	atomic_dec(&mm->context.active_cpus);
+	cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm));
+
+out_flush:
 	_tlbiel_pid(pid, RIC_FLUSH_ALL);
 }
 
@@ -672,7 +682,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm)
 	 */
 	smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb,
 				(void *)mm, 1);
-	mm_reset_thread_local(mm);
 }
 
 void radix__flush_tlb_mm(struct mm_struct *mm)
Signature · 1 line
2.23.0
#2 lore raw
In reply to Nicholas Piggin
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Nicholas,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on sparc/master]
[also build test ERROR on powerpc/next sparc-next/master v5.9-rc1 next-20200817] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Nicholas-Piggin/sparc64-remove-mm_cpumask-clearing-to-fix-kthread_use_mm-race/20200818-113600
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc.git master
config: sparc-allyesconfig (attached as .config)
compiler: sparc64-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=sparc

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

arch/sparc/kernel/smp_64.c: In function 'smp_flush_tlb_mm':

>> arch/sparc/kernel/smp_64.c:1053:6: error: unused variable 'cpu' [-Werror=unused-variable]
    1053 |  int cpu = get_cpu();
         |      ^~~
   arch/sparc/kernel/smp_64.c: In function 'smp_flush_tlb_pending':
   arch/sparc/kernel/smp_64.c:1081:6: error: unused variable 'cpu' [-Werror=unused-variable]
    1081 |  int cpu = get_cpu();
         |      ^~~
   arch/sparc/kernel/smp_64.c: In function 'smp_flush_tlb_page':
   arch/sparc/kernel/smp_64.c:1098:6: error: unused variable 'cpu' [-Werror=unused-variable]
    1098 |  int cpu = get_cpu();
         |      ^~~

cc1: all warnings being treated as errors

# https://github.com/0day-ci/linux/commit/9af8f1e046bb13612593bd8fcd3f98347223203b
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Nicholas-Piggin/sparc64-remove-mm_cpumask-clearing-to-fix-kthread_use_mm-race/20200818-113600 git checkout 9af8f1e046bb13612593bd8fcd3f98347223203b
vim +/cpu +1053 arch/sparc/kernel/smp_64.c

916ca14aaf12a71 arch/sparc/kernel/smp_64.c David S. Miller 2012-10-16  1035  
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1036  /* We know that the window frames of the user have been flushed
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1037   * to the stack before we get here because all callers of us
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1038   * are flush_tlb_*() routines, and these run after flush_cache_*()
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1039   * which performs the flushw.
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1040   *
9af8f1e046bb136 arch/sparc/kernel/smp_64.c Nicholas Piggin 2020-08-18  1041   * mm->cpu_vm_mask is a bit mask of which cpus an address
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1042   * space has (potentially) executed on, this is the heuristic
9af8f1e046bb136 arch/sparc/kernel/smp_64.c Nicholas Piggin 2020-08-18  1043   * we use to limit cross calls.
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1044   */
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1045  
62dbec78be652c2 arch/sparc64/kernel/smp.c  David S. Miller 2005-11-07  1046  /* This currently is only used by the hugetlb arch pre-fault
62dbec78be652c2 arch/sparc64/kernel/smp.c  David S. Miller 2005-11-07  1047   * hook on UltraSPARC-III+ and later when changing the pagesize
62dbec78be652c2 arch/sparc64/kernel/smp.c  David S. Miller 2005-11-07  1048   * bits of the context register for an address space.
62dbec78be652c2 arch/sparc64/kernel/smp.c  David S. Miller 2005-11-07  1049   */
62dbec78be652c2 arch/sparc64/kernel/smp.c  David S. Miller 2005-11-07  1050  void smp_flush_tlb_mm(struct mm_struct *mm)
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1051  {
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1052  	u32 ctx = CTX_HWBITS(mm->context);
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16 @1053  	int cpu = get_cpu();
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1054  
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1055  	smp_cross_call_masked(&xcall_flush_tlb_mm,
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1056  			      ctx, 0, 0,
81f1adf01224f5c arch/sparc/kernel/smp_64.c Rusty Russell   2009-03-16  1057  			      mm_cpumask(mm));
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1058  
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1059  	__flush_tlb_mm(ctx, SECONDARY_CONTEXT);
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1060  
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1061  	put_cpu();
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1062  }
^1da177e4c3f415 arch/sparc64/kernel/smp.c  Linus Torvalds  2005-04-16  1063  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org