mm, sched: Rework lazy mm handling

Hi all-

Sorry I've been sitting on this so long. I think it's in decent shape, it has no *known* bugs, and I think it's time to get the show on the road. This series needs more eyeballs, too.

The overall point of this series is to get rid of the scalability problems with mm_count, and my goal is to solve it once and for all, for all architectures, in a way that doesn't have any gotchas for unwary users of ->active_mm.

Most of this series is just cleanup, though. mmgrab(), mmdrop(), and ->active_mm are a mess. A number of ->active_mm users are simply wrong. kthread lazy mm handling is inconsistent with user thread lazy mm handling (by accident, as far as I can tell). And membarrier() relies on the barrier semantics of mmdrop() and mmgrab(), such that anything that gets rid of those barriers risks breaking membarrier(). x86 is sometimes non-lazy when the core thinks it's lazy because the core mm code didn't offer any mechanism by which x86 could tell the core that it's exiting lazy mode.

So most of this series is just cleanup. Bogus users of ->active_mm are fixed, and membarrier() is reworked so that its barriers are explicit instead of depending on mmdrop() and mmgrab(). x86 lazy handling is extensively tidied up, and x86's EFI mm code gets tidied up a bit too. I think I've done this all in a way that introduces little or no overhead.

Additionally, all the code paths that change current->mm are consolidated so that there is only one path to start using an mm and only one path to stop using it.

Once that's done, the actual meat (the hazard pointers) isn't so bad, and the x86 optimization on top that should eliminate scanning of remote CPUs in __mmput() is about two lines of code. Other architectures with sufficiently accurate mm_cpumask() tracking should be able to do the same thing.

akpm, this is intended to mostly replace Nick Piggin's lazy shootdown series. This series implements lazy shootdown on x86 implicitly, and powerpc should be able to do the same thing in just a couple lines of code if it wants to. The result is IMO much cleaner and more maintainable.

Once this is all reviewed, I'm hoping it can go in -tip (and -next) after the merge window or go in -mm. This is not intended for v5.16. I suspect -tip is easier in case other arch maintainers want to optimize their code in the same release.

Andy Lutomirski (23):
membarrier: Document why membarrier() works
x86/mm: Handle unlazying membarrier core sync in the arch code
membarrier: Remove membarrier_arch_switch_mm() prototype in core code
membarrier: Make the post-switch-mm barrier explicit membarrier, kthread: Use _ONCE accessors for task->mm powerpc/membarrier: Remove special barrier on mm switch
membarrier: Rewrite sync_core_before_usermode() and improve
documentation
membarrier: Remove redundant clear of mm->membarrier_state in

    exec_mmap()

membarrier: Fix incorrect barrier positions during exec and
kthread_use_mm()
x86/events, x86/insn-eval: Remove incorrect active_mm references sched/scs: Initialize shadow stack on idle thread bringup, not
shutdown
Rework "sched/core: Fix illegal RCU from offline CPUs"
exec: Remove unnecessary vmacache_seqnum clear in exec_mmap() sched, exec: Factor current mm changes out from exec
kthread: Switch to __change_current_mm()
sched: Use lightweight hazard pointers to grab lazy mms x86/mm: Make use/unuse_temporary_mm() non-static x86/mm: Allow temporary mms when IRQs are on
x86/efi: Make efi_enter/leave_mm use the temporary_mm machinery x86/mm: Remove leave_mm() in favor of unlazy_mm_irqs_off() x86/mm: Use unlazy_mm_irqs_off() in TLB flush IPIs x86/mm: Optimize for_each_possible_lazymm_cpu() x86/mm: Opt in to IRQs-off activate_mm()

 .../membarrier-sync-core/arch-support.txt     |  69 +--
 arch/arm/include/asm/membarrier.h             |  21 +
 arch/arm/kernel/smp.c                         |   2 -
 arch/arm64/include/asm/membarrier.h           |  19 +
 arch/arm64/kernel/smp.c                       |   2 -
 arch/csky/kernel/smp.c                        |   2 -
 arch/ia64/kernel/process.c                    |   1 -
 arch/mips/cavium-octeon/smp.c                 |   1 -
 arch/mips/kernel/smp-bmips.c                  |   2 -
 arch/mips/kernel/smp-cps.c                    |   1 -
 arch/mips/loongson64/smp.c                    |   2 -
 arch/powerpc/include/asm/membarrier.h         |  28 +-
 arch/powerpc/mm/mmu_context.c                 |   1 -
 arch/powerpc/platforms/85xx/smp.c             |   2 -
 arch/powerpc/platforms/powermac/smp.c         |   2 -
 arch/powerpc/platforms/powernv/smp.c          |   1 -
 arch/powerpc/platforms/pseries/hotplug-cpu.c  |   2 -
 arch/powerpc/platforms/pseries/pmem.c         |   1 -
 arch/riscv/kernel/cpu-hotplug.c               |   2 -
 arch/s390/kernel/smp.c                        |   1 -
 arch/sh/kernel/smp.c                          |   1 -
 arch/sparc/kernel/smp_64.c                    |   2 -
 arch/x86/Kconfig                              |   2 +-
 arch/x86/events/core.c                        |   9 +-
 arch/x86/include/asm/membarrier.h             |  25 ++
 arch/x86/include/asm/mmu.h                    |   6 +-
 arch/x86/include/asm/mmu_context.h            |  15 +-
 arch/x86/include/asm/sync_core.h              |  20 -
 arch/x86/kernel/alternative.c                 |  67 +--
 arch/x86/kernel/cpu/mce/core.c                |   2 +-
 arch/x86/kernel/smpboot.c                     |   2 -
 arch/x86/lib/insn-eval.c                      |  13 +-
 arch/x86/mm/tlb.c                             | 155 +++++--
 arch/x86/platform/efi/efi_64.c                |   9 +-
 arch/x86/xen/mmu_pv.c                         |   2 +-
 arch/xtensa/kernel/smp.c                      |   1 -
 drivers/cpuidle/cpuidle.c                     |   2 +-
 drivers/idle/intel_idle.c                     |   4 +-
 drivers/misc/sgi-gru/grufault.c               |   2 +-
 drivers/misc/sgi-gru/gruhandles.c             |   2 +-
 drivers/misc/sgi-gru/grukservices.c           |   2 +-
 fs/exec.c                                     |  28 +-
 include/linux/mmu_context.h                   |   4 +-
 include/linux/sched/hotplug.h                 |   6 -
 include/linux/sched/mm.h                      |  58 ++-
 include/linux/sync_core.h                     |  21 -
 init/Kconfig                                  |   3 -
 kernel/cpu.c                                  |  21 +-
 kernel/exit.c                                 |   2 +-
 kernel/fork.c                                 |  11 +
 kernel/kthread.c                              |  50 +--
 kernel/sched/core.c                           | 409 +++++++++++++++---
 kernel/sched/idle.c                           |   1 +
 kernel/sched/membarrier.c                     |  97 ++++-
 kernel/sched/sched.h                          |  11 +-
 55 files changed, 745 insertions(+), 482 deletions(-)
 create mode 100644 arch/arm/include/asm/membarrier.h
 create mode 100644 arch/arm64/include/asm/membarrier.h
 create mode 100644 arch/x86/include/asm/membarrier.h
 delete mode 100644 include/linux/sync_core.h

2.33.1