x86,tlb,mm: more lazy TLB cleanups & optimizations

Clarify exactly what the memory barrier synchronizes with.

The code in on_each_cpu_cond sets CPUs in a locally allocated bitmask, which should never be used by other CPUs simultaneously. There is no need to use locked memory accesses to set the bits in this bitmap.

Switch to __cpumask_set_cpu.

Introduce a variant of on_each_cpu_cond that iterates only over the CPUs in a cpumask, in order to avoid making callbacks for every single CPU in the system when we only need to test a subset.

Quoted 4 lines · Rik van Riel

Introduce a variant of on_each_cpu_cond that iterates only over the CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.

Nice.

Although, if you want to be really fancy, you could optimize this (or add a variant) that does the callback on the local CPU in parallel with the remote ones. That would give a small boost to TLB flushes.

Quoted 16 lines · Rik van Riel

Quoted 7 lines · Andy Lutomirski

Quoted 5 lines · Rik van Riel

Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.

Quoted 7 lines

Nice.
Although, if you want to be really fancy, you could optimize this (or
add a variant) that does the callback on the local CPU in parallel
with the remote ones. That would give a small boost to TLB flushes.

Quoted 8 lines

The test_func callbacks are not run remotely, but on the local CPU, before deciding who to send callbacks to.

The actual IPIs are sent in parallel, if the cpumask allocation succeeds (it always should in many kernel configurations, and almost always in the rest).

What I meant is that on_each_cpu_mask does:

smp_call_function_many(mask, func, info, wait);

if (cpumask_test_cpu(cpu, mask)) {
   unsigned long flags;
   local_irq_save(flags); func(info);
   local_irq_restore(flags);
}

So it IPIs all the remote CPUs in parallel, then waits, then does the local work. In principle, the local flush could be done after triggering the IPIs but before they all finish.

Quoted 16 lines · Rik van Riel

Quoted 7 lines · Andy Lutomirski

Quoted 5 lines · Rik van Riel

Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.

Quoted 7 lines

Nice.
Although, if you want to be really fancy, you could optimize this (or
add a variant) that does the callback on the local CPU in parallel
with the remote ones. That would give a small boost to TLB flushes.

Quoted 8 lines

The test_func callbacks are not run remotely, but on the local CPU, before deciding who to send callbacks to.

The actual IPIs are sent in parallel, if the cpumask allocation succeeds (it always should in many kernel configurations, and almost always in the rest).

What I meant is that on_each_cpu_mask does:

smp_call_function_many(mask, func, info, wait);

if (cpumask_test_cpu(cpu, mask)) {
   unsigned long flags;
   local_irq_save(flags); func(info);
   local_irq_restore(flags);
}

So it IPIs all the remote CPUs in parallel, then waits, then does the local work. In principle, the local flush could be done after triggering the IPIs but before they all finish.

Quoted 38 lines · Andy Lutomirski

Quoted 18 lines · Rik van Riel

Quoted 9 lines · Andy Lutomirski

Quoted 5 lines · Rik van Riel

Introduce a variant of on_each_cpu_cond that iterates only over the
CPUs in a cpumask, in order to avoid making callbacks for every single
CPU in the system when we only need to test a subset.

Quoted 7 lines

Nice.
Although, if you want to be really fancy, you could optimize this (or
add a variant) that does the callback on the local CPU in parallel
with the remote ones. That would give a small boost to TLB flushes.

Quoted 8 lines

The test_func callbacks are not run remotely, but on the local CPU, before deciding who to send callbacks to.

The actual IPIs are sent in parallel, if the cpumask allocation succeeds (it always should in many kernel configurations, and almost always in the rest).

What I meant is that on_each_cpu_mask does:

smp_call_function_many(mask, func, info, wait);
if (cpumask_test_cpu(cpu, mask)) {
unsigned long flags;
local_irq_save(flags); func(info);
local_irq_restore(flags);
}

Quoted 3 lines

So it IPIs all the remote CPUs in parallel, then waits, then does the local work. In principle, the local flush could be done after triggering the IPIs but before they all finish.

Grepping around the code, I found a few examples where the calling code appears to expect that smp_call_function_many also calls "func" on the local CPU.

For example, kvm_emulate_wbinvd_noskip has this:

       if (kvm_x86_ops->has_wbinvd_exit()) {
               int cpu = get_cpu();

               cpumask_set_cpu(cpu, vcpu->arch.wbinvd_dirty_mask);
               smp_call_function_many(vcpu->arch.wbinvd_dirty_mask,
                               wbinvd_ipi, NULL, 1);
               put_cpu();
               cpumask_clear(vcpu->arch.wbinvd_dirty_mask);
       } else
               wbinvd();

This seems to result in systems with ->has_wbinvd_exit only calling wbinvd_ipi on OTHER CPUs, and not on the CPU where the guest exited with wbinvd?

This seems unintended.

I guess looking into on_each_cpu_mask might be a little higher priority than waiting until the next Outreachy season :)

Quoted 6 lines · Rik van Riel

This seems to result in systems with ->has_wbinvd_exit only calling wbinvd_ipi on OTHER CPUs, and not on the CPU where the guest exited with wbinvd?

Quoted 5 lines

This seems unintended.

I guess looking into on_each_cpu_mask might be a little higher priority than waiting until the next Outreachy season :)

The right approach might be a tree wise rename from smp_call_... to on_other_cpus_mask() it similar. The current naming and semantics are extremely confusing.

Instead of open coding bitmap magic, use on_each_cpu_cond to determine which CPUs to send TLB flush IPIs to.

This might be a little bit slower than examining the bitmaps, but it should be a lot easier to maintain in the long run.

Quoted 3 lines · Rik van Riel

Instead of open coding bitmap magic, use on_each_cpu_cond to determine which CPUs to send TLB flush IPIs to.

Quoted 2 lines

This might be a little bit slower than examining the bitmaps, but it should be a lot easier to maintain in the long run.

Looks good.

i assume it's not easy to get the remove-tables case to do a single on_each_cpu_cond() instead of two? Currently it's doing the lazy ones and the non-lazy ones separately.

When switching back from lazy TLB mode to a thread of the same process that switched into lazy TLB mode, we still have the cr4 (and sometimes LDT) of that process loaded, and there is no need to reload it.

When there was no TLB flush while the CPU was in lazy TLB mode, the current code in switch_mm_irqs_off already avoids the reload, by returning early.

However, when the TLB contents on the CPU are out of date, and we flush the TLB for the task, we fall through to the regular context switching code. This patch teaches that code to skip the cr4 and LDT flushes when switching back to the same mm after a flush.

Conditionally skip lazy TLB mm refcounting. When an architecture has CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in lazy TLB mode anywhere will get shot down from exit_mmap, and there in no need to incur the cache line bouncing overhead of refcounting a lazy TLB mm.

Quoted 6 lines · Rik van Riel

Conditionally skip lazy TLB mm refcounting. When an architecture has
CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in lazy TLB mode anywhere will get shot down from exit_mmap, and there in no need to incur the cache line bouncing overhead of refcounting a lazy TLB mm.

Unless I've misunderstood something, this patch results in idle tasks whose active_mm has been freed still having active_mm pointing at freed memory.

Quoted 11 lines · Andy Lutomirski

Quoted 6 lines · Rik van Riel

Conditionally skip lazy TLB mm refcounting. When an architecture has
CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in lazy TLB mode anywhere will get shot down from exit_mmap, and there in no need to incur the cache line bouncing overhead of refcounting a lazy TLB mm.

Quoted 3 lines

Unless I've misunderstood something, this patch results in idle tasks whose active_mm has been freed still having active_mm pointing at freed memory.

Patch 9/10 is supposed to ensure that the lazy TLB CPUs get switched to init_mm before an mm is freed. No CPU should ever have its active_mm pointing at a freed mm.

Your message made me re-read the code, and now I realize that leave_mm does not actually do that.

Looking at the other callers of leave_mm, I might not be the only one surprised by that; xen_drop_mm_ref comes to mind.

I guess I should some code to leave_mm to have it actually clear active_mm and call the conditional refcount drop helper function.

Does that clear up the confusion?

Quoted 35 lines · Rik van Riel

Quoted 15 lines · Andy Lutomirski

Quoted 10 lines · Rik van Riel

Conditionally skip lazy TLB mm refcounting. When an architecture
has
CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in
lazy TLB mode anywhere will get shot down from exit_mmap, and there
in no need to incur the cache line bouncing overhead of refcounting
a lazy TLB mm.

Quoted 4 lines

Unless I've misunderstood something, this patch results in idle tasks
whose active_mm has been freed still having active_mm pointing at freed memory.

Patch 9/10 is supposed to ensure that the lazy TLB CPUs get switched to init_mm before an mm is freed. No CPU should ever have its active_mm pointing at a freed mm.

Your message made me re-read the code, and now I realize that leave_mm does not actually do that.

Looking at the other callers of leave_mm, I might not be the only one surprised by that; xen_drop_mm_ref comes to mind.

I guess I should some code to leave_mm to have it actually clear active_mm and call the conditional refcount drop helper function.

Quoted 1 line

Does that clear up the confusion?

Kind of. But what’s the point of keeping active_mm? On architectures that opt in to the new mode, there won’t be any code that cares about it’s value. What’s the benefit of keeping it around? If you ifdef it out, then it can’t possibly point to freed memory, and there’s nothing to worry about.

Quoted 5 lines · Rik van Riel

Conditionally skip lazy TLB mm refcounting. When an architecture has CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in lazy TLB mode anywhere will get shot down from exit_mmap, and there in no need to incur the cache line bouncing overhead of refcounting a lazy TLB mm.

Unless I've misunderstood something, this patch results in idle tasks whose active_mm has been freed still having active_mm pointing at freed memory.

Quoted 5 lines · Rik van Riel

Conditionally skip lazy TLB mm refcounting. When an architecture has CONFIG_ARCH_NO_ACTIVE_MM_REFCOUNTING enabled, an mm that is used in lazy TLB mode anywhere will get shot down from exit_mmap, and there in no need to incur the cache line bouncing overhead of refcounting a lazy TLB mm.

Unless I've misunderstood something, this patch results in idle tasks whose active_mm has been freed still having active_mm pointing at freed memory.

kernel/sched/core.c +1 -1

index c45de46fdf10..11724c9e88b0 100644
@@ -2691,7 +2691,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 */
 	if (mm) {
 		membarrier_mm_sync_core_before_usermode(mm);
-		mmdrop(mm);
+		drop_lazy_mm(mm);
 	}
 	if (unlikely(prev_state == TASK_DEAD)) {
 		if (prev->sched_class->task_dead)

Quoted 21 lines · Rik van Riel

kernel/sched/core.c +1 -1

index c45de46fdf10..11724c9e88b0 100644
@@ -2691,7 +2691,7 @@ static struct rq *finish_task_switch(struct
 task_struct *prev)
 	 */
 	if (mm) {
 		membarrier_mm_sync_core_before_usermode(mm);
-		mmdrop(mm);
+		drop_lazy_mm(mm);
 	}
 	if (unlikely(prev_state == TASK_DEAD)) {

kernel/sched/core.c +1 -1

 		if (prev->sched_class->task_dead)

@@ -2805,7 +2805,7 @@ context_switch(struct rq *rq, struct
 task_struct *prev,
 	 */
 	if (!mm) {
 		next->active_mm = oldmm;
-		mmgrab(oldmm);
+		grab_lazy_mm(oldmm);

Quoted 3 lines

kernel/sched/core.c +0 -0

 		enter_lazy_tlb(oldmm, next);
 	} else

switch_mm_irqs_off(oldmm, mm, next);

What happened to the rework I did there? That not only avoided fiddling
with active_mm, but also avoids grab/drop cycles for the other architectures when doing task->kthread->kthread->task things.

I agree with Andy that if you avoid the refcount fiddling, then you should also not muck with active_mm.

That is, if you keep active_mm for now (which seems a reasonable first
step) then at least ensure you keep ->mm == ->active_mm at all times.

Quoted 4 lines

What happened to the rework I did there? That not only avoided fiddling
with active_mm, but also avoids grab/drop cycles for the other architectures when doing task->kthread->kthread->task things.

I don't think I saw that. I only saw your email from July 20th with this fragment of code, which does not appear to avoid the grab/drop cycles, and still fiddles with active_mm:

(quoted diff) +6 -0

+               if (!prev->mm) {                        // from kernel
+                       /* will mmdrop() in finish_task_switch(). */
+                       rq->prev_mm = prev->active_mm;
+                       prev->active_mm = NULL;
+               }
+#endif

Quoted 9 lines · Rik van Riel

Quoted 4 lines

What happened to the rework I did there? That not only avoided fiddling
with active_mm, but also avoids grab/drop cycles for the other architectures when doing task->kthread->kthread->task things.

I don't think I saw that. I only saw your email from July 20th with this fragment of code, which does not appear to avoid the grab/drop cycles, and still fiddles with active_mm:

Yeah, that's it. Note how it doesn't do a grab+drop for kernel-

Quoted 1 line

kernel,

where the current could would have.

And also note that it only fiddles with active_mm if it does the grab+drop thing (the below should have s/ifdef/ifndef/ to make more sense maybe).

So for ARCH_NO_ACTIVE_MM we never touch ->active_mm and therefore ->active_mm == ->mm.

Review

Quoted 11 lines

(quoted diff) +8 -0

+       /*
+        * kernel -> kernel   lazy + transfer active
+        *   user -> kernel   lazy + mmgrab() active
+        *
+        * kernel ->   user   switch + mmdrop() active
+        *   user ->   user   switch
+        */
+       if (!next->mm) {                                // to

kernel

(quoted diff) +2 -0

+               enter_lazy_tlb(prev->active_mm, next);
+

#ifndef ARCH_NO_ACTIVE_MM

Quoted 4 lines

(quoted diff) +2 -0

+               next->active_mm = prev->active_mm;
+               if (prev->mm)                           // from

user
+ mmgrab(prev->active_mm);

		else
			prev->active_mm = NULL;

Quoted 5 lines

(quoted diff) +3 -0

+#endif
+       } else {                                        // to user
+               switch_mm_irqs_off(prev->active_mm, next->mm,

next);
+

#ifndef ARCH_NO_ACTIVE_MM

Quoted 8 lines

+ if (!prev->mm) { // from
kernel
+ /* will mmdrop() in finish_task_switch().
*/

(quoted diff) +4 -0

+                       rq->prev_mm = prev->active_mm;
+                       prev->active_mm = NULL;
+               }
+#endif

Quoted 12 lines · Peter Zijlstra

Quoted 9 lines · Rik van Riel

Quoted 4 lines

What happened to the rework I did there? That not only avoided fiddling
with active_mm, but also avoids grab/drop cycles for the other architectures when doing task->kthread->kthread->task things.

I don't think I saw that. I only saw your email from July 20th with this fragment of code, which does not appear to avoid the grab/drop cycles, and still fiddles with active_mm:

Quoted 7 lines

Yeah, that's it. Note how it doesn't do a grab+drop for kernel-

Quoted 1 line

kernel,

where the current could would have.

And also note that it only fiddles with active_mm if it does the grab+drop thing (the below should have s/ifdef/ifndef/ to make more sense maybe).

I'll kick off a test with your variant. I don't think we will see any performance difference on x86 (due to not using a refcount at all any more), but unless Ingo is in a hurry I guess there's no issue rewriting this part of the patch series :)

Do the other patches look ok to you and Andy?

Quoted 31 lines · Rik van Riel

Quoted 15 lines · Peter Zijlstra

Quoted 11 lines · Rik van Riel

Quoted 6 lines

What happened to the rework I did there? That not only avoided
fiddling
with active_mm, but also avoids grab/drop cycles for the other
architectures when doing task->kthread->kthread->task things.

I don't think I saw that. I only saw your email from July 20th with this fragment of code, which does not appear to avoid the grab/drop cycles, and still fiddles with active_mm:

Yeah, that's it. Note how it doesn't do a grab+drop for kernel-

Quoted 8 lines

Quoted 1 line

kernel,

where the current could would have.

And also note that it only fiddles with active_mm if it does the grab+drop thing (the below should have s/ifdef/ifndef/ to make more
sense maybe).

I'll kick off a test with your variant. I don't think we will see any performance difference on x86 (due to not using a refcount at all any more), but unless Ingo is in a hurry I guess there's no issue rewriting this part of the patch series :)

Quoted 2 lines

Do the other patches look ok to you and Andy?

The whole series other than the active_mm stuff looked okay to me.

Quoted 33 lines · Andy Lutomirski

Quoted 31 lines · Rik van Riel

Quoted 15 lines · Peter Zijlstra

Quoted 11 lines · Rik van Riel

Quoted 6 lines

What happened to the rework I did there? That not only avoided
fiddling
with active_mm, but also avoids grab/drop cycles for the other
architectures when doing task->kthread->kthread->task things.

I don't think I saw that. I only saw your email from July 20th with this fragment of code, which does not appear to avoid the grab/drop cycles, and still fiddles with active_mm:

Yeah, that's it. Note how it doesn't do a grab+drop for kernel-

Quoted 8 lines

Quoted 1 line

kernel,

where the current could would have.

And also note that it only fiddles with active_mm if it does the grab+drop thing (the below should have s/ifdef/ifndef/ to make more
sense maybe).

I'll kick off a test with your variant. I don't think we will see any performance difference on x86 (due to not using a refcount at all any more), but unless Ingo is in a hurry I guess there's no issue rewriting this part of the patch series :)

Quoted 4 lines

Quoted 2 lines

Do the other patches look ok to you and Andy?

The whole series other than the active_mm stuff looked okay to me.

Does the active_mm stuff look like a step in the right direction with the bugfix, or would you prefer the code to go in an entirely different direction?

If this looks like a step in the right direction, it may make sense to make this step before the merge window opens, and continue with more patches in this direction later.

--
All Rights Reversed.

I think it's a big step in the right direction, but it still makes be nervous. I'd be more comfortable with it if you at least had a functional set of patches that result in active_mm being gone, because
that will mean that you actually audited the whole mess and fixed anything that might rely on active_mm pointing somewhere or that might
be putting a value you didn't take into account into active_mm. IOW I'm not totally thrilled by applying the patches as is if we're still a bit unsure as to what might have gotten missed.

I don't think it's at all necessary to redo the patches.

Does that seem reasonable?

Quoted 15 lines · Andy Lutomirski

I think it's a big step in the right direction, but it still makes be nervous. I'd be more comfortable with it if you at least had a functional set of patches that result in active_mm being gone, because
that will mean that you actually audited the whole mess and fixed anything that might rely on active_mm pointing somewhere or that might
be putting a value you didn't take into account into active_mm. IOW I'm not totally thrilled by applying the patches as is if we're still a bit unsure as to what might have gotten missed.

Quoted 3 lines

I don't think it's at all necessary to redo the patches.

Does that seem reasonable?

Absolutely. I tried to keep ->active_mm very similar to before for exactly that reason.

Lets go through all the places where it is used, in x86 and architecture independent code. I have not checked other architectures.

It looks like we should be able to get rid of ->active_mm at some point, but a lot of it depends on other architecture maintainers.

arch/x86/events/core.c:
- get_segment_base: get current->active_mm->context.ldt, this appears to be for TIF_IA32 user programs only, so we should be able to use current->mm here

arch/x86/kernel/cpu/common.c:
- current task's ->active_mm assigned in two places, never read

arch/x86/lib/insn-eval.c:
- get_desc() gets current->active_mm->context.ldt, this appears to be only for user space programs

arch/x86/mm/tlb.c:
- this series adds two places where current->active_mm is written, it is never read

arch/x86/platform/efi/efi_64.c:
- current->active_mm is set to efi_mm for a little bit, with irqs disabled, and then changed back, with irqs still disabled; we should be able to get rid of ->active_mm here
- in the init code, ->active_mm is set to efi_mm as well, presumably the kernel automatically switches that back on the next context switch; this may be buggy, since preemption is enabled and a GFP_KERNEL allocation is just a few lines below

arch/x86/power/cpu.c:
- fix_processor_context() calls load_mm_ldt(current->active_mm);, we should be able to use cpu_tlbstate.loaded_mm instead

drivers/cpufreq/pmac32-cpufreq.c:
- pmu_set_cpu_speed() restores current->active_mm - don't know if anyone still cares about 32 bit PPC :)

drivers/firmware/efi/arm-runtime.c:
- efi_virtmap_unload switches back the pgd to current->active_mm from &efi_mm; that mm could be stored elsewhere if we excised ->active_mm everywhere

So for ARCH_NO_ACTIVE_MM we never touch ->active_mm and therefore ->active_mm == ->mm.

Review

Quoted 11 lines

(quoted diff) +8 -0

+       /*
+        * kernel -> kernel   lazy + transfer active
+        *   user -> kernel   lazy + mmgrab() active
+        *
+        * kernel ->   user   switch + mmdrop() active
+        *   user ->   user   switch
+        */
+       if (!next->mm) {                                // to

kernel

(quoted diff) +2 -0

+               enter_lazy_tlb(prev->active_mm, next);
+

#ifndef ARCH_NO_ACTIVE_MM

Quoted 4 lines

(quoted diff) +2 -0

+               next->active_mm = prev->active_mm;
+               if (prev->mm)                           // from

user
+ mmgrab(prev->active_mm);

		else
			prev->active_mm = NULL;

Quoted 5 lines

(quoted diff) +3 -0

+#endif
+       } else {                                        // to user
+               switch_mm_irqs_off(prev->active_mm, next->mm,

next);
+

#ifndef ARCH_NO_ACTIVE_MM

Quoted 8 lines

+ if (!prev->mm) { // from
kernel
+ /* will mmdrop() in finish_task_switch().
*/

(quoted diff) +4 -0

+                       rq->prev_mm = prev->active_mm;
+                       prev->active_mm = NULL;
+               }
+#endif

Quoted 3 lines · Peter Zijlstra

So for ARCH_NO_ACTIVE_MM we never touch ->active_mm and therefore ->active_mm == ->mm.

Close, but not true for kernel threads, which have a NULL ->mm, but a non-null ->active_mm that gets passed to enter_lazy_tlb().

Quoted 7 lines · Rik van Riel

Quoted 4 lines · Peter Zijlstra

So for ARCH_NO_ACTIVE_MM we never touch ->active_mm and therefore ->active_mm == ->mm.

Quoted 3 lines

Close, but not true for kernel threads, which have a NULL ->mm, but a non-null ->active_mm that gets passed to enter_lazy_tlb().

I'm confused on the need for this. We mark the CPU lazy, why do we still care about this?

Quoted 14 lines

Quoted 10 lines · Peter Zijlstra

Quoted 5 lines · Rik van Riel

On Mon, 2018-07-30 at 18:26 +0200, Peter Zijlstra wrote:

So for ARCH_NO_ACTIVE_MM we never touch ->active_mm and therefore
->active_mm == ->mm.

Close, but not true for kernel threads, which have a NULL ->mm, but a non-null ->active_mm that gets passed to enter_lazy_tlb().

Quoted 3 lines

I'm confused on the need for this. We mark the CPU lazy, why do we still
care about this?

I have considered renaming enter_lazy_tlb() to something like lazy_switch_to_kernel_mm() (or an irqs_off variant) and making it take no parameters or maybe just task pointer parameters.

Quoted 14 lines · Andy Lutomirski

Quoted 14 lines

Quoted 10 lines · Peter Zijlstra

Quoted 5 lines · Rik van Riel

On Mon, 2018-07-30 at 18:26 +0200, Peter Zijlstra wrote:

So for ARCH_NO_ACTIVE_MM we never touch ->active_mm and therefore
->active_mm == ->mm.

Close, but not true for kernel threads, which have a NULL ->mm, but a non-null ->active_mm that gets passed to enter_lazy_tlb().

Quoted 7 lines

Quoted 3 lines

I'm confused on the need for this. We mark the CPU lazy, why do we still
care about this?

I have considered renaming enter_lazy_tlb() to something like lazy_switch_to_kernel_mm() (or an irqs_off variant) and making it take no parameters or maybe just task pointer parameters.

Of all the architectures, only Alpha uses the "mm" parameter to enter_lazy_tlb.

It uses it to store the pgd address of the current active_mm context into the thread info of the next task.

If it can get that from a per CPU variable, we can get rid of the mm parameter to enter_lazy_tlb.

This patch series implements the cleanups suggested by Peter and Andy, removes lazy TLB mm refcounting on x86, and shows how other architectures could implement that same optimization.

The previous patch series already seems to have removed most of the cache line contention I was seeing at context switch time, so CPU use of the memcache and memcache-like workloads has not changed measurably with this patch series.

However, the memory bandwidth used by the memcache system has been reduced by about 1%, to serve the same number of queries per second.

This happens on two socket Haswell and Broadwell systems. Maybe on larger systems (4 or 8 socket) one might also see a measurable drop in the amount of CPU time used, with workloads where the previous patch series does not remove all cache line contention on the mm.

This is against the latest -tip tree, and seems to be stable (on top of another tree) with workloads that do over a million context switches a second.