alpha: fix stale TLB translations breaking copy-on-write and writeback
Subsystems (2)
Magnus Lindholm
maintainerv3 cover alpha: fix stale TLB translations breaking copy-on-write and writeback
On Alpha, stale TLB translations can break copy-on-write and shared-mapping
writeback: a multi-threaded process can lose stores to its own private memory, read data belonging to its own child, and lose data written through a shared file mapping. The copy-on-write failures need more than one CPU; the writeback failure also happens on a uniprocessor.
Three related problems are fixed here.
Patch 1 is stranded deferred-ASN bookkeeping. check_mmu_context() clears asn_lock and acts on need_new_asn, but it runs only as the tail of switch_to(), after alpha_switch_to() returns. A newly forked task never gets there: its first context switch resumes at ret_from_fork instead. asn_lock is left set and the task goes on to run user space with it set and interrupts enabled, so a shootdown IPI arriving in that window takes the deferred path, and the need_new_asn handshake meant to cover it never runs. finish_task_switch() calls finish_arch_post_lock_switch() with preemption disabled, on the CPU that ran switch_mm(), which is where that bookkeeping can be completed.
kthread_use_mm() and sched_force_init_mm() also reach the same hook, outside the scheduler's preemption-disabled switch tail. check_mmu_context() acts on per-CPU state, so it can only complete this bookkeeping while still on the CPU that ran switch_mm(). The hook therefore tests preemptible() directly: where migration is possible it does nothing, while where preemption remains disabled, or is not configured, completing the bookkeeping is safe. alpha selects ARCH_NO_PREEMPT, so in an ordinary build preemptible() is a compile-time 0 and the hook runs everywhere; the test only takes effect where something turns on PREEMPT_COUNT.
The second problem is a targeted tbi() issued against the wrong context. tbi() acts on the address space context currently loaded on a CPU, so it is only guaranteed to reach an mm's translations when a thread of that mm is current. current->active_mm is not sufficient: under lazy TLB an idle or kernel task keeps an mm as its active_mm while a different ASN is loaded, so the invalidate is issued against the wrong context and the mm's stale translations can survive - and nothing retires the old ASN either, mm->context[cpu] still being valid, so the resuming thread reuses it. Patch 2 fixes the shootdown IPI handler, patch 3 the local side of flush_tlb_page(), and patch 5 the uniprocessor flush_tlb_page().
The third is an omitted caller-side invalidate, covered by patches 4, 6 and 7: when the target mm is not the calling CPU's active_mm, nothing invalidates that CPU at all, because smp_call_function() does not call back into the caller. The UP flush_tlb_mm() and flush_icache_user_page() already contain exactly the missing branch. Their active_mm tests are left alone:
both load a fresh context through __load_new_mm_context() rather than a targeted tbi() against whatever ASN happened to be loaded, and only the targeted tbi() depends on which context is loaded.
None of these patches covers a task that borrowed an mm through kthread_use_mm(), and patch 1 makes that case worse. Such a task has current->mm set while ev5_switch_mm() has only prepared the PCB and nothing has installed it. Today asn_lock stays set for the whole of the borrow, because check_mmu_context() runs only from switch_to(), so a shootdown IPI for that mm finds asn_locked() true and takes the conservative flush_tlb_other() path, retiring mm->context[cpu]. With patch 1's hook the lock is cleared when kthread_use_mm() returns, so the IPI instead issues a targeted tbi() against the context that is actually loaded and leaves the slot valid. Loading the context on the direct switch closes that window; where both are applied, that change belongs before patch 1. It is the third patch of the follow-up series:
https://lore.kernel.org/linux-alpha/[email protected]/
No user-space data race is involved in the reproducers: every slot is written and read by one thread only, and the main thread inspects them only after joining the workers. The other writes come from forked children with their own address space, so observing one of those in the parent is the bug.
Deferred-path behaviour after the fixes, counted with the counters reset before each 6-second run of the lost-store reproducer:
lost stores runs entering the window
no fixes 11 of 40 13 of 40, 11 of them lost
patched 0 of 400 15 of 400, none lost
Where the remaining patches are reached, counted the same way, on a CONFIG_COMPACTION=n kernel:
thread of mm lazily
current borrowing
writeback of a shared mapping 51577 50688
reclaim under memory pressure 17609 14827
anonymous COW / fork 144237 0
About half the calls during writeback, and none at all on anonymous memory, which is why patches 1 and 2 did not cover it. flush_tlb_mm() was entered with the mm not this CPU's active_mm 2934 times over a fork-heavy run and 632 times while otherwise idle.
The writeback row is config-dependent, which v2 did not say. With CONFIG_COMPACTION=y, asm/pgtable.h overrides ptep_clear_flush() to call migrate_flush_tlb_page(), which rendezvouses with every CPU and handles the context itself, so folio_mkclean() does not reach flush_tlb_page() at all. (That the override is guarded by CONFIG_COMPACTION rather than CONFIG_MIGRATION looks like a defect of its own - MEMORY_HOTREMOVE, NUMA_MIGRATION, MEMORY_FAILURE and CMA all select MIGRATION without it - but that is a separate patch.)
Reproducing it. Two self-contained tests were written for this. Source for both can be made available on request.
alpha-cow-smoketest.c covers patches 1 and 2 (pthreads only, ~40s, needs more than one CPU; confined to one with taskset -c 0 it does not fail). A failing run reports:
stale read after COW fault FAIL thread 2 read 0xdeadbeefcafebabe, expected 0x1000002 <- the CHILD's value
Two details in it matter, because getting either wrong hides the bug:
slots are 128 bytes apart so several threads share a page, and a thread writes its slot once then reads it many times, since a thread that keeps writing refreshes its own translation. Its lost-stores check fires in at best a quarter of runs; the stale-read check is the reliable one.
mkclean4.c exercises the combined SMP fix in patches 3 and 4, and the UP fix in patch 5. It needs root and CONFIG_COMPACTION=n. On SMP it takes
both: the flusher kworker has no mm of its own, so patch 3's current->mm test sends it to the branch patch 4 adds, and without patch 4 the calling CPU - the one holding the writer's translations, and the one smp_call_function() does not call back into - is still left alone. The UP implementation already has that branch, so patch 5 is the whole fix there.
A single thread writes a small MAP_SHARED file while background writeback cleans it, and the file is then compared against the mapping. It forces two conditions that are rare in normal operation on SMP: the flusher kworker and the writer on the same CPU, and a working set small enough to stay resident in the data TLB. That is also why this is hard to hit in the field - any faulting write from any CPU repairs the dirty state. On a uniprocessor the first condition holds by construction, so no pinning is needed there.
On COMPACTION=y the test does not exercise this defect, for the reason above, and passes on an unpatched kernel: 0 failures in 103 rounds across three unpatched
COMPACTION=y configurations. v2's Testing section did not record which kernel its mkclean4 numbers came from, and this is the correction.
Nothing isolates patch 4 from patch 3; the two are exercised together above. Patches 6 and 7 have no reproducer for the bug they fix; they are justified by the contract of the functions, by the UP implementations already having the missing branch, and by the counts above. Patch 7's path was exercised for regressions by driving gdb to set and clear a breakpoint several hundred times, reaching copy_to_user_page() -> flush_icache_user_page().
Originally found as intermittent heap corruption in glibc's malloc/tst-malloc-fork-deadlock-malloc-check. glibc is not at fault: with MALLOC_CHECK_=3 it is simply a very effective detector, and every fork() runs __malloc_fork_unlock_child() in the child, which writes to allocator state.
Testing. ES40, EV68AL (21264C) Tsunami, 3 CPUs, on v7.2-rc2 and v7.2-rc6.
before after smoke test, stale-read check 7 of 9 rounds 0 of 9 lost stores 11 of 40 0 of 400 writeback (mkclean4), SMP [*] 10 of 10 0 of 10 writeback (mkclean4), UP [*] every round 0 of 8
tst-malloc-fork-deadlock-
malloc-check 8 of 10 fail 25/25 pass
ptrace breakpoint exerciser - result matches
[*] CONFIG_COMPACTION=n; see above.
Also 10/10 pass each for tst-malloc-fork-deadlock, tst-malloc-check, tst-tcfree1-malloc-check and tst-tcfree2-malloc-check. No measurable
cost: 1470/1038/206 forks per second with 0/2/8 sibling threads against 1472/1043/199 unpatched. Patch 6 changes a function none of the reproducers exercise, so it is covered for regressions only.
Also run with CONFIG_COMPACTION and CONFIG_MIGRATION enabled, with compaction forced continuously underneath the tests: 368522 folios migrated during the run, no failures.
Also built and tested with CONFIG_ALPHA_GENERIC and CONFIG_SMP=n. That is how patch 5 was found: arch/alpha/kernel/smp.c is not built there, so patches 2, 3, 4, 6 and 7 are absent and patch 1 is inert, and mkclean4.c failed on every round because the uniprocessor flush_tlb_page() carries the same defect. With patch 5 it passes 8 of 8, and the rest of the tests above pass there too.
Matt Turner tested this series on an ES47 (EV7) on v7.3-rc1, one CPU
online: no regressions in a gdb breakpoint exerciser, a fork/COW check, or a writeback test, on both an SMP-enabled and a CONFIG_SMP=n kernel. He could not reproduce the writeback failure on that machine on an unpatched kernel with CONFIG_COMPACTION either way, while confirming with counters that the unpatched flush_tlb_page() does issue the targeted tbi() against a foreign context a few hundred times per run. One reading is that the EV7 PALcode invalidates by VA without matching the ASN, which would make the same defect unobservable there. Patches 3 and 5 rest on the contract of tbi() either way.
Changes since v2:
- patch 1 no longer claims that the hook does nothing after kthread_use_mm(). alpha selects ARCH_NO_PREEMPT, so preemptible() is a compile-time 0 in an ordinary build and the hook runs there too, clearing asn_lock; the claim held only for PREEMPT_COUNT=y. Patch 2's last paragraph carried the same error and is corrected with it. The shootdown handlers therefore do not cover a borrowed mm, which is now said plainly in both patches and in this cover letter.
- patch 1 records that moving the call from switch_to() to finish_arch_post_lock_switch() means the bookkeeping now runs with interrupts enabled, and why that window is safe. v2 described this as no functional change, which undersold it.
- patch 1 now records that it makes the borrowed-mm case worse, rather than v2's claim that swapping the active_mm test for current->mm leaves it unchanged. That comparison was right about the two tests and wrong about the series, because patch 1 also changes when asn_lock is cleared. Patch 2 no longer draws the "no worse" conclusion either.
- patch 3 says that its folio_mkclean() counts are from a CONFIG_COMPACTION=n kernel, and that COMPACTION=y routes folio_mkclean() away from flush_tlb_page() entirely.
- the reproducer is described as exercising patches 3 and 4 together on SMP, and patch 5 on UP. v2 credited it to patches 3 and 5 and listed patch 4 as having no reproducer, which was wrong: patch 3 alone leaves the calling CPU without any local invalidate.
- patches 2, 3 and 5 describe the targeted tbi() as guaranteed only when the target mm's context is loaded, rather than as failing on every implementation. Matt's EV7 result is that a mismatched loaded ASN did not produce the failure there, and the argument for these patches does not depend on which way that goes.
- the mkclean4 rows in Testing are labelled with the config they were measured on, and the COMPACTION=y result is recorded.
- rebased from v7.2-rc6 onto v7.3-rc1.
- no code changes.
Thanks to Matt Turner for the review, the EV7 testing, and for finding the CONFIG_COMPACTION dependency.
v2: https://lore.kernel.org/linux-alpha/[email protected]/
Magnus Lindholm (7):
alpha: run check_mmu_context() from finish_arch_post_lock_switch()
alpha: only use a targeted tbi() when the target mm is really current
alpha: fix the local TLB invalidate in flush_tlb_page()
alpha: invalidate the local context in flush_tlb_page()
alpha: fix the local TLB invalidate in the UP flush_tlb_page()
alpha: invalidate the local context in flush_tlb_mm()
alpha: invalidate the local context in flush_icache_user_page()
arch/alpha/include/asm/mmu_context.h | 8 ++++++++ arch/alpha/include/asm/switch_to.h | 1 - arch/alpha/include/asm/tlbflush.h | 3 ++- arch/alpha/kernel/smp.c | 15 +++++++++++++-- 4 files changed, 23 insertions(+), 4 deletions(-)
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935 --
2.43.0
Magnus Lindholm
maintainerPATCH v3 1/7 alpha: run check_mmu_context() from finish_arch_post_lock_switch()
check_mmu_context() clears asn_lock and acts on need_new_asn, but it runs only as the tail of switch_to(), after alpha_switch_to() returns. A newly forked task never gets there: its first context switch resumes at ret_from_fork, which goes to schedule_tail() and then to user space rather than returning to the code following alpha_switch_to(). A new kernel thread reaches schedule_tail() the same way, through ret_from_kernel_thread().
asn_lock is left set on that CPU, so the forked task runs user space with it set and interrupts enabled. A TLB shootdown IPI arriving in that window takes the deferred path, and the need_new_asn handshake meant to cover that never runs.
finish_task_switch() calls finish_arch_post_lock_switch() with preemption disabled, on the CPU that ran switch_mm(), so hooking check_mmu_context() there completes the bookkeeping for both. The existing call from switch_to() then becomes redundant, since finish_task_switch() runs immediately afterwards and does the same work, so drop it.
kthread_use_mm() and sched_force_init_mm() reach the same hook outside the scheduler's preemption-disabled switch tail, where the CPU may have changed since switch_mm(). check_mmu_context() acts on per-CPU state, so testing preemptible() expresses the required condition directly rather than naming particular callers. alpha selects ARCH_NO_PREEMPT, so unless something else turns on PREEMPT_COUNT the test is a compile-time 0 and the hook runs everywhere, including at the end of kthread_use_mm(); it only takes effect in PREEMPT_COUNT builds.
Moving the call changes when it runs. switch_to() ran it with the rq lock held and interrupts off, while finish_arch_post_lock_switch() runs after finish_lock_switch() has dropped that lock and re-enabled interrupts, with preemption still disabled. A shootdown IPI taken in that window either finds asn_lock still set and defers, or finds it already cleared and flushes directly and by then PAL_swpctx has installed the incoming context, so the direct flush acts on the right one. need_new_asn is only ever set while asn_lock is 1.
It also changes one case for the worse, which the series that follows does not fix. Before this patch nothing cleared asn_lock during a kthread_use_mm() borrow, since check_mmu_context() ran only from switch_to(), so a shootdown IPI for the borrowed mm found asn_locked() true and took the conservative flush_tlb_other() path, retiring mm->context[cpu]. With the hook in place asn_lock is cleared when kthread_use_mm() returns, so such an IPI instead issues a targeted tbi() against a context that ev5_switch_mm() prepared but never installed, and leaves the slot valid. Loading the context on a direct mm switch closes that window; where both changes are applied, that one belongs first.
--- arch/alpha/include/asm/mmu_context.h | 8 ++++++++ arch/alpha/include/asm/switch_to.h | 1 - 2 files changed, 8 insertions(+), 1 deletion(-)
arch/alpha/include/asm/mmu_context.h +8 -0
@@ -181,6 +181,14 @@ do { \#define check_mmu_context () do { } while(0)#endif +/* Per-CPU state: only safe while still on the switching CPU. */ +#define finish_arch_post_lock_switch finish_arch_post_lock_switch +static inline void finish_arch_post_lock_switch (void ) +{ +if (!preemptible ()) +check_mmu_context (); +} + __EXTERN_INLINE voidev5_activate_mm (struct mm_struct *prev_mm,struct mm_struct *next_mm) {
arch/alpha/include/asm/switch_to.h +0 -1
@@ -9,7 +9,6 @@ extern struct task_struct *alpha_switch_to(unsigned long, struct task_struct *);#define switch_to (P ,N ,L ) \ do { \ (L) = alpha_switch_to(virt_to_phys(&task_thread_info(N)->pcb), (P)); \ - check_mmu_context(); \ } while (0)#endif /* __ALPHA_SWITCH_TO_H */
Signature · 1 line
2.43.0
Matt Turner
maintainerRe: [PATCH v3 1/7] alpha: run check_mmu_context() from finish_arch_post_lock_switch()
Quoted 5 lines · Magnus Lindholm
so testing
preemptible()expresses the required condition directly rather than naming particular callers. alpha selectsARCH_NO_PREEMPT, so unless something else turns onPREEMPT_COUNTthe test is a compile-time 0 and the hook runs everywhere, including at the end ofkthread_use_mm(); it only takes effect inPREEMPT_COUNTbuilds.
The only way I can find to get PREEMPT_COUNT on alpha is RCU_STRICT_GRACE_PERIOD. DEBUG_ATOMIC_SLEEP depends on !ARCH_NO_PREEMPT and PROVE_LOCKING only selects it if !ARCH_NO_PREEMPT.
In that config preemptible() is true at the end of kthread_use_mm(), but the task still cannot migrate there: there is no kernel preemption and nothing sleeps between switch_mm_irqs_off() and the hook. So the test never prevents anything, and its only effect is that one debug config leaves asn_lock set where every other config clears it. I would drop it.
Quoted 5 lines
A shootdown IPI taken in
that window either findsasn_lockstill set and defers, or finds it already cleared and flushes directly and by thenPAL_swpctxhas installed the incoming context, so the direct flush acts on the right one.need_new_asnis only ever set whileasn_lockis 1.
The deferred case has two halves. If ev5_switch_mm() reused the ASN, need_new_asn is set and check_mmu_context() reloads. If it allocated a new one, need_new_asn is 0, the IPI zeroes mm->context[cpu], and the task runs on with a zero slot. Nothing is stale at that point, but the follow-up series reads a zero slot as "not running here". More in my reply to v2 3/3 there.
Magnus Lindholm
maintainerPATCH v3 2/7 alpha: only use a targeted tbi() when the target mm is really current
In reply to Magnus Lindholmipi_flush_tlb_page() gates a targeted tbi() on current->active_mm. tbi() acts on the address space context currently loaded on the CPU, so it is only guaranteed to reach an mm's translations when a thread of that mm is running there. current->active_mm is not sufficient: under lazy TLB an idle or kernel task keeps an mm as its active_mm while a different ASN is loaded, so the invalidate is issued against the wrong context and the stale entry can survive. Nothing retires the old ASN afterwards either, mm->context[cpu] still being valid, so the resuming thread can reuse it.
Test current->mm instead and otherwise fall back to flush_tlb_other(), which clears mm->context[cpu] and forces a fresh ASN at the next switch whatever is loaded now.
This does not cover a task that borrowed an mm through kthread_use_mm(): both comparisons match such a task, which has current->mm set while ev5_switch_mm() has only prepared the PCB and nothing has issued the PAL_swpctx that installs it. asn_locked() does not identify that window either, for the reason given in the previous patch. Loading the context on a direct mm switch is what fixes that case, and is a separate change.
--- arch/alpha/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/alpha/kernel/smp.c +2 -1
@@ -669,7 +669,8 @@ ipi_flush_tlb_page(void *x)struct flush_tlb_page_struct *data = x;struct mm_struct * mm = data->mm; -if (mm == current->active_mm && !asn_locked ()) +/* A targeted tbi() needs a thread of MM to be current. */ +if (mm == current->mm && !asn_locked ())flush_tlb_current_page (mm, data->vma, data->addr);else flush_tlb_other (mm);
Signature · 1 line
2.43.0
Magnus Lindholm
maintainerPATCH v3 3/7 alpha: fix the local TLB invalidate in flush_tlb_page()
In reply to Magnus Lindholmflush_tlb_page() invalidates the calling CPU itself before asking the others, and gates that on current->active_mm. For a non-executable vma that means a targeted tbi(2, addr), which acts on the context currently loaded and is only guaranteed to invalidate the intended translations when the target mm's context is the loaded one. A lazy active_mm does not establish that, so as in ipi_flush_tlb_page() the target mm's stale translations can survive, and nothing forces the old ASN to be retired afterwards.
Test current->mm instead. The calling CPU then has no local invalidate at all when the mm is only lazily borrowed; the next patch adds it.
Which callers reach here with a foreign mm depends on the configuration. folio_mkclean(), from the writeback flusher kworker that has no mm of its own, accounted for about half the calls during writeback of a shared mapping and none at all on anonymous memory. Those counts were measured with CONFIG_COMPACTION=n: with COMPACTION=y, asm/pgtable.h overrides ptep_clear_flush() to call migrate_flush_tlb_page(), which rendezvouses with every CPU and handles the context itself, so folio_mkclean() does not reach this function.
--- arch/alpha/kernel/smp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/alpha/kernel/smp.c +2 -1
@@ -684,7 +684,8 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)preempt_disable (); -if (mm == current->active_mm) { +/* As in ipi_flush_tlb_page(): a targeted tbi() needs MM current. */ +if (mm == current->mm) {flush_tlb_current_page (mm, vma, addr);if (atomic_read (&mm->mm_users) <=1 ) {int cpu, this_cpu =smp_processor_id ();
Signature · 1 line
2.43.0
Matt Turner
maintainerRe: [PATCH v3 3/7] alpha: fix the local TLB invalidate in flush_tlb_page()
Quoted 2 lines · Magnus Lindholm
Test
current->mminstead. The calling CPU then has no local invalidate at all when the mm is only lazily borrowed; the next patch adds it.
Please swap this with 4/7. The else branch is correct on its own, since it only covers mm != active_mm until this patch widens it. In the current order a bisect that lands here has a lazy caller doing nothing locally, which is worse than before on a machine where the tbi() against the wrong ASN happens to work, as it appears to on EV7.
Magnus Lindholm
maintainerPATCH v3 4/7 alpha: invalidate the local context in flush_tlb_page()
In reply to Magnus LindholmWhen the target mm is not current, flush_tlb_page() does nothing locally, and smp_call_function() reaches only the other CPUs. This CPU may still hold translations for the mm and can later reuse the old ASN together with them.
Add the missing else branch. flush_tlb_other() clears mm->context[cpu] so a fresh ASN is taken at the next switch. With the previous patch this also covers a lazily borrowed mm, which no longer takes the targeted path.
The uniprocessor implementations of flush_tlb_mm() and flush_icache_user_page() in asm/tlbflush.h and asm/cacheflush.h already contain exactly this branch.
--- arch/alpha/kernel/smp.c | 3 +++ 1 file changed, 3 insertions(+)
arch/alpha/kernel/smp.c +3 -0
@@ -698,6 +698,9 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)preempt_enable ();return ; } + }else { +/* smp_call_function() does not call back into this CPU. */ +flush_tlb_other (mm); } data.vma = vma;
Signature · 1 line
2.43.0
Magnus Lindholm
maintainerPATCH v3 5/7 alpha: fix the local TLB invalidate in the UP flush_tlb_page()
In reply to Magnus LindholmThe uniprocessor flush_tlb_page() in asm/tlbflush.h has the same wrong test as the SMP one: a targeted tbi() is only guaranteed to invalidate the intended translations when the target mm's context is loaded, but the gate is current->active_mm, which a lazily borrowed mm also satisfies.
Test current->mm instead. The else branch it falls to already exists here, so unlike the SMP side this is the whole fix. arch/alpha/kernel/smp.c is not built with CONFIG_SMP=n, so this is how the same defect reaches a uniprocessor, where the flusher kworker necessarily shares the only CPU with the writer.
--- arch/alpha/include/asm/tlbflush.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/alpha/include/asm/tlbflush.h +2 -1
@@ -87,7 +87,8 @@ flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) {struct mm_struct *mm = vma->vm_mm; -if (mm == current->active_mm) +/* A targeted tbi() needs a thread of MM to be current. */ +if (mm == current->mm)flush_tlb_current_page (mm, vma, addr);else flush_tlb_other (mm);
Signature · 1 line
2.43.0
Magnus Lindholm
maintainerPATCH v3 6/7 alpha: invalidate the local context in flush_tlb_mm()
In reply to Magnus Lindholmflush_tlb_mm() has the same caller-CPU omission that was fixed in flush_tlb_page(): when the target mm is not the calling CPU's active_mm nothing happens locally, and smp_call_function() handles only the other CPUs, so this CPU may later reuse the old ASN together with the translations it still holds.
The active_mm test itself is left alone here. That path calls flush_tlb_current(), which loads a fresh context through __load_new_mm_context() rather than issuing a targeted tbi() against whatever ASN happens to be loaded, so it does not depend on which context is current.
The uniprocessor implementation in asm/tlbflush.h already has the missing branch. Counted over a fork-heavy run, flush_tlb_mm() was entered with the mm not this CPU's active_mm 2934 times, and 632 times while otherwise idle.
--- arch/alpha/kernel/smp.c | 3 +++ 1 file changed, 3 insertions(+)
arch/alpha/kernel/smp.c +3 -0
@@ -649,6 +649,9 @@ flush_tlb_mm(struct mm_struct *mm)preempt_enable ();return ; } + }else { +/* smp_call_function() does not call back into this CPU. */ +flush_tlb_other (mm); }smp_call_function (ipi_flush_tlb_mm, mm,1 );
Signature · 1 line
2.43.0
Re: [PATCH v3 6/7] alpha: invalidate the local context in flush_tlb_mm()
Quoted 3 lines · Magnus Lindholm
The uniprocessor implementation in
asm/tlbflush.halready has the missing branch. Counted over a fork-heavy run,flush_tlb_mm()was entered with the mm not this CPU'sactive_mm2934 times, and 632 times while otherwise idle.
With the hugetlb series applied this path gets busier: every hugetlb flush there goes through flush_tlb_range(), so unmapping or migrating a hugetlb folio from another context ends up here with a foreign mm. That might also give you the reproducer this patch lacks: move_pages() on another process's hugetlb mapping, with the caller pinned to the CPU the target last ran on.
Magnus Lindholm
maintainerPATCH v3 7/7 alpha: invalidate the local context in flush_icache_user_page()
In reply to Magnus Lindholmflush_icache_user_page() has the same caller-CPU omission that the previous patch fixed in flush_tlb_mm(): when the target mm is not the calling CPU's active_mm nothing happens locally, and smp_call_function() handles only the other CPUs, so this CPU may later reuse the old ASN together with the translations it still holds.
This matters here in particular because the function exists for operating on another process's mappings: the comment above it describes setting breakpoints through ptrace, and access_remote_vm() reaches it through copy_to_user_page(). The calling CPU is therefore often running something other than the target mm.
As in flush_tlb_mm(), the uniprocessor implementation in asm/cacheflush.h already has the missing case.
No imb() is needed, here or in ipi_flush_icache_page(). Alpha's user-space I-cache flush works by allocating a new ASN rather than by invalidating the I-cache: the entries stay, but they are tagged with the old ASN and can no longer match. An imb() is only required when the ASN space wraps and numbers are reused, and __get_new_mm_context() already does one in that case.
--- arch/alpha/kernel/smp.c | 3 +++ 1 file changed, 3 insertions(+)
arch/alpha/kernel/smp.c +3 -0
@@ -758,6 +758,9 @@ flush_icache_user_page(struct vm_area_struct *vma, struct page *page,preempt_enable ();return ; } + }else { +/* smp_call_function() does not call back into this CPU. */ +flush_tlb_other (mm); }smp_call_function (ipi_flush_icache_page, mm,1 );
Signature · 1 line
2.43.0
Quoted 234 lines · Magnus Lindholm
On Alpha, stale TLB translations can break copy-on-write and shared-mapping
writeback: a multi-threaded process can lose stores to its own private memory, read data belonging to its own child, and lose data written through a shared file mapping. The copy-on-write failures need more than one CPU; the writeback failure also happens on a uniprocessor.Three related problems are fixed here.
Patch 1 is stranded deferred-ASN bookkeeping.
check_mmu_context()clearsasn_lockand acts onneed_new_asn, but it runs only as the tail ofswitch_to(), afteralpha_switch_to()returns. A newly forked task never gets there: its first context switch resumes atret_from_forkinstead.asn_lockis left set and the task goes on to run user space with it set and interrupts enabled, so a shootdown IPI arriving in that window takes the deferred path, and theneed_new_asnhandshake meant to cover it never runs.finish_task_switch()callsfinish_arch_post_lock_switch()with preemption disabled, on the CPU that ranswitch_mm(), which is where that bookkeeping can be completed.
kthread_use_mm()andsched_force_init_mm()also reach the same hook, outside the scheduler's preemption-disabled switch tail.check_mmu_context()acts on per-CPU state, so it can only complete this bookkeeping while still on the CPU that ranswitch_mm(). The hook therefore testspreemptible()directly: where migration is possible it does nothing, while where preemption remains disabled, or is not configured, completing the bookkeeping is safe. alpha selectsARCH_NO_PREEMPT, so in an ordinary buildpreemptible()is a compile-time 0 and the hook runs everywhere; the test only takes effect where something turns onPREEMPT_COUNT.The second problem is a targeted
tbi()issued against the wrong context.tbi()acts on the address space context currently loaded on a CPU, so it is only guaranteed to reach an mm's translations when a thread of that mm is current.current->active_mmis not sufficient: under lazy TLB an idle or kernel task keeps an mm as itsactive_mmwhile a different ASN is loaded, so the invalidate is issued against the wrong context and the mm's stale translations can survive - and nothing retires the old ASN either, mm->context[cpu] still being valid, so the resuming thread reuses it. Patch 2 fixes the shootdown IPI handler, patch 3 the local side offlush_tlb_page(), and patch 5 the uniprocessorflush_tlb_page().The third is an omitted caller-side invalidate, covered by patches 4, 6 and 7: when the target mm is not the calling CPU's
active_mm, nothing invalidates that CPU at all, becausesmp_call_function()does not call back into the caller. The UPflush_tlb_mm()andflush_icache_user_page()already contain exactly the missing branch. Theiractive_mmtests are left alone:
both load a fresh context through__load_new_mm_context()rather than a targetedtbi()against whatever ASN happened to be loaded, and only the targetedtbi()depends on which context is loaded.None of these patches covers a task that borrowed an mm through
kthread_use_mm(), and patch 1 makes that case worse. Such a task hascurrent->mmset whileev5_switch_mm()has only prepared the PCB and nothing has installed it. Todayasn_lockstays set for the whole of the borrow, becausecheck_mmu_context()runs only fromswitch_to(), so a shootdown IPI for that mm findsasn_locked()true and takes the conservativeflush_tlb_other()path, retiring mm->context[cpu]. With patch 1's hook the lock is cleared whenkthread_use_mm()returns, so the IPI instead issues a targetedtbi()against the context that is actually loaded and leaves the slot valid. Loading the context on the direct switch closes that window; where both are applied, that change belongs before patch 1. It is the third patch of the follow-up series:https://lore.kernel.org/linux-alpha/[email protected]/
No user-space data race is involved in the reproducers: every slot is written and read by one thread only, and the main thread inspects them only after joining the workers. The other writes come from forked children with their own address space, so observing one of those in the parent is the bug.
Deferred-path behaviour after the fixes, counted with the counters reset before each 6-second run of the lost-store reproducer:
lost stores runs entering the window
no fixes 11 of 40 13 of 40, 11 of them lost
patched 0 of 400 15 of 400, none lostWhere the remaining patches are reached, counted the same way, on a
CONFIG_COMPACTION=nkernel:thread of mm lazily
current borrowing
writeback of a shared mapping 51577 50688
reclaim under memory pressure 17609 14827
anonymous COW / fork 144237 0About half the calls during writeback, and none at all on anonymous memory, which is why patches 1 and 2 did not cover it.
flush_tlb_mm()was entered with the mm not this CPU'sactive_mm2934 times over a fork-heavy run and 632 times while otherwise idle.The writeback row is config-dependent, which v2 did not say. With
CONFIG_COMPACTION=y,asm/pgtable.hoverridesptep_clear_flush()to callmigrate_flush_tlb_page(), which rendezvouses with every CPU and handles the context itself, sofolio_mkclean()does not reachflush_tlb_page()at all. (That the override is guarded byCONFIG_COMPACTIONrather thanCONFIG_MIGRATIONlooks like a defect of its own -MEMORY_HOTREMOVE,NUMA_MIGRATION,MEMORY_FAILUREand CMA all select MIGRATION without it - but that is a separate patch.)Reproducing it. Two self-contained tests were written for this. Source for both can be made available on request.
alpha-cow-smoketest.ccovers patches 1 and 2 (pthreads only, ~40s, needs more than one CPU; confined to one with taskset -c 0 it does not fail). A failing run reports:stale read after COW fault FAIL
thread 2 read0xdeadbeefcafebabe, expected0x1000002<- the CHILD's valueTwo details in it matter, because getting either wrong hides the bug:
slots are 128 bytes apart so several threads share a page, and a thread writes its slot once then reads it many times, since a thread that keeps writing refreshes its own translation. Its lost-stores check fires in at best a quarter of runs; the stale-read check is the reliable one.
mkclean4.cexercises the combined SMP fix in patches 3 and 4, and the UP fix in patch 5. It needs root andCONFIG_COMPACTION=n. On SMP it takes
both: the flusher kworker has no mm of its own, so patch 3'scurrent->mmtest sends it to the branch patch 4 adds, and without patch 4 the calling CPU - the one holding the writer's translations, and the onesmp_call_function()does not call back into - is still left alone. The UP implementation already has that branch, so patch 5 is the whole fix there.A single thread writes a small
MAP_SHAREDfile while background writeback cleans it, and the file is then compared against the mapping. It forces two conditions that are rare in normal operation on SMP: the flusher kworker and the writer on the same CPU, and a working set small enough to stay resident in the data TLB. That is also why this is hard to hit in the field - any faulting write from any CPU repairs the dirty state. On a uniprocessor the first condition holds by construction, so no pinning is needed there.On COMPACTION=y the test does not exercise this defect, for the reason above, and passes on an unpatched kernel: 0 failures in 103 rounds across three unpatched
COMPACTION=y configurations. v2's Testing section did not record which kernel its mkclean4 numbers came from, and this is the correction.Nothing isolates patch 4 from patch 3; the two are exercised together above. Patches 6 and 7 have no reproducer for the bug they fix; they are justified by the contract of the functions, by the UP implementations already having the missing branch, and by the counts above. Patch 7's path was exercised for regressions by driving gdb to set and clear a breakpoint several hundred times, reaching
copy_to_user_page()->flush_icache_user_page().Originally found as intermittent heap corruption in glibc's malloc/tst-malloc-fork-deadlock-malloc-check. glibc is not at fault: with MALLOC_CHECK_=3 it is simply a very effective detector, and every
fork()runs__malloc_fork_unlock_child()in the child, which writes to allocator state.Testing. ES40, EV68AL (21264C) Tsunami, 3 CPUs, on v7.2-rc2 and v7.2-rc6.
before after smoke test, stale-read check 7 of 9 rounds 0 of 9 lost stores 11 of 40 0 of 400 writeback (mkclean4), SMP [*] 10 of 10 0 of 10 writeback (mkclean4), UP [*] every round 0 of 8tst-malloc-fork-deadlock-
malloc-check 8 of 10 fail 25/25 pass
ptrace breakpoint exerciser - result matches[*]
CONFIG_COMPACTION=n; see above.Also 10/10 pass each for tst-malloc-fork-deadlock, tst-malloc-check, tst-tcfree1-malloc-check and tst-tcfree2-malloc-check. No measurable
cost: 1470/1038/206 forks per second with 0/2/8 sibling threads against 1472/1043/199 unpatched. Patch 6 changes a function none of the reproducers exercise, so it is covered for regressions only.Also run with
CONFIG_COMPACTIONandCONFIG_MIGRATIONenabled, with compaction forced continuously underneath the tests: 368522 folios migrated during the run, no failures.Also built and tested with
CONFIG_ALPHA_GENERICandCONFIG_SMP=n. That is how patch 5 was found:arch/alpha/kernel/smp.cis not built there, so patches 2, 3, 4, 6 and 7 are absent and patch 1 is inert, andmkclean4.cfailed on every round because the uniprocessorflush_tlb_page()carries the same defect. With patch 5 it passes 8 of 8, and the rest of the tests above pass there too.Matt Turner tested this series on an ES47 (EV7) on v7.3-rc1, one CPU
online: no regressions in a gdb breakpoint exerciser, a fork/COW check, or a writeback test, on both an SMP-enabled and aCONFIG_SMP=nkernel. He could not reproduce the writeback failure on that machine on an unpatched kernel withCONFIG_COMPACTIONeither way, while confirming with counters that the unpatchedflush_tlb_page()does issue the targetedtbi()against a foreign context a few hundred times per run. One reading is that the EV7 PALcode invalidates by VA without matching the ASN, which would make the same defect unobservable there. Patches 3 and 5 rest on the contract oftbi()either way.Changes since v2:
- patch 1 no longer claims that the hook does nothing afterkthread_use_mm(). alpha selectsARCH_NO_PREEMPT, sopreemptible()is a compile-time 0 in an ordinary build and the hook runs there too, clearingasn_lock; the claim held only for PREEMPT_COUNT=y. Patch 2's last paragraph carried the same error and is corrected with it. The shootdown handlers therefore do not cover a borrowed mm, which is now said plainly in both patches and in this cover letter.
- patch 1 records that moving the call fromswitch_to()tofinish_arch_post_lock_switch()means the bookkeeping now runs with interrupts enabled, and why that window is safe. v2 described this as no functional change, which undersold it.
- patch 1 now records that it makes the borrowed-mm case worse, rather than v2's claim that swapping theactive_mmtest forcurrent->mmleaves it unchanged. That comparison was right about the two tests and wrong about the series, because patch 1 also changes whenasn_lockis cleared. Patch 2 no longer draws the "no worse" conclusion either.
- patch 3 says that itsfolio_mkclean()counts are from aCONFIG_COMPACTION=nkernel, and that COMPACTION=y routesfolio_mkclean()away fromflush_tlb_page()entirely.
- the reproducer is described as exercising patches 3 and 4 together on SMP, and patch 5 on UP. v2 credited it to patches 3 and 5 and listed patch 4 as having no reproducer, which was wrong: patch 3 alone leaves the calling CPU without any local invalidate.
- patches 2, 3 and 5 describe the targetedtbi()as guaranteed only when the target mm's context is loaded, rather than as failing on every implementation. Matt's EV7 result is that a mismatched loaded ASN did not produce the failure there, and the argument for these patches does not depend on which way that goes.
- the mkclean4 rows in Testing are labelled with the config they were measured on, and the COMPACTION=y result is recorded.
- rebased from v7.2-rc6 onto v7.3-rc1.
- no code changes.Thanks to Matt Turner for the review, the EV7 testing, and for finding the
CONFIG_COMPACTIONdependency.
Quoted 8 lines
Magnus Lindholm (7):
alpha: runcheck_mmu_context()fromfinish_arch_post_lock_switch()
alpha: only use a targetedtbi()when the target mm is really current
alpha: fix the local TLB invalidate inflush_tlb_page()
alpha: invalidate the local context inflush_tlb_page()
alpha: fix the local TLB invalidate in the UPflush_tlb_page()
alpha: invalidate the local context inflush_tlb_mm()
alpha: invalidate the local context inflush_icache_user_page()
on patches 2, 5, 7 (and 6 with a change, see separate reply).