PREEMPT_RT: livelock between lockless page cache lookup and the folio freeze->removal window
Subsystems (2)
Hello,
We hit a permanent livelock on a PREEMPT_RT system and traced it to an interaction between the lockless page cache lookup retry loop in filemap_get_entry() and the folio freeze -> xarray removal window in __remove_mapping() (and the migrate equivalents). On non-RT that window is atomic (IRQs disabled) and therefore bounded; on PREEMPT_RT it is preemptible and can become unbounded, which turns the "refcount == 0 is transient" assumption of the lookup path into a permanent spin.
The affected code is vanilla upstream (our mm/filemap.c, mm/vmscan.c and mm/migrate.c match 6.12 exactly); nothing vendor-specific is involved.
The obvious fix -- making the window non-preemptible on RT -- does not work for the reclaim path, which is the one we hit (section 4.1). We are deliberately not sending a patch yet: we would like to hear which fix direction upstream prefers before going further.
1. Symptom
----------
- 6.12.73-rt15 (PREEMPT_RT), arm64, 20 CPUs.
- A SCHED_FIFO 99 userspace workload was page-faulting on file-backed code pages of a shared library (libc.so.6 in our case) while the system was under memory pressure.
- One CPU reported rcu_preempt self-detected stall every ~63 s from then
on: 670+ reports over the ~12 hours covered by our captured logs.
- The next day, and again about two days after the first stall, we inspected the machine over JTAG. It was still stuck, with the same register values as in the first reports. It never recovered on its own; only a reboot brought it back.
- On that CPU the running task was spinning in filemap_get_entry() on a folio whose _refcount was 0 (frozen) while the folio was still reachable from the mapping's xarray.
- kswapd0 was found preempted inside __remove_mapping(), its stack ending in __filemap_remove_folio() -> filemap_unaccount_folio() -> ... -> preempt_schedule(): the folio was already frozen (refcount 0, off-LRU, unmapped) but not yet removed from the xarray, and the task still held the xarray lock.
All 670 stall reports show the same task and the same folio. Their PC samples fall inside this retry loop, mostly in xas_load(); the path taken after a successful refcount increment was never sampled. A second, non-RT process later hung in the same loop on the same folio, showing that once such an entry exists, every reader of that page is a victim.
2. Root cause
-------------
2.1 Reader: unbounded retry on refcount == 0
/* mm/filemap.c, filemap_get_entry() */
rcu_read_lock();
repeat:
xas_reset(&xas);
folio = xas_load(&xas);
if (xas_retry(&xas, folio))
goto repeat;
/*
* A shadow entry of a recently evicted page, or a swap entry from
* shmem/tmpfs. Return it without attempting to raise page count.
*/
if (!folio || xa_is_value(folio))
goto out;
if (!folio_try_get(folio))
goto repeat;The loop treats refcount == 0 as a transient state and retries. It never sleeps or yields while doing so, and there is no retry bound, no timeout and no warning.
2.2 Writer: the window is bounded on non-RT
/* mm/vmscan.c, __remove_mapping() */
if (!folio_test_swapcache(folio))
spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
...
refcount = 1 + folio_nr_pages(folio);
if (!folio_ref_freeze(folio, refcount))
goto cannot_free;
/* note: atomic_cmpxchg in folio_ref_freeze provides the smp_rmb */
...
__filemap_remove_folio(folio, shadow);
xa_unlock_irq(&mapping->i_pages);
...
cannot_free:
xa_unlock_irq(&mapping->i_pages);Readers on other CPUs can briefly observe the frozen folio even on non-RT -- that is what the retry loop is for. On non-RT, however, xa_lock_irq() disables IRQs and preemption, so the interval from folio_ref_freeze() to __filemap_remove_folio() cannot be preempted and cannot outlive the locked code path: the window is bounded, and the retry terminates after a few iterations.
2.3 On PREEMPT_RT the window is unbounded -- this is the break
On PREEMPT_RT xa_lock_irq() maps to rt_spin_lock(): an rt_mutex plus migrate_disable() and rcu_read_lock(). It does not disable preemption. A task inside the freeze -> removal window can therefore be preempted, and the frozen folio stays visible in the xarray with refcount == 0 for as long as the holder is off-CPU -- the window is no longer bounded by the locked code path, but by scheduling.
migrate_disable() aggravates this: the holder is pinned to the CPU it was preempted on. The holder does not have to be kswapd: the same shrink_folio_list() -> __remove_mapping() path is used by direct reclaim, so any task allocating memory can be the victim.
2.4 How it becomes permanent: priority inversion
In our case a SCHED_FIFO 99 task preempted the holder on the same CPU, then faulted on the very page being removed and entered the filemap_get_entry() spin. Two further SCHED_FIFO 99 threads of the same workload blocked on the same xarray lock; through priority inheritance the holder (kswapd) was boosted into the RT scheduling class at the same effective priority as the spinner. SCHED_FIFO does not time-slice among equal priorities, so a spinner that never blocks never gives the CPU back to the holder.
This is where self-healing is lost. While the holder remains CFS, the RT bandwidth limit (sched_rt_runtime_us, 950 ms per 1 s period by default) reserves CPU time for CFS tasks exactly so that a run-away RT task cannot lock up the machine (see Documentation/scheduler/sched-rt-group.rst); the holder can still run and close the window. Once PI pulls the holder into the RT class it is throttled together with the spinner and no longer benefits from that reserve. (With sched_rt_runtime_us = -1 even the CFS escape does not exist.)
The spinner is invisible to PI: it waits on the folio, not on the lock, so nothing ever boosts the holder on its behalf.
The four necessary conditions:
(a) frozen-but-visible folio -> reader spins
where: filemap_get_entry() retry loop
(b) window unbounded (holder preemptible, pinned)
where: PREEMPT_RT xa_lock_irq() semantics
(c) unyielding spinner on the same CPU
where: SCHED_FIFO 99 in the retry loop
(d) PI removes the holder's CFS bandwidth reserve
where: RT waiters on the same xarray lock
3. Blast radius
---------------
We went through every folio_ref_freeze()/page_ref_freeze() user in 6.12. Two of them can freeze a regular file folio and leave it reachable from the page cache xarray, and on PREEMPT_RT both can be preempted there:
- mm/vmscan.c, __remove_mapping()
trigger: reclaim -- kswapd or direct reclaim
- mm/migrate.c, __folio_migrate_mapping()
trigger: compaction / CMA / hotplug
Both hold the xarray lock from before the freeze until the folio has been removed, or replaced and unfrozen.
The unbounded refcount == 0 spin exists in several lookup loops, not just filemap_get_entry():
- filemap_get_entry() -- goto repeat, no bound
- find_get_entry() -- goto reset/goto retry, no bound
- filemap_get_folios_contig(), filemap_get_read_batch() -- goto retry, no bound
- ksm_get_folio() -- while (!folio_try_get(folio)) cpu_relax() with a swapcache-flag escape
ksm.c documents the transient state explicitly ("however, it might mean that the page is under page_ref_freeze()..."), which suggests the frozen-but-visible window is a known state of the design -- the filemap loops simply have no bound on it.
4. Fix directions we have considered
------------------------------------
4.1 Direction A (writer side): restore non-RT atomicity of the window
The idea: on PREEMPT_RT, wrap the freeze -> xarray-removal interval in preempt_disable()/preempt_enable(), keeping the lock acquisition itself RT-semantic (it may sleep while waiting). Readers stay untouched and "refcount == 0 is transient" holds again, for every reader.
By inspection this works for __folio_migrate_mapping(). Its window, from the freeze to the xas_store() of the new folio, contains nothing that sleeps on RT: that xas_store() call is not wrapped by mapping_set_update(), so no update hook fires.
It does not work for __remove_mapping(), the site we hit. The removal itself takes a spinlock_t whenever shadow-node tracking changes:
__filemap_remove_folio()
page_cache_delete()
mapping_set_update() /* xa_update = workingset_update_node */
xas_store(&xas, shadow)
update_node()
xas_update()
workingset_update_node()
list_lru_add_obj() / list_lru_del_obj()
spin_lock(&nlru->lock) /* spinlock_t */__delete_from_swap_cache() installs the same hook, so the swapcache branch is affected as well. This happens whenever an xa_node becomes shadow-only or stops being so, which is routine when reclaim evicts a run of file pages. shadow_nodes is a single list_lru whose per-node lock is shared by all mappings and by the shadow-node shrinker, so it can be contended. Unlike the other sleeping call we found, it cannot be moved out of the window: it is taken from inside the very xas_store() the window exists to cover.
(The other sleeping call in this function, mem_cgroup_swapout() in the swapcache branch, could be moved: with memsw accounting it takes swap_cgroup's ctrl->lock in swap_cgroup_record(). It would have to stay under the xarray lock though, because memcg1_swapout() relies on interrupts being disabled by it on non-RT.)
Making direction A work for reclaim would need the shadow-node list_lru update to be decoupled from xas_store(), for example deferred until after xa_unlock_irq(). That is an xarray/workingset change affecting every page cache and swap cache deletion, which we do not want to attempt without guidance.
4.2 Direction B (reader side): bounded retry + blocking fallback
Every freezer listed in section 3 (Blast radius) holds the xarray lock until the folio is no longer frozen-and-visible. So a reader that keeps failing can simply wait for that lock: on PREEMPT_RT it sleeps on the rt_mutex, gives the CPU back, and priority inheritance boosts the holder. Once the lock has been acquired the window is closed, and the retry sees the final state of the slot.
This is the same fallback commit 0f8821da4845 uses in mnt_get_write_access() (fs/namespace.c): on PREEMPT_RT the spinner on MNT_WRITE_HOLD stops spinning and takes the writer's lock instead, specifically to avoid this livelock.
The change would be small (mm/filemap.c only, we estimate about 80 lines):
- Applies to the four filemap loops listed in section 3 (Blast radius).
- Counts failed attempts; every 64 of them the reader calls rcu_read_unlock(), then xas_lock_irq() + xas_unlock_irq(), then rcu_read_lock(), resets the xa_state and retries the same index. Nothing is skipped and nothing is returned early, so semantics do not
change: callers such as truncation still see every entry.
- Only on PREEMPT_RT: the check compiles away otherwise, so non-RT behavior is identical.
- Only when preemptible(): callers that cannot sleep, such as some BPF/perf paths reaching build_id_parse_nofault(), keep retrying as before. An outer rcu_read_lock() does not need special handling:
PREEMPT_RT uses preemptible RCU, which allows blocking in a read-side section (at the cost of a longer grace period).
We have not prototyped or tested this yet; the sketch is here so the trade-off can be judged before we invest in a patch series.
- Pros: covers the reclaim site that direction A cannot; covers every freezer that follows the "hold the xarray lock" rule, including ones added later; adds no non-preemptible window.
- Cons: the lookups can now sleep on PREEMPT_RT in preemptible context, and we have not audited all of their callers. Two things need checking:
that no caller holds a lock which is taken inside the xarray lock (we spot-checked list_lru walkers and found no such caller, but this was not exhaustive), and that every caller keeps the mapping alive across the rcu_read_unlock(), not just by RCU.
- Not covered: callers that cannot sleep can still spin, and so can ksm_get_folio(), which has no mapping at hand to wait on. KSM is disabled on our system, so we did not look further into it.
Our current view is that B addresses the path we hit, while A, at best, narrows the problem to the migrate path -- but we may be missing a reason why upstream would rather restructure the writer side.
5. Questions
------------
1. Is the "refcount == 0 is transient" assumption of the lockless lookup path documented as depending on the freeze -> removal window being non-preemptible? If so, PREEMPT_RT breaking it is arguably an RT-side invariant break; if not, should the lookup loops grow a bound?
2. Given that the reclaim site cannot be made non-preemptible as long as workingset_update_node() takes the list_lru spinlock_t from inside xas_store(): is a reader-side fallback like direction B acceptable, or would upstream rather decouple the shadow-node LRU update from xas_store() and then fix the writers?
3. For direction B: is it acceptable that these lookups may sleep on the xarray lock on PREEMPT_RT, in preemptible context only (the same fallback commit 0f8821da4845 uses in mnt_get_write_access())? Is
"every freezer holds the xarray lock until the folio is no longer frozen-and-visible" an invariant we may rely on, and should it be documented next to the lockless page cache protocol comment?
4. Are there other freeze sites we have missed?
5. Would a WARN_ON_ONCE() on excessive retry be acceptable as a diagnostic, independent of the chosen fix?
6. Reproducer sketch
--------------------
1. Pin a SCHED_FIFO 99 task to CPU X; have it repeatedly page-fault on file-backed code pages of a shared library under reclaim pressure.
2. Drive reclaim (memory pressure is enough; direct reclaim works too) so a task runs __remove_mapping() for one of those pages on CPU X.
3. The FIFO task preempts the holder inside the freeze -> removal window and faults on the frozen page -> spin.
4. Two more FIFO 99 threads of the same process fault on other pages of the same mapping to block on the xarray lock and trigger PI.
Anonymized evidence (register sampling, stack traces of the three RT threads and kswapd, decoded structure dumps) is available on request. If direction B is the preferred one, we will turn the sketch above into a patch series and send it for review.
Best regards,
Donghui
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
Quoted 8 lines · 董辉
We hit a permanent livelock on a
PREEMPT_RTsystem and traced it to an interaction between the lockless page cache lookup retry loop infilemap_get_entry()and the folio freeze -> xarray removal window in__remove_mapping()(and the migrate equivalents). On non-RT that window is atomic (IRQs disabled) and therefore bounded; onPREEMPT_RTit is preemptible and can become unbounded, which turns the "refcount == 0 is transient" assumption of the lookup path into a permanent spin.
Quoted 8 lines
The affected code is vanilla upstream (our
mm/filemap.c,mm/vmscan.candmm/migrate.cmatch 6.12 exactly); nothing vendor-specific is involved.The obvious fix -- making the window non-preemptible on RT -- does not work for the reclaim path, which is the one we hit (section 4.1). We are deliberately not sending a patch yet: we would like to hear which fix direction upstream prefers before going further.
Thanks for bringing it to us early so we can discuss the right solution. Much easier than trying to infer from a patch what the problem was originally ;-)
Quoted 51 lines
1. Symptom
----------- 6.12.73-rt15 (
PREEMPT_RT), arm64, 20 CPUs.
- ASCHED_FIFO99 userspace workload was page-faulting on file-backed code pages of a shared library (libc.so.6 in our case) while the system was under memory pressure.
- One CPU reportedrcu_preemptself-detected stall every ~63 s from then
on: 670+ reports over the ~12 hours covered by our captured logs.
- The next day, and again about two days after the first stall, we inspected the machine over JTAG. It was still stuck, with the same register values as in the first reports. It never recovered on its own; only a reboot brought it back.
- On that CPU the running task was spinning infilemap_get_entry()on a folio whose_refcountwas 0 (frozen) while the folio was still reachable from the mapping's xarray.
- kswapd0 was found preempted inside__remove_mapping(), its stack ending in__filemap_remove_folio()->filemap_unaccount_folio()-> ... ->preempt_schedule(): the folio was already frozen (refcount 0, off-LRU, unmapped) but not yet removed from the xarray, and the task still held the xarray lock.All 670 stall reports show the same task and the same folio. Their PC samples fall inside this retry loop, mostly in
xas_load(); the path taken after a successful refcount increment was never sampled. A second, non-RT process later hung in the same loop on the same folio, showing that once such an entry exists, every reader of that page is a victim.2. Root cause
-------------2.1 Reader: unbounded retry on refcount == 0
/*
mm/filemap.c,filemap_get_entry()*/rcu_read_lock(); repeat: xas_reset(&xas); folio = xas_load(&xas); if (xas_retry(&xas, folio)) goto repeat; /* * A shadow entry of a recently evicted page, or a swap entry from * shmem/tmpfs. Return it without attempting to raise page count. */ if (!folio || xa_is_value(folio)) goto out; if (!folio_try_get(folio)) goto repeat;
Quoted 3 lines
The loop treats refcount == 0 as a transient state and retries. It never sleeps or yields while doing so, and there is no retry bound, no timeout and no warning.
Right. As you suggest later, this is intended to be a transient state that des not need to be bounded, slept on or warned about; usually finding a folio with a zero refcount isn't from a freeze but a stale folio pointer.
Quoted 60 lines
2.2 Writer: the window is bounded on non-RT
/*
mm/vmscan.c,__remove_mapping()*/if (!folio_test_swapcache(folio)) spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); ... refcount = 1 + folio_nr_pages(folio); if (!folio_ref_freeze(folio, refcount)) goto cannot_free; /* note: atomic_cmpxchg in folio_ref_freeze provides the smp_rmb */ ... __filemap_remove_folio(folio, shadow); xa_unlock_irq(&mapping->i_pages); ... cannot_free: xa_unlock_irq(&mapping->i_pages);Readers on other CPUs can briefly observe the frozen folio even on non-RT -- that is what the retry loop is for. On non-RT, however,
xa_lock_irq()disables IRQs and preemption, so the interval fromfolio_ref_freeze()to__filemap_remove_folio()cannot be preempted and cannot outlive the locked code path: the window is bounded, and the retry terminates after a few iterations.2.3 On
PREEMPT_RTthe window is unbounded -- this is the breakOn
PREEMPT_RTxa_lock_irq()maps tort_spin_lock(): anrt_mutexplusmigrate_disable()andrcu_read_lock(). It does not disable preemption. A task inside the freeze -> removal window can therefore be preempted, and the frozen folio stays visible in the xarray with refcount == 0 for as long as the holder is off-CPU -- the window is no longer bounded by the locked code path, but by scheduling.
migrate_disable()aggravates this: the holder is pinned to the CPU it was preempted on. The holder does not have to be kswapd: the sameshrink_folio_list()->__remove_mapping()path is used by direct reclaim, so any task allocating memory can be the victim.2.4 How it becomes permanent: priority inversion
In our case a
SCHED_FIFO99 task preempted the holder on the same CPU, then faulted on the very page being removed and entered thefilemap_get_entry()spin. Two furtherSCHED_FIFO99 threads of the same workload blocked on the same xarray lock; through priority inheritance the holder (kswapd) was boosted into the RT scheduling class at the same effective priority as the spinner.SCHED_FIFOdoes not time-slice among equal priorities, so a spinner that never blocks never gives the CPU back to the holder.This is where self-healing is lost. While the holder remains CFS, the RT bandwidth limit (
sched_rt_runtime_us, 950 ms per 1 s period by default) reserves CPU time for CFS tasks exactly so that a run-away RT task cannot lock up the machine (seeDocumentation/scheduler/sched-rt-group.rst); the holder can still run and close the window. Once PI pulls the holder into the RT class it is throttled together with the spinner and no longer benefits from that reserve. (Withsched_rt_runtime_us= -1 even the CFS escape does not exist.)
Quoted 2 lines
The spinner is invisible to PI: it waits on the folio, not on the lock, so nothing ever boosts the holder on its behalf.
I think you mean that it spins on the folio refcount, rather than it waiting on the folio (when people say "wait on the folio", usually they're talking abut folio_lock()).
Quoted 19 lines
The four necessary conditions:
(a) frozen-but-visible folio -> reader spins
where:filemap_get_entry()retry loop
(b) window unbounded (holder preemptible, pinned)
where:PREEMPT_RTxa_lock_irq()semantics
(c) unyielding spinner on the same CPU
where:SCHED_FIFO99 in the retry loop
(d) PI removes the holder's CFS bandwidth reserve
where: RT waiters on the same xarray lock3. Blast radius
---------------We went through every folio_ref_freeze()/page_ref_freeze() user in 6.12. Two of them can freeze a regular file folio and leave it reachable from the page cache xarray, and on
PREEMPT_RTboth can be preempted there:
Quoted 4 lines
-
mm/vmscan.c,__remove_mapping()
trigger: reclaim -- kswapd or direct reclaim
-mm/migrate.c,__folio_migrate_mapping()
trigger: compaction / CMA / hotplug
There's also __folio_freeze_split_file(). I know PREEMPT_RT disables TRANSPARENT_HUGEPAGE at the moment, but hopefully that restriction will be lifted.
Quoted 46 lines
Both hold the xarray lock from before the freeze until the folio has been removed, or replaced and unfrozen.
The unbounded refcount == 0 spin exists in several lookup loops, not just
filemap_get_entry():-
filemap_get_entry()-- goto repeat, no bound
-find_get_entry()-- goto reset/goto retry, no bound
-filemap_get_folios_contig(),filemap_get_read_batch()-- goto retry, no bound
-ksm_get_folio()-- while (!folio_try_get(folio))cpu_relax()with a swapcache-flag escape
ksm.cdocuments the transient state explicitly ("however, it might mean that the page is underpage_ref_freeze()..."), which suggests the frozen-but-visible window is a known state of the design -- the filemap loops simply have no bound on it.4. Fix directions we have considered
------------------------------------4.1 Direction A (writer side): restore non-RT atomicity of the window
The idea: on
PREEMPT_RT, wrap the freeze -> xarray-removal interval in preempt_disable()/preempt_enable(), keeping the lock acquisition itself RT-semantic (it may sleep while waiting). Readers stay untouched and "refcount == 0 is transient" holds again, for every reader.By inspection this works for
__folio_migrate_mapping(). Its window, from the freeze to thexas_store()of the new folio, contains nothing that sleeps on RT: thatxas_store()call is not wrapped bymapping_set_update(), so no update hook fires.It does not work for
__remove_mapping(), the site we hit. The removal itself takes aspinlock_twhenever shadow-node tracking changes:__filemap_remove_folio() page_cache_delete() mapping_set_update() /* xa_update = workingset_update_node */ xas_store(&xas, shadow) update_node() xas_update() workingset_update_node() list_lru_add_obj() / list_lru_del_obj() spin_lock(&nlru->lock) /* spinlock_t */
Quoted 8 lines
__delete_from_swap_cache()installs the same hook, so the swapcache branch is affected as well. This happens whenever anxa_nodebecomes shadow-only or stops being so, which is routine when reclaim evicts a run of file pages.shadow_nodesis a singlelist_lruwhose per-node lock is shared by all mappings and by the shadow-node shrinker, so it can be contended. Unlike the other sleeping call we found, it cannot be moved out of the window: it is taken from inside the veryxas_store()the window exists to cover.
Is it possible / wise to convert the nlru->lock to be a raw spinlock?
In the old days when the BKL was magic, we could take a mutex and choose to sleep. I don't think the preempt_disable() works like that. It's a bit annying because we'd be fine with being preempted after the folio is replaced in the XArray. It feels unutterably ugly to insert calls to preempt_disable() / preempt_enable() / preempt_disable() / preempt_enable() around all of this though.
Quoted 11 lines
(The other sleeping call in this function,
mem_cgroup_swapout()in the swapcache branch, could be moved: with memsw accounting it takes swap_cgroup'sctrl->lockinswap_cgroup_record(). It would have to stay under the xarray lock though, becausememcg1_swapout()relies on interrupts being disabled by it on non-RT.)Making direction A work for reclaim would need the shadow-node
list_lruupdate to be decoupled fromxas_store(), for example deferred until afterxa_unlock_irq(). That is an xarray/workingset change affecting every page cache and swap cache deletion, which we do not want to attempt without guidance.
Yeah, I don't think this is a great approach either.
Quoted 46 lines
4.2 Direction B (reader side): bounded retry + blocking fallback
Every freezer listed in section 3 (Blast radius) holds the xarray lock until the folio is no longer frozen-and-visible. So a reader that keeps failing can simply wait for that lock: on
PREEMPT_RTit sleeps on thert_mutex, gives the CPU back, and priority inheritance boosts the holder. Once the lock has been acquired the window is closed, and the retry sees the final state of the slot.This is the same fallback commit
0f8821da4845uses inmnt_get_write_access()(fs/namespace.c): onPREEMPT_RTthe spinner onMNT_WRITE_HOLDstops spinning and takes the writer's lock instead, specifically to avoid this livelock.The change would be small (
mm/filemap.conly, we estimate about 80 lines):- Applies to the four filemap loops listed in section 3 (Blast radius).
- Counts failed attempts; every 64 of them the reader callsrcu_read_unlock(), thenxas_lock_irq()+xas_unlock_irq(), thenrcu_read_lock(), resets thexa_stateand retries the same index. Nothing is skipped and nothing is returned early, so semantics do not
change: callers such as truncation still see every entry.
- Only onPREEMPT_RT: the check compiles away otherwise, so non-RT behavior is identical.
- Only whenpreemptible(): callers that cannot sleep, such as some BPF/perf paths reachingbuild_id_parse_nofault(), keep retrying as before. An outerrcu_read_lock()does not need special handling:
PREEMPT_RTuses preemptible RCU, which allows blocking in a read-side section (at the cost of a longer grace period).We have not prototyped or tested this yet; the sketch is here so the trade-off can be judged before we invest in a patch series.
- Pros: covers the reclaim site that direction A cannot; covers every freezer that follows the "hold the xarray lock" rule, including ones added later; adds no non-preemptible window.
- Cons: the lookups can now sleep onPREEMPT_RTin preemptible context, and we have not audited all of their callers. Two things need checking:
that no caller holds a lock which is taken inside the xarray lock (we spot-checkedlist_lruwalkers and found no such caller, but this was not exhaustive), and that every caller keeps the mapping alive across thercu_read_unlock(), not just by RCU.
- Not covered: callers that cannot sleep can still spin, and so canksm_get_folio(), which has no mapping at hand to wait on. KSM is disabled on our system, so we did not look further into it.
Quoted 3 lines
Our current view is that B addresses the path we hit, while A, at best, narrows the problem to the migrate path -- but we may be missing a reason why upstream would rather restructure the writer side.
But I don't like this either ;-( Hope somebody comes up with a nicer idea.
Quoted 8 lines
5. Questions
------------1. Is the "refcount == 0 is transient" assumption of the lockless lookup path documented as depending on the freeze -> removal window being non-preemptible? If so,
PREEMPT_RTbreaking it is arguably an RT-side invariant break; if not, should the lookup loops grow a bound?
I don't think it's _documented_. But I do think it's intentional.
Quoted 7 lines
2. Given that the reclaim site cannot be made non-preemptible as long as
workingset_update_node()takes thelist_lruspinlock_tfrom insidexas_store(): is a reader-side fallback like direction B acceptable, or would upstream rather decouple the shadow-node LRU update fromxas_store()and then fix the writers?
3. For direction B: is it acceptable that these lookups may sleep on the xarray lock onPREEMPT_RT, in preemptible context only (the
Quoted 8 lines
same fallback commit
0f8821da4845uses inmnt_get_write_access())? Is
"every freezer holds the xarray lock until the folio is no longer frozen-and-visible" an invariant we may rely on, and should it be documented next to the lockless page cache protocol comment?
4. Are there other freeze sites we have missed?
5. Would aWARN_ON_ONCE()on excessive retry be acceptable as a diagnostic, independent of the chosen fix?
No, I don't think we should do that.
Quoted 20 lines
6. Reproducer sketch
--------------------1. Pin a
SCHED_FIFO99 task to CPU X; have it repeatedly page-fault on file-backed code pages of a shared library under reclaim pressure.
2. Drive reclaim (memory pressure is enough; direct reclaim works too) so a task runs__remove_mapping()for one of those pages on CPU X.
3. The FIFO task preempts the holder inside the freeze -> removal window and faults on the frozen page -> spin.
4. Two more FIFO 99 threads of the same process fault on other pages of the same mapping to block on the xarray lock and trigger PI.Anonymized evidence (register sampling, stack traces of the three RT threads and kswapd, decoded structure dumps) is available on request. If direction B is the preferred one, we will turn the sketch above into a patch series and send it for review.
Best regards,
Donghui
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#