mm: restrict can_spin_trylock() to preemptible context on PREEMPT_RT
Subsystems (4)
RFC PATCH 1/1 mm: restrict can_spin_trylock() to preemptible context on PREEMPT_RT
On PREEMPT_RT kmalloc_nolock() deadlocks the machine when the caller holds a pi_lock. A BPF program which creates task local storage from the sched_waking tracepoint is enough. With it mainline hangs in 10 of 10 boots on a Raspberry Pi 500+ and in 7 of 28 boots in a 4 CPU x86-64 QEMU guest. The CPUs spin on raw locks with interrupts disabled and there is no output.
sched_waking fires in try_to_wake_up() with p->pi_lock held. can_spin_trylock() refuses only NMI and hard interrupt context on PREEMPT_RT, so kmalloc_nolock() does spin_trylock_irqsave() on n->list_lock, which is a sleeping lock. Two things go wrong:
- rt_spin_trylock() takes the rtmutex wait_lock under the pi_lock. A regular kmalloc() on another CPU holds wait_lock and takes a pi_lock in try_to_take_rt_mutex().
- If the trylock succeeded and a waiter arrived, rt_spin_unlock() has to wake it. That is a try_to_wake_up() inside try_to_wake_up():
try_to_wake_up raw_spin_lock_irqsave(&p->pi_lock)
rt_mutex_slowunlock
rt_spin_unlock
spin_unlock_irqrestore(&n->list_lock)
get_from_partial_node
___slab_alloc
__kmalloc_nolock_noprof
bpf_local_storage_alloc
...
bpf_trace_run1
try_to_wake_up sched_waking, a pi_lock is held The pi_lock held there belongs to the woken task, not to current.
Commit 99a3e3a1cfc9 ("slab: fix kmalloc_nolock() context check for PREEMPT_RT") closed this for v6.19 with a !preemptible() check. v7.0-rc1 relaxed it again, see the Fixes tag.
Allow preemptible context only, as v6.19 did. A held raw spinlock implies !preemptible(), so the locks of the caller do not have to be known. With this change both machines pass 10 of 10 boots.
The nolock allocations then fail on PREEMPT_RT from every context with preemption or interrupts disabled, also where no scheduler lock is held. Creation of BPF local storage from such a context fails, as it did in v6.19.
---
Notes:
RFC because I do not know if that last paragraph is acceptable for BPF. 073d5f156292 relaxed the check on purpose.
Harry wrote that the _nolock() helpers can be called under pi_lock at least in theory, that it was tough to reproduce, and proposed a check of current's pi_lock:
https://lore.kernel.org/r/apfwshahN_C6jBSn@nixos
https://lore.kernel.org/r/apfyeYF8IE03aazT@nixos
That check would not see the pi_lock in the stack above. Sebastian described the wait_lock and pi_lock order here:
https://lore.kernel.org/r/[email protected]
Same config, program and load for all rows. x86-64 QEMU (TCG), 4 CPUs, PREEMPT_RT, no lockdep, 60 seconds of load per boot:
v6.18 10 pass, 0 hang
parent of f484f4a3e058 10 pass, 0 hang
f484f4a3e058 0 pass, 10 hang
v6.19 10 pass, 0 hang
parent of 073d5f156292 10 pass, 0 hang
073d5f156292 0 pass, 10 hang
mainline 40288c9206c1 (v7.3-rc3) 21 pass, 7 hang
mainline with this change 10 pass, 0 hang
f484f4a3e058 ("bpf: Replace bpf memory allocator with kmalloc_nolock() in local storage") made the path reachable.
Raspberry Pi 500+, arm64 defconfig plus PREEMPT_RT, same mainline
commit: the output stops 1 to 2 seconds after the load starts. 4 of the 10 boots were then reset by the hardware watchdog. The other 6 made no more progress but still answered ping.
The stacks are from gdb on the hung QEMU guests, I have none from the Pi. gdb does not unwind through the JITed program. The frames below it are from a raw dump of the stacks. In that capture three of four CPUs wait like this for the pi_lock of the same task. The first case looks like this, two CPUs from the BPF program against one regular kmalloc():
rt_mutex_slowtrylock raw_spin_lock(&lock->wait_lock)
rt_spin_trylock
spin_trylock_irqsave(&n->list_lock)
get_from_partial_node
___slab_alloc
__kmalloc_nolock_noprof
try_to_take_rt_mutex raw_spin_lock(&task->pi_lock)
rtlock_slowlock_locked
rtlock_slowlock
rt_spin_lock
spin_lock(&n->list_lock)
get_from_partial_node
___slab_alloc
With lockdep and DEBUG_RT_MUTEXES the same program gives "possible circular locking dependency detected" for wait_lock and pi_lock on the first event.
The program:
struct {
__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
__uint(map_flags, BPF_F_NO_PREALLOC);
__type(key, int);
__type(value, long);
} per_task SEC(".maps");
SEC("tp_btf/sched_waking")
int BPF_PROG(on_waking, struct task_struct *p) {
struct task_struct *cur = bpf_get_current_task_btf(); long *v;
v = bpf_task_storage_get(&per_task, cur, 0,
BPF_LOCAL_STORAGE_GET_F_CREATE);
if (v)
bpf_task_storage_delete(&per_task, cur);
return 0;
}
The delete only makes every event allocate again. The load is a few loopback ping floods, tmpfs and block writes, /proc walks and fork/pipe loops, so that list_lock sees contention. The program is more aggressive than a real tool and the load is synthetic.
can_spin_trylock() is also used by the page allocator. Its nolock free paths fall back to their deferred lists. I did not test those separately. Kernels before v7.3-rc1 have no can_spin_trylock() and need the check in kmalloc_nolock() instead.
mm/internal.h | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
mm/internal.h +7 -3
@@ -1641,10 +1641,14 @@ static inline bool can_spin_trylock(void)* confuse PI logic, so return immediately if called from hard IRQ or * NMI. * -* Note, irqs_disabled() case is ok. spin_trylock() can be called -* from raw_spin_lock_irqsave region. +* Task context with a raw spinlock held is not safe either. The +* caller may hold a pi_lock, like a BPF program on a tracepoint in +* try_to_wake_up(). rt_spin_trylock() takes the rtmutex wait_lock +* and can take a pi_lock under it. rt_spin_unlock() wakes a waiter +* if there is one, which takes a pi_lock again. The locks held by +* the caller are not known here, so allow preemptible context only. */ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ())return false;/* On UP, spin_trylock() always succeeds even when it is locked */
base-commit: 40288c9206c17eb66a603262e06a58d300d0f279
Signature · 1 line
2.53.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] mm: can_spin_trylock() uses !preemptible() causing persistent bailout --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] mm: can_spin_trylock() uses !preemptible() causing persistent bailout --
commit da2bda4aca43320be916d709a95e77705d7bd75f
Author: Karl Mehltretter <[email protected]>
mm: restrict can_spin_trylock() to preemptible context on PREEMPT_RT
This patch resolves a deadlock on PREEMPT_RT during kmalloc_nolock() by restricting can_spin_trylock() to preemptible contexts only. It prevents nested lock acquisition of the pi_lock when called from sched_waking tracepoints inside try_to_wake_up().
Quoted 3 lines
mm/internal.h +0
-0
[ ... ]
Quoted 5 lines
mm/internal.h +0
-1@@ -1641,10 +1641,14 @@ static inline bool can_spin_trylock(void)* confuse PI logic, so return immediately if called from hard IRQ or * NMI. * -* Note, irqs_disabled() case is ok. spin_trylock() can be called
mm/internal.h static inline bool can_spin_trylock(void) · line 1651
Quoted 10 lines
mm/internal.h +7
-2-* from raw_spin_lock_irqsave region. +* Task context with a raw spinlock held is not safe either. The +* caller may hold a pi_lock, like a BPF program on a tracepoint in +* try_to_wake_up(). rt_spin_trylock() takes the rtmutex wait_lock +* and can take a pi_lock under it. rt_spin_unlock() wakes a waiter +* if there is one, which takes a pi_lock again. The locks held by +* the caller are not known here, so allow preemptible context only. */ -if (IS_ENABLED (CONFIG_PREEMPT_RT ) && (in_nmi () ||in_hardirq ())) +if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ())
[Severity: Medium]
Does this !preemptible() check create a persistent bailout for ALLOC_TRYLOCK paths on PREEMPT_RT?
BPF programs attached to non-faultable tracepoints run with preemption disabled via guard(preempt_notrace). When they attempt to create task local storage using kmalloc_nolock(), it calls can_spin_trylock().
Because preemption is always disabled inside the tracepoint, this check acts as a persistent bailout rather than a transient one, unconditionally failing the allocation on PREEMPT_RT. Will this permanently break BPF task storage creation in tracepoints on PREEMPT_RT?
Quoted 3 lines
mm/internal.h +0
-0return false;/* On UP, spin_trylock() always succeeds even when it is locked */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 8 lines · Karl Mehltretter
Allow preemptible context only, as v6.19 did. A held raw spinlock implies !preemptible(), so the locks of the caller do not have to be known. With this change both machines pass 10 of 10 boots.
The nolock allocations then fail on
PREEMPT_RTfrom every context with preemption or interrupts disabled, also where no scheduler lock is held. Creation of BPF local storage from such a context fails, as it did in v6.19.
6.19 had this check in kmalloc_nolock() only. alloc_pages_nolock() and free_pages_nolock() allowed irqs disabled since they were introduced, and arena was sleepable only under a mutex back then.
[...]
Quoted 5 lines
mm/internal.h +0
-0@@ -1641,10 +1641,14 @@ static inline bool can_spin_trylock(void)
[...]
mm/internal.h
Quoted 1 line
mm/internal.h +0
-1- if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi()
|| in_hardirq()))
mm/internal.h static inline bool can_spin_trylock(void) · line 1652
Quoted 2 lines
mm/internal.h +1
-0+if (IS_ENABLED (CONFIG_PREEMPT_RT ) && !preemptible ())return false;
No. This kills bpf arena on RT.
Since 7.0 arena_alloc_pages() and arena_vm_fault() take raw_res_spin_lock_irqsave(&arena->spinlock) and call alloc_pages_nolock() and kmalloc_nolock() (from range_tree) under it. With !preemptible() here bpf_arena_alloc_pages() returns NULL for every prog, sleepable included, and a user space fault in arena gets SIGSEGV.
As Sebastian said in
https://lore.kernel.org/r/[email protected]raw_spinlock_t is fine in general. pi_lock is special. rq lock too, I think, since rt_spin_unlock() can end up in try_to_wake_up(). The check has to be about those and not about every irq/preempt disabled section.
pw-bot: cr
Quoted 3 lines · Alexei Starovoitov
6.19 had this check in
kmalloc_nolock()only.alloc_pages_nolock()andfree_pages_nolock()allowed irqs disabled since they were introduced, and arena was sleepable only under a mutex back then.
Hi Alexei,
Thanks for the review!
Quoted 1 line
No. This kills bpf arena on RT.
I have now confirmed by testing that the RFC breaks both BPF arena paths on PREEMPT_RT.
Quoted 6 lines
As Sebastian said in
https://lore.kernel.org/r/[email protected]raw_spinlock_tis fine in general.pi_lockis special. rq lock too, I think, sincert_spin_unlock()can end up intry_to_wake_up(). The check has to be about those and not about every irq/preempt disabled section.
This leaves me unsure where the boundary between the MM and BPF fixes should be. I see four possible directions:
a) Change MM so _nolock() allocation remains safe and can still succeed while pi_lock or an rq lock is held. Existing BPF behavior would remain unchanged. Is this feasible, or would it require substantial allocator changes?
b) Restore BPF local storage's dedicated allocator, and document, with debug checks if possible, that _nolock() must not be used in those scheduler-lock contexts. Since the allocator change would restore the behavior before f484f4a3e058 and be confined to BPF, could this also be suitable for stable kernels?
c) Have MM detect those contexts and return NULL. This prevents the deadlock while preserving ordinary raw-lock callers such as arena, but task-storage creation from scheduler tracepoints then fails deterministically.
d) Combine (b) and (c) restore BPF functionality first, then add the MM restriction for other and future callers.
Are these the right alternatives? In particular, what behavior is intended from the _nolock() API in these scheduler-lock contexts?
Thanks,
Karl
Quoted 4 lines · Karl Mehltretter
a) Change MM so
_nolock()allocation remains safe and can still succeed whilepi_lockor an rq lock is held. Existing BPF behavior would remain unchanged. Is this feasible, or would it require substantial allocator changes?
I don't think it's feasible. On RT the sheaves lock, n->list_lock and zone->lock are all rtmutex based. When trylock succeeded and a waiter showed up rt_spin_unlock() has to wake it.
Quoted 3 lines
b) Restore BPF local storage's dedicated allocator, and document, with debug checks if possible, that
_nolock()must not be used in those scheduler-lock contexts.
No. Local storage is not the only one. bpf_arena_alloc_pages(), bpf_stream_vprintk(), bpf_task_work_schedule_*() call kmalloc_nolock() as well and the same sched_waking prog can call them. bpf_mem_alloc was removed from local storage, because it wastes memory on preallocation. Same answer for d).
Quoted 4 lines
c) Have MM detect those contexts and return NULL. This prevents the deadlock while preserving ordinary raw-lock callers such as arena, but task-storage creation from scheduler tracepoints then fails deterministically.
This one.bpf_task_storage_get() with F_CREATE can return NULL and the progs have to check for it anyway.
Quoted 2 lines
Are these the right alternatives? In particular, what behavior is intended from the
_nolock()API in these scheduler-lock contexts?
It can be called from any context and it can return NULL. That's what it does from NMI and hardirq on RT already.
Vlastimil Babka (SUSE)
reviewerQuoted 20 lines · Alexei Starovoitov
Quoted 4 lines · Karl Mehltretter
a) Change MM so
_nolock()allocation remains safe and can still succeed whilepi_lockor an rq lock is held. Existing BPF behavior would remain unchanged. Is this feasible, or would it require substantial allocator changes?I don't think it's feasible. On RT the sheaves lock,
n->list_lockandzone->lockare all rtmutex based. When trylock succeeded and a waiter showed uprt_spin_unlock()has to wake it.Quoted 3 lines
b) Restore BPF local storage's dedicated allocator, and document, with debug checks if possible, that
_nolock()must not be used in those scheduler-lock contexts.No. Local storage is not the only one.
bpf_arena_alloc_pages(),bpf_stream_vprintk(), bpf_task_work_schedule_*() callkmalloc_nolock()as well and the samesched_wakingprog can call them.bpf_mem_allocwas removed from local storage, because it wastes memory on preallocation. Same answer for d).
Quoted 6 lines
Quoted 4 lines
c) Have MM detect those contexts and return NULL. This prevents the deadlock while preserving ordinary raw-lock callers such as arena, but task-storage creation from scheduler tracepoints then fails deterministically.
This one.
Yeah, but how exactly to do that detection :)
Quoted 2 lines
bpf_task_storage_get()withF_CREATEcan return NULL and the progs have to check for it anyway.
e) Introduce raw_local_trylock_t...
Quoted 5 lines
Quoted 2 lines
Are these the right alternatives? In particular, what behavior is intended from the
_nolock()API in these scheduler-lock contexts?It can be called from any context and it can return NULL. That's what it does from NMI and hardirq on RT already.
Quoted 26 lines · Vlastimil Babka (SUSE)
Quoted 25 lines · Alexei Starovoitov
Quoted 4 lines · Karl Mehltretter
a) Change MM so
_nolock()allocation remains safe and can still succeed whilepi_lockor an rq lock is held. Existing BPF behavior would remain unchanged. Is this feasible, or would it require substantial allocator changes?I don't think it's feasible. On RT the sheaves lock,
n->list_lockandzone->lockare all rtmutex based. When trylock succeeded and a waiter showed uprt_spin_unlock()has to wake it.Quoted 3 lines
b) Restore BPF local storage's dedicated allocator, and document, with debug checks if possible, that
_nolock()must not be used in those scheduler-lock contexts.No. Local storage is not the only one.
bpf_arena_alloc_pages(),bpf_stream_vprintk(), bpf_task_work_schedule_*() callkmalloc_nolock()as well and the samesched_wakingprog can call them.bpf_mem_allocwas removed from local storage, because it wastes memory on preallocation. Same answer for d).Quoted 4 lines
c) Have MM detect those contexts and return NULL. This prevents the deadlock while preserving ordinary raw-lock callers such as arena, but task-storage creation from scheduler tracepoints then fails deterministically.
Quoted 8 lines
Quoted 1 line
This one.
Yeah, but how exactly to do that detection :)
Quoted 2 lines
bpf_task_storage_get()withF_CREATEcan return NULL and the progs have to check for it anyway.e) Introduce
raw_local_trylock_t...
+1 on this :)
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Are these the right alternatives? In particular, what behavior is intended from the
_nolock()API in these scheduler-lock contexts?It can be called from any context and it can return NULL. That's what it does from NMI and hardirq on RT already.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Sebastian Andrzej Siewior
maintainerQuoted 3 lines · Harry Yoo
Quoted 1 line
e) Introduce
raw_local_trylock_t...+1 on this :)
maybe consolidate first the existing variants :)
Sebastian