bpf: fix and improve open-coded task_vma iterator
Status from this message: This series was applied to bpf/bpf-next.git (master)
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v6 Latest3 patches 1 replyApplied 3/3
- v5 posting 2/23 patches 5 repliesSuperseded
- v5 posting 1/20 of 3 patches 0 repliesSuperseded
Earlier versions · 4
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v4 | 3 patches | 2 replies | Superseded | |
| v3 | 3 patches | 5 replies | Superseded | |
| v2 | 4 patches | 11 replies | Superseded | |
| v1 | 3 patches | 7 replies | Superseded |
Changes in v6 from the cover letter
- Replace local_irq_disable() + get_task_mm() with spin_trylock() on alloc_lock to avoid a softirq deadlock: if the target task holds its alloc_lock and gets interrupted, a softirq BPF program iterating that task would deadlock on task_lock() (Gemini)
- Gate on CONFIG_MMU in patch 1 so that the mmput() fallback in bpf_iter_mmput_async() cannot sleep in non-sleepable BPF context on NOMMU; patch 2 tightens this to CONFIG_PER_VMA_LOCK (Gemini)
- Merge the split if (irq_work_busy) / if (!mmap_read_trylock()) back into a single if statement in patch 1 (Andrii)
- Flip comparison direction in bpf_iter_task_vma_find_next() so both the locked and unlocked VMA failure cases read consistently:
end <= next_addr → PAGE_SIZE, else - use end (Andrii)
- Add Acked-by from Andrii on patch 3
Patches in v6 All 3 applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | bpf: fix mm lifecycle in open-coded task_vma iteratorkernel/ | +51 | 0 replies | Applied |
| 2/3 | bpf: switch task_vma iterator from mmap_lock to per-VMA lockskernel/ | +73 | 0 replies | Applied |
| 3/3 | bpf: return VMA snapshot from task_vma iteratorkernel/ | +30 | 1 reply | Applied |
Reviews trailers per patch
1 of 3 patches has a Reviewed-by or Acked-by.
| Patch | Andrii Nakryiko | Mykyta Yatsenko |
|---|---|---|
| 1/3 | ||
| 2/3 | ||
| 3/3 | A-b | A-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH bpf v6 0/3] bpf: fix and improve open-coded task_vma iterator
Changelog:
v5: https://lore.kernel.org/all/[email protected]/
Changes in v6:
- Replace local_irq_disable() + get_task_mm() with spin_trylock() on alloc_lock to avoid a softirq deadlock: if the target task holds its alloc_lock and gets interrupted, a softirq BPF program iterating that task would deadlock on task_lock() (Gemini)
- Gate on CONFIG_MMU in patch 1 so that the mmput() fallback in bpf_iter_mmput_async() cannot sleep in non-sleepable BPF context on NOMMU; patch 2 tightens this to CONFIG_PER_VMA_LOCK (Gemini)
- Merge the split if (irq_work_busy) / if (!mmap_read_trylock()) back into a single if statement in patch 1 (Andrii)
- Flip comparison direction in bpf_iter_task_vma_find_next() so both the locked and unlocked VMA failure cases read consistently:
end <= next_addr → PAGE_SIZE, else - use end (Andrii)
- Add Acked-by from Andrii on patch 3
v4: https://lore.kernel.org/all/[email protected]/
Changes in v5:
- Use get_task_mm() instead of a lockless task->mm read followed by mmget_not_zero() to fix a use-after-free: mm_struct is not SLAB_TYPESAFE_BY_RCU, so the lockless pointer can go stale (AI)
- Add a local bpf_iter_mmput_async() wrapper with #ifdef CONFIG_MMU to avoid modifying fork.c and sched/mm.h outside the BPF tree
- Drop the fork.c and sched/mm.h changes that widened the mmput_async() #if guard
- Disable IRQs around get_task_mm() to prevent raw tracepoint re-entrancy from deadlocking on task_lock()
v3: https://lore.kernel.org/all/[email protected]/
Changes in v4:
- Disable task_vma iterator in irq_disabled() contexts to mitigate deadlocks (Alexei)
- Use a helper function to reset the snapshot (Andrii)
- Remove the redundant snap->vm_mm = kit->data->mm; (Andrii)
- Remove all irq_work deferral as the iterator will not work in irq_disabled() sections anymore and _new() will return -EBUSY early.
v2: https://lore.kernel.org/all/[email protected]/
Changes in v3:
- Remove the rename patch 1 (Andrii)
- Put the irq_work in the iter data, per-cpu slot is not needed (Andrii)
- Remove the unnecessary !in_hardirq() in the deferral path (Alexei)
- Use PAGE_SIZE advancement in case vma shrinks back to maintain the forward progress guarantee (AI)
v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Add a preparatory patch to rename mmap_unlock_irq_work to bpf_iter_mm_irq_work (Mykyta)
- Fix bpf_iter_mmput() to also defer for IRQ disabled regions (Alexei)
- Fix a build issue where mmpu_async() is not available without CONFIG_MMU (kernel test robot)
- Reuse mmap_unlock_irq_work (after rename) for mmput (Mykyta)
- Move vma lookup (retry block) to a separate function (Mykyta)
This series fixes the mm lifecycle handling in the open-coded task_vma BPF iterator and switches it from mmap_lock to per-VMA locking to reduce contention. It then fixes a deadlock that is caused by holding locks accross the body of the iterator where faulting is allowed.
Patch 1 fixes a use-after-free where task->mm was read locklessly and could be freed before the iterator used it. It uses a trylock on alloc_lock to safely read task->mm and acquire an mm reference, and disables the iterator in irq_disabled() contexts by returning -EBUSY from _new().
Patch 2 switches from holding mmap_lock for the entire iteration to per-VMA locking via lock_vma_under_rcu(). This still doesn't fix the deadlock problem because holding the per-vma lock for the whole iteration can still cause lock ordering issues when a faultable helper is called in the body of the iterator.
Patch 3 resolves the lock ordering problems caused by holding the per-VMA lock or the mmap_lock (not applicable after patch 2) across BPF program execution. It snapshots VMA fields under the lock, then drops the lock before returning to the BPF program. File references are managed via get_file()/fput() across iterations.
Puranjay Mohan (3):
bpf: fix mm lifecycle in open-coded task_vma iterator
bpf: switch task_vma iterator from mmap_lock to per-VMA locks
bpf: return VMA snapshot from task_vma iterator
kernel/bpf/task_iter.c | 151 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 136 insertions(+), 15 deletions(-)
base-commit: d8a9a4b11a137909e306e50346148fc5c3b63f9d
Signature · 1 line
2.52.0