Make sleepable arena paths use sleepable alloc_pages
Status from this message: This series was applied to bpf/bpf-next.git (master)
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v6 Latest posting 2/27 patches 0 repliesApplied 7/7
- v6 posting 1/27 patches 2 repliesSuperseded
- v57 patches 1 replySuperseded
Earlier versions · 4
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v4 | 7 patches | 2 replies | Superseded | |
| v3 | 7 patches | 3 replies | Superseded | |
| v2 | 2 patches | 5 replies | Superseded | |
| v1 | 3 patches | 3 replies | Superseded |
Patches in v6 posting 2/2 All 7 applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/7 | bpf: Use an llist for page allocationsinclude/ kernel/ kernel/ | +67 | 0 replies | Applied |
| 2/7 | bpf: Add sleepable argument to bpf_alloc_pages()include/ kernel/ kernel/ | +10 | 0 replies | Applied |
| 3/7 | bpf: Add sleepable arena page allocation pathkernel/ | +46 | 0 replies | Applied |
| 4/7 | selftests/bpf: Test large allocations for both sleepable/nonsleepable arena userstools/ | +50 | 0 replies | Applied |
| 5/7 | bpf: Directly store kfunc desc index in instruction off fieldinclude/ include/ kernel/ +1 more | +33 | 0 replies | Applied |
| 6/7 | bpf: Support per-call-site kfunc specializationinclude/ kernel/ | +91 | 0 replies | Applied |
| 7/7 | selftests/bpf: Test per-call site function specializationtools/ tools/ | +144 | 0 replies | Applied |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v6 yet.
Cover letter [RESEND PATCH bpf-next v6 0/7] Make sleepable arena paths use sleepable alloc_pages
The arena_alloc_pages() call takes a sleepable argument based on whether its caller is a sleepable BPF function. This flag, along with the context the kfunc is called in, decides whether the call will try to fulfill the allocation using the regular or the _nolock variant of the alloc_pages API, by means of bpf_map_alloc_pages().
However, the arena_alloc_pages() call currently only makes allocations inside an IRQ-disabled critical section. This forces all allocations to use the _nolock() API, which may eagerly fail where the regular variant would eventually succeed. There have been reports of this happening for sched-ext schedulers.
Restructure arena_alloc_pages() to use the _nolock() page allocation API only when necessary. This requires moving allocations outside of the spinlock critical section for sleepable calls, which in turn requires slightly different logic in the allocation path. Replace the page list allocation with logic that reuses pcp_llist to chain allocated pages together, allowing us to merge the code paths for both sleepable and nonsleepable arena allocations.
Also fix kfunc specialization to not unnecessarily force the nonsleepable version of bpf_arena_alloc_pages() for call sites that do not need it. This requires making specialization per-call site instead of overwriting the descriptor during fixups.
v1 -> v2 (https://lore.kernel.org/bpf/[email protected]/)
- Keep the sleepable and non-sleepable allocation paths within arena_alloc_pages (Alexei)
- Incorporate bot feedback on selftests (bot-ci)
v2 -> v3 (https://lore.kernel.org/bpf/[email protected]/)
- Remove the intermediate page array allocation in bpf_arena_alloc_pages() and use the pcp_llist pointer instead (Alexei)
- Fix function specialization to only specialize to the nonsleepable version when necessary
v3 -> v4 (https://lore.kernel.org/bpf/[email protected]/)
- Remove in-flight pages tracking (Alexei)
- Remove stale split sleepable/nonsleepable error handling (Alexei)
v4 -> v5 (https://lore.kernel.org/bpf/[email protected]/)
- Remove unnecessary imm-based sorting for kfunc desc table (Alexei)
- Skip all nonspecialized kfunc descs during the linear scan done by function specialization
v5 -> v6 (https://lore.kernel.org/bpf/[email protected]/)
- Use verifier_bug() for diagnostics (bot)
Emil Tsalapatis (7):
bpf: Use an llist for page allocations
bpf: Add sleepable argument to bpf_alloc_pages()
bpf: Add sleepable arena page allocation path selftests/bpf: Test large allocations for both sleepable/nonsleepable
arena users
bpf: Directly store kfunc desc index in instruction off field
bpf: Support per-call-site kfunc specialization selftests/bpf: Test per-call site function specialization
include/linux/bpf.h | 10 +- include/linux/bpf_verifier.h | 15 +- kernel/bpf/arena.c | 158 +++++++++--------- kernel/bpf/fixups.c | 70 +------- kernel/bpf/syscall.c | 43 +++-- kernel/bpf/verifier.c | 107 +++++++++++- .../selftests/bpf/prog_tests/file_reader.c | 15 ++ .../testing/selftests/bpf/progs/file_reader.c | 129 ++++++++++++++ .../bpf/progs/verifier_arena_large.c | 64 +++++-- 9 files changed, 423 insertions(+), 188 deletions(-)
Signature · 1 line
2.52.0