Skip to content

Remove KF_SLEEPABLE from arena kfuncs

Puranjay Mohan bpf 8 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v8 Latest4 patches 8 repliesIn review
  2. v74 patches 1 replySuperseded
  3. v64 patches 1 replySuperseded
Earlier versions · 5
VersionPostedPatchesRepliesStatus
v54 patches1 replySuperseded
v44 patches2 repliesSuperseded
v34 patches4 repliesSuperseded
v24 patches12 repliesSuperseded
v14 patches2 repliesSuperseded

Changes in v8 from the cover letter

- Use clear_lo32(arena->user_vm_start) in place of user_vm_start in patch 3

Patches in v8 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4bpf: arena: populate vm_area without allocating memorykernel/bpf/arena.c+90 -100 replies
2/4bpf: arena: use kmalloc_nolock() in place of kvcalloc()kernel/bpf/arena.c+55 -290 replies
3/4bpf: arena: make arena kfuncs any context safeinclude/linux/bpf.h kernel/bpf/arena.c kernel/bpf/verifier.c+233 -410 replies
4/4selftests: bpf: test non-sleepable arena allocationstools/testing/selftests/bpf/prog_tests/arena_list.c tools/testing/selftests/bpf/progs/arena_list.c tools/testing/selftests/bpf/progs/verifier_arena.c +1 more+240 -58 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v8 yet.

Read the v8 thread

Cover letter [PATCH bpf-next v8 0/4] Remove KF_SLEEPABLE from arena kfuncs

V7: https://lore.kernel.org/all/[email protected]/
Changes in V7->v8:
- Use clear_lo32(arena->user_vm_start) in place of user_vm_start in patch 3

V6: https://lore.kernel.org/all/[email protected]/
Changes in v6->v7:
- Fix a deadlock in patch 1, that was being fixed in patch 2. Move the fix to patch 1.
- Call flush_cache_vmap() after setting up the mappings as it is required by some architectures.

V5: https://lore.kernel.org/all/[email protected]/
Changes in v5->v6:
Patch 1:
- Add a missing ; to make sure this patch builds individually. (AI)

V4: https://lore.kernel.org/all/[email protected]/
Changes in v4->v5:
Patch 1:
- Fix a memory leak in arena_alloc_pages(), it was being fixed in Patch 3 but, every patch should be complete in itself. (AI)
Patch 3:
- Don't do useless addition in arena_alloc_pages() (Alexei)
- Add a comment about kmalloc_nolock() failure and expectations.

v3: https://lore.kernel.org/all/[email protected]/
Changes in v3->v4:
- Coding style changes related to comments in Patch 2/3 (Alexei)

v2: https://lore.kernel.org/all/[email protected]/
Changes in v2->v3:
Patch 1:
- Call range_tree_destroy() in error path of populate_pgtable_except_pte() in arena_map_alloc() (AI)
Patch 2:
- Fix double mutex_unlock() in the error path of arena_alloc_pages() (AI)
- Fix coding style issues (Alexei)
Patch 3:
- Unlock spinlock before returning from arena_vm_fault() in case BPF_F_SEGV_ON_FAULT is set by user. (AI)
- Use __llist_del_all() in place of llist_del_all for on-stack llist (free_pages) (Alexei)
- Fix build issues on 32-bit systems where arena.c is not compiled. (kernel test robot)
- Make bpf_arena_alloc_pages() polymorphic so it knows if it has been called in sleepable or non-sleepable context. This information is passed to arena_free_pages() in the error path.
Patch 4:
- Add a better comment for the big_alloc3() test that triggers kmalloc_nolock()'s limit and if bpf_arena_alloc_pages() works correctly above this limit.

v1: https://lore.kernel.org/all/[email protected]/
Changes in v1->v2:
Patch 1:
- Import tlbflush.h to fix build issue in loongarch. (kernel test robot)
- Fix unused variable error in apply_range_clear_cb() (kernel test robot)
- Call bpf_map_area_free() on error path of populate_pgtable_except_pte() (AI)
- Use PAGE_SIZE in apply_to_existing_page_range() (AI)
Patch 2:
- Cap allocation made by kmalloc_nolock() for pages array to KMALLOC_MAX_CACHE_SIZE and reuse the array in an explicit loop to overcome this limit. (AI)
Patch 3:
- Do page_ref_add(page, 1); under the spinlock to mitigate a race (AI)
Patch 4:
- Add a new testcase big_alloc3() verifier_arena_large.c that tries to allocate a large number of pages at once, this is to trigger the kmalloc_nolock() limit in Patch 2 and see if the loop logic works correctly.

This set allows arena kfuncs to be called from non-sleepable contexts. It is acheived by the following changes:

The range_tree is now protected with a rqspinlock and not a mutex, this change is enough to make bpf_arena_reserve_pages() any context safe.

bpf_arena_alloc_pages() had four points where it could sleep:

1. Mutex to protect range_tree: now replaced with rqspinlock

2. kvcalloc() for allocations: now replaced with kmalloc_nolock()

3. Allocating pages with bpf_map_alloc_pages(): this already calls alloc_pages_nolock() in non-sleepable contexts and therefore is safe.

4. Setting up kernel page tables with vm_area_map_pages():
vm_area_map_pages() may allocate memory while inserting pages into bpf arena's vm_area. Now, at arena creation time populate all page table levels except the last level and when new pages need to be inserted call apply_to_page_range() again which will only do set_pte_at() for those pages and will not allocate memory.

The above four changes make bpf_arena_alloc_pages() any context safe.

bpf_arena_free_pages() has to do the following steps:

1. Update the range_tree
2. vm_area_unmap_pages(): to unmap pages from kernel vm_area
3. flush the tlb: done in step 2, already.
4. zap_pages(): to unmap pages from user page tables
5. free pages.

The third patch in this set makes bpf_arena_free_pages() polymorphic using the specialize_kfunc() mechanism. When called from a sleepable context, arena_free_pages() remains mostly unchanged except the following:
1. rqspinlock is taken now instead of the mutex for the range tree
2. Instead of using vm_area_unmap_pages() that can free intermediate page table levels, apply_to_existing_page_range() with a callback is used that only does pte_clear() on the last level and leaves the intermediate page table levels intact. This is needed to make sure that bpf_arena_alloc_pages() can safely do set_pte_at() without allocating intermediate page tables.

When arena_free_pages() is called from a non-sleepable context or it fails to acquire the rqspinlock in the sleepable case, a lock-less list of struct arena_free_span is used to queue the uaddr and page cnt. kmalloc_nolock() is used to allocate this arena_free_span, this can fail but we need to make this trade-off for frees done from non-sleepable contexts.

arena_free_pages() then raises an irq_work whose handler in turn schedules work that iterate this list and clears ptes, flushes tlbs, zap pages, and frees pages for the queued uaddr and page cnts.

apply_range_clear_cb() with apply_to_existing_page_range() is used to clear PTEs and collect pages to be freed, struct llist_node pcp_llist; in the struct page is used to do this.

Puranjay Mohan (4):
bpf: arena: populate vm_area without allocating memory
bpf: arena: use kmalloc_nolock() in place of kvcalloc()
bpf: arena: make arena kfuncs any context safe
selftests: bpf: test non-sleepable arena allocations

 include/linux/bpf.h                           |  16 +
 kernel/bpf/arena.c                            | 380 +++++++++++++++---
 kernel/bpf/verifier.c                         |  10 +
 .../selftests/bpf/prog_tests/arena_list.c     |  20 +-
 .../testing/selftests/bpf/progs/arena_list.c  |  11 +
 .../selftests/bpf/progs/verifier_arena.c      | 185 +++++++++
 .../bpf/progs/verifier_arena_large.c          |  29 ++
 7 files changed, 592 insertions(+), 59 deletions(-)

base-commit: f785a31395d9cafb8b2c42c7358fad72a6463142

Signature · 1 line
2.47.3