Remove KF_SLEEPABLE from arena kfuncs
Subsystems (4)
v3 cover Remove KF_SLEEPABLE from arena kfuncs
v2: https://lore.kernel.org/all/[email protected]/
Changes in v2->v3:
Patch 1:
- Call range_tree_destroy() in error path of populate_pgtable_except_pte() in arena_map_alloc() (AI)
Patch 2:
- Fix double mutex_unlock() in the error path of arena_alloc_pages() (AI)
- Fix coding style issues (Alexei)
Patch 3:
- Unlock spinlock before returning from arena_vm_fault() in case BPF_F_SEGV_ON_FAULT is set by user. (AI)
- Use __llist_del_all() in place of llist_del_all for on-stack llist (free_pages) (Alexei)
- Fix build issues on 32-bit systems where arena.c is not compiled. (kernel test robot)
- Make bpf_arena_alloc_pages() polymorphic so it knows if it has been called in sleepable or non-sleepable context. This information is passed to arena_free_pages() in the error path.
Patch 4:
- Add a better comment for the big_alloc3() test that triggers kmalloc_nolock()'s limit and if bpf_arena_alloc_pages() works correctly above this limit.
v1: https://lore.kernel.org/all/[email protected]/
Changes in v1->v2:
Patch 1:
- Import tlbflush.h to fix build issue in loongarch. (kernel test robot)
- Fix unused variable error in apply_range_clear_cb() (kernel test robot)
- Call bpf_map_area_free() on error path of populate_pgtable_except_pte() (AI)
- Use PAGE_SIZE in apply_to_existing_page_range() (AI)
Patch 2:
- Cap allocation made by kmalloc_nolock() for pages array to KMALLOC_MAX_CACHE_SIZE and reuse the array in an explicit loop to overcome this limit. (AI)
Patch 3:
- Do page_ref_add(page, 1); under the spinlock to mitigate a race (AI)
Patch 4:
- Add a new testcase big_alloc3() verifier_arena_large.c that tries to allocate a large number of pages at once, this is to trigger the kmalloc_nolock() limit in Patch 2 and see if the loop logic works correctly.
This set allows arena kfuncs to be called from non-sleepable contexts. It is acheived by the following changes:
The range_tree is now protected with a rqspinlock and not a mutex, this change is enough to make bpf_arena_reserve_pages() any context safe.
bpf_arena_alloc_pages() had four points where it could sleep:
1. Mutex to protect range_tree: now replaced with rqspinlock
2. kvcalloc() for allocations: now replaced with kmalloc_nolock()
3. Allocating pages with bpf_map_alloc_pages(): this already calls alloc_pages_nolock() in non-sleepable contexts and therefore is safe.
4. Setting up kernel page tables with vm_area_map_pages():
vm_area_map_pages() may allocate memory while inserting pages into bpf arena's vm_area. Now, at arena creation time populate all page table levels except the last level and when new pages need to be inserted call apply_to_page_range() again which will only do set_pte_at() for those pages and will not allocate memory.
The above four changes make bpf_arena_alloc_pages() any context safe.
bpf_arena_free_pages() has to do the following steps:
1. Update the range_tree
2. vm_area_unmap_pages(): to unmap pages from kernel vm_area
3. flush the tlb: done in step 2, already.
4. zap_pages(): to unmap pages from user page tables
5. free pages.
The third patch in this set makes bpf_arena_free_pages() polymorphic using the specialize_kfunc() mechanism. When called from a sleepable context, arena_free_pages() remains mostly unchanged except the following:
1. rqspinlock is taken now instead of the mutex for the range tree
2. Instead of using vm_area_unmap_pages() that can free intermediate page table levels, apply_to_existing_page_range() with a callback is used that only does pte_clear() on the last level and leaves the intermediate page table levels intact. This is needed to make sure that bpf_arena_alloc_pages() can safely do set_pte_at() without allocating intermediate page tables.
When arena_free_pages() is called from a non-sleepable context or it fails to acquire the rqspinlock in the sleepable case, a lock-less list of struct arena_free_span is used to queue the uaddr and page cnt. kmalloc_nolock() is used to allocate this arena_free_span, this can fail but we need to make this trade-off for frees done from non-sleepable contexts.
arena_free_pages() then raises an irq_work whose handler in turn schedules work that iterate this list and clears ptes, flushes tlbs, zap pages, and frees pages for the queued uaddr and page cnts.
apply_range_clear_cb() with apply_to_existing_page_range() is used to clear PTEs and collect pages to be freed, struct llist_node pcp_llist; in the struct page is used to do this.
Puranjay Mohan (4):
bpf: arena: populate vm_area without allocating memory
bpf: arena: use kmalloc_nolock() in place of kvcalloc()
bpf: arena: make arena kfuncs any context safe
selftests: bpf: test non-sleepable arena allocations
include/linux/bpf.h | 15 + kernel/bpf/arena.c | 371 +++++++++++++++--- kernel/bpf/verifier.c | 10 + .../selftests/bpf/prog_tests/arena_list.c | 20 +- .../testing/selftests/bpf/progs/arena_list.c | 11 + .../selftests/bpf/progs/verifier_arena.c | 185 +++++++++ .../bpf/progs/verifier_arena_large.c | 29 ++ 7 files changed, 582 insertions(+), 59 deletions(-)
Signature · 1 line
2.47.1
PATCH v3 1/4 bpf: arena: populate vm_area without allocating memory
vm_area_map_pages() may allocate memory while inserting pages into bpf arena's vm_area. In order to make bpf_arena_alloc_pages() kfunc non-sleepable change bpf arena to populate pages without allocating memory:
- at arena creation time populate all page table levels except the last level
- when new pages need to be inserted call apply_to_page_range() again with apply_range_set_cb() which will only set_pte_at() those pages and will not allocate memory.
- when freeing pages call apply_to_existing_page_range with apply_range_clear_cb() to clear the pte for the page to be removed. This doesn't free intermediate page table levels.
--- kernel/bpf/arena.c | 77 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 71 insertions(+), 6 deletions(-)
kernel/bpf/arena.c +71 -6
@@ -7,6 +7,7 @@#include <linux/btf_ids.h> #include <linux/vmalloc.h> #include <linux/pagemap.h> +#include <asm/tlbflush.h> #include "range_tree.h" /*
@@ -92,6 +93,62 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr)return (u32)(uaddr - (u32 )arena->user_vm_start) >>PAGE_SHIFT ; } +struct apply_range_data { +struct page **pages; +int i; +}; + +static int apply_range_set_cb (pte_t *pte,unsigned long addr,void *data) +{ +struct apply_range_data *d = data; +struct page *page; + +if (!data) +return 0 ; +/* sanity check */ +if (unlikely (!pte_none (ptep_get (pte)))) +return -EBUSY ; + + page = d->pages[d->i++]; +/* paranoia, similar to vmap_pages_pte_range() */ +if (WARN_ON_ONCE (!pfn_valid (page_to_pfn (page)))) +return -EINVAL ; + +set_pte_at (&init_mm, addr, pte,mk_pte (page,PAGE_KERNEL )); +return 0 ; +} + +static int apply_range_clear_cb (pte_t *pte,unsigned long addr,void *data) +{ +pte_t old_pte; +struct page *page; + +/* sanity check */ + old_pte =ptep_get (pte); +if (pte_none (old_pte) || !pte_present (old_pte)) +return 0 ;/* nothing to do */ + +/* get page and free it */ + page =pte_page (old_pte); +if (WARN_ON_ONCE (!page)) +return -EINVAL ; + +pte_clear (&init_mm, addr, pte); + +/* ensure no stale TLB entries */ +flush_tlb_kernel_range (addr, addr +PAGE_SIZE ); + +__free_page (page); + +return 0 ; +} + +static int populate_pgtable_except_pte (struct bpf_arena *arena) +{ +return apply_to_page_range (&init_mm,bpf_arena_get_kern_vm_start (arena), +KERN_VM_SZ -GUARD_SZ , apply_range_set_cb,NULL ); +} +static struct bpf_map *arena_map_alloc (union bpf_attr *attr) {struct vm_struct *kern_vm;
@@ -144,6 +201,12 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) goto err; }mutex_init (&arena->lock); + err =populate_pgtable_except_pte (arena); +if (err) { +range_tree_destroy (&arena->rt); +bpf_map_area_free (arena); + goto err; + }return &arena->map; err:
@@ -286,6 +349,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)if (ret)return VM_FAULT_SIGSEGV ; +struct apply_range_data data = { .pages = &page, .i =0 };/* Account into memcg of the process that created bpf_arena */ ret =bpf_map_alloc_pages (map,NUMA_NO_NODE ,1 , &page);if (ret) {
@@ -293,7 +357,7 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)return VM_FAULT_SIGSEGV ; } - ret =vm_area_map_pages (arena->kern_vm, kaddr, kaddr +PAGE_SIZE , &page); + ret =apply_to_page_range (&init_mm, kaddr,PAGE_SIZE , apply_range_set_cb, &data);if (ret) {range_tree_set (&arena->rt, vmf->pgoff,1 );__free_page (page);
@@ -428,7 +492,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* user_vm_end/start are fixed before bpf prog runs */ long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); -struct page **pages; +struct page **pages =NULL ;long pgoff =0 ;u32 uaddr32;int ret, i;
@@ -465,6 +529,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntif (ret) goto out_free_pages; +struct apply_range_data data = { .pages = pages, .i =0 }; ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages);if (ret) goto out;
@@ -477,8 +542,8 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow * lower 32-bit and it's ok. */ - ret =vm_area_map_pages (arena->kern_vm, kern_vm_start + uaddr32, - kern_vm_start + uaddr32 + page_cnt *PAGE_SIZE , pages); + ret =apply_to_page_range (&init_mm, kern_vm_start + uaddr32, + page_cnt <<PAGE_SHIFT , apply_range_set_cb, &data);if (ret) {for (i =0 ; i < page_cnt; i++)__free_page (pages[i]);
@@ -545,8 +610,8 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt)* page_cnt is big it's faster to do the batched zap. */ zap_pages (arena, full_uaddr,1 ); -vm_area_unmap_pages (arena->kern_vm, kaddr, kaddr +PAGE_SIZE ); -__free_page (page); +apply_to_existing_page_range (&init_mm, kaddr,PAGE_SIZE , apply_range_clear_cb, +NULL ); } }
Signature · 1 line
2.47.1
PATCH v3 2/4 bpf: arena: use kmalloc_nolock() in place of kvcalloc()
In reply to Puranjay MohanTo make arena_alloc_pages() safe to be called from any context, replace kvcalloc() with kmalloc_nolock() so as it doesn't sleep or take any locks. kmalloc_nolock() returns NULL for allocations larger than KMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE * 2) = 8KB on systems with 4KB pages. So, round down the allocation done by kmalloc_nolock to 1024
* 8 and reuse the array in a loop.
--- kernel/bpf/arena.c | 83 +++++++++++++++++++++++++++++++--------------- 1 file changed, 57 insertions(+), 26 deletions(-)
kernel/bpf/arena.c +57 -26
@@ -43,6 +43,8 @@#define GUARD_SZ round_up(1ull << sizeof_field(struct bpf_insn, off) * 8, PAGE_SIZE << 1)#define KERN_VM_SZ (SZ_4G + GUARD_SZ) +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt); +struct bpf_arena {struct bpf_map map;u64 user_vm_start;
@@ -492,7 +494,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* user_vm_end/start are fixed before bpf prog runs */ long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data;struct page **pages =NULL ; +long remaining, mapped =0 ; +long alloc_pages;long pgoff =0 ;u32 uaddr32;int ret, i;
@@ -509,17 +514,21 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntreturn 0 ; } -/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ - pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL ); +/* +* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. +*/ + alloc_pages =min (page_cnt,KMALLOC_MAX_CACHE_SIZE /sizeof (struct page *)); + pages =kmalloc_nolock (alloc_pages *sizeof (struct page *),0 ,NUMA_NO_NODE );if (!pages)return 0 ; + data.pages = pages; -guard (mutex)(&arena->lock); +mutex_lock (&arena->lock);if (uaddr) { ret =is_range_tree_set (&arena->rt, pgoff, page_cnt);if (ret) - goto out_free_pages; + goto out_unlock_free_pages; ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }else { ret = pgoff =range_tree_find (&arena->rt, page_cnt);
@@ -527,34 +536,56 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }if (ret) - goto out_free_pages; - -struct apply_range_data data = { .pages = pages, .i =0 }; - ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); -if (ret) - goto out; + goto out_unlock_free_pages; + remaining = page_cnt; uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); -/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 -* will not overflow 32-bit. Lower 32-bit need to represent -* contiguous user address range. -* Map these pages at kern_vm_start base. -* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow -* lower 32-bit and it's ok. -*/ - ret =apply_to_page_range (&init_mm, kern_vm_start + uaddr32, - page_cnt <<PAGE_SHIFT , apply_range_set_cb, &data); -if (ret) { -for (i =0 ; i < page_cnt; i++) -__free_page (pages[i]); - goto out; + +while (remaining) { +long this_batch =min (remaining, alloc_pages); + +/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ +memset (pages,0 , this_batch *sizeof (struct page *)); + data.i =0 ; + + ret =bpf_map_alloc_pages (&arena->map, node_id, this_batch, pages); +if (ret) + goto out; + +/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 +* will not overflow 32-bit. Lower 32-bit need to represent +* contiguous user address range. +* Map these pages at kern_vm_start base. +* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow +* lower 32-bit and it's ok. +*/ + ret =apply_to_page_range (&init_mm, + kern_vm_start + uaddr32 + (mapped <<PAGE_SHIFT ), + this_batch <<PAGE_SHIFT , apply_range_set_cb, &data); +if (ret) { +/* data.i pages were mapped, account them and free the remaining */ + mapped += data.i; +for (i = data.i; i < this_batch; i++) +__free_page (pages[i]); + goto out; + } + + mapped += this_batch; + remaining -= this_batch; } -kvfree (pages); +mutex_unlock (&arena->lock); +kfree_nolock (pages);return clear_lo32 (arena->user_vm_start) + uaddr32; out: -range_tree_set (&arena->rt, pgoff, page_cnt); +range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped); +mutex_unlock (&arena->lock); +if (mapped) +arena_free_pages (arena,clear_lo32 (arena->user_vm_start) + uaddr32, mapped); + goto out_free_pages; +out_unlock_free_pages: +mutex_unlock (&arena->lock); out_free_pages: -kvfree (pages); +kfree_nolock (pages);return 0 ; }
Signature · 1 line
2.47.1
PATCH v3 3/4 bpf: arena: make arena kfuncs any context safe
Make arena related kfuncs any context safe by the following changes:
bpf_arena_alloc_pages() and bpf_arena_reserve_pages():
Replace the usage of the mutex with a rqspinlock for range tree and use kmalloc_nolock() wherever needed. Use free_pages_nolock() to free pages from any context.
apply_range_set/clear_cb() with apply_to_page_range() has already made populating the vm_area in bpf_arena_alloc_pages() any context safe.
bpf_arena_free_pages(): defer the main logic to a workqueue if it is called from a non-sleepable context.
specialize_kfunc() is used to replace the sleepable arena_free_pages() with bpf_arena_free_pages_non_sleepable() when the verifier detects the call is from a non-sleepable context.
In the non-sleepable case, arena_free_pages() queues the address and the page count to be freed to a lock-less list of struct arena_free_spans and raises an irq_work. The irq_work handler calls schedules_work() as it is safe to be called from irq context. arena_free_worker() (the work queue handler) iterates these spans and clears ptes, flushes tlb, zaps pages, and calls __free_page().
--- include/linux/bpf.h | 15 +++ kernel/bpf/arena.c | 249 +++++++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 10 ++ 3 files changed, 233 insertions(+), 41 deletions(-)
include/linux/bpf.h +15 -0
@@ -673,6 +673,21 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT ) +void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id, +u64 flags); +void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt); +#else +static inline void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt, +int node_id,u64 flags) +{ +} + +static inline void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt) +{ +} +#endif +extern const struct bpf_map_ops bpf_map_offload_ops;/* bpf_type_flag contains a set of flags that are applicable to the values of
kernel/bpf/arena.c +208 -41
455 lines not on this page · raw message
kernel/bpf/verifier.c +10 -0
@@ -12349,6 +12349,8 @@ enum special_kfunc_type { KF___bpf_trap, KF_bpf_task_work_schedule_signal_impl, KF_bpf_task_work_schedule_resume_impl, + KF_bpf_arena_alloc_pages, + KF_bpf_arena_free_pages, };BTF_ID_LIST (special_kfunc_list)
@@ -12423,6 +12425,8 @@ BTF_ID(func, bpf_dynptr_file_discard) BTF_ID(func, __bpf_trap) BTF_ID(func, bpf_task_work_schedule_signal_impl) BTF_ID(func, bpf_task_work_schedule_resume_impl) +BTF_ID(func, bpf_arena_alloc_pages) +BTF_ID(func, bpf_arena_free_pages)static bool is_task_work_add_kfunc (u32 func_id) {
@@ -22380,6 +22384,12 @@ static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc }else if (func_id == special_kfunc_list[KF_bpf_dynptr_from_file]) {if (!env->insn_aux_data[insn_idx].non_sleepable) addr = (unsigned long )bpf_dynptr_from_file_sleepable; + }else if (func_id == special_kfunc_list[KF_bpf_arena_alloc_pages]) { +if (env->insn_aux_data[insn_idx].non_sleepable) + addr = (unsigned long )bpf_arena_alloc_pages_non_sleepable; + }else if (func_id == special_kfunc_list[KF_bpf_arena_free_pages]) { +if (env->insn_aux_data[insn_idx].non_sleepable) + addr = (unsigned long )bpf_arena_free_pages_non_sleepable; } desc->addr = addr;return 0 ;
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v3 3/4] bpf: arena: make arena kfuncs any context safe
Quoted 114 lines · Puranjay Mohan
Make arena related kfuncs any context safe by the following changes:
bpf_arena_alloc_pages()andbpf_arena_reserve_pages():
Replace the usage of the mutex with a rqspinlock for range tree and usekmalloc_nolock()wherever needed. Usefree_pages_nolock()to free pages from any context.
apply_range_set/clear_cb() withapply_to_page_range()has already made populating thevm_areainbpf_arena_alloc_pages()any context safe.
bpf_arena_free_pages(): defer the main logic to a workqueue if it is called from a non-sleepable context.
specialize_kfunc()is used to replace the sleepablearena_free_pages()withbpf_arena_free_pages_non_sleepable()when the verifier detects the call is from a non-sleepable context.In the non-sleepable case,
arena_free_pages()queues the address and the page count to be freed to a lock-less list of structarena_free_spansand raises anirq_work. Theirq_workhandler callsschedules_work()as it is safe to be called from irq context.arena_free_worker()(the work queue handler) iterates these spans and clears ptes, flushes tlb, zaps pages, and calls__free_page().Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/bpf.h | 15 +++ kernel/bpf/arena.c | 249 +++++++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 10 ++ 3 files changed, 233 insertions(+), 41 deletions(-)include/linux/bpf.h +15
-0@@ -673,6 +673,21 @@ void bpf_map_free_internal_structs(struct bpf_map *map, void *obj);int bpf_dynptr_from_file_sleepable (struct file *file,u32 flags,struct bpf_dynptr *ptr__uninit); +#if defined(CONFIG_MMU ) && defined(CONFIG_64BIT ) +void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt,int node_id, +u64 flags); +void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt); +#else +static inline void *bpf_arena_alloc_pages_non_sleepable (void *p__map,void *addr__ign,u32 page_cnt, +int node_id,u64 flags) +{ +} + +static inline void bpf_arena_free_pages_non_sleepable (void *p__map,void *ptr__ign,u32 page_cnt) +{ +} +#endif +extern const struct bpf_map_ops bpf_map_offload_ops;/* bpf_type_flag contains a set of flags that are applicable to the values of kernel/bpf/arena.c +19
-2@@ -3,7 +3,9 @@#include <linux/bpf.h> #include <linux/btf.h> #include <linux/err.h> +#include <linux/irq_work.h> #include "linux/filter.h" +#include <linux/llist.h> #include <linux/btf_ids.h> #include <linux/vmalloc.h> #include <linux/pagemap.h> @@ -43,7 +45,7 @@#define GUARD_SZ round_up(1ull << sizeof_field(struct bpf_insn, off) * 8, PAGE_SIZE << 1)#define KERN_VM_SZ (SZ_4G + GUARD_SZ) -static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt); +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt,bool sleepable);struct bpf_arena {struct bpf_map map;@@ -51,8 +53,23 @@ struct bpf_arena {u64 user_vm_end;struct vm_struct *kern_vm;struct range_tree rt; +/* protects rt */ +rqspinlock_t spinlock;struct list_head vma_list; +/* protects vma_list */ struct mutex lock; +struct irq_work free_irq; +struct work_struct free_work; +struct llist_head free_spans; +}; + +static void arena_free_worker (struct work_struct *work); +static void arena_free_irq (struct irq_work *iw); + +struct arena_free_span { +struct llist_node node; +unsigned long uaddr; +u32 page_cnt; };u64 bpf_arena_get_kern_vm_start (struct bpf_arena *arena)@@ -120,7 +137,7 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)return 0 ; } -static int apply_range_clear_cb (pte_t *pte,unsigned long addr,void *data) +static int apply_range_clear_cb (pte_t *pte,unsigned long addr,void *free_pages) {pte_t old_pte;struct page *page;
kernel/bpf/arena.c static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data) · line 150
Quoted 5 lines
kernel/bpf/arena.c +1
-1@@ -130,17 +147,16 @@ static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)if (pte_none (old_pte) || !pte_present (old_pte))return 0 ;/* nothing to do */ -/* get page and free it */ +/* get page and clear pte */
Just delete the comment. The code is obvious enough.
Quoted 51 lines
kernel/bpf/arena.c +12
-4page =pte_page (old_pte);if (WARN_ON_ONCE (!page))return -EINVAL ;pte_clear (&init_mm, addr, pte); -/* ensure no stale TLB entries */ -flush_tlb_kernel_range (addr, addr +PAGE_SIZE ); - -__free_page (page); +/* Add page to the list so it is freed later */ +if (free_pages) +__llist_add (&page->pcp_llist, free_pages);return 0 ; }@@ -195,6 +211,9 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) arena->user_vm_end = arena->user_vm_start + vm_range;INIT_LIST_HEAD (&arena->vma_list); +init_llist_head (&arena->free_spans); +init_irq_work (&arena->free_irq, arena_free_irq); +INIT_WORK (&arena->free_work, arena_free_worker);bpf_map_init_from_attr (&arena->map, attr);range_tree_init (&arena->rt); err =range_tree_set (&arena->rt,0 , attr->max_entries);@@ -203,6 +222,7 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) goto err; }mutex_init (&arena->lock); +raw_res_spin_lock_init (&arena->spinlock); err =populate_pgtable_except_pte (arena);if (err) {range_tree_destroy (&arena->rt);@@ -249,6 +269,10 @@ static void arena_map_free(struct bpf_map *map)if (WARN_ON_ONCE (!list_empty (&arena->vma_list)))return ; +/* Ensure no pending deferred frees */ +irq_work_sync (&arena->free_irq); +flush_work (&arena->free_work); +/* * free_vm_area() calls remove_vm_area() that calls free_unmap_vmap_area(). * It unmaps everything from vmalloc area and clears pgtables. @@ -332,12 +356,19 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) struct bpf_arena *arena = container_of(map, struct bpf_arena, map);struct page *page;long kbase, kaddr; +unsigned long flags;int ret;
kernel/bpf/arena.c static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · line 369
Quoted 10 lines
kernel/bpf/arena.c +5
-1kbase =bpf_arena_get_kern_vm_start (arena); kaddr = kbase + (u32)(vmf->address); -guard (mutex)(&arena->lock); +if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) +/* +* This is an impossible case and would only trigger if res_spin_lock is buggy or +* due to another kernel bug. +*/
Looking at this comment again I'm starting to feel that it opens a can of worms, since what does it mean to retry when there is a kernel bug or rqspinlock bug? Should it be BUG_ON? etc.
Let's just say
/* Make a reasonable effort to address impossible case */
Quoted 138 lines
kernel/bpf/arena.c +32
-14+return VM_FAULT_RETRY ; + page =vmalloc_to_page ((void *)kaddr);if (page)/* already have a page vmap-ed */ @@ -345,30 +376,34 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)if (arena->map.map_flags &BPF_F_SEGV_ON_FAULT )/* User space requested to segfault when page is not allocated by bpf prog */ -return VM_FAULT_SIGSEGV ; + goto out_unlock_sigsegv; ret =range_tree_clear (&arena->rt, vmf->pgoff,1 );if (ret) -return VM_FAULT_SIGSEGV ; + goto out_unlock_sigsegv;struct apply_range_data data = { .pages = &page, .i =0 };/* Account into memcg of the process that created bpf_arena */ ret =bpf_map_alloc_pages (map,NUMA_NO_NODE ,1 , &page);if (ret) {range_tree_set (&arena->rt, vmf->pgoff,1 ); -return VM_FAULT_SIGSEGV ; + goto out_unlock_sigsegv; } ret =apply_to_page_range (&init_mm, kaddr,PAGE_SIZE , apply_range_set_cb, &data);if (ret) {range_tree_set (&arena->rt, vmf->pgoff,1 ); -__free_page (page); -return VM_FAULT_SIGSEGV ; +free_pages_nolock (page,0 ); + goto out_unlock_sigsegv; } out:page_ref_add (page,1 ); +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); vmf->page = page;return 0 ; +out_unlock_sigsegv: +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); +return VM_FAULT_SIGSEGV ; }static const struct vm_operations_struct arena_vm_ops = {@@ -489,7 +524,8 @@ static u64 clear_lo32(u64 val)* Allocate pages and vmap them into kernel vmalloc area. * Later the pages will be mmaped into user space vma. */ -static long arena_alloc_pages (struct bpf_arena *arena,long uaddr,long page_cnt,int node_id) +static long arena_alloc_pages (struct bpf_arena *arena,long uaddr,long page_cnt,int node_id, +bool sleepable) {/* user_vm_end/start are fixed before bpf prog runs */ long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;@@ -498,6 +534,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntstruct page **pages =NULL ;long remaining, mapped =0 ;long alloc_pages; +unsigned long flags;long pgoff =0 ;u32 uaddr32;int ret, i;@@ -523,7 +560,8 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntreturn 0 ; data.pages = pages; -mutex_lock (&arena->lock); +if (raw_res_spin_lock_irqsave (&arena->spinlock, flags)) + goto out_free_pages;if (uaddr) { ret =is_range_tree_set (&arena->rt, pgoff, page_cnt);@@ -566,24 +604,25 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* data.i pages were mapped, account them and free the remaining */ mapped += data.i;for (i = data.i; i < this_batch; i++) -__free_page (pages[i]); +free_pages_nolock (pages[i],0 ); goto out; } mapped += this_batch; remaining -= this_batch; } -mutex_unlock (&arena->lock); +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);kfree_nolock (pages);return clear_lo32 (arena->user_vm_start) + uaddr32; out:range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped); -mutex_unlock (&arena->lock); +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags);if (mapped) -arena_free_pages (arena,clear_lo32 (arena->user_vm_start) + uaddr32, mapped); +arena_free_pages (arena,clear_lo32 (arena->user_vm_start) + uaddr32, mapped, + sleepable); goto out_free_pages; out_unlock_free_pages: -mutex_unlock (&arena->lock); +raw_res_spin_unlock_irqrestore (&arena->spinlock, flags); out_free_pages:kfree_nolock (pages);return 0 ;@@ -598,42 +637,65 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) {struct vma_list *vml; +guard (mutex)(&arena->lock); +/* iterate link list under lock */ list_for_each_entry (vml, &arena->vma_list, head)zap_page_range_single (vml->vma, uaddr,PAGE_SIZE * page_cnt,NULL ); } -static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt) +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt,bool sleepable) {u64 full_uaddr, uaddr_end; -long kaddr, pgoff, i; +long kaddr, pgoff;struct page *page; +struct llist_head free_pages; +struct llist_node *pos, *t; +struct arena_free_span *s; +unsigned long flags; +int ret =0 ;/* only aligned lower 32-bit are relevant */ uaddr = (u32 )uaddr; uaddr &=PAGE_MASK ; + kaddr =bpf_arena_get_kern_vm_start (arena) + uaddr; full_uaddr =clear_lo32 (arena->user_vm_start) + uaddr; uaddr_end =min (arena->user_vm_end, full_uaddr + (page_cnt <<PAGE_SHIFT ));if (full_uaddr >= uaddr_end)return ; page_cnt = (uaddr_end - full_uaddr) >>PAGE_SHIFT ;
kernel/bpf/arena.c static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) · line 676
Quoted 10 lines
kernel/bpf/arena.c +8
-1+ pgoff =compute_pgoff (arena, uaddr); -guard (mutex)(&arena->lock); +if (!sleepable) + goto defer; + + ret =raw_res_spin_lock_irqsave (&arena->spinlock, flags); +/* +* Can't proceed without holding the spinlock so defer the free +*/
This can be a single line comment like /* Can't proceed ... */
PATCH v3 4/4 selftests: bpf: test non-sleepable arena allocations
In reply to Puranjay MohanAs arena kfuncs can now be called from non-sleepable contexts, test this by adding non-sleepable copies of tests in verifier_arena, this is done by using a socket program instead of syscall.
Add a new test case in verifier_arena_large to check that the bpf_arena_alloc_pages() works for more than 1024 pages.
1024 * sizeof(struct page *) is the upper limit of kmalloc_nolock() but bpf_arena_alloc_pages() should still succeed because it re-uses this array in a loop.
Augment the arena_list selftest to also run in non-sleepable context by taking rcu_read_lock.
--- .../selftests/bpf/prog_tests/arena_list.c | 20 +- .../testing/selftests/bpf/progs/arena_list.c | 11 ++ .../selftests/bpf/progs/verifier_arena.c | 185 ++++++++++++++++++ .../bpf/progs/verifier_arena_large.c | 29 +++ 4 files changed, 240 insertions(+), 5 deletions(-)
tools/testing/selftests/bpf/prog_tests/arena_list.c +15 -5
@@ -27,17 +27,23 @@ static int list_sum(struct arena_list_head *head)return sum; } -static void test_arena_list_add_del (int cnt) +static void test_arena_list_add_del (int cnt,bool nonsleepable) {LIBBPF_OPTS (bpf_test_run_opts, opts);struct arena_list *skel;int expected_sum = (u64 )cnt * (cnt -1 ) /2 ;int ret, sum; - skel =arena_list__open_and_load (); -if (!ASSERT_OK_PTR (skel,"arena_list__open_and_load" )) + skel =arena_list__open (); +if (!ASSERT_OK_PTR (skel,"arena_list__open" ))return ; + skel->rodata->nonsleepable = nonsleepable; + + ret =arena_list__load (skel); +if (!ASSERT_OK (ret,"arena_list__load" )) + goto out; + skel->bss->cnt = cnt; ret =bpf_prog_test_run_opts (bpf_program__fd (skel->progs.arena_list_add), &opts);ASSERT_OK (ret,"ret_add" );
@@ -65,7 +71,11 @@ static void test_arena_list_add_del(int cnt)void test_arena_list (void ) {if (test__start_subtest ("arena_list_1" )) -test_arena_list_add_del (1 ); +test_arena_list_add_del (1 , false);if (test__start_subtest ("arena_list_1000" )) -test_arena_list_add_del (1000 ); +test_arena_list_add_del (1000 , false); +if (test__start_subtest ("arena_list_1_nonsleepable" )) +test_arena_list_add_del (1 , true); +if (test__start_subtest ("arena_list_1000_nonsleepable" )) +test_arena_list_add_del (1000 , true); }
tools/testing/selftests/bpf/progs/arena_list.c +11 -0
@@ -30,6 +30,7 @@ struct arena_list_head __arena *list_head;int list_sum;int cnt;bool skip = false; +const volatile bool nonsleepable = false;#ifdef __BPF_FEATURE_ADDR_SPACE_CASTlong __arena arena_sum;
@@ -42,6 +43,9 @@ int test_val SEC(".addr_space.1");int zero; +void bpf_rcu_read_lock(void) __ksym; +void bpf_rcu_read_unlock(void) __ksym; + SEC("syscall")int arena_list_add (void *ctx) {
@@ -71,6 +75,10 @@ int arena_list_del(void *ctx) struct elem __arena *n;int sum =0 ; +/* Take rcu_read_lock to test non-sleepable context */ +if (nonsleepable) +bpf_rcu_read_lock (); + arena_sum =0 ;list_for_each_entry (n, list_head, node) { sum += n->value;
@@ -79,6 +87,9 @@ int arena_list_del(void *ctx) bpf_free(n); } list_sum = sum; + +if (nonsleepable) +bpf_rcu_read_unlock ();#else skip = true;#endif
tools/testing/selftests/bpf/progs/verifier_arena.c +185 -0
227 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_arena_large.c +29 -0
@@ -270,5 +270,34 @@ int big_alloc2(void *ctx)return 9 ;return 0 ; } + +SEC("socket") +__success __retval(0) +int big_alloc3(void *ctx) +{ +#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) + char __arena *pages; +u64 i; + +/* +* Allocate 2051 pages in one go to check how kmalloc_nolock() handles large requests. +* Since kmalloc_nolock() can allocate up to 1024 struct page * at a time, this call should +* result in three batches: two batches of 1024 pages each, followed by a final batch of 3 +* pages. +*/ + pages =bpf_arena_alloc_pages (&arena,NULL ,2051 ,NUMA_NO_NODE ,0 ); +if (!pages) +return -1 ; + +bpf_for (i,0 ,2051 ) + pages[i * PAGE_SIZE] = 123; + bpf_for(i, 0, 2051) +if (pages[i *PAGE_SIZE ] !=123 ) +return i; + +bpf_arena_free_pages (&arena, pages,2051 ); +#endif +return 0 ; +}#endif char _license[] SEC("license") = "GPL";
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v3 2/4] bpf: arena: use kmalloc_nolock() in place of kvcalloc()
In reply to Puranjay MohanQuoted 87 lines · Puranjay Mohan
To make
arena_alloc_pages()safe to be called from any context, replacekvcalloc()withkmalloc_nolock()so as it doesn't sleep or take any locks.kmalloc_nolock()returns NULL for allocations larger thanKMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE* 2) = 8KB on systems with 4KB pages. So, round down the allocation done bykmalloc_nolockto 1024
* 8 and reuse the array in a loop.Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/arena.c | 83 +++++++++++++++++++++++++++++++--------------- 1 file changed, 57 insertions(+), 26 deletions(-)kernel/bpf/arena.c +15
-20@@ -43,6 +43,8 @@#define GUARD_SZ round_up(1ull << sizeof_field(struct bpf_insn, off) * 8, PAGE_SIZE << 1)#define KERN_VM_SZ (SZ_4G + GUARD_SZ) +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt); +struct bpf_arena {struct bpf_map map;u64 user_vm_start;@@ -492,7 +494,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* user_vm_end/start are fixed before bpf prog runs */ long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data;struct page **pages =NULL ; +long remaining, mapped =0 ; +long alloc_pages;long pgoff =0 ;u32 uaddr32;int ret, i;@@ -509,17 +514,21 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntreturn 0 ; } -/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ - pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL ); +/* +* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. +*/ + alloc_pages =min (page_cnt,KMALLOC_MAX_CACHE_SIZE /sizeof (struct page *)); + pages =kmalloc_nolock (alloc_pages *sizeof (struct page *),0 ,NUMA_NO_NODE );if (!pages)return 0 ; + data.pages = pages; -guard (mutex)(&arena->lock); +mutex_lock (&arena->lock);if (uaddr) { ret =is_range_tree_set (&arena->rt, pgoff, page_cnt);if (ret) - goto out_free_pages; + goto out_unlock_free_pages; ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }else { ret = pgoff =range_tree_find (&arena->rt, page_cnt);@@ -527,34 +536,56 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }if (ret) - goto out_free_pages; - -struct apply_range_data data = { .pages = pages, .i =0 }; - ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); -if (ret) - goto out; + goto out_unlock_free_pages; + remaining = page_cnt; uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); -/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 -* will not overflow 32-bit. Lower 32-bit need to represent -* contiguous user address range. -* Map these pages at kern_vm_start base. -* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow -* lower 32-bit and it's ok. -*/ - ret =apply_to_page_range (&init_mm, kern_vm_start + uaddr32, - page_cnt <<PAGE_SHIFT , apply_range_set_cb, &data); -if (ret) {
kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 549
Quoted 10 lines
kernel/bpf/arena.c +7
-3-for (i =0 ; i < page_cnt; i++) -__free_page (pages[i]); - goto out; + +while (remaining) { +long this_batch =min (remaining, alloc_pages); + +/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ +memset (pages,0 , this_batch *sizeof (struct page *)); + data.i =0 ;
Pls move data.i = 0 further down to be done right before apply_to_page_range() since it's one logical operation. Here it's done too early.
kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 555
Quoted 6 lines
kernel/bpf/arena.c +6
-0+ + ret =bpf_map_alloc_pages (&arena->map, node_id, this_batch, pages); +if (ret) + goto out; + +/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1
Pls reformat the comment as you move them as
/*
* ...
*/
Quoted 20 lines
kernel/bpf/arena.c +19
-0+* will not overflow 32-bit. Lower 32-bit need to represent +* contiguous user address range. +* Map these pages at kern_vm_start base. +* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow +* lower 32-bit and it's ok. +*/ + ret =apply_to_page_range (&init_mm, + kern_vm_start + uaddr32 + (mapped <<PAGE_SHIFT ), + this_batch <<PAGE_SHIFT , apply_range_set_cb, &data); +if (ret) { +/* data.i pages were mapped, account them and free the remaining */ + mapped += data.i; +for (i = data.i; i < this_batch; i++) +__free_page (pages[i]); + goto out; + } + + mapped += this_batch; + remaining -= this_batch; }
kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 583
Quoted 10 lines
kernel/bpf/arena.c +6
-2-kvfree (pages); +mutex_unlock (&arena->lock); +kfree_nolock (pages);return clear_lo32 (arena->user_vm_start) + uaddr32; out: -range_tree_set (&arena->rt, pgoff, page_cnt); +range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped); +mutex_unlock (&arena->lock); +if (mapped) +arena_free_pages (arena,clear_lo32 (arena->user_vm_start) + uaddr32, mapped);
This doesn't look right.
The first thing arena_free_pages() does is:
uaddr = (u32)uaddr;
uaddr &= PAGE_MASK;
full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;pw-bot: cr
Re: [PATCH bpf-next v3 2/4] bpf: arena: use kmalloc_nolock() in place of kvcalloc()
Quoted 147 lines · Alexei Starovoitov
Quoted 87 lines · Puranjay Mohan
To make
arena_alloc_pages()safe to be called from any context, replacekvcalloc()withkmalloc_nolock()so as it doesn't sleep or take any locks.kmalloc_nolock()returns NULL for allocations larger thanKMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE* 2) = 8KB on systems with 4KB pages. So, round down the allocation done bykmalloc_nolockto 1024
* 8 and reuse the array in a loop.Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/arena.c | 83 +++++++++++++++++++++++++++++++--------------- 1 file changed, 57 insertions(+), 26 deletions(-)kernel/bpf/arena.c +15
-20@@ -43,6 +43,8 @@#define GUARD_SZ round_up(1ull << sizeof_field(struct bpf_insn, off) * 8, PAGE_SIZE << 1)#define KERN_VM_SZ (SZ_4G + GUARD_SZ) +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt); +struct bpf_arena {struct bpf_map map;u64 user_vm_start;@@ -492,7 +494,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* user_vm_end/start are fixed before bpf prog runs */ long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data;struct page **pages =NULL ; +long remaining, mapped =0 ; +long alloc_pages;long pgoff =0 ;u32 uaddr32;int ret, i;@@ -509,17 +514,21 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntreturn 0 ; } -/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ - pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL ); +/* +* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. +*/ + alloc_pages =min (page_cnt,KMALLOC_MAX_CACHE_SIZE /sizeof (struct page *)); + pages =kmalloc_nolock (alloc_pages *sizeof (struct page *),0 ,NUMA_NO_NODE );if (!pages)return 0 ; + data.pages = pages; -guard (mutex)(&arena->lock); +mutex_lock (&arena->lock);if (uaddr) { ret =is_range_tree_set (&arena->rt, pgoff, page_cnt);if (ret) - goto out_free_pages; + goto out_unlock_free_pages; ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }else { ret = pgoff =range_tree_find (&arena->rt, page_cnt);@@ -527,34 +536,56 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }if (ret) - goto out_free_pages; - -struct apply_range_data data = { .pages = pages, .i =0 }; - ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); -if (ret) - goto out; + goto out_unlock_free_pages; + remaining = page_cnt; uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); -/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 -* will not overflow 32-bit. Lower 32-bit need to represent -* contiguous user address range. -* Map these pages at kern_vm_start base. -* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow -* lower 32-bit and it's ok. -*/ - ret =apply_to_page_range (&init_mm, kern_vm_start + uaddr32, - page_cnt <<PAGE_SHIFT , apply_range_set_cb, &data); -if (ret) {kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 549
Quoted 10 lines
kernel/bpf/arena.c +7
-3-for (i =0 ; i < page_cnt; i++) -__free_page (pages[i]); - goto out; + +while (remaining) { +long this_batch =min (remaining, alloc_pages); + +/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ +memset (pages,0 , this_batch *sizeof (struct page *)); + data.i =0 ;Pls move data.i = 0 further down to be done right before
apply_to_page_range()since it's one logical operation. Here it's done too early.kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 555
Quoted 6 lines
kernel/bpf/arena.c +6
-0+ + ret =bpf_map_alloc_pages (&arena->map, node_id, this_batch, pages); +if (ret) + goto out; + +/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 Pls reformat the comment as you move them as
/*
* ...
*/Quoted 20 lines
kernel/bpf/arena.c +19
-0+* will not overflow 32-bit. Lower 32-bit need to represent +* contiguous user address range. +* Map these pages at kern_vm_start base. +* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow +* lower 32-bit and it's ok. +*/ + ret =apply_to_page_range (&init_mm, + kern_vm_start + uaddr32 + (mapped <<PAGE_SHIFT ), + this_batch <<PAGE_SHIFT , apply_range_set_cb, &data); +if (ret) { +/* data.i pages were mapped, account them and free the remaining */ + mapped += data.i; +for (i = data.i; i < this_batch; i++) +__free_page (pages[i]); + goto out; + } + + mapped += this_batch; + remaining -= this_batch; }kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 583
Quoted 10 lines
kernel/bpf/arena.c +6
-2-kvfree (pages); +mutex_unlock (&arena->lock); +kfree_nolock (pages);return clear_lo32 (arena->user_vm_start) + uaddr32; out: -range_tree_set (&arena->rt, pgoff, page_cnt); +range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped); +mutex_unlock (&arena->lock); +if (mapped) +arena_free_pages (arena,clear_lo32 (arena->user_vm_start) + uaddr32, mapped);
Quoted 5 lines
This doesn't look right.
The first thingarena_free_pages()does is:uaddr = (u32)uaddr; uaddr &= PAGE_MASK; full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
So, arena_free_pages() should be called with what arena_alloc_pages() returns, we return:
return clear_lo32(arena->user_vm_start) + uaddr32;
from arena_alloc_pages(); few lines above.
This usage looks correct to me.
Thanks,
Puranjay
Re: [PATCH bpf-next v3 2/4] bpf: arena: use kmalloc_nolock() in place of kvcalloc()
Quoted 156 lines · Puranjay Mohan
Quoted 147 lines · Alexei Starovoitov
Quoted 87 lines · Puranjay Mohan
To make
arena_alloc_pages()safe to be called from any context, replacekvcalloc()withkmalloc_nolock()so as it doesn't sleep or take any locks.kmalloc_nolock()returns NULL for allocations larger thanKMALLOC_MAX_CACHE_SIZE, which is (PAGE_SIZE* 2) = 8KB on systems with 4KB pages. So, round down the allocation done bykmalloc_nolockto 1024
* 8 and reuse the array in a loop.Signed-off-by Puranjay Mohan <[email protected]>--- kernel/bpf/arena.c | 83 +++++++++++++++++++++++++++++++--------------- 1 file changed, 57 insertions(+), 26 deletions(-)kernel/bpf/arena.c +15
-20@@ -43,6 +43,8 @@#define GUARD_SZ round_up(1ull << sizeof_field(struct bpf_insn, off) * 8, PAGE_SIZE << 1)#define KERN_VM_SZ (SZ_4G + GUARD_SZ) +static void arena_free_pages (struct bpf_arena *arena,long uaddr,long page_cnt); +struct bpf_arena {struct bpf_map map;u64 user_vm_start;@@ -492,7 +494,10 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt/* user_vm_end/start are fixed before bpf prog runs */ long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >>PAGE_SHIFT ;u64 kern_vm_start =bpf_arena_get_kern_vm_start (arena); +struct apply_range_data data;struct page **pages =NULL ; +long remaining, mapped =0 ; +long alloc_pages;long pgoff =0 ;u32 uaddr32;int ret, i;@@ -509,17 +514,21 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cntreturn 0 ; } -/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ - pages =kvcalloc (page_cnt,sizeof (struct page *),GFP_KERNEL ); +/* +* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. +*/ + alloc_pages =min (page_cnt,KMALLOC_MAX_CACHE_SIZE /sizeof (struct page *)); + pages =kmalloc_nolock (alloc_pages *sizeof (struct page *),0 ,NUMA_NO_NODE );if (!pages)return 0 ; + data.pages = pages; -guard (mutex)(&arena->lock); +mutex_lock (&arena->lock);if (uaddr) { ret =is_range_tree_set (&arena->rt, pgoff, page_cnt);if (ret) - goto out_free_pages; + goto out_unlock_free_pages; ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }else { ret = pgoff =range_tree_find (&arena->rt, page_cnt);@@ -527,34 +536,56 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt ret =range_tree_clear (&arena->rt, pgoff, page_cnt); }if (ret) - goto out_free_pages; - -struct apply_range_data data = { .pages = pages, .i =0 }; - ret =bpf_map_alloc_pages (&arena->map, node_id, page_cnt, pages); -if (ret) - goto out; + goto out_unlock_free_pages; + remaining = page_cnt; uaddr32 = (u32)(arena->user_vm_start + pgoff *PAGE_SIZE ); -/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 -* will not overflow 32-bit. Lower 32-bit need to represent -* contiguous user address range. -* Map these pages at kern_vm_start base. -* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow -* lower 32-bit and it's ok. -*/ - ret =apply_to_page_range (&init_mm, kern_vm_start + uaddr32, - page_cnt <<PAGE_SHIFT , apply_range_set_cb, &data); -if (ret) {kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 549
Quoted 10 lines
kernel/bpf/arena.c +7
-3-for (i =0 ; i < page_cnt; i++) -__free_page (pages[i]); - goto out; + +while (remaining) { +long this_batch =min (remaining, alloc_pages); + +/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */ +memset (pages,0 , this_batch *sizeof (struct page *)); + data.i =0 ;Pls move data.i = 0 further down to be done right before
apply_to_page_range()since it's one logical operation. Here it's done too early.kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 555
Quoted 6 lines
kernel/bpf/arena.c +6
-0+ + ret =bpf_map_alloc_pages (&arena->map, node_id, this_batch, pages); +if (ret) + goto out; + +/* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1 Pls reformat the comment as you move them as
/*
* ...
*/Quoted 20 lines
kernel/bpf/arena.c +19
-0+* will not overflow 32-bit. Lower 32-bit need to represent +* contiguous user address range. +* Map these pages at kern_vm_start base. +* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow +* lower 32-bit and it's ok. +*/ + ret =apply_to_page_range (&init_mm, + kern_vm_start + uaddr32 + (mapped <<PAGE_SHIFT ), + this_batch <<PAGE_SHIFT , apply_range_set_cb, &data); +if (ret) { +/* data.i pages were mapped, account them and free the remaining */ + mapped += data.i; +for (i = data.i; i < this_batch; i++) +__free_page (pages[i]); + goto out; + } + + mapped += this_batch; + remaining -= this_batch; }kernel/bpf/arena.c static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · line 583
Quoted 10 lines
kernel/bpf/arena.c +6
-2-kvfree (pages); +mutex_unlock (&arena->lock); +kfree_nolock (pages);return clear_lo32 (arena->user_vm_start) + uaddr32; out: -range_tree_set (&arena->rt, pgoff, page_cnt); +range_tree_set (&arena->rt, pgoff + mapped, page_cnt - mapped); +mutex_unlock (&arena->lock); +if (mapped) +arena_free_pages (arena,clear_lo32 (arena->user_vm_start) + uaddr32, mapped);Quoted 5 lines
This doesn't look right.
The first thingarena_free_pages()does is:uaddr = (u32)uaddr; uaddr &= PAGE_MASK; full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
Quoted 6 lines
So,
arena_free_pages()should be called with whatarena_alloc_pages()returns, we return:return clear_lo32(arena->user_vm_start) + uaddr32;from
arena_alloc_pages(); few lines above.
yes, that's what this kfunc returns to bpf prog,
but we shouldn't do pointless math to call arena_free_pages(). Just arena_free_pages(..., uaddr32, ...) will do.