Share user memory to BPF program through task storage map
Subsystems (7)
Martin KaFai Lau
reviewerv6 cover Share user memory to BPF program through task storage map
From: Martin KaFai Lau <[email protected]>
It is the v6 of this series. Starting from v5, it is a continuation work of the RFC v4.
Changes in v6:
1. In patch 1, reject t->size == 0 in btf_check_and_fixup_fields. Reject a uptr pointing to an empty struct.
A test is added to patch 12 to test this case.
2. In patch 6, when checking if the uptr struct spans across pages, there was an off by one error in calculating the "end" such that the uptr will be rejected by error if the object is located exactly at the end of a page.
This is fixed by adding t->size "- 1" to "start".
A test is added to patch 9 to test this case.
3. In patch 6, check for PageHighMem(page) and return -EOPNOTSUPP. The 32 bit arch jit is missing other crucial bpf features (e.g. kfunc). Patch 6 commit message has been updated to include this change.
4. The selftests are cleaned up such that "struct user_data *dummy_data" global ptr is used instead of the whole "struct user_data dummy_data" object. Still a hack to avoid generating fwd btf type for the uptr struct but somewhat lighter than a full blown global object.
Changes in v5:
1. The original patch 1 and patch 2 are combined.
2. Patch 3, 4, and 5 are new. They get the bpf_local_storage ready to handle the __uptr in the map_value.
3. Patch 6 is mostly new, so I reset the sob.
4. There are some changes in the carry over patch 1 and 2 also. They are mentioned at the individual patch.
5. More tests are added.
The following is the original cover letter and the earlier change log. The bpf prog example has been removed. Please find a similar example in the selftests task_ls_uptr.c.
~~~~~~~~
Some of BPF schedulers (sched_ext) need hints from user programs to do a better job. For example, a scheduler can handle a task in a different way if it knows a task is doing GC. So, we need an efficient way to share the information between user programs and BPF programs. Sharing memory between user programs and BPF programs is what this patchset does.
== REQUIREMENT ==
This patchset enables every task in every process to share a small chunk of memory of it's own with a BPF scheduler. So, they can update the hints without expensive overhead of syscalls. It also wants every task sees only the data/memory belong to the task/or the task's process.
== DESIGN ==
This patchset enables BPF prorams to embed __uptr; uptr in the values of task storage maps. A uptr field can only be set by user programs by updating map element value through a syscall. A uptr points to a block of memory allocated by the user program updating the element value. The memory will be pinned to ensure it staying in the core memory and to avoid a page fault when the BPF program accesses it.
Please see the selftests task_ls_uptr.c for an example.
== MEMORY ==
In order to use memory efficiently, we don't want to pin a large number of pages. To archieve that, user programs should collect the memory blocks pointed by uptrs together to share memory pages if possible. It avoid the situation that pin one page for each thread in a process. Instead, we can have several threads pointing their uptrs to the same page but with different offsets.
Although it is not necessary, avoiding the memory pointed by an uptr crossing the boundary of a page can prevent an additional mapping in the kernel address space.
== RESTRICT ==
The memory pointed by a uptr should reside in one memory page. Crossing multi-pages is not supported at the moment.
Only task storage map have been supported at the moment.
The values of uptrs can only be updated by user programs through syscalls.
bpf_map_lookup_elem() from userspace returns zeroed values for uptrs to prevent leaking information of the kernel.
---
Changes from v3:
- Merge part 4 and 5 as the new part 4 in order to cease the warning of unused functions from CI.
Changes from v1:
- Rename BPF_KPTR_USER to BPF_UPTR.
- Restrict uptr to one page.
- Mark uptr with PTR_TO_MEM | PTR_MAY_BE_NULL and with the size of the target type.
- Move uptr away from bpf_obj_memcpy() by introducing bpf_obj_uptrcpy() and copy_map_uptr_locked().
- Remove the BPF_FROM_USER flag.
- Align the meory pointed by an uptr in the test case. Remove the uptr of mmapped memory.
Kui-Feng Lee (4):
bpf: Support __uptr type tag in BTF
bpf: Handle BPF_UPTR in verifier
libbpf: define __uptr.
selftests/bpf: Some basic __uptr tests
Martin KaFai Lau (8):
bpf: Add "bool swap_uptrs" arg to bpf_local_storage_update() and
bpf_selem_alloc()
bpf: Postpone bpf_selem_free() in bpf_selem_unlink_storage_nolock()
bpf: Postpone bpf_obj_free_fields to the rcu callback
bpf: Add uptr support in the map_value of the task local storage. selftests/bpf: Test a uptr struct spanning across pages. selftests/bpf: Add update_elem failure test for task storage uptr selftests/bpf: Add uptr failure verifier tests selftests/bpf: Create task_local_storage map with invalid uptr's
struct
include/linux/bpf.h | 25 ++ include/linux/bpf_local_storage.h | 12 +- kernel/bpf/bpf_cgrp_storage.c | 4 +- kernel/bpf/bpf_inode_storage.c | 4 +- kernel/bpf/bpf_local_storage.c | 79 ++++- kernel/bpf/bpf_task_storage.c | 7 +- kernel/bpf/btf.c | 34 ++- kernel/bpf/syscall.c | 108 ++++++- kernel/bpf/verifier.c | 39 ++- net/core/bpf_sk_storage.c | 6 +- tools/lib/bpf/bpf_helpers.h | 1 + .../bpf/prog_tests/task_local_storage.c | 278 ++++++++++++++++++ .../selftests/bpf/progs/task_ls_uptr.c | 63 ++++ .../selftests/bpf/progs/uptr_failure.c | 105 +++++++ .../selftests/bpf/progs/uptr_map_failure.c | 27 ++ .../selftests/bpf/progs/uptr_update_failure.c | 42 +++ tools/testing/selftests/bpf/test_progs.h | 8 + .../testing/selftests/bpf/uptr_test_common.h | 63 ++++ 18 files changed, 862 insertions(+), 43 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/task_ls_uptr.c create mode 100644 tools/testing/selftests/bpf/progs/uptr_failure.c create mode 100644 tools/testing/selftests/bpf/progs/uptr_map_failure.c create mode 100644 tools/testing/selftests/bpf/progs/uptr_update_failure.c create mode 100644 tools/testing/selftests/bpf/uptr_test_common.h
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 1/12 bpf: Support __uptr type tag in BTF
From: Kui-Feng Lee <[email protected]>
This patch introduces the "__uptr" type tag to BTF. It is to define a pointer pointing to the user space memory. This patch adds BTF logic to pass the "__uptr" type tag.
btf_find_kptr() is reused for the "__uptr" tag. The "__uptr" will only be supported in the map_value of the task storage map. However, btf_parse_struct_meta() also uses btf_find_kptr() but it is not interested in "__uptr". This patch adds a "field_mask" argument to btf_find_kptr() which will return BTF_FIELD_IGNORE if the caller is not interested in a “__uptr” field.
btf_parse_kptr() is also reused to parse the uptr.
The btf_check_and_fixup_fields() is changed to do extra checks on the uptr to ensure that its struct size is not larger than PAGE_SIZE. It is not clear how a uptr pointing to a CO-RE supported kernel struct will be used, so it is also not allowed now.
---
Changes in v5:
- A field_mask arg is added to btf_find_kptr
- Some uptr enforcement is added to btf_check_and_fixup_fields()
- The "case MAP_UPTR:" addition to bpf_obj_init_field() is moved to the later patch together with other bpf_obj_*() changes when BPF_UPTR is finally enabled in task storage map.
include/linux/bpf.h | 5 +++++ kernel/bpf/btf.c | 34 +++++++++++++++++++++++++++++----- kernel/bpf/syscall.c | 2 ++ 3 files changed, 36 insertions(+), 5 deletions(-)
include/linux/bpf.h +5 -0
@@ -203,6 +203,7 @@ enum btf_field_type {BPF_GRAPH_ROOT =BPF_RB_ROOT |BPF_LIST_HEAD ,BPF_REFCOUNT = (1 <<9 ),BPF_WORKQUEUE = (1 <<10 ), + BPF_UPTR = (1 << 11), };typedef void (*btf_dtor_kfunc_t )(void *);
@@ -322,6 +323,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)return "kptr" ;case BPF_KPTR_PERCPU :return "percpu_kptr" ; +case BPF_UPTR : +return "uptr" ;case BPF_LIST_HEAD :return "bpf_list_head" ;case BPF_LIST_NODE :
@@ -350,6 +353,7 @@ static inline u32 btf_field_type_size(enum btf_field_type type)case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :return sizeof (u64);case BPF_LIST_HEAD :return sizeof (struct bpf_list_head );
@@ -379,6 +383,7 @@ static inline u32 btf_field_type_align(enum btf_field_type type)case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :return __alignof__(u64 );case BPF_LIST_HEAD :return __alignof__(struct bpf_list_head );
kernel/bpf/btf.c +29 -5
@@ -3334,7 +3334,7 @@ static int btf_find_struct(const struct btf *btf, const struct btf_type *t, }static int btf_find_kptr (const struct btf *btf,const struct btf_type *t, -u32 off,int sz,struct btf_field_info *info) +u32 off,int sz,struct btf_field_info *info,u32 field_mask) {enum btf_field_type type;u32 res_id;
@@ -3358,9 +3358,14 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t, type =BPF_KPTR_REF ; else if (!strcmp("percpu_kptr", __btf_name_by_offset(btf, t->name_off))) type =BPF_KPTR_PERCPU ; + else if (!strcmp("uptr", __btf_name_by_offset(btf, t->name_off))) + type =BPF_UPTR ; else return -EINVAL; +if (!(type & field_mask)) +return BTF_FIELD_IGNORE ; +/* Get the base type */ t =btf_type_skip_modifiers (btf, t->type, &res_id);/* Only pointer to struct is allowed */
@@ -3502,7 +3507,7 @@ static int btf_get_field_type(const struct btf *btf, const struct btf_type *var_field_mask_test_name (BPF_REFCOUNT ,"bpf_refcount" );/* Only return BPF_KPTR when all other types with matchable names fail */ -if (field_mask &BPF_KPTR && !__btf_type_is_struct (var_type)) { +if (field_mask & (BPF_KPTR |BPF_UPTR ) && !__btf_type_is_struct (var_type)) { type =BPF_KPTR_REF ; goto end; }
@@ -3535,6 +3540,7 @@ static int btf_repeat_fields(struct btf_field_info *info,case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :case BPF_LIST_HEAD :case BPF_RB_ROOT :break ;
@@ -3661,8 +3667,9 @@ static int btf_find_field_one(const struct btf *btf,case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR : ret =btf_find_kptr (btf, var_type, off, sz, - info_cnt ? &info[0 ] : &tmp); + info_cnt ? &info[0 ] : &tmp, field_mask);if (ret <0 )return ret;break ;
@@ -3985,6 +3992,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_typecase BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR : ret =btf_parse_kptr (btf, &rec->fields[i], &info_arr[i]);if (ret <0 ) goto end;
@@ -4044,12 +4052,28 @@ int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec)* Hence we only need to ensure that bpf_{list_head,rb_root} ownership * does not form cycles. */ -if (IS_ERR_OR_NULL (rec) || !(rec->field_mask &BPF_GRAPH_ROOT )) +if (IS_ERR_OR_NULL (rec) || !(rec->field_mask & (BPF_GRAPH_ROOT |BPF_UPTR )))return 0 ;for (i =0 ; i < rec->cnt; i++) {struct btf_struct_meta *meta; +const struct btf_type *t;u32 btf_id; +if (rec->fields[i].type ==BPF_UPTR ) { +/* The uptr only supports pinning one page and cannot +* point to a kernel struct +*/ +if (btf_is_kernel (rec->fields[i].kptr.btf)) +return -EINVAL ; + t =btf_type_by_id (rec->fields[i].kptr.btf, + rec->fields[i].kptr.btf_id); +if (!t->size) +return -EINVAL ; +if (t->size >PAGE_SIZE ) +return -E2BIG ; +continue ; + } +if (!(rec->fields[i].type &BPF_GRAPH_ROOT ))continue ; btf_id = rec->fields[i].graph_root.value_btf_id;
@@ -5560,7 +5584,7 @@ btf_parse_struct_metas(struct bpf_verifier_log *log, struct btf *btf) goto free_aof; } - ret =btf_find_kptr (btf, t,0 ,0 , &tmp); + ret =btf_find_kptr (btf, t,0 ,0 , &tmp,BPF_KPTR );if (ret !=BTF_FIELD_FOUND )continue ;
kernel/bpf/syscall.c +2 -0
@@ -548,6 +548,7 @@ void btf_record_free(struct btf_record *rec)case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :if (rec->fields[i].kptr.module)module_put (rec->fields[i].kptr.module);if (btf_is_kernel (rec->fields[i].kptr.btf))
@@ -597,6 +598,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec)case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :if (btf_is_kernel (fields[i].kptr.btf))btf_get (fields[i].kptr.btf);if (fields[i].kptr.module && !try_module_get (fields[i].kptr.module)) {
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 2/12 bpf: Handle BPF_UPTR in verifier
In reply to Martin KaFai LauFrom: Kui-Feng Lee <[email protected]>
This patch adds BPF_UPTR support to the verifier. Not that only the map_value will support the "__uptr" type tag.
This patch enforces only BPF_LDX is allowed to the value of an uptr. After BPF_LDX, it will mark the dst_reg as PTR_TO_MEM | PTR_MAYBE_NULL with size deduced from the field.kptr.btf_id. This will make the dst_reg pointed memory to be readable and writable as scalar.
There is a redundant "val_reg = reg_state(env, value_regno);" statement in the check_map_kptr_access(). This patch takes this chance to remove it also.
---
Changes in v5:
- The "if (kptr_field->type == BPF_UPTR)" addition in v4 in map_kptr_match_type() is removed. It will not be reached. meta->kptr_field is for kptr only.
- Use btf_field_type_name() for the verbose() log in check_map_access().
- Directly use t->size in mark_uptr_ld_reg. "t" must be a struct.
kernel/bpf/verifier.c | 39 +++++++++++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +33 -6
@@ -5485,6 +5485,22 @@ static u32 btf_ld_kptr_type(struct bpf_verifier_env *env, struct btf_field *kptrreturn ret; } +static int mark_uptr_ld_reg (struct bpf_verifier_env *env,u32 regno, +struct btf_field *field) +{ +struct bpf_reg_state *reg; +const struct btf_type *t; + + t =btf_type_by_id (field->kptr.btf, field->kptr.btf_id); +mark_reg_known_zero (env,cur_regs (env), regno); + reg =reg_state (env, regno); + reg->type =PTR_TO_MEM |PTR_MAYBE_NULL ; + reg->mem_size = t->size; + reg->id = ++env->id_gen; + +return 0 ; +} +static int check_map_kptr_access (struct bpf_verifier_env *env,u32 regno,int value_regno,int insn_idx,struct btf_field *kptr_field)
@@ -5513,9 +5529,15 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,verbose (env,"store to referenced kptr disallowed\n" );return -EACCES ; } +if (class !=BPF_LDX && kptr_field->type ==BPF_UPTR ) { +verbose (env,"store to uptr disallowed\n" ); +return -EACCES ; + }if (class ==BPF_LDX ) { - val_reg =reg_state (env, value_regno); +if (kptr_field->type ==BPF_UPTR ) +return mark_uptr_ld_reg (env, value_regno, kptr_field); +/* We can simply mark the value_regno receiving the pointer * value from map as PTR_TO_BTF_ID, with the correct type. */
@@ -5573,21 +5595,26 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :if (src !=ACCESS_DIRECT ) { -verbose (env,"kptr cannot be accessed indirectly by helper\n" ); +verbose (env,"%s cannot be accessed indirectly by helper\n" , +btf_field_type_name (field->type));return -EACCES ; }if (!tnum_is_const (reg->var_off)) { -verbose (env,"kptr access cannot have variable offset\n" ); +verbose (env,"%s access cannot have variable offset\n" , +btf_field_type_name (field->type));return -EACCES ; }if (p != off + reg->var_off.value) { -verbose (env,"kptr access misaligned expected=%u off=%llu\n" , +verbose (env,"%s access misaligned expected=%u off=%llu\n" , +btf_field_type_name (field->type), p, off + reg->var_off.value);return -EACCES ; }if (size !=bpf_size_to_bytes (BPF_DW )) { -verbose (env,"kptr access size must be BPF_DW\n" ); +verbose (env,"%s access size must be BPF_DW\n" , +btf_field_type_name (field->type));return -EACCES ; }break ;
@@ -6953,7 +6980,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnreturn err;if (tnum_is_const (reg->var_off)) kptr_field =btf_record_find (reg->map_ptr->record, - off + reg->var_off.value,BPF_KPTR ); + off + reg->var_off.value,BPF_KPTR |BPF_UPTR );if (kptr_field) { err =check_map_kptr_access (env, regno, value_regno, insn_idx, kptr_field); }else if (t ==BPF_READ && value_regno >=0 ) {
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 3/12 bpf: Add "bool swap_uptrs" arg to bpf_local_storage_update() and bpf_selem_alloc()
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
In a later patch, the task local storage will only accept uptr from the syscall update_elem and will not accept uptr from the bpf prog. The reason is the bpf prog does not have a way to provide a valid user space address.
bpf_local_storage_update() and bpf_selem_alloc() are used by both bpf prog bpf_task_storage_get(BPF_LOCAL_STORAGE_GET_F_CREATE) and bpf syscall update_elem. "bool swap_uptrs" arg is added to bpf_local_storage_update() and bpf_selem_alloc() to tell if it is called by the bpf prog or by the bpf syscall. When swap_uptrs==true, it is called by the syscall.
The arg is named (swap_)uptrs because the later patch will swap the uptrs between the newly allocated selem and the user space provided map_value. It will make error handling easier in case map->ops->map_update_elem() fails and the caller can decide if it needs to unpin the uptr in the user space provided map_value or the bpf_local_storage_update() has already taken the uptr ownership and will take care of unpinning it also.
Only swap_uptrs==false is passed now. The logic to handle the true case will be added in a later patch.
--- include/linux/bpf_local_storage.h | 4 ++-- kernel/bpf/bpf_cgrp_storage.c | 4 ++-- kernel/bpf/bpf_inode_storage.c | 4 ++-- kernel/bpf/bpf_local_storage.c | 8 ++++---- kernel/bpf/bpf_task_storage.c | 4 ++-- net/core/bpf_sk_storage.c | 6 +++--- 6 files changed, 15 insertions(+), 15 deletions(-)
include/linux/bpf_local_storage.h +2 -2
@@ -181,7 +181,7 @@ void bpf_selem_link_map(struct bpf_local_storage_map *smap,struct bpf_local_storage_elem *bpf_selem_alloc (struct bpf_local_storage_map *smap,void *owner,void *value, -bool charge_mem,gfp_t gfp_flags); +bool charge_mem,bool swap_uptrs,gfp_t gfp_flags); void bpf_selem_free(struct bpf_local_storage_elem *selem, struct bpf_local_storage_map *smap,
@@ -195,7 +195,7 @@ bpf_local_storage_alloc(void *owner,struct bpf_local_storage_data *bpf_local_storage_update (void *owner,struct bpf_local_storage_map *smap, -void *value,u64 map_flags,gfp_t gfp_flags); +void *value,u64 map_flags,bool swap_uptrs,gfp_t gfp_flags);u64 bpf_local_storage_map_mem_usage (const struct bpf_map *map);
kernel/bpf/bpf_cgrp_storage.c +2 -2
@@ -107,7 +107,7 @@ static long bpf_cgrp_storage_update_elem(struct bpf_map *map, void *key,bpf_cgrp_storage_lock (); sdata =bpf_local_storage_update (cgroup, (struct bpf_local_storage_map *)map, - value, map_flags,GFP_ATOMIC ); + value, map_flags, false,GFP_ATOMIC );bpf_cgrp_storage_unlock ();cgroup_put (cgroup);return PTR_ERR_OR_ZERO (sdata);
@@ -181,7 +181,7 @@ BPF_CALL_5(bpf_cgrp_storage_get, struct bpf_map *, map, struct cgroup *, cgroup,if (!percpu_ref_is_dying (&cgroup->self.refcnt) && (flags &BPF_LOCAL_STORAGE_GET_F_CREATE )) sdata =bpf_local_storage_update (cgroup, (struct bpf_local_storage_map *)map, - value,BPF_NOEXIST , gfp_flags); + value,BPF_NOEXIST , false, gfp_flags); unlock:bpf_cgrp_storage_unlock ();
kernel/bpf/bpf_inode_storage.c +2 -2
@@ -100,7 +100,7 @@ static long bpf_fd_inode_storage_update_elem(struct bpf_map *map, void *key, sdata =bpf_local_storage_update (file_inode (fd_file (f)), (struct bpf_local_storage_map *)map, - value, map_flags,GFP_ATOMIC ); + value, map_flags, false,GFP_ATOMIC );return PTR_ERR_OR_ZERO (sdata); }
@@ -154,7 +154,7 @@ BPF_CALL_5(bpf_inode_storage_get, struct bpf_map *, map, struct inode *, inode,if (flags &BPF_LOCAL_STORAGE_GET_F_CREATE ) { sdata =bpf_local_storage_update ( inode, (struct bpf_local_storage_map *)map, value, -BPF_NOEXIST , gfp_flags); +BPF_NOEXIST , false, gfp_flags);return IS_ERR (sdata) ? (unsigned long )NULL : (unsigned long )sdata->data; }
kernel/bpf/bpf_local_storage.c +4 -4
@@ -73,7 +73,7 @@ static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)struct bpf_local_storage_elem *bpf_selem_alloc (struct bpf_local_storage_map *smap,void *owner, -void *value,bool charge_mem,gfp_t gfp_flags) +void *value,bool charge_mem,bool swap_uptrs,gfp_t gfp_flags) {struct bpf_local_storage_elem *selem;
@@ -524,7 +524,7 @@ int bpf_local_storage_alloc(void *owner,*/ struct bpf_local_storage_data *bpf_local_storage_update (void *owner,struct bpf_local_storage_map *smap, -void *value,u64 map_flags,gfp_t gfp_flags) +void *value,u64 map_flags,bool swap_uptrs,gfp_t gfp_flags) {struct bpf_local_storage_data *old_sdata =NULL ;struct bpf_local_storage_elem *alloc_selem, *selem =NULL ;
@@ -550,7 +550,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,if (err)return ERR_PTR (err); - selem =bpf_selem_alloc (smap, owner, value, true, gfp_flags); + selem =bpf_selem_alloc (smap, owner, value, true, swap_uptrs, gfp_flags);if (!selem)return ERR_PTR (-ENOMEM );
@@ -584,7 +584,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,/* A lookup has just been done before and concluded a new selem is * needed. The chance of an unnecessary alloc is unlikely. */ - alloc_selem = selem =bpf_selem_alloc (smap, owner, value, true, gfp_flags); + alloc_selem = selem =bpf_selem_alloc (smap, owner, value, true, swap_uptrs, gfp_flags);if (!alloc_selem)return ERR_PTR (-ENOMEM );
kernel/bpf/bpf_task_storage.c +2 -2
@@ -147,7 +147,7 @@ static long bpf_pid_task_storage_update_elem(struct bpf_map *map, void *key,bpf_task_storage_lock (); sdata =bpf_local_storage_update ( task, (struct bpf_local_storage_map *)map, value, map_flags, -GFP_ATOMIC ); + false,GFP_ATOMIC );bpf_task_storage_unlock (); err =PTR_ERR_OR_ZERO (sdata);
@@ -219,7 +219,7 @@ static void *__bpf_task_storage_get(struct bpf_map *map, (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) && nobusy) { sdata =bpf_local_storage_update ( task, (struct bpf_local_storage_map *)map, value, -BPF_NOEXIST , gfp_flags); +BPF_NOEXIST , false, gfp_flags);return IS_ERR (sdata) ?NULL : sdata->data; }
net/core/bpf_sk_storage.c +3 -3
@@ -106,7 +106,7 @@ static long bpf_fd_sk_storage_update_elem(struct bpf_map *map, void *key,if (sock) { sdata =bpf_local_storage_update ( sock->sk, (struct bpf_local_storage_map *)map, value, - map_flags,GFP_ATOMIC ); + map_flags, false,GFP_ATOMIC );sockfd_put (sock);return PTR_ERR_OR_ZERO (sdata); }
@@ -137,7 +137,7 @@ bpf_sk_storage_clone_elem(struct sock *newsk, {struct bpf_local_storage_elem *copy_selem; - copy_selem =bpf_selem_alloc (smap, newsk,NULL , true,GFP_ATOMIC ); + copy_selem =bpf_selem_alloc (smap, newsk,NULL , true, false,GFP_ATOMIC );if (!copy_selem)return NULL ;
@@ -243,7 +243,7 @@ BPF_CALL_5(bpf_sk_storage_get, struct bpf_map *, map, struct sock *, sk, refcount_inc_not_zero(&sk->sk_refcnt)) { sdata =bpf_local_storage_update ( sk, (struct bpf_local_storage_map *)map, value, -BPF_NOEXIST , gfp_flags); +BPF_NOEXIST , false, gfp_flags);/* sk must be a fullsock (guaranteed by verifier), * so sock_gen_put() is unnecessary. */
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 4/12 bpf: Postpone bpf_selem_free() in bpf_selem_unlink_storage_nolock()
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
In a later patch, bpf_selem_free() will call unpin_user_page() through bpf_obj_free_fields(). unpin_user_page() may take spin_lock. However, some bpf_selem_free() call paths have held a raw_spin_lock. Like this:
raw_spin_lock_irqsave()
bpf_selem_unlink_storage_nolock()
bpf_selem_free()
unpin_user_page()
spin_lock()To avoid spinlock nested in raw_spinlock, bpf_selem_free() should be done after releasing the raw_spinlock. The "bool reuse_now" arg is replaced with "struct hlist_head *free_selem_list" in bpf_selem_unlink_storage_nolock(). The bpf_selem_unlink_storage_nolock() will append the to-be-free selem at the free_selem_list. The caller of bpf_selem_unlink_storage_nolock() will need to call the new bpf_selem_free_list(free_selem_list, reuse_now) to free the selem after releasing the raw_spinlock.
Note that the selem->snode cannot be reused for linking to the free_selem_list because the selem->snode is protected by the raw_spinlock that we want to avoid holding. A new
"struct hlist_node free_node;" is union-ized with
the rcu_head. Only the first one successfully hlist_del_init_rcu(&selem->snode) will be able
to use the free_node. After succeeding hlist_del_init_rcu(&selem->snode), the free_node and rcu_head usage is serialized such that they can share the 16 bytes in a union.
--- include/linux/bpf_local_storage.h | 8 ++++++- kernel/bpf/bpf_local_storage.c | 35 ++++++++++++++++++++++++++----- 2 files changed, 37 insertions(+), 6 deletions(-)
include/linux/bpf_local_storage.h +7 -1
@@ -77,7 +77,13 @@ struct bpf_local_storage_elem {struct hlist_node map_node;/* Linked to bpf_local_storage_map */ struct hlist_node snode;/* Linked to bpf_local_storage */ struct bpf_local_storage __rcu *local_storage; -struct rcu_head rcu; +union { +struct rcu_head rcu; +struct hlist_node free_node;/* used to postpone +* bpf_selem_free +* after raw_spin_unlock +*/ + };/* 8 bytes hole */ /* The data is stored in another cacheline to minimize * the number of cachelines access during a cache hit.
kernel/bpf/bpf_local_storage.c +30 -5
@@ -246,13 +246,30 @@ void bpf_selem_free(struct bpf_local_storage_elem *selem, } } +static void bpf_selem_free_list (struct hlist_head *list,bool reuse_now) +{ +struct bpf_local_storage_elem *selem; +struct bpf_local_storage_map *smap; +struct hlist_node *n; + +/* The "_safe" iteration is needed. +* The loop is not removing the selem from the list +* but bpf_selem_free will use the selem->rcu_head +* which is union-ized with the selem->free_node. +*/ +hlist_for_each_entry_safe (selem, n, list, free_node) { + smap =rcu_dereference_check (SDATA (selem)->smap,bpf_rcu_lock_held ()); +bpf_selem_free (selem, smap, reuse_now); + } +} +/* local_storage->lock must be held and selem->local_storage == local_storage. * The caller must ensure selem->smap is still valid to be * dereferenced for its smap->elem_size and smap->cache_idx. */ static bool bpf_selem_unlink_storage_nolock (struct bpf_local_storage *local_storage,struct bpf_local_storage_elem *selem, -bool uncharge_mem,bool reuse_now) +bool uncharge_mem,struct hlist_head *free_selem_list) {struct bpf_local_storage_map *smap;bool free_local_storage;
@@ -296,7 +313,7 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor SDATA(selem))RCU_INIT_POINTER (local_storage->cache[smap->cache_idx],NULL ); -bpf_selem_free (selem, smap, reuse_now); +hlist_add_head (&selem->free_node, free_selem_list);if (rcu_access_pointer (local_storage->smap) == smap)RCU_INIT_POINTER (local_storage->smap,NULL );
@@ -345,6 +362,7 @@ static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,struct bpf_local_storage_map *storage_smap;struct bpf_local_storage *local_storage;bool bpf_ma, free_local_storage = false; +HLIST_HEAD (selem_free_list);unsigned long flags;if (unlikely (!selem_linked_to_storage_lockless (selem)))
@@ -360,9 +378,11 @@ static void bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem,raw_spin_lock_irqsave (&local_storage->lock, flags);if (likely (selem_linked_to_storage (selem))) free_local_storage =bpf_selem_unlink_storage_nolock ( - local_storage, selem, true, reuse_now); + local_storage, selem, true, &selem_free_list);raw_spin_unlock_irqrestore (&local_storage->lock, flags); +bpf_selem_free_list (&selem_free_list, reuse_now); +if (free_local_storage)bpf_local_storage_free (local_storage, storage_smap, bpf_ma, reuse_now); }
@@ -529,6 +549,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,struct bpf_local_storage_data *old_sdata =NULL ;struct bpf_local_storage_elem *alloc_selem, *selem =NULL ;struct bpf_local_storage *local_storage; +HLIST_HEAD (old_selem_free_list);unsigned long flags;int err;
@@ -624,11 +645,12 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,if (old_sdata) {bpf_selem_unlink_map (SELEM (old_sdata));bpf_selem_unlink_storage_nolock (local_storage,SELEM (old_sdata), - true, false); + true, &old_selem_free_list); } unlock:raw_spin_unlock_irqrestore (&local_storage->lock, flags); +bpf_selem_free_list (&old_selem_free_list, false);if (alloc_selem) {mem_uncharge (smap, owner, smap->elem_size);bpf_selem_free (alloc_selem, smap, true);
@@ -706,6 +728,7 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)struct bpf_local_storage_map *storage_smap;struct bpf_local_storage_elem *selem;bool bpf_ma, free_storage = false; +HLIST_HEAD (free_selem_list);struct hlist_node *n;unsigned long flags;
@@ -734,10 +757,12 @@ void bpf_local_storage_destroy(struct bpf_local_storage *local_storage)* of the loop will set the free_cgroup_storage to true. */ free_storage =bpf_selem_unlink_storage_nolock ( - local_storage, selem, true, true); + local_storage, selem, true, &free_selem_list); }raw_spin_unlock_irqrestore (&local_storage->lock, flags); +bpf_selem_free_list (&free_selem_list, true); +if (free_storage)bpf_local_storage_free (local_storage, storage_smap, bpf_ma, true); }
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 5/12 bpf: Postpone bpf_obj_free_fields to the rcu callback
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
A later patch will enable the uptr usage in the task_local_storage map. This will require the unpin_user_page() to be done after the rcu task trace gp for the cases that the uptr may still be used by a bpf prog. The bpf_obj_free_fields() will be the one doing unpin_user_page(), so this patch is to postpone calling bpf_obj_free_fields() to the rcu callback.
The bpf_obj_free_fields() is only required to be done in the rcu callback when bpf->bpf_ma==true and reuse_now==false.
bpf->bpf_ma==true case is because uptr will only be enabled in task storage which has already been moved to bpf_mem_alloc. The bpf->bpf_ma==false case can be supported in the future also if there is a need.
reuse_now==false when the selem (aka storage) is deleted by bpf prog (bpf_task_storage_delete) or by syscall delete_elem(). In both cases, bpf_obj_free_fields() needs to wait for rcu gp.
A few words on reuse_now==true. reuse_now==true when the storage's owner (i.e. the task_struct) is destructing or the map itself is doing map_free(). In both cases, no bpf prog should have a hold on the selem and its uptrs, so there is no need to postpone bpf_obj_free_fields(). reuse_now==true should be the common case for local storage usage where the storage exists throughout the lifetime of its owner (task_struct).
The bpf_obj_free_fields() needs to use the map->record. Doing bpf_obj_free_fields() in a rcu callback will require the bpf_local_storage_map_free() to wait for rcu_barrier. An optimization could be only waiting for rcu_barrier when the map has uptr in its map_value. This will require either yet another rcu callback function or adding a bool in the selem to flag if the SDATA(selem)->smap is still valid. This patch chooses to keep it simple and wait for rcu_barrier for maps that use bpf_mem_alloc.
--- kernel/bpf/bpf_local_storage.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-)
kernel/bpf/bpf_local_storage.c +24 -5
@@ -209,8 +209,12 @@ static void __bpf_selem_free(struct bpf_local_storage_elem *selem,static void bpf_selem_free_rcu (struct rcu_head *rcu) {struct bpf_local_storage_elem *selem; +struct bpf_local_storage_map *smap; selem = container_of(rcu, struct bpf_local_storage_elem, rcu); +/* The bpf_local_storage_map_free will wait for rcu_barrier */ + smap =rcu_dereference_check (SDATA (selem)->smap,1 ); +bpf_obj_free_fields (smap->map.record,SDATA (selem)->data);bpf_mem_cache_raw_free (selem); }
@@ -226,16 +230,25 @@ void bpf_selem_free(struct bpf_local_storage_elem *selem, struct bpf_local_storage_map *smap, bool reuse_now) { - bpf_obj_free_fields(smap->map.record, SDATA(selem)->data); -if (!smap->bpf_ma) { +/* Only task storage has uptrs and task storage +* has moved to bpf_mem_alloc. Meaning smap->bpf_ma == true +* for task storage, so this bpf_obj_free_fields() won't unpin +* any uptr. +*/ +bpf_obj_free_fields (smap->map.record,SDATA (selem)->data);__bpf_selem_free (selem, reuse_now);return ; } -if (!reuse_now) { -call_rcu_tasks_trace (&selem->rcu, bpf_selem_free_trace_rcu); - }else { +if (reuse_now) { +/* reuse_now == true only happens when the storage owner +* (e.g. task_struct) is being destructed or the map itself +* is being destructed (ie map_free). In both cases, +* no bpf prog can have a hold on the selem. It is +* safe to unpin the uptrs and free the selem now. +*/ +bpf_obj_free_fields (smap->map.record,SDATA (selem)->data);/* Instead of using the vanilla call_rcu(), * bpf_mem_cache_free will be able to reuse selem * immediately.
@@ -243,7 +256,10 @@ void bpf_selem_free(struct bpf_local_storage_elem *selem,migrate_disable ();bpf_mem_cache_free (&smap->selem_ma, selem);migrate_enable (); +return ; } + +call_rcu_tasks_trace (&selem->rcu, bpf_selem_free_trace_rcu); }static void bpf_selem_free_list (struct hlist_head *list,bool reuse_now)
@@ -908,6 +924,9 @@ void bpf_local_storage_map_free(struct bpf_map *map,synchronize_rcu ();if (smap->bpf_ma) { +rcu_barrier_tasks_trace (); +if (!rcu_trace_implies_rcu_gp ()) +rcu_barrier ();bpf_mem_alloc_destroy (&smap->selem_ma);bpf_mem_alloc_destroy (&smap->storage_ma); }
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 6/12 bpf: Add uptr support in the map_value of the task local storage.
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch adds uptr support in the map_value of the task local storage.
struct map_value {
struct user_data __uptr *uptr;
};struct {
__uint(type, BPF_MAP_TYPE_TASK_STORAGE);
__uint(map_flags, BPF_F_NO_PREALLOC);
__type(key, int);
__type(value, struct value_type);
} datamap SEC(".maps");A new bpf_obj_pin_uptrs() is added to pin the user page and also stores the kernel address back to the uptr for the bpf prog to use later. It currently does not support the uptr pointing to a user struct across two pages. It also excludes PageHighMem support to keep it simple. As of now, the 32bit bpf jit is missing other more crucial bpf features. For example, many important bpf features depend on bpf kfunc now but so far only one arch (x86-32) supports it which was added by me as an example when kfunc was first introduced to bpf.
The uptr can only be stored to the task local storage by the syscall update_elem. Meaning the uptr will not be considered if it is provided by the bpf prog through bpf_task_storage_get(BPF_LOCAL_STORAGE_GET_F_CREATE). This is enforced by only calling bpf_local_storage_update(swap_uptrs==true) in bpf_pid_task_storage_update_elem. Everywhere else will have swap_uptrs==false.
This will pump down to bpf_selem_alloc(swap_uptrs==true). It is the only case that bpf_selem_alloc() will take the uptr value when updating the newly allocated selem. bpf_obj_swap_uptrs() is added to swap the uptr between the SDATA(selem)->data and the user provided map_value in "void *value". bpf_obj_swap_uptrs() makes the SDATA(selem)->data takes the ownership of the uptr and the user space provided map_value will have NULL in the uptr.
The bpf_obj_unpin_uptrs() is called after map->ops->map_update_elem() returning error. If the map->ops->map_update_elem has reached a state that the local storage has taken the uptr ownership, the bpf_obj_unpin_uptrs() will be a no op because the uptr is NULL. A "__"bpf_obj_unpin_uptrs is added to make this error path unpin easier such that it does not have to check the map->record is NULL or not.
BPF_F_LOCK is not supported when the map_value has uptr. This can be revisited later if there is a use case. A similar swap_uptrs idea can be considered.
The final bit is to do unpin_user_page in the bpf_obj_free_fields(). The earlier patch has ensured that the bpf_obj_free_fields() has gone through the rcu gp when needed.
--- include/linux/bpf.h | 20 +++++++ kernel/bpf/bpf_local_storage.c | 7 ++- kernel/bpf/bpf_task_storage.c | 5 +- kernel/bpf/syscall.c | 106 +++++++++++++++++++++++++++++++-- 4 files changed, 131 insertions(+), 7 deletions(-)
include/linux/bpf.h +20 -0
@@ -424,6 +424,7 @@ static inline void bpf_obj_init_field(const struct btf_field *field, void *addr)case BPF_KPTR_UNREF :case BPF_KPTR_REF :case BPF_KPTR_PERCPU : +case BPF_UPTR :break ;default :WARN_ON_ONCE (1 );
@@ -512,6 +513,25 @@ static inline void copy_map_value_long(struct bpf_map *map, void *dst, void *srcbpf_obj_memcpy (map->record, dst, src, map->value_size, true); } +static inline void bpf_obj_swap_uptrs (const struct btf_record *rec,void *dst,void *src) +{ +unsigned long *src_uptr, *dst_uptr; +const struct btf_field *field; +int i; + +if (!btf_record_has_field (rec,BPF_UPTR )) +return ; + +for (i =0 , field = rec->fields; i < rec->cnt; i++, field++) { +if (field->type !=BPF_UPTR ) +continue ; + + src_uptr = src + field->offset; + dst_uptr = dst + field->offset; +swap (*src_uptr, *dst_uptr); + } +} +static inline void bpf_obj_memzero (struct btf_record *rec,void *dst,u32 size) {u32 curr_off =0 ;
kernel/bpf/bpf_local_storage.c +5 -2
@@ -99,9 +99,12 @@ bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner, }if (selem) { -if (value) +if (value) { +/* No need to call check_and_init_map_value as memory is zero init */ copy_map_value (&smap->map,SDATA (selem)->data, value); -/* No need to call check_and_init_map_value as memory is zero init */ +if (swap_uptrs) +bpf_obj_swap_uptrs (smap->map.record,SDATA (selem)->data, value); + }return selem; }
kernel/bpf/bpf_task_storage.c +4 -1
@@ -129,6 +129,9 @@ static long bpf_pid_task_storage_update_elem(struct bpf_map *map, void *key,struct pid *pid;int fd, err; +if ((map_flags &BPF_F_LOCK ) &&btf_record_has_field (map->record,BPF_UPTR )) +return -EOPNOTSUPP ; + fd = *(int *)key; pid =pidfd_get_pid (fd, &f_flags);if (IS_ERR (pid))
@@ -147,7 +150,7 @@ static long bpf_pid_task_storage_update_elem(struct bpf_map *map, void *key,bpf_task_storage_lock (); sdata =bpf_local_storage_update ( task, (struct bpf_local_storage_map *)map, value, map_flags, - false,GFP_ATOMIC ); + true,GFP_ATOMIC );bpf_task_storage_unlock (); err =PTR_ERR_OR_ZERO (sdata);
kernel/bpf/syscall.c +102 -4
136 lines not on this page · raw message
Signature · 1 line
2.43.5
Re: [PATCH v6 bpf-next 06/12] bpf: Add uptr support in the map_value of the task local storage.
Quoted 55 lines · Martin KaFai Lau
From: Martin KaFai Lau <[email protected]>
This patch adds uptr support in the
map_valueof the task local storage.struct map_value { struct user_data __uptr *uptr; };struct { __uint(type, BPF_MAP_TYPE_TASK_STORAGE); __uint(map_flags, BPF_F_NO_PREALLOC); __type(key, int); __type(value, struct value_type); } datamap SEC(".maps");A new
bpf_obj_pin_uptrs()is added to pin the user page and also stores the kernel address back to the uptr for the bpf prog to use later. It currently does not support the uptr pointing to a user struct across two pages. It also excludes PageHighMem support to keep it simple. As of now, the 32bit bpf jit is missing other more crucial bpf features. For example, many important bpf features depend on bpf kfunc now but so far only one arch (x86-32) supports it which was added by me as an example when kfunc was first introduced to bpf.The uptr can only be stored to the task local storage by the syscall
update_elem. Meaning the uptr will not be considered if it is provided by the bpf prog throughbpf_task_storage_get(BPF_LOCAL_STORAGE_GET_F_CREATE). This is enforced by only callingbpf_local_storage_update(swap_uptrs==true)inbpf_pid_task_storage_update_elem. Everywhere else will have swap_uptrs==false.This will pump down to
bpf_selem_alloc(swap_uptrs==true). It is the only case thatbpf_selem_alloc()will take the uptr value when updating the newly allocated selem.bpf_obj_swap_uptrs()is added to swap the uptr between the SDATA(selem)->data and the user providedmap_valuein "void *value".bpf_obj_swap_uptrs()makes the SDATA(selem)->data takes the ownership of the uptr and the user space providedmap_valuewill have NULL in the uptr.The
bpf_obj_unpin_uptrs()is called aftermap->ops->map_update_elem()returning error. If themap->ops->map_update_elemhas reached a state that the local storage has taken the uptr ownership, thebpf_obj_unpin_uptrs()will be a no op because the uptr is NULL. A "__"bpf_obj_unpin_uptrs is added to make this error path unpin easier such that it does not have to check themap->recordis NULL or not.
BPF_F_LOCKis not supported when themap_valuehas uptr. This can be revisited later if there is a use case. A similarswap_uptrsidea can be considered.
Quoted 7 lines
The final bit is to do
unpin_user_pagein thebpf_obj_free_fields(). The earlier patch has ensured that thebpf_obj_free_fields()has gone through the rcu gp when needed.Cc Shakeel Butt <[email protected]>Signed-off-by Martin KaFai Lau <[email protected]>
Martin KaFai Lau
reviewerPATCH v6 7/12 libbpf: define __uptr.
In reply to Martin KaFai LauFrom: Kui-Feng Lee <[email protected]>
Make __uptr available to BPF programs to enable them to define uptrs.
--- tools/lib/bpf/bpf_helpers.h | 1 + 1 file changed, 1 insertion(+)
tools/lib/bpf/bpf_helpers.h +1 -0
@@ -185,6 +185,7 @@ enum libbpf_tristate {#define __kptr_untrusted __attribute__((btf_type_tag("kptr_untrusted")))#define __kptr __attribute__((btf_type_tag("kptr")))#define __percpu_kptr __attribute__((btf_type_tag("percpu_kptr"))) +#define __uptr __attribute__((btf_type_tag("uptr")))#if defined (__clang__)#define bpf_ksym_exists (sym) ({ \
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 8/12 selftests/bpf: Some basic __uptr tests
In reply to Martin KaFai LauFrom: Kui-Feng Lee <[email protected]>
Make sure the memory of uptrs have been mapped to the kernel properly. Also ensure the values of uptrs in the kernel haven't been copied to userspace.
It also has the syscall update_elem/delete_elem test to test the pin/unpin code paths.
--- .../bpf/prog_tests/task_local_storage.c | 142 ++++++++++++++++++ .../selftests/bpf/progs/task_ls_uptr.c | 63 ++++++++ .../testing/selftests/bpf/uptr_test_common.h | 35 +++++ 3 files changed, 240 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/task_ls_uptr.c create mode 100644 tools/testing/selftests/bpf/uptr_test_common.h
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +142 -0
@@ -7,12 +7,15 @@#include <pthread.h> #include <sys/syscall.h> /* For SYS_xxx definitions */ #include <sys/types.h> +#include <sys/eventfd.h> #include <test_progs.h> #include "task_local_storage_helpers.h" #include "task_local_storage.skel.h" #include "task_local_storage_exit_creds.skel.h" #include "task_ls_recursion.skel.h" #include "task_storage_nodeadlock.skel.h" +#include "uptr_test_common.h" +#include "task_ls_uptr.skel.h" static void test_sys_enter_exit (void ) {
@@ -227,6 +230,143 @@ static void test_nodeadlock(void)sched_setaffinity (getpid (),sizeof (old), &old); } +static struct user_data udata __attribute__((aligned(16))) = { + .a = 1, + .b = 2, +}; + +static struct user_data udata2 __attribute__((aligned(16))) = { + .a = 3, + .b = 4, +}; + +static void check_udata2 (int expected) +{ + udata2.result = udata2.nested_result =0 ; +usleep (1 ); +ASSERT_EQ (udata2.result, expected,"udata2.result" ); +ASSERT_EQ (udata2.nested_result, expected,"udata2.nested_result" ); +} + +static void test_uptr_basic (void ) +{ +int map_fd, parent_task_fd, ev_fd; +struct value_type value = {}; +struct task_ls_uptr *skel; +pid_t child_pid, my_tid; +__u64 ev_dummy_data =1 ; +int err; + + my_tid =syscall (SYS_gettid); + parent_task_fd =sys_pidfd_open (my_tid,0 ); +if (!ASSERT_OK_FD (parent_task_fd,"parent_task_fd" )) +return ; + + ev_fd =eventfd (0 ,0 ); +if (!ASSERT_OK_FD (ev_fd,"ev_fd" )) { +close (parent_task_fd); +return ; + } + + skel =task_ls_uptr__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open_and_load" )) + goto out; + + map_fd =bpf_map__fd (skel->maps.datamap); + value.udata = &udata; + value.nested.udata = &udata; + err =bpf_map_update_elem (map_fd, &parent_task_fd, &value,BPF_NOEXIST ); +if (!ASSERT_OK (err,"update_elem(udata)" )) + goto out; + + err =task_ls_uptr__attach (skel); +if (!ASSERT_OK (err,"skel_attach" )) + goto out; + + child_pid =fork (); +if (!ASSERT_NEQ (child_pid,-1 ,"fork" )) + goto out; + +/* Call syscall in the child process, but access the map value of +* the parent process in the BPF program to check if the user kptr +* is translated/mapped correctly. +*/ +if (child_pid ==0 ) { +/* child */ + +/* Overwrite the user_data in the child process to check if +* the BPF program accesses the user_data of the parent. +*/ + udata.a =0 ; + udata.b =0 ; + +/* Wait for the parent to set child_pid */ +read (ev_fd, &ev_dummy_data,sizeof (ev_dummy_data)); +exit (0 ); + } + + skel->bss->parent_pid = my_tid; + skel->bss->target_pid = child_pid; + +write (ev_fd, &ev_dummy_data,sizeof (ev_dummy_data)); + + err =waitpid (child_pid,NULL ,0 ); +ASSERT_EQ (err, child_pid,"waitpid" ); +ASSERT_EQ (udata.result,MAGIC_VALUE + udata.a + udata.b,"udata.result" ); +ASSERT_EQ (udata.nested_result,MAGIC_VALUE + udata.a + udata.b,"udata.nested_result" ); + + skel->bss->target_pid = my_tid; + +/* update_elem: uptr changes from udata1 to udata2 */ + value.udata = &udata2; + value.nested.udata = &udata2; + err =bpf_map_update_elem (map_fd, &parent_task_fd, &value,BPF_EXIST ); +if (!ASSERT_OK (err,"update_elem(udata2)" )) + goto out; +check_udata2 (MAGIC_VALUE + udata2.a + udata2.b); + +/* update_elem: uptr changes from udata2 uptr to NULL */ +memset (&value,0 ,sizeof (value)); + err =bpf_map_update_elem (map_fd, &parent_task_fd, &value,BPF_EXIST ); +if (!ASSERT_OK (err,"update_elem(udata2)" )) + goto out; +check_udata2 (0 ); + +/* update_elem: uptr changes from NULL to udata2 */ + value.udata = &udata2; + value.nested.udata = &udata2; + err =bpf_map_update_elem (map_fd, &parent_task_fd, &value,BPF_EXIST ); +if (!ASSERT_OK (err,"update_elem(udata2)" )) + goto out; +check_udata2 (MAGIC_VALUE + udata2.a + udata2.b); + +/* Check if user programs can access the value of user kptrs +* through bpf_map_lookup_elem(). Make sure the kernel value is not +* leaked. +*/ + err =bpf_map_lookup_elem (map_fd, &parent_task_fd, &value); +if (!ASSERT_OK (err,"bpf_map_lookup_elem" )) + goto out; +ASSERT_EQ (value.udata,NULL ,"value.udata" ); +ASSERT_EQ (value.nested.udata,NULL ,"value.nested.udata" ); + +/* delete_elem */ + err =bpf_map_delete_elem (map_fd, &parent_task_fd); +ASSERT_OK (err,"delete_elem(udata2)" ); +check_udata2 (0 ); + +/* update_elem: add uptr back to test map_free */ + value.udata = &udata2; + value.nested.udata = &udata2; + err =bpf_map_update_elem (map_fd, &parent_task_fd, &value,BPF_NOEXIST ); +ASSERT_OK (err,"update_elem(udata2)" ); + +out: +task_ls_uptr__destroy (skel); +close (ev_fd); +close (parent_task_fd); +} +void test_task_local_storage (void ) {if (test__start_subtest ("sys_enter_exit" ))
@@ -237,4 +377,6 @@ void test_task_local_storage(void)test_recursion ();if (test__start_subtest ("nodeadlock" ))test_nodeadlock (); +if (test__start_subtest ("uptr_basic" )) +test_uptr_basic (); }
tools/testing/selftests/bpf/progs/task_ls_uptr.c new file +63 -0
63 lines not on this page · raw message
tools/testing/selftests/bpf/uptr_test_common.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#ifndef _UPTR_TEST_COMMON_H +#define _UPTR_TEST_COMMON_H + +#define MAGIC_VALUE 0xabcd1234 + +#ifdef __BPF__ +/* Avoid fwd btf type being generated for the following struct */ +struct user_data *dummy_data; +struct cgroup *dummy_cgrp; +#else +#define __uptr +#define __kptr +#endif + +struct user_data { +int a; +int b; +int result; +int nested_result; +}; + +struct nested_udata { + struct user_data __uptr *udata; +}; + +struct value_type { + struct user_data __uptr *udata; + struct cgroup __kptr *cgrp; +struct nested_udata nested; +}; + +#endif
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 9/12 selftests/bpf: Test a uptr struct spanning across pages.
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch tests the case when uptr has a struct spanning across two pages. It is not supported now and EOPNOTSUPP is expected from the syscall update_elem.
It also tests the whole uptr struct located exactly at the end of a page and ensures that this case is accepted by update_elem.
--- .../bpf/prog_tests/task_local_storage.c | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+)
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +43 -0
@@ -8,6 +8,7 @@#include <sys/syscall.h> /* For SYS_xxx definitions */ #include <sys/types.h> #include <sys/eventfd.h> +#include <sys/mman.h> #include <test_progs.h> #include "task_local_storage_helpers.h" #include "task_local_storage.skel.h"
@@ -367,6 +368,46 @@ static void test_uptr_basic(void)close (parent_task_fd); } +static void test_uptr_across_pages (void ) +{ +int page_size =getpagesize (); +struct value_type value = {}; +struct task_ls_uptr *skel; +int err, task_fd, map_fd; +void *mem; + + task_fd =sys_pidfd_open (getpid (),0 ); +if (!ASSERT_OK_FD (task_fd,"task_fd" )) +return ; + + mem =mmap (NULL , page_size *2 ,PROT_READ |PROT_WRITE , +MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +if (!ASSERT_OK_PTR (mem,"mmap(page_size * 2)" )) { +close (task_fd); +return ; + } + + skel =task_ls_uptr__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open_and_load" )) + goto out; + + map_fd =bpf_map__fd (skel->maps.datamap); + value.udata = mem + page_size - offsetof(struct user_data , b); + err =bpf_map_update_elem (map_fd, &task_fd, &value,0 ); +if (!ASSERT_ERR (err,"update_elem(udata)" )) + goto out; +ASSERT_EQ (errno,EOPNOTSUPP ,"errno" ); + + value.udata = mem + page_size -sizeof (struct user_data ); + err =bpf_map_update_elem (map_fd, &task_fd, &value,0 ); +ASSERT_OK (err,"update_elem(udata)" ); + +out: +task_ls_uptr__destroy (skel); +close (task_fd); +munmap (mem, page_size *2 ); +} +void test_task_local_storage (void ) {if (test__start_subtest ("sys_enter_exit" ))
@@ -379,4 +420,6 @@ void test_task_local_storage(void)test_nodeadlock ();if (test__start_subtest ("uptr_basic" ))test_uptr_basic (); +if (test__start_subtest ("uptr_across_pages" )) +test_uptr_across_pages (); }
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 10/12 selftests/bpf: Add update_elem failure test for task storage uptr
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch test the following failures in syscall update_elem
1. The first update_elem(BPF_F_LOCK) should be EOPNOTSUPP. syscall.c takes care of unpinning the uptr.
2. The second update_elem(BPF_EXIST) fails. syscall.c takes care of unpinning the uptr.
3. The forth update_elem(BPF_NOEXIST) fails. bpf_local_storage_update takes care of unpinning the uptr.
--- .../bpf/prog_tests/task_local_storage.c | 45 +++++++++++++++++++ .../selftests/bpf/progs/uptr_update_failure.c | 42 +++++++++++++++++ .../testing/selftests/bpf/uptr_test_common.h | 5 +++ 3 files changed, 92 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/uptr_update_failure.c
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +45 -0
@@ -17,6 +17,7 @@#include "task_storage_nodeadlock.skel.h" #include "uptr_test_common.h" #include "task_ls_uptr.skel.h" +#include "uptr_update_failure.skel.h" static void test_sys_enter_exit (void ) {
@@ -408,6 +409,48 @@ static void test_uptr_across_pages(void)munmap (mem, page_size *2 ); } +static void test_uptr_update_failure (void ) +{ +struct value_lock_type value = {}; +struct uptr_update_failure *skel; +int err, task_fd, map_fd; + + task_fd =sys_pidfd_open (getpid (),0 ); +if (!ASSERT_OK_FD (task_fd,"task_fd" )) +return ; + + skel =uptr_update_failure__open_and_load (); +if (!ASSERT_OK_PTR (skel,"skel_open_and_load" )) + goto out; + + map_fd =bpf_map__fd (skel->maps.datamap); + + value.udata = &udata; + err =bpf_map_update_elem (map_fd, &task_fd, &value,BPF_F_LOCK ); +if (!ASSERT_ERR (err,"update_elem(udata, BPF_F_LOCK)" )) + goto out; +ASSERT_EQ (errno,EOPNOTSUPP ,"errno" ); + + err =bpf_map_update_elem (map_fd, &task_fd, &value,BPF_EXIST ); +if (!ASSERT_ERR (err,"update_elem(udata, BPF_EXIST)" )) + goto out; +ASSERT_EQ (errno,ENOENT ,"errno" ); + + err =bpf_map_update_elem (map_fd, &task_fd, &value,BPF_NOEXIST ); +if (!ASSERT_OK (err,"update_elem(udata, BPF_NOEXIST)" )) + goto out; + + value.udata = &udata2; + err =bpf_map_update_elem (map_fd, &task_fd, &value,BPF_NOEXIST ); +if (!ASSERT_ERR (err,"update_elem(udata2, BPF_NOEXIST)" )) + goto out; +ASSERT_EQ (errno,EEXIST ,"errno" ); + +out: +uptr_update_failure__destroy (skel); +close (task_fd); +} +void test_task_local_storage (void ) {if (test__start_subtest ("sys_enter_exit" ))
@@ -422,4 +465,6 @@ void test_task_local_storage(void)test_uptr_basic ();if (test__start_subtest ("uptr_across_pages" ))test_uptr_across_pages (); +if (test__start_subtest ("uptr_update_failure" )) +test_uptr_update_failure (); }
tools/testing/selftests/bpf/progs/uptr_update_failure.c new file +42 -0
42 lines not on this page · raw message
tools/testing/selftests/bpf/uptr_test_common.h +5 -0
@@ -32,4 +32,9 @@ struct value_type {struct nested_udata nested; }; +struct value_lock_type { + struct user_data __uptr *udata; +struct bpf_spin_lock lock; +}; +#endif
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 11/12 selftests/bpf: Add uptr failure verifier tests
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
Add verifier tests to ensure invalid uptr usages are rejected.
--- .../bpf/prog_tests/task_local_storage.c | 2 + .../selftests/bpf/progs/uptr_failure.c | 105 ++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/uptr_failure.c
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +2 -0
@@ -18,6 +18,7 @@#include "uptr_test_common.h" #include "task_ls_uptr.skel.h" #include "uptr_update_failure.skel.h" +#include "uptr_failure.skel.h" static void test_sys_enter_exit (void ) {
@@ -467,4 +468,5 @@ void test_task_local_storage(void)test_uptr_across_pages ();if (test__start_subtest ("uptr_update_failure" ))test_uptr_update_failure (); +RUN_TESTS (uptr_failure); }
tools/testing/selftests/bpf/progs/uptr_failure.c new file +105 -0
105 lines not on this page · raw message
Signature · 1 line
2.43.5
Martin KaFai Lau
reviewerPATCH v6 12/12 selftests/bpf: Create task_local_storage map with invalid uptr's struct
In reply to Martin KaFai LauFrom: Martin KaFai Lau <[email protected]>
This patch tests the map creation failure when the map_value has unsupported uptr. The three cases are the struct is larger than one page, the struct is empty, and the struct is a kernel struct.
--- .../bpf/prog_tests/task_local_storage.c | 46 +++++++++++++++++++ .../selftests/bpf/progs/uptr_map_failure.c | 27 +++++++++++ tools/testing/selftests/bpf/test_progs.h | 8 ++++ .../testing/selftests/bpf/uptr_test_common.h | 23 ++++++++++ 4 files changed, 104 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/uptr_map_failure.c
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +46 -0
@@ -10,6 +10,7 @@#include <sys/eventfd.h> #include <sys/mman.h> #include <test_progs.h> +#include <bpf/btf.h> #include "task_local_storage_helpers.h" #include "task_local_storage.skel.h" #include "task_local_storage_exit_creds.skel.h"
@@ -19,6 +20,7 @@#include "task_ls_uptr.skel.h" #include "uptr_update_failure.skel.h" #include "uptr_failure.skel.h" +#include "uptr_map_failure.skel.h" static void test_sys_enter_exit (void ) {
@@ -452,6 +454,40 @@ static void test_uptr_update_failure(void)close (task_fd); } +static void test_uptr_map_failure (const char *map_name,int expected_errno) +{ +LIBBPF_OPTS (bpf_map_create_opts, create_attr); +struct uptr_map_failure *skel; +struct bpf_map *map; +struct btf *btf; +int map_fd, err; + + skel =uptr_map_failure__open (); +if (!ASSERT_OK_PTR (skel,"uptr_map_failure__open" )) +return ; + + map =bpf_object__find_map_by_name (skel->obj, map_name); + btf =bpf_object__btf (skel->obj); + err =btf__load_into_kernel (btf); +if (!ASSERT_OK (err,"btf__load_into_kernel" )) + goto done; + + create_attr.map_flags =bpf_map__map_flags (map); + create_attr.btf_fd =btf__fd (btf); + create_attr.btf_key_type_id =bpf_map__btf_key_type_id (map); + create_attr.btf_value_type_id =bpf_map__btf_value_type_id (map); + map_fd =bpf_map_create (bpf_map__type (map), map_name, +bpf_map__key_size (map),bpf_map__value_size (map), +0 , &create_attr); +if (ASSERT_ERR_FD (map_fd,"map_create" )) +ASSERT_EQ (errno, expected_errno,"errno" ); +else +close (map_fd); + +done: +uptr_map_failure__destroy (skel); +} +void test_task_local_storage (void ) {if (test__start_subtest ("sys_enter_exit" ))
@@ -468,5 +504,15 @@ void test_task_local_storage(void)test_uptr_across_pages ();if (test__start_subtest ("uptr_update_failure" ))test_uptr_update_failure (); +if (test__start_subtest ("uptr_map_failure_e2big" )) { +if (getpagesize () ==PAGE_SIZE ) +test_uptr_map_failure ("large_uptr_map" ,E2BIG ); +else +test__skip (); + } +if (test__start_subtest ("uptr_map_failure_size0" )) +test_uptr_map_failure ("empty_uptr_map" ,EINVAL ); +if (test__start_subtest ("uptr_map_failure_kstruct" )) +test_uptr_map_failure ("kstruct_uptr_map" ,EINVAL );RUN_TESTS (uptr_failure); }
tools/testing/selftests/bpf/progs/uptr_map_failure.c new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "uptr_test_common.h" + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct large_uptr); +} large_uptr_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct empty_uptr); +} empty_uptr_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_TASK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct kstruct_uptr); +} kstruct_uptr_map SEC(".maps");
tools/testing/selftests/bpf/test_progs.h +8 -0
@@ -390,6 +390,14 @@ int test__join_cgroup(const char *path); ___ok; \ }) +#define ASSERT_ERR_FD (fd, name) ({ \ + static int duration = 0; \ + int ___fd = (fd); \ + bool ___ok = ___fd < 0; \ + CHECK(!___ok, (name), "unexpected fd: %d\n", ___fd); \ + ___ok; \ +}) +#define SYS (goto_label, fmt, ...) \ ({ \ char cmd[1024]; \
tools/testing/selftests/bpf/uptr_test_common.h +23 -0
@@ -5,9 +5,12 @@#define _UPTR_TEST_COMMON_H#define MAGIC_VALUE 0xabcd1234 +#define PAGE_SIZE 4096#ifdef __BPF__/* Avoid fwd btf type being generated for the following struct */ +struct large_data *dummy_large_data; +struct empty_data *dummy_empty_data;struct user_data *dummy_data;struct cgroup *dummy_cgrp;#else
@@ -37,4 +40,24 @@ struct value_lock_type {struct bpf_spin_lock lock; }; +struct large_data { +__u8 one_page[PAGE_SIZE ]; +int a; +}; + +struct large_uptr { + struct large_data __uptr *udata; +}; + +struct empty_data { +}; + +struct empty_uptr { + struct empty_data __uptr *udata; +}; + +struct kstruct_uptr { + struct cgroup __uptr *cgrp; +}; +#endif
Signature · 1 line
2.43.5
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Wed, 23 Oct 2024 16:47:47 -0700 you wrote:
From: Martin KaFai Lau <[email protected]>
It is the v6 of this series. Starting from v5, it is a continuation work of the RFC v4.
Changes in v6:
1. In patch 1, rejectt->size== 0 inbtf_check_and_fixup_fields. Reject a uptr pointing to an empty struct.[...]
Here is the summary with links:
- [v6,bpf-next,01/12] bpf: Support __uptr type tag in BTF
https://git.kernel.org/bpf/bpf-next/c/1cb80d9e93f8 - [v6,bpf-next,02/12] bpf: Handle BPF_UPTR in verifier https://git.kernel.org/bpf/bpf-next/c/99dde42e3749 - [v6,bpf-next,03/12] bpf: Add "bool swap_uptrs" arg to bpf_local_storage_update() and bpf_selem_alloc() https://git.kernel.org/bpf/bpf-next/c/b9a5a07aeaa2 - [v6,bpf-next,04/12] bpf: Postpone bpf_selem_free() in bpf_selem_unlink_storage_nolock() https://git.kernel.org/bpf/bpf-next/c/5bd5bab76669 - [v6,bpf-next,05/12] bpf: Postpone bpf_obj_free_fields to the rcu callback https://git.kernel.org/bpf/bpf-next/c/9bac675e6368 - [v6,bpf-next,06/12] bpf: Add uptr support in the map_value of the task local storage. https://git.kernel.org/bpf/bpf-next/c/ba512b00e5ef - [v6,bpf-next,07/12] libbpf: define __uptr. https://git.kernel.org/bpf/bpf-next/c/7aa12b8d9f24 - [v6,bpf-next,08/12] selftests/bpf: Some basic __uptr tests https://git.kernel.org/bpf/bpf-next/c/4579b4a4279e - [v6,bpf-next,09/12] selftests/bpf: Test a uptr struct spanning across pages. https://git.kernel.org/bpf/bpf-next/c/51fff4083372 - [v6,bpf-next,10/12] selftests/bpf: Add update_elem failure test for task storage uptr https://git.kernel.org/bpf/bpf-next/c/cbf9f849a3e8 - [v6,bpf-next,11/12] selftests/bpf: Add uptr failure verifier tests https://git.kernel.org/bpf/bpf-next/c/898cbca4a757 - [v6,bpf-next,12/12] selftests/bpf: Create task_local_storage map with invalid uptr's struct https://git.kernel.org/bpf/bpf-next/c/bd5879a6fe4b
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html