Introduce kfuncs for memory reads into dynptrs
Subsystems (7)
v2 cover Introduce kfuncs for memory reads into dynptrs
From: Mykyta Yatsenko <[email protected]>
This patch adds new kfuncs that enable reading variable-length user or kernel data directly into dynptrs.
These kfuncs provide a way to perform dynamically-sized reads while maintaining memory safety. Unlike existing bpf_probe_read_{user|kernel} APIs, which are limited to constant-sized reads, these new kfuncs allow for more flexible data access.
Mykyta Yatsenko (3):
helpers: make few bpf helpers public
bpf: implement dynptr copy kfuncs selftests/bpf: introduce tests for dynptr copy kfuncs
include/linux/bpf.h | 7 + kernel/bpf/helpers.c | 14 +- kernel/trace/bpf_trace.c | 199 +++++++++++++++++ tools/testing/selftests/bpf/DENYLIST | 1 + .../testing/selftests/bpf/prog_tests/dynptr.c | 13 ++ .../selftests/bpf/progs/dynptr_success.c | 201 ++++++++++++++++++ 6 files changed, 432 insertions(+), 3 deletions(-)
Signature · 1 line
2.49.0
PATCH v2 1/3 helpers: make few bpf helpers public
From: Mykyta Yatsenko <[email protected]>
Make bpf_dynptr_slice_rdwr, bpf_dynptr_check_off_len and __bpf_dynptr_write available outside of the helpers.c by adding their prototypes into linux/include/bpf.h.
These functions are going to be used from bpf_trace.c in the next patch of this series.
--- include/linux/bpf.h | 7 +++++++ kernel/bpf/helpers.c | 6 +++--- 2 files changed, 10 insertions(+), 3 deletions(-)
include/linux/bpf.h +7 -0
@@ -1349,6 +1349,13 @@ u32 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr);const void *__bpf_dynptr_data (const struct bpf_dynptr_kern *ptr,u32 len);void *__bpf_dynptr_data_rw (const struct bpf_dynptr_kern *ptr,u32 len);bool __bpf_dynptr_is_rdonly (const struct bpf_dynptr_kern *ptr); +int __bpf_dynptr_write (const struct bpf_dynptr_kern *dst,u32 offset, +void *src,u32 len,u64 flags); +int bpf_dynptr_check_off_len (const struct bpf_dynptr_kern *ptr,u32 offset,u32 len); +void *bpf_dynptr_slice_rdwr (const struct bpf_dynptr *p,u32 offset, +void *buffer__opt,u32 buffer__szk); +int bpf_dynptr_check_off_len (const struct bpf_dynptr_kern *ptr, +u32 offset,u32 len);#ifdef CONFIG_BPF_JIT int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
kernel/bpf/helpers.c +3 -3
@@ -1713,7 +1713,7 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr)memset (ptr,0 ,sizeof (*ptr)); } -static int bpf_dynptr_check_off_len (const struct bpf_dynptr_kern *ptr,u32 offset,u32 len) +int bpf_dynptr_check_off_len (const struct bpf_dynptr_kern *ptr,u32 offset,u32 len) {u32 size =__bpf_dynptr_size (ptr);
@@ -1809,8 +1809,8 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = { .arg5_type = ARG_ANYTHING, }; -static int __bpf_dynptr_write (const struct bpf_dynptr_kern *dst,u32 offset,void *src, -u32 len,u64 flags) +int __bpf_dynptr_write (const struct bpf_dynptr_kern *dst,u32 offset,void *src, +u32 len,u64 flags) {enum bpf_dynptr_type type;int err;
Signature · 1 line
2.49.0
PATCH v2 2/3 bpf: implement dynptr copy kfuncs
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
This patch introduces a new set of kfuncs for working with dynptrs in BPF programs, enabling reading variable-length user or kernel data into dynptr directly. To enable memory-safety, verifier allows only constant-sized reads via existing bpf_probe_read_{user|kernel} etc. kfuncs, dynptr-based kfuncs allow dynamically-sized reads without memory safety shortcomings.
The following kfuncs are introduced:
* bpf_probe_read_kernel_dynptr(): probes kernel-space data into a dynptr
* bpf_probe_read_user_dynptr(): probes user-space data into a dynptr
* bpf_probe_read_kernel_str_dynptr(): probes kernel-space string into a dynptr
* bpf_probe_read_user_str_dynptr(): probes user-space string into a dynptr
* bpf_copy_from_user_dynptr(): sleepable, copies user-space data into a dynptr for the current task
* bpf_copy_from_user_str_dynptr(): sleepable, copies user-space string into a dynptr for the current task
* bpf_copy_from_user_task_dynptr(): sleepable, copies user-space data of the task into a dynptr
* bpf_copy_from_user_task_str_dynptr(): sleepable, copies user-space string of the task into a dynptr
The implementation is built on two generic functions:
* __bpf_dynptr_copy
* __bpf_dynptr_copy_str
These functions take function pointers as arguments, enabling the copying of data from various sources, including both kernel and user space. Notably, these indirect calls are typically inlined.
--- kernel/bpf/helpers.c | 8 ++ kernel/trace/bpf_trace.c | 199 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+)
kernel/bpf/helpers.c +8 -0
@@ -3294,6 +3294,14 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr) +BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/trace/bpf_trace.c +199 -0
209 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH bpf-next v2 2/3] bpf: implement dynptr copy kfuncs
Quoted 46 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
This patch introduces a new set of kfuncs for working with dynptrs in BPF programs, enabling reading variable-length user or kernel data into dynptr directly. To enable memory-safety, verifier allows only constant-sized reads via existing bpf_probe_read_{user|kernel} etc. kfuncs, dynptr-based kfuncs allow dynamically-sized reads without memory safety shortcomings.
The following kfuncs are introduced:
*bpf_probe_read_kernel_dynptr(): probes kernel-space data into a dynptr
*bpf_probe_read_user_dynptr(): probes user-space data into a dynptr
*bpf_probe_read_kernel_str_dynptr(): probes kernel-space string into a dynptr
*bpf_probe_read_user_str_dynptr(): probes user-space string into a dynptr
*bpf_copy_from_user_dynptr(): sleepable, copies user-space data into a dynptr for the current task
*bpf_copy_from_user_str_dynptr(): sleepable, copies user-space string into a dynptr for the current task
*bpf_copy_from_user_task_dynptr(): sleepable, copies user-space data of the task into a dynptr
*bpf_copy_from_user_task_str_dynptr(): sleepable, copies user-space string of the task into a dynptrThe implementation is built on two generic functions:
*__bpf_dynptr_copy
*__bpf_dynptr_copy_str
These functions take function pointers as arguments, enabling the copying of data from various sources, including both kernel and user space. Notably, these indirect calls are typically inlined.Reviewed-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 8 ++ kernel/trace/bpf_trace.c | 199 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+)kernel/bpf/helpers.c +0
-0@@ -3294,6 +3294,14 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEBTF_ID_FLAGS (func, bpf_iter_kmem_cache_destroy,KF_ITER_DESTROY |KF_SLEEPABLE )
kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE · line 3304
Quoted 10 lines
kernel/bpf/helpers.c +8
-0BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr) +BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS (func, bpf_copy_from_user_task_str_dynptr,KF_SLEEPABLE )
They need to have KF_TRUSTED_ARGS, otherwise legacy ptr_to_btf_id can be passed in.
Quoted 8 lines
kernel/bpf/helpers.c +0
-0BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {kernel/trace/bpf_trace.c +0
-0
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3504
Quoted 6 lines
kernel/trace/bpf_trace.c +3
-0@@ -3499,6 +3499,147 @@ static int __init bpf_kprobe_multi_kfuncs_init(void)late_initcall (bpf_kprobe_multi_kfuncs_init); +typedef int (*copy_fn_t )(void *dst,const void *src,u32 size,struct task_struct *tsk); + +static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u32 doff, u32 size,
why always_inline?
patch 1 already adds overhead in non-LTO build,
since small helper bpf_dynptr_check_off_len() will not be inlined. This __always_inline looks odd and probably wrong
from optimizations pov.
All other __always_inline look wrong too.
Quoted 84 lines
kernel/trace/bpf_trace.c +84
-0+const void __user *unsafe_src, + copy_fn_t str_copy_fn, + struct task_struct *tsk) +{ + struct bpf_dynptr_kern *dst; +u32 chunk_sz, off; +void *dst_slice; +int cnt, err; +char buf[256 ]; + + dst_slice =bpf_dynptr_slice_rdwr (dptr, doff,NULL , size); +if (likely (dst_slice)) +return str_copy_fn (dst_slice, unsafe_src, size, tsk); + + dst = (struct bpf_dynptr_kern *)dptr; +if (bpf_dynptr_check_off_len (dst, doff, size)) +return -E2BIG ; + +for (off =0 ; off < size; off += chunk_sz -1 ) { + chunk_sz =min_t (u32,sizeof (buf), size - off); +/* Expect str_copy_fn to return count of copied bytes, including +* zero terminator. Next iteration increment off by chunk_sz - 1 to +* overwrite NUL. +*/ + cnt =str_copy_fn (buf, unsafe_src + off, chunk_sz, tsk); +if (cnt <0 ) +return cnt; + err =__bpf_dynptr_write (dst, doff + off, buf, cnt,0 ); +if (err) +return err; +if (cnt < chunk_sz || chunk_sz ==1 )/* we are done */ +return off + cnt; + } +return off; +} + +static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u32 doff, +u32 size,const void __user *unsafe_src, +copy_fn_t copy_fn,struct task_struct *tsk) +{ +struct bpf_dynptr_kern *dst; +void *dst_slice; +char buf[256 ]; +u32 off, chunk_sz; +int err; + + dst_slice =bpf_dynptr_slice_rdwr (dptr, doff,NULL , size); +if (likely (dst_slice)) +return copy_fn (dst_slice, unsafe_src, size, tsk); + + dst = (struct bpf_dynptr_kern *)dptr; +if (bpf_dynptr_check_off_len (dst, doff, size)) +return -E2BIG ; + +for (off =0 ; off < size; off += chunk_sz) { + chunk_sz =min_t (u32,sizeof (buf), size - off); + err =copy_fn (buf, unsafe_src + off, chunk_sz, tsk); +if (err) +return err; + err =__bpf_dynptr_write (dst, doff + off, buf, chunk_sz,0 ); +if (err) +return err; + } +return 0 ; +} + +static __always_inline int copy_user_data_nofault(void *dst, const void __user *unsafe_src, +u32 size,struct task_struct *tsk) +{ +if (WARN_ON_ONCE (tsk)) +return -EFAULT ; + +return copy_from_user_nofault (dst, unsafe_src, size); +} + +static __always_inline int copy_user_data_sleepable(void *dst, const void __user *unsafe_src, +u32 size,struct task_struct *tsk) +{ +int ret; + +if (!tsk)/* Read from the current task */ +return copy_from_user (dst, unsafe_src, size); + + ret =access_process_vm (tsk, (unsigned long )unsafe_src, dst, size,0 );
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3598
Quoted 10 lines
kernel/trace/bpf_trace.c +10
-0+if (ret != size) +return -EFAULT ; +return 0 ; +} + +static __always_inline int copy_kernel_data_nofault(void *dst, const void *unsafe_src, +u32 size,struct task_struct *tsk) +{ +if (WARN_ON_ONCE (tsk)) +return -EFAULT ;
Why ? Let's not do defensive programming.
The caller clearly passes NULL.
Just drop this line.
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3607
Quoted 9 lines
kernel/trace/bpf_trace.c +9
-0+ +return copy_from_kernel_nofault (dst, unsafe_src, size); +} + +static __always_inline int copy_user_str_nofault(void *dst, const void __user *unsafe_src, +u32 size,struct task_struct *tsk) +{ +if (WARN_ON_ONCE (tsk)) +return -EFAULT ;
same here and further down.
pw-bot: cr
Re: [PATCH bpf-next v2 2/3] bpf: implement dynptr copy kfuncs
Quoted 69 lines · Alexei Starovoitov
Quoted 45 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
This patch introduces a new set of kfuncs for working with dynptrs in BPF programs, enabling reading variable-length user or kernel data into dynptr directly. To enable memory-safety, verifier allows only constant-sized reads via existing bpf_probe_read_{user|kernel} etc. kfuncs, dynptr-based kfuncs allow dynamically-sized reads without memory safety shortcomings.
The following kfuncs are introduced:
*bpf_probe_read_kernel_dynptr(): probes kernel-space data into a dynptr
*bpf_probe_read_user_dynptr(): probes user-space data into a dynptr
*bpf_probe_read_kernel_str_dynptr(): probes kernel-space string into a dynptr
*bpf_probe_read_user_str_dynptr(): probes user-space string into a dynptr
*bpf_copy_from_user_dynptr(): sleepable, copies user-space data into a dynptr for the current task
*bpf_copy_from_user_str_dynptr(): sleepable, copies user-space string into a dynptr for the current task
*bpf_copy_from_user_task_dynptr(): sleepable, copies user-space data of the task into a dynptr
*bpf_copy_from_user_task_str_dynptr(): sleepable, copies user-space string of the task into a dynptrThe implementation is built on two generic functions:
*__bpf_dynptr_copy
*__bpf_dynptr_copy_str
These functions take function pointers as arguments, enabling the copying of data from various sources, including both kernel and user space. Notably, these indirect calls are typically inlined.Reviewed-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 8 ++ kernel/trace/bpf_trace.c | 199 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+)kernel/bpf/helpers.c +0
-0@@ -3294,6 +3294,14 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEBTF_ID_FLAGS (func, bpf_iter_kmem_cache_destroy,KF_ITER_DESTROY |KF_SLEEPABLE )kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE · line 3304
Quoted 10 lines
kernel/bpf/helpers.c +8
-0BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr) +BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS (func, bpf_copy_from_user_task_str_dynptr,KF_SLEEPABLE )They need to have
KF_TRUSTED_ARGS, otherwise legacyptr_to_btf_idcan be passed in.Quoted 9 lines
kernel/bpf/helpers.c +0
-0BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {kernel/trace/bpf_trace.c +0
-0@@ -3499,6 +3499,147 @@ static int __init bpf_kprobe_multi_kfuncs_init(void)
Quoted 6 lines
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3504
Quoted 5 lines
kernel/trace/bpf_trace.c +3
-0late_initcall (bpf_kprobe_multi_kfuncs_init); +typedef int (*copy_fn_t )(void *dst,const void *src,u32 size,struct task_struct *tsk); + +static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u32 doff, u32 size,why
always_inline?
Just wanted to get rid of the overhead of having implementation in the generic __bpf_dynptr_copy() (Removes one jmp, perhaps not a big deal)
Quoted 3 lines
patch 1 already adds overhead in non-LTO build,
since small helperbpf_dynptr_check_off_len()will not be inlined.
Thanks for pointing this one, I guess bpf_dynptr_check_off_len() implementation should be moved to header and be inlined?
Quoted 95 lines
This
__always_inlinelooks odd and probably wrong
from optimizations pov.All other
__always_inlinelook wrong too.Quoted 89 lines
kernel/trace/bpf_trace.c +89
-0+const void __user *unsafe_src, + copy_fn_t str_copy_fn, + struct task_struct *tsk) +{ + struct bpf_dynptr_kern *dst; +u32 chunk_sz, off; +void *dst_slice; +int cnt, err; +char buf[256 ]; + + dst_slice =bpf_dynptr_slice_rdwr (dptr, doff,NULL , size); +if (likely (dst_slice)) +return str_copy_fn (dst_slice, unsafe_src, size, tsk); + + dst = (struct bpf_dynptr_kern *)dptr; +if (bpf_dynptr_check_off_len (dst, doff, size)) +return -E2BIG ; + +for (off =0 ; off < size; off += chunk_sz -1 ) { + chunk_sz =min_t (u32,sizeof (buf), size - off); +/* Expect str_copy_fn to return count of copied bytes, including +* zero terminator. Next iteration increment off by chunk_sz - 1 to +* overwrite NUL. +*/ + cnt =str_copy_fn (buf, unsafe_src + off, chunk_sz, tsk); +if (cnt <0 ) +return cnt; + err =__bpf_dynptr_write (dst, doff + off, buf, cnt,0 ); +if (err) +return err; +if (cnt < chunk_sz || chunk_sz ==1 )/* we are done */ +return off + cnt; + } +return off; +} + +static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u32 doff, +u32 size,const void __user *unsafe_src, +copy_fn_t copy_fn,struct task_struct *tsk) +{ +struct bpf_dynptr_kern *dst; +void *dst_slice; +char buf[256 ]; +u32 off, chunk_sz; +int err; + + dst_slice =bpf_dynptr_slice_rdwr (dptr, doff,NULL , size); +if (likely (dst_slice)) +return copy_fn (dst_slice, unsafe_src, size, tsk); + + dst = (struct bpf_dynptr_kern *)dptr; +if (bpf_dynptr_check_off_len (dst, doff, size)) +return -E2BIG ; + +for (off =0 ; off < size; off += chunk_sz) { + chunk_sz =min_t (u32,sizeof (buf), size - off); + err =copy_fn (buf, unsafe_src + off, chunk_sz, tsk); +if (err) +return err; + err =__bpf_dynptr_write (dst, doff + off, buf, chunk_sz,0 ); +if (err) +return err; + } +return 0 ; +} + +static __always_inline int copy_user_data_nofault(void *dst, const void __user *unsafe_src, +u32 size,struct task_struct *tsk) +{ +if (WARN_ON_ONCE (tsk)) +return -EFAULT ; + +return copy_from_user_nofault (dst, unsafe_src, size); +} + +static __always_inline int copy_user_data_sleepable(void *dst, const void __user *unsafe_src, +u32 size,struct task_struct *tsk) +{ +int ret; + +if (!tsk)/* Read from the current task */ +return copy_from_user (dst, unsafe_src, size); + + ret =access_process_vm (tsk, (unsigned long )unsafe_src, dst, size,0 ); +if (ret != size) +return -EFAULT ; +return 0 ; +} +
Quoted 8 lines
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3598
Quoted 5 lines
kernel/trace/bpf_trace.c +5
-0+static __always_inline int copy_kernel_data_nofault(void *dst, const void *unsafe_src, +u32 size,struct task_struct *tsk) +{ +if (WARN_ON_ONCE (tsk)) +return -EFAULT ;Why ? Let's not do defensive programming.
The caller clearly passes NULL.
Just drop this line.
got it.
Quoted 12 lines
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3607
Quoted 9 lines
kernel/trace/bpf_trace.c +9
-0+ +return copy_from_kernel_nofault (dst, unsafe_src, size); +} + +static __always_inline int copy_user_str_nofault(void *dst, const void __user *unsafe_src, +u32 size,struct task_struct *tsk) +{ +if (WARN_ON_ONCE (tsk)) +return -EFAULT ;same here and further down.
pw-bot: cr
Re: [PATCH bpf-next v2 2/3] bpf: implement dynptr copy kfuncs
Quoted 73 lines · Mykyta Yatsenko
Quoted 71 lines · Alexei Starovoitov
Quoted 45 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
This patch introduces a new set of kfuncs for working with dynptrs in BPF programs, enabling reading variable-length user or kernel data into dynptr directly. To enable memory-safety, verifier allows only constant-sized reads via existing bpf_probe_read_{user|kernel} etc. kfuncs, dynptr-based kfuncs allow dynamically-sized reads without memory safety shortcomings.
The following kfuncs are introduced:
*bpf_probe_read_kernel_dynptr(): probes kernel-space data into a dynptr
*bpf_probe_read_user_dynptr(): probes user-space data into a dynptr
*bpf_probe_read_kernel_str_dynptr(): probes kernel-space string into a dynptr
*bpf_probe_read_user_str_dynptr(): probes user-space string into a dynptr
*bpf_copy_from_user_dynptr(): sleepable, copies user-space data into a dynptr for the current task
*bpf_copy_from_user_str_dynptr(): sleepable, copies user-space string into a dynptr for the current task
*bpf_copy_from_user_task_dynptr(): sleepable, copies user-space data of the task into a dynptr
*bpf_copy_from_user_task_str_dynptr(): sleepable, copies user-space string of the task into a dynptrThe implementation is built on two generic functions:
*__bpf_dynptr_copy
*__bpf_dynptr_copy_str
These functions take function pointers as arguments, enabling the copying of data from various sources, including both kernel and user space. Notably, these indirect calls are typically inlined.Reviewed-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 8 ++ kernel/trace/bpf_trace.c | 199 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+)kernel/bpf/helpers.c +0
-0@@ -3294,6 +3294,14 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEBTF_ID_FLAGS (func, bpf_iter_kmem_cache_destroy,KF_ITER_DESTROY |KF_SLEEPABLE )kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE · line 3304
Quoted 10 lines
kernel/bpf/helpers.c +8
-0BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr) +BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS (func, bpf_copy_from_user_task_str_dynptr,KF_SLEEPABLE )They need to have
KF_TRUSTED_ARGS, otherwise legacyptr_to_btf_idcan be passed in.Quoted 11 lines
kernel/bpf/helpers.c +0
-0BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {kernel/trace/bpf_trace.c +0
-0@@ -3499,6 +3499,147 @@ static int __init bpf_kprobe_multi_kfuncs_init(void)late_initcall (bpf_kprobe_multi_kfuncs_init);
Quoted 7 lines
Quoted 4 lines
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3504
Quoted 3 lines
kernel/trace/bpf_trace.c +3
-0+typedef int (*copy_fn_t )(void *dst,const void *src,u32 size,struct task_struct *tsk); + +static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u32 doff, u32 size,why
always_inline?Just wanted to get rid of the overhead of having implementation in the generic
__bpf_dynptr_copy()
(Removes one jmp, perhaps not a big deal)
This __always_inline is there to make sure the compiler doesn't generate indirect calls into callbacks, which is more expensive, especially on some kernel configurations. So this is less about inlining implementation of __bpf_dyntpr_copy() logic itself for performance reasons, rather than to allow compiler to put direct calls into all the specific callback implementations (copy_user_data_sleepable, copy_user_data_nofault, and so on).
Please put this explanation in the commit message for the next revision, and maybe also leave a small comment next to __bpf_dynptr_copy_str and __bpf_dynptr_copy.
Quoted 12 lines
Quoted 3 lines
patch 1 already adds overhead in non-LTO build,
since small helperbpf_dynptr_check_off_len()will not be inlined.Thanks for pointing this one, I guess
bpf_dynptr_check_off_len()implementation should be moved to header
and be inlined?Quoted 6 lines
This
__always_inlinelooks odd and probably wrong
from optimizations pov.All other
__always_inlinelook wrong too.
[...]
Re: [PATCH bpf-next v2 2/3] bpf: implement dynptr copy kfuncs
In reply to Mykyta YatsenkoQuoted 73 lines · Mykyta Yatsenko
Quoted 71 lines · Alexei Starovoitov
Quoted 45 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
This patch introduces a new set of kfuncs for working with dynptrs in BPF programs, enabling reading variable-length user or kernel data into dynptr directly. To enable memory-safety, verifier allows only constant-sized reads via existing bpf_probe_read_{user|kernel} etc. kfuncs, dynptr-based kfuncs allow dynamically-sized reads without memory safety shortcomings.
The following kfuncs are introduced:
*bpf_probe_read_kernel_dynptr(): probes kernel-space data into a dynptr
*bpf_probe_read_user_dynptr(): probes user-space data into a dynptr
*bpf_probe_read_kernel_str_dynptr(): probes kernel-space string into a dynptr
*bpf_probe_read_user_str_dynptr(): probes user-space string into a dynptr
*bpf_copy_from_user_dynptr(): sleepable, copies user-space data into a dynptr for the current task
*bpf_copy_from_user_str_dynptr(): sleepable, copies user-space string into a dynptr for the current task
*bpf_copy_from_user_task_dynptr(): sleepable, copies user-space data of the task into a dynptr
*bpf_copy_from_user_task_str_dynptr(): sleepable, copies user-space string of the task into a dynptrThe implementation is built on two generic functions:
*__bpf_dynptr_copy
*__bpf_dynptr_copy_str
These functions take function pointers as arguments, enabling the copying of data from various sources, including both kernel and user space. Notably, these indirect calls are typically inlined.Reviewed-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- kernel/bpf/helpers.c | 8 ++ kernel/trace/bpf_trace.c | 199 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 207 insertions(+)kernel/bpf/helpers.c +0
-0@@ -3294,6 +3294,14 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEBTF_ID_FLAGS (func, bpf_iter_kmem_cache_destroy,KF_ITER_DESTROY |KF_SLEEPABLE )kernel/bpf/helpers.c BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE · line 3304
Quoted 10 lines
kernel/bpf/helpers.c +8
-0BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_probe_read_user_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_user_str_dynptr) +BTF_ID_FLAGS(func, bpf_probe_read_kernel_str_dynptr) +BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE) +BTF_ID_FLAGS (func, bpf_copy_from_user_task_str_dynptr,KF_SLEEPABLE )They need to have
KF_TRUSTED_ARGS, otherwise legacyptr_to_btf_idcan be passed in.Quoted 11 lines
kernel/bpf/helpers.c +0
-0BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {kernel/trace/bpf_trace.c +0
-0@@ -3499,6 +3499,147 @@ static int __init bpf_kprobe_multi_kfuncs_init(void)late_initcall (bpf_kprobe_multi_kfuncs_init);
Quoted 7 lines
Quoted 4 lines
kernel/trace/bpf_trace.c static int __init bpf_kprobe_multi_kfuncs_init(void) · line 3504
Quoted 3 lines
kernel/trace/bpf_trace.c +3
-0+typedef int (*copy_fn_t )(void *dst,const void *src,u32 size,struct task_struct *tsk); + +static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u32 doff, u32 size,why
always_inline?Just wanted to get rid of the overhead of having implementation in the generic
__bpf_dynptr_copy()
(Removes one jmp, perhaps not a big deal)
As Andrii suggested please add a comment right in the code to explain what this always_inline is for.
Quoted 6 lines
Quoted 3 lines
patch 1 already adds overhead in non-LTO build,
since small helperbpf_dynptr_check_off_len()will not be inlined.Thanks for pointing this one, I guess
bpf_dynptr_check_off_len()implementation should be moved to header
and be inlined?
Yes. Please. Otherwise saving on indirect call, but wasting a direct call for a tiny function looks odd.
PATCH v2 3/3 selftests/bpf: introduce tests for dynptr copy kfuncs
In reply to Mykyta YatsenkoFrom: Mykyta Yatsenko <[email protected]>
Introduce selftests verifying newly-added dynptr copy kfuncs. Covering contiguous and non-contiguous memory backed dynptrs.
Disable test_probe_read_user_str_dynptr that triggers bug in strncpy_from_user_nofault. Patch to fix the issue [1].
[1] https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/
--- tools/testing/selftests/bpf/DENYLIST | 1 + .../testing/selftests/bpf/prog_tests/dynptr.c | 13 ++ .../selftests/bpf/progs/dynptr_success.c | 201 ++++++++++++++++++ 3 files changed, 215 insertions(+)
tools/testing/selftests/bpf/DENYLIST +1 -0
@@ -1,5 +1,6 @@ # TEMPORARY # Alphabetical order +dynptr/test_probe_read_user_str_dynptr # disabled until https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/ makes it into the bpf-next get_stack_raw_tp # spams with kernel warnings until next bpf -> bpf-next merge stacktrace_build_id stacktrace_build_id_nmi
tools/testing/selftests/bpf/prog_tests/dynptr.c +13 -0
@@ -33,10 +33,19 @@ static struct { {"test_dynptr_skb_no_buff", SETUP_SKB_PROG}, {"test_dynptr_skb_strcmp", SETUP_SKB_PROG}, {"test_dynptr_skb_tp_btf", SETUP_SKB_PROG_TP}, + {"test_probe_read_user_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_user_str_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_str_dynptr", SETUP_XDP_PROG}, + {"test_copy_from_user_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_str_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_str_dynptr", SETUP_SYSCALL_SLEEP}, };static void verify_success (const char *prog_name,enum test_setup_type setup_type) { +char user_data[384 ] = {[0 ...382 ] ='a' ,'\0' };struct dynptr_success *skel;struct bpf_program *prog;struct bpf_link *link;
@@ -58,6 +67,10 @@ static void verify_success(const char *prog_name, enum test_setup_type setup_typif (!ASSERT_OK (err,"dynptr_success__load" )) goto cleanup; + skel->bss->user_ptr = user_data; + skel->data->test_len[0 ] =sizeof (user_data); +memcpy (skel->bss->expected_str, user_data,sizeof (user_data)); +switch (setup_type) {case SETUP_SYSCALL_SLEEP : link =bpf_program__attach (prog);
tools/testing/selftests/bpf/progs/dynptr_success.c +201 -0
204 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH bpf-next v2 3/3] selftests/bpf: introduce tests for dynptr copy kfuncs
Quoted 70 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Introduce selftests verifying newly-added dynptr copy kfuncs. Covering contiguous and non-contiguous memory backed dynptrs.
Disable
test_probe_read_user_str_dynptrthat triggers bug instrncpy_from_user_nofault. Patch to fix the issue [1].[1] https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- tools/testing/selftests/bpf/DENYLIST | 1 + .../testing/selftests/bpf/prog_tests/dynptr.c | 13 ++ .../selftests/bpf/progs/dynptr_success.c | 201 ++++++++++++++++++ 3 files changed, 215 insertions(+)tools/testing/selftests/bpf/DENYLIST +1
-0@@ -1,5 +1,6 @@ # TEMPORARY # Alphabetical order +dynptr/test_probe_read_user_str_dynptr # disabled until https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/ makes it into the bpf-next get_stack_raw_tp # spams with kernel warnings until next bpf -> bpf-next merge stacktrace_build_id stacktrace_build_id_nmitools/testing/selftests/bpf/prog_tests/dynptr.c +13
-0@@ -33,10 +33,19 @@ static struct { {"test_dynptr_skb_no_buff", SETUP_SKB_PROG}, {"test_dynptr_skb_strcmp", SETUP_SKB_PROG}, {"test_dynptr_skb_tp_btf", SETUP_SKB_PROG_TP}, + {"test_probe_read_user_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_user_str_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_str_dynptr", SETUP_XDP_PROG}, + {"test_copy_from_user_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_str_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_str_dynptr", SETUP_SYSCALL_SLEEP}, };static void verify_success (const char *prog_name,enum test_setup_type setup_type) { +char user_data[384 ] = {[0 ...382 ] ='a' ,'\0' };struct dynptr_success *skel;struct bpf_program *prog;struct bpf_link *link;@@ -58,6 +67,10 @@ static void verify_success(const char *prog_name, enum test_setup_type setup_typif (!ASSERT_OK (err,"dynptr_success__load" )) goto cleanup; + skel->bss->user_ptr = user_data; + skel->data->test_len[0 ] =sizeof (user_data); +memcpy (skel->bss->expected_str, user_data,sizeof (user_data)); +switch (setup_type) {case SETUP_SYSCALL_SLEEP : link =bpf_program__attach (prog);tools/testing/selftests/bpf/progs/dynptr_success.c +0
-0
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 685
Quoted 6 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +3
-0@@ -680,3 +680,204 @@ int test_dynptr_copy_xdp(struct xdp_md *xdp)bpf_ringbuf_discard_dynptr (&ptr_buf,0 );return XDP_DROP ; } + +void *user_ptr; +char expected_str[384 ];/* Contains the copy of the data pointed by user_ptr */
what so magic about 384?
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 691
Quoted 6 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +6
-0+__u32 test_len[7 ] = {0 /* placeholder */ ,0 ,1 ,2 ,255 ,256 ,257 }; + +typedef int (*bpf_read_dynptr_fn_t )(struct bpf_dynptr *dptr,u32 off, +u32 size,const void *unsafe_ptr); + +static __always_inline void test_dynptr_probe(void *ptr, bpf_read_dynptr_fn_t bpf_read_dynptr_fn)
More __always_inline in the prog too?
Why?
Quoted 51 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +51
-0+{ +char buf[sizeof (expected_str)]; +struct bpf_dynptr ptr_buf; +int i; + + err =bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (buf),0 , &ptr_buf); + + bpf_for(i, 0, ARRAY_SIZE(test_len)) { +__u32 len = test_len[i]; + + err = err ?:bpf_read_dynptr_fn (&ptr_buf,0 , test_len[i], ptr); +if (len >sizeof (buf)) +break ; + err = err ?:bpf_dynptr_read (&buf, len, &ptr_buf,0 ,0 ); + +if (err ||bpf_memcmp (expected_str, buf, len)) + err =1 ; + +/* Reset buffer and dynptr */ +__builtin_memset (buf,0 ,sizeof (buf)); + err = err ?:bpf_dynptr_write (&ptr_buf,0 , buf, len,0 ); + } +bpf_ringbuf_discard_dynptr (&ptr_buf,0 ); +} + +static __always_inline void test_dynptr_probe_str(void *ptr, +bpf_read_dynptr_fn_t bpf_read_dynptr_fn) +{ +char buf[sizeof (expected_str)]; +struct bpf_dynptr ptr_buf; +__u32 cnt, i; + +bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (buf),0 , &ptr_buf); + + bpf_for(i, 0, ARRAY_SIZE(test_len)) { +__u32 len = test_len[i]; + + cnt =bpf_read_dynptr_fn (&ptr_buf,0 , len, ptr); +if (cnt != len) + err =1 ; + +if (len >sizeof (buf)) +continue ; + err = err ?:bpf_dynptr_read (&buf, len, &ptr_buf,0 ,0 ); +if (!len) +continue ; +if (err ||bpf_memcmp (expected_str, buf, len -1 ) || buf[len -1 ] !='\0' ) + err =1 ; + } +bpf_ringbuf_discard_dynptr (&ptr_buf,0 ); +}
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 752
Quoted 10 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +10
-0+ +static __always_inline void test_dynptr_probe_xdp(struct xdp_md *xdp, void *ptr, +bpf_read_dynptr_fn_t bpf_read_dynptr_fn) +{ +struct bpf_dynptr ptr_xdp; +char buf[sizeof (expected_str)]; +__u32 off, i; + +/* Set offset near the seam between buffers */ + off =3500 ;
what is 3500 ?
Re: [PATCH bpf-next v2 3/3] selftests/bpf: introduce tests for dynptr copy kfuncs
Quoted 70 lines · Alexei Starovoitov
Quoted 68 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Introduce selftests verifying newly-added dynptr copy kfuncs. Covering contiguous and non-contiguous memory backed dynptrs.
Disable
test_probe_read_user_str_dynptrthat triggers bug instrncpy_from_user_nofault. Patch to fix the issue [1].[1] https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- tools/testing/selftests/bpf/DENYLIST | 1 + .../testing/selftests/bpf/prog_tests/dynptr.c | 13 ++ .../selftests/bpf/progs/dynptr_success.c | 201 ++++++++++++++++++ 3 files changed, 215 insertions(+)tools/testing/selftests/bpf/DENYLIST +1
-0@@ -1,5 +1,6 @@ # TEMPORARY # Alphabetical order +dynptr/test_probe_read_user_str_dynptr # disabled until https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/ makes it into the bpf-next get_stack_raw_tp # spams with kernel warnings until next bpf -> bpf-next merge stacktrace_build_id stacktrace_build_id_nmitools/testing/selftests/bpf/prog_tests/dynptr.c +13
-0@@ -33,10 +33,19 @@ static struct { {"test_dynptr_skb_no_buff", SETUP_SKB_PROG}, {"test_dynptr_skb_strcmp", SETUP_SKB_PROG}, {"test_dynptr_skb_tp_btf", SETUP_SKB_PROG_TP}, + {"test_probe_read_user_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_user_str_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_str_dynptr", SETUP_XDP_PROG}, + {"test_copy_from_user_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_str_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_str_dynptr", SETUP_SYSCALL_SLEEP}, };static void verify_success (const char *prog_name,enum test_setup_type setup_type) { +char user_data[384 ] = {[0 ...382 ] ='a' ,'\0' };struct dynptr_success *skel;struct bpf_program *prog;struct bpf_link *link;@@ -58,6 +67,10 @@ static void verify_success(const char *prog_name, enum test_setup_type setup_typif (!ASSERT_OK (err,"dynptr_success__load" )) goto cleanup; + skel->bss->user_ptr = user_data; + skel->data->test_len[0 ] =sizeof (user_data); +memcpy (skel->bss->expected_str, user_data,sizeof (user_data)); +switch (setup_type) {case SETUP_SYSCALL_SLEEP : link =bpf_program__attach (prog);tools/testing/selftests/bpf/progs/dynptr_success.c +0
-0
Quoted 8 lines
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 685
Quoted 7 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +3
-0@@ -680,3 +680,204 @@ int test_dynptr_copy_xdp(struct xdp_md *xdp)bpf_ringbuf_discard_dynptr (&ptr_buf,0 );return XDP_DROP ; } + +void *user_ptr; +char expected_str[384 ];/* Contains the copy of the data pointed by user_ptr */ what so magic about 384?
It's MAX_BPF_STACK, but leaving some space for other things (temp buffer of this size is allocated on stack of each bpf prog) The reason to use large buffer, is because when kfunc implementation falls back onto copying data chunk by chunk, the length of chunk is 256 bytes, so I'm trying to force it copy more than 1 chunk.
Quoted 8 lines
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 691
Quoted 6 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +6
-0+__u32 test_len[7 ] = {0 /* placeholder */ ,0 ,1 ,2 ,255 ,256 ,257 }; + +typedef int (*bpf_read_dynptr_fn_t )(struct bpf_dynptr *dptr,u32 off, +u32 size,const void *unsafe_ptr); + +static __always_inline void test_dynptr_probe(void *ptr, bpf_read_dynptr_fn_t bpf_read_dynptr_fn)More
__always_inlinein the prog too?
Why?
This one is a little bit tricky:
When removing always_inline, the generated bpf code does indirect call into bpf_read_dynptr_fn_t, which errors out with "unknown opcode 8d", when trying to load the program.
An example of the generated code that fails to load:
89: 79 a5 58 fe 00 00 00 00 r5 = *(u64 *)(r10 - 0x1a8) 90: 8d 05 00 00 00 00 00 00 callx r5
I could not find another way of fixing this, I'll appreciate any alternative.
Quoted 54 lines
Quoted 54 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +54
-0+{ +char buf[sizeof (expected_str)]; +struct bpf_dynptr ptr_buf; +int i; + + err =bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (buf),0 , &ptr_buf); + + bpf_for(i, 0, ARRAY_SIZE(test_len)) { +__u32 len = test_len[i]; + + err = err ?:bpf_read_dynptr_fn (&ptr_buf,0 , test_len[i], ptr); +if (len >sizeof (buf)) +break ; + err = err ?:bpf_dynptr_read (&buf, len, &ptr_buf,0 ,0 ); + +if (err ||bpf_memcmp (expected_str, buf, len)) + err =1 ; + +/* Reset buffer and dynptr */ +__builtin_memset (buf,0 ,sizeof (buf)); + err = err ?:bpf_dynptr_write (&ptr_buf,0 , buf, len,0 ); + } +bpf_ringbuf_discard_dynptr (&ptr_buf,0 ); +} + +static __always_inline void test_dynptr_probe_str(void *ptr, +bpf_read_dynptr_fn_t bpf_read_dynptr_fn) +{ +char buf[sizeof (expected_str)]; +struct bpf_dynptr ptr_buf; +__u32 cnt, i; + +bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (buf),0 , &ptr_buf); + + bpf_for(i, 0, ARRAY_SIZE(test_len)) { +__u32 len = test_len[i]; + + cnt =bpf_read_dynptr_fn (&ptr_buf,0 , len, ptr); +if (cnt != len) + err =1 ; + +if (len >sizeof (buf)) +continue ; + err = err ?:bpf_dynptr_read (&buf, len, &ptr_buf,0 ,0 ); +if (!len) +continue ; +if (err ||bpf_memcmp (expected_str, buf, len -1 ) || buf[len -1 ] !='\0' ) + err =1 ; + } +bpf_ringbuf_discard_dynptr (&ptr_buf,0 ); +} + +static __always_inline void test_dynptr_probe_xdp(struct xdp_md *xdp, void *ptr, +bpf_read_dynptr_fn_t bpf_read_dynptr_fn)
Quoted 8 lines
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 752
Quoted 7 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +7
-0+{ +struct bpf_dynptr ptr_xdp; +char buf[sizeof (expected_str)]; +__u32 off, i; + +/* Set offset near the seam between buffers */ + off =3500 ;what is 3500 ?
This is an offset into the xdp-backed dynptr that is close to the end of the first fragment. As far as I understand the size of the fragment is: max_data_sz = 4096 - headroom - tailroom;, I found it in bpf_prog_test_run_xdp() from net/bpf/test_run.c. I'd like my writes to go into multiple fragments so that dynptr does not have a contiguous buffer of the requested size and chunking logic is hit.
Re: [PATCH bpf-next v2 3/3] selftests/bpf: introduce tests for dynptr copy kfuncs
Quoted 99 lines · Mykyta Yatsenko
Quoted 70 lines · Alexei Starovoitov
Quoted 68 lines · Mykyta Yatsenko
From: Mykyta Yatsenko <[email protected]>
Introduce selftests verifying newly-added dynptr copy kfuncs. Covering contiguous and non-contiguous memory backed dynptrs.
Disable
test_probe_read_user_str_dynptrthat triggers bug instrncpy_from_user_nofault. Patch to fix the issue [1].[1] https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/
Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Mykyta Yatsenko <[email protected]>--- tools/testing/selftests/bpf/DENYLIST | 1 + .../testing/selftests/bpf/prog_tests/dynptr.c | 13 ++ .../selftests/bpf/progs/dynptr_success.c | 201 ++++++++++++++++++ 3 files changed, 215 insertions(+)tools/testing/selftests/bpf/DENYLIST +1
-0@@ -1,5 +1,6 @@ # TEMPORARY # Alphabetical order +dynptr/test_probe_read_user_str_dynptr # disabled until https://patchwork.kernel.org/project/linux-mm/patch/[email protected]/ makes it into the bpf-next get_stack_raw_tp # spams with kernel warnings until next bpf -> bpf-next merge stacktrace_build_id stacktrace_build_id_nmitools/testing/selftests/bpf/prog_tests/dynptr.c +13
-0@@ -33,10 +33,19 @@ static struct { {"test_dynptr_skb_no_buff", SETUP_SKB_PROG}, {"test_dynptr_skb_strcmp", SETUP_SKB_PROG}, {"test_dynptr_skb_tp_btf", SETUP_SKB_PROG_TP}, + {"test_probe_read_user_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_user_str_dynptr", SETUP_XDP_PROG}, + {"test_probe_read_kernel_str_dynptr", SETUP_XDP_PROG}, + {"test_copy_from_user_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_str_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_dynptr", SETUP_SYSCALL_SLEEP}, + {"test_copy_from_user_task_str_dynptr", SETUP_SYSCALL_SLEEP}, };static void verify_success (const char *prog_name,enum test_setup_type setup_type) { +char user_data[384 ] = {[0 ...382 ] ='a' ,'\0' };struct dynptr_success *skel;struct bpf_program *prog;struct bpf_link *link;@@ -58,6 +67,10 @@ static void verify_success(const char *prog_name, enum test_setup_type setup_typif (!ASSERT_OK (err,"dynptr_success__load" )) goto cleanup; + skel->bss->user_ptr = user_data; + skel->data->test_len[0 ] =sizeof (user_data); +memcpy (skel->bss->expected_str, user_data,sizeof (user_data)); +switch (setup_type) {case SETUP_SYSCALL_SLEEP : link =bpf_program__attach (prog);tools/testing/selftests/bpf/progs/dynptr_success.c +0
-0Quoted 8 lines
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 685
Quoted 7 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +3
-0@@ -680,3 +680,204 @@ int test_dynptr_copy_xdp(struct xdp_md *xdp)bpf_ringbuf_discard_dynptr (&ptr_buf,0 );return XDP_DROP ; } + +void *user_ptr; +char expected_str[384 ];/* Contains the copy of the data pointed by user_ptr */ what so magic about 384?
It's
MAX_BPF_STACK, but leaving some space for other things (temp buffer of this size is allocated on stack of each bpf prog)
The reason to use large buffer, is because when kfunc implementation falls back onto copying data chunk by chunk, the length of chunk is 256 bytes, so I'm trying to force it copy more than 1 chunk.Quoted 8 lines
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 691
Quoted 6 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +6
-0+__u32 test_len[7 ] = {0 /* placeholder */ ,0 ,1 ,2 ,255 ,256 ,257 }; + +typedef int (*bpf_read_dynptr_fn_t )(struct bpf_dynptr *dptr,u32 off, +u32 size,const void *unsafe_ptr); + +static __always_inline void test_dynptr_probe(void *ptr, bpf_read_dynptr_fn_t bpf_read_dynptr_fn)More
__always_inlinein the prog too?
Why?This one is a little bit tricky:
When removingalways_inline, the generated bpf code does indirect call intobpf_read_dynptr_fn_t,
which errors out with "unknown opcode 8d", when trying to load the program. An example of the generated code that fails to load:
Quoted 5 lines
89: 79 a5 58 fe 00 00 00 00 r5 = *(u64 *)(r10 -
0x1a8) 90: 8d 05 00 00 00 00 00 00 callx r5I could not find another way of fixing this, I'll appreciate any alternative.
So the always inline is to avoid indirect call?
That makes sense. Please add a comment.
Quoted 63 lines
Quoted 55 lines
Quoted 55 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +54
-0+{ +char buf[sizeof (expected_str)]; +struct bpf_dynptr ptr_buf; +int i; + + err =bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (buf),0 , &ptr_buf); + + bpf_for(i, 0, ARRAY_SIZE(test_len)) { +__u32 len = test_len[i]; + + err = err ?:bpf_read_dynptr_fn (&ptr_buf,0 , test_len[i], ptr); +if (len >sizeof (buf)) +break ; + err = err ?:bpf_dynptr_read (&buf, len, &ptr_buf,0 ,0 ); + +if (err ||bpf_memcmp (expected_str, buf, len)) + err =1 ; + +/* Reset buffer and dynptr */ +__builtin_memset (buf,0 ,sizeof (buf)); + err = err ?:bpf_dynptr_write (&ptr_buf,0 , buf, len,0 ); + } +bpf_ringbuf_discard_dynptr (&ptr_buf,0 ); +} + +static __always_inline void test_dynptr_probe_str(void *ptr, +bpf_read_dynptr_fn_t bpf_read_dynptr_fn) +{ +char buf[sizeof (expected_str)]; +struct bpf_dynptr ptr_buf; +__u32 cnt, i; + +bpf_ringbuf_reserve_dynptr (&ringbuf,sizeof (buf),0 , &ptr_buf); + + bpf_for(i, 0, ARRAY_SIZE(test_len)) { +__u32 len = test_len[i]; + + cnt =bpf_read_dynptr_fn (&ptr_buf,0 , len, ptr); +if (cnt != len) + err =1 ; + +if (len >sizeof (buf)) +continue ; + err = err ?:bpf_dynptr_read (&buf, len, &ptr_buf,0 ,0 ); +if (!len) +continue ; +if (err ||bpf_memcmp (expected_str, buf, len -1 ) || buf[len -1 ] !='\0' ) + err =1 ; + } +bpf_ringbuf_discard_dynptr (&ptr_buf,0 ); +} + +static __always_inline void test_dynptr_probe_xdp(struct xdp_md *xdp, void *ptr, +bpf_read_dynptr_fn_t bpf_read_dynptr_fn)Quoted 8 lines
tools/testing/selftests/bpf/progs/dynptr_success.c int test_dynptr_copy_xdp(struct xdp_md *xdp) · line 752
Quoted 7 lines
tools/testing/selftests/bpf/progs/dynptr_success.c +7
-0+{ +struct bpf_dynptr ptr_xdp; +char buf[sizeof (expected_str)]; +__u32 off, i; + +/* Set offset near the seam between buffers */ + off =3500 ;what is 3500 ?
Quoted 8 lines
This is an offset into the xdp-backed dynptr that is close to the end of the first fragment.
As far as I understand the size of the fragment is: `max_data_sz = 4096
- headroom - tailroom;`,
I found it inbpf_prog_test_run_xdp()fromnet/bpf/test_run.c. I'd like my writes to go into multiple
fragments so that dynptr does not have a contiguous buffer of the requested size and chunking logic is hit.
That's far from obvious. A detailed comment is necessary. Also, I recall, there were some patches to fix:max_data_sz = 4096 - headroom - tailroom;
since it's wrong on archs with different page size.
Would be good to make it more robust.