bpf: Add user-space-publisher ring buffer map type
Subsystems (7)
v6 cover bpf: Add user-space-publisher ring buffer map type
This patch set defines a new map type, BPF_MAP_TYPE_USER_RINGBUF, which provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer. Along with the new map type, a helper function called bpf_user_ringbuf_drain() is added which allows a BPF program to specify a callback with the following signature, to which samples are posted by the helper:
void (struct bpf_dynptr *dynptr, void *context);
The program can then use the bpf_dynptr_read() or bpf_dynptr_data() helper functions to safely read the sample from the dynptr. There are currently no helpers available to determine the size of the sample, but one could easily be added if required.
On the user-space side, libbpf has been updated to export a new 'struct ring_buffer_user' type, along with the following symbols:
struct ring_buffer_user *
ring_buffer_user__new(int map_fd,
const struct ring_buffer_user_opts *opts); void ring_buffer_user__free(struct ring_buffer_user *rb); void *ring_buffer_user__reserve(struct ring_buffer_user *rb, uint32_t size); void *ring_buffer_user__poll(struct ring_buffer_user *rb, uint32_t size, int timeout_ms);
void ring_buffer_user__discard(struct ring_buffer_user *rb, void *sample); void ring_buffer_user__submit(struct ring_buffer_user *rb, void *sample);
These symbols are exported for inclusion in libbpf version 1.0.0.
--
v5 -> v6:
- Fixed s/BPF_MAP_TYPE_RINGBUF/BPF_MAP_TYPE_USER_RINGBUF typo in the libbpf user ringbuf doxygen header comment for ring_buffer_user__new() (Andrii).
- Specify that pointer returned from ring_buffer_user__reserve() and its blocking counterpart is 8-byte aligned (Andrii).
- Renamed user_ringbuf__commit() to user_ringbuf_commit(), as it's static (Andrii).
- Another slight reworking of user_ring_buffer__reserve_blocking() to remove some extraneous nanosecond variables + checking (Andrii).
- Add a final check of user_ring_buffer__reserve() in user_ring_buffer__reserve_blocking().
- Moved busy bit lock / unlock logic from __bpf_user_ringbuf_peek() to bpf_user_ringbuf_drain() (Andrii).
- -ENOSPC -> -ENODATA for an empty ring buffer in __bpf_user_ringbuf_peek() (Andrii).
- Updated BPF_RB_FORCE_WAKEUP to only force a wakeup notification to be sent if even if no sample was drained.
- Changed a bit of the wording in the UAPI header for bpf_user_ringbuf_drain() to mention the BPF_RB_FORCE_WAKEUP behavior.
- Remove extra space after return in ringbuf_map_poll_user() (Andrii).
- Removed now-extraneous paragraph from the commit summary of patch 2/4 (Andrii).
v4 -> v5:
- DENYLISTed the user-ringbuf test suite on s390x. We have a number of functions in the progs/user_ringbuf_success.c prog that user-space fires by invoking a syscall. Not all of these syscalls are available on s390x. If and when we add the ability to kick the kernel from user-space, or if we end up using iterators for that per Hao's suggestion, we could re-enable this test suite on s390x.
- Fixed a few more places that needed ringbuffer -> ring buffer. v3 -> v4:
- Update BPF_MAX_USER_RINGBUF_SAMPLES to not specify a bit, and instead just specify a number of samples. (Andrii)
- Update "ringbuffer" in comments and commit summaries to say "ring buffer". (Andrii)
- Return -E2BIG from bpf_user_ringbuf_drain() both when a sample can't fit into the ring buffer, and when it can't fit into a dynptr. (Andrii)
- Don't loop over samples in __bpf_user_ringbuf_peek() if a sample was discarded. Instead, return -EAGAIN so the caller can deal with it. Also updated the caller to detect -EAGAIN and skip over it when iterating. (Andrii)
- Removed the heuristic for notifying user-space when a sample is drained, causing the ring buffer to no longer be full. This may be useful in the future, but is being removed now because it's strictly a heuristic.
- Re-add BPF_RB_FORCE_WAKEUP flag to bpf_user_ringbuf_drain(). (Andrii)
- Remove helper_allocated_dynptr tracker from verifier. (Andrii)
- Add libbpf function header comments to tools/lib/bpf/libbpf.h, so that they will be included in rendered libbpf docs. (Andrii)
- Add symbols to a new LIBBPF_1.1.0 section in linker version script, rather than including them in LIBBPF_1.0.0. (Andrii)
- Remove libbpf_err() calls from static libbpf functions. (Andrii)
- Check user_ring_buffer_opts instead of ring_buffer_opts in user_ring_buffer__new(). (Andrii)
- Avoid an extra if in the hot path in user_ringbuf__commit(). (Andrii)
- Use ENOSPC rather than ENODATA if no space is available in the ring buffer. (Andrii)
- Don't round sample size in header to 8, but still round size that is reserved and written to 8, and validate positions are multiples of 8 (Andrii).
- Use nanoseconds for most calculations in user_ring_buffer__reserve_blocking(). (Andrii)
- Don't use CHECK() in testcases, instead use ASSERT_*. (Andrii)
- Use SEC("?raw_tp") instead of SEC("?raw_tp/sys_nanosleep") in negative test. (Andrii)
- Move test_user_ringbuf.h header to live next to BPF program instead of a directory up from both it and the user-space test program. (Andrii)
- Update bpftool help message / docs to also include user_ringbuf. v2 -> v3:
- Lots of formatting fixes, such as keeping things on one line if they fit within 100 characters, and removing some extraneous newlines. Applies to all diffs in the patch-set. (Andrii)
- Renamed ring_buffer_user__* symbols to user_ring_buffer__*. (Andrii)
- Added a missing smb_mb__before_atomic() in __bpf_user_ringbuf_sample_release(). (Hao)
- Restructure how and when notification events are sent from the kernel to the user-space producers via the .map_poll() callback for the BPF_MAP_TYPE_USER_RINGBUF map. Before, we only sent a notification when the ringbuffer was fully drained. Now, we guarantee user-space that we'll send an event at least once per bpf_user_ringbuf_drain(), as long as at least one sample was drained, and BPF_RB_NO_WAKEUP was not passed. As a heuristic, we also send a notification event any time a sample being drained causes the ringbuffer to no longer be full. (Andrii)
- Continuing on the above point, updated user_ring_buffer__reserve_blocking() to loop around epoll_wait() until a sufficiently large sample is found. (Andrii)
- Communicate BPF_RINGBUF_BUSY_BIT and BPF_RINGBUF_DISCARD_BIT in sample headers. The ringbuffer implementation still only supports single-producer semantics, but we can now add synchronization support in user_ring_buffer__reserve(), and will automatically get multi-producer semantics. (Andrii)
- Updated some commit summaries, specifically adding more details where warranted. (Andrii)
- Improved function documentation for bpf_user_ringbuf_drain(), more clearly explaining all function arguments and return types, as well as the semantics for waking up user-space producers.
- Add function header comments for user_ring_buffer__reserve{_blocking}(). (Andrii)
- Rounding-up all samples to 8-bytes in the user-space producer, and enforcing that all samples are properly aligned in the kernel. (Andrii)
- Added testcases that verify that bpf_user_ringbuf_drain() properly validates samples, and returns error conditions if any invalid samples are encountered. (Andrii)
- Move atomic_t busy field out of the consumer page, and into the struct bpf_ringbuf. (Andrii)
- Split ringbuf_map_{mmap, poll}_{kern, user}() into separate implementations. (Andrii)
- Don't silently consume errors in bpf_user_ringbuf_drain(). (Andrii)
- Remove magic number of samples (4096) from bpf_user_ringbuf_drain(), and instead use BPF_MAX_USER_RINGBUF_SAMPLES macro, which allows 128k samples. (Andrii)
- Remove MEM_ALLOC modifier from PTR_TO_DYNPTR register in verifier, and instead rely solely on the register being PTR_TO_DYNPTR. (Andrii)
- Move freeing of atomic_t busy bit to before we invoke irq_work_queue() in __bpf_user_ringbuf_sample_release(). (Andrii)
- Only check for BPF_RB_NO_WAKEUP flag in bpf_ringbuf_drain().
- Remove libbpf function names from kernel smp_{load, store}* comments in the kernel. (Andrii)
- Don't use double-underscore naming convention in libbpf functions. (Andrii)
- Use proper __u32 and __u64 for types where we need to guarantee their size. (Andrii)
v1 -> v2:
- Following Joanne landing 883743422ced ("bpf: Fix ref_obj_id for dynptr data slices in verifier") [0], removed [PATCH 1/5] bpf: Clear callee saved regs after updating REG0 [1]. (Joanne)
- Following the above adjustment, updated check_helper_call() to not store a reference for bpf_dynptr_data() if the register containing the dynptr is of type MEM_ALLOC. (Joanne)
- Fixed casting issue pointed out by kernel test robot by adding a missing (uintptr_t) cast. (lkp)
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
David Vernet (4):
bpf: Define new BPF_MAP_TYPE_USER_RINGBUF map type
bpf: Add bpf_user_ringbuf_drain() helper
bpf: Add libbpf logic for user-space ring buffer selftests/bpf: Add selftests validating the user ringbuf
include/linux/bpf.h | 11 +- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 39 + kernel/bpf/helpers.c | 2 + kernel/bpf/ringbuf.c | 248 +++++- kernel/bpf/verifier.c | 62 +- .../bpf/bpftool/Documentation/bpftool-map.rst | 2 +- tools/bpf/bpftool/map.c | 2 +- tools/include/uapi/linux/bpf.h | 39 + tools/lib/bpf/libbpf.c | 11 +- tools/lib/bpf/libbpf.h | 106 +++ tools/lib/bpf/libbpf.map | 10 + tools/lib/bpf/libbpf_probes.c | 1 + tools/lib/bpf/libbpf_version.h | 2 +- tools/lib/bpf/ringbuf.c | 268 +++++++ tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/user_ringbuf.c | 754 ++++++++++++++++++ .../selftests/bpf/progs/test_user_ringbuf.h | 35 + .../selftests/bpf/progs/user_ringbuf_fail.c | 177 ++++ .../bpf/progs/user_ringbuf_success.c | 218 +++++ 20 files changed, 1968 insertions(+), 21 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/user_ringbuf.c create mode 100644 tools/testing/selftests/bpf/progs/test_user_ringbuf.h create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_fail.c create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_success.c
Signature · 1 line
2.37.3
PATCH v6 1/4 bpf: Define new BPF_MAP_TYPE_USER_RINGBUF map type
We want to support a ringbuf map type where samples are published from user-space, to be consumed by BPF programs. BPF currently supports a kernel -> user-space circular ring buffer via the BPF_MAP_TYPE_RINGBUF map type. We'll need to define a new map type for user-space -> kernel, as none of the helpers exported for BPF_MAP_TYPE_RINGBUF will apply to a user-space producer ring buffer, and we'll want to add one or more helper functions that would not apply for a kernel-producer ring buffer.
This patch therefore adds a new BPF_MAP_TYPE_USER_RINGBUF map type definition. The map type is useless in its current form, as there is no way to access or use it for anything until we one or more BPF helpers. A follow-on patch will therefore add a new helper function that allows BPF programs to run callbacks on samples that are published to the ring buffer.
--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/ringbuf.c | 62 +++++++++++++++++-- kernel/bpf/verifier.c | 3 + .../bpf/bpftool/Documentation/bpftool-map.rst | 2 +- tools/bpf/bpftool/map.c | 2 +- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/libbpf.c | 1 + 8 files changed, 65 insertions(+), 8 deletions(-)
include/linux/bpf_types.h +1 -0
@@ -126,6 +126,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)#endif BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) +BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)BPF_LINK_TYPE (BPF_LINK_TYPE_TRACING , tracing)
include/uapi/linux/bpf.h +1 -0
@@ -928,6 +928,7 @@ enum bpf_map_type {BPF_MAP_TYPE_INODE_STORAGE ,BPF_MAP_TYPE_TASK_STORAGE ,BPF_MAP_TYPE_BLOOM_FILTER , + BPF_MAP_TYPE_USER_RINGBUF, };/* Note that tracing related programs such as
kernel/bpf/ringbuf.c +56 -6
92 lines not on this page · raw message
kernel/bpf/verifier.c +3 -0
@@ -6240,6 +6240,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, func_id != BPF_FUNC_ringbuf_discard_dynptr) goto error;break ; +case BPF_MAP_TYPE_USER_RINGBUF : + goto error;case BPF_MAP_TYPE_STACK_TRACE :if (func_id != BPF_FUNC_get_stackid) goto error;
@@ -12635,6 +12637,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,case BPF_MAP_TYPE_ARRAY_OF_MAPS :case BPF_MAP_TYPE_HASH_OF_MAPS :case BPF_MAP_TYPE_RINGBUF : +case BPF_MAP_TYPE_USER_RINGBUF :case BPF_MAP_TYPE_INODE_STORAGE :case BPF_MAP_TYPE_SK_STORAGE :case BPF_MAP_TYPE_TASK_STORAGE :
tools/bpf/bpftool/Documentation/bpftool-map.rst +1 -1
@@ -55,7 +55,7 @@ MAP COMMANDS | | **devmap** | **devmap_hash** | **sockmap** | **cpumap** | **xskmap** | **sockhash** | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage** | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage** -| | **task_storage** | **bloom_filter** } +| | **task_storage** | **bloom_filter** | **user_ringbuf** } DESCRIPTION ===========
tools/bpf/bpftool/map.c +1 -1
@@ -1459,7 +1459,7 @@ static int do_help(int argc, char **argv)" devmap | devmap_hash | sockmap | cpumap | xskmap | sockhash |\n" " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n" " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n" -" task_storage | bloom_filter }\n" +" task_storage | bloom_filter | user_ringbuf }\n" " " HELP_SPEC_OPTIONS " |\n" " {-f|--bpffs} | {-n|--nomount} }\n" "",
tools/include/uapi/linux/bpf.h +1 -0
@@ -928,6 +928,7 @@ enum bpf_map_type {BPF_MAP_TYPE_INODE_STORAGE ,BPF_MAP_TYPE_TASK_STORAGE ,BPF_MAP_TYPE_BLOOM_FILTER , + BPF_MAP_TYPE_USER_RINGBUF, };/* Note that tracing related programs such as
tools/lib/bpf/libbpf.c +1 -0
@@ -163,6 +163,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_INODE_STORAGE] = "inode_storage", [BPF_MAP_TYPE_TASK_STORAGE] = "task_storage", [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter", + [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", };static const char *const prog_type_name[] = {
Signature · 1 line
2.37.3
PATCH v6 2/4 bpf: Add bpf_user_ringbuf_drain() helper
In reply to David VernetIn a prior change, we added a new BPF_MAP_TYPE_USER_RINGBUF map type which will allow user-space applications to publish messages to a ring buffer that is consumed by a BPF program in kernel-space. In order for this map-type to be useful, it will require a BPF helper function that BPF programs can invoke to drain samples from the ring buffer, and invoke callbacks on those samples. This change adds that capability via a new BPF helper function:
bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx,
u64 flags)
BPF programs may invoke this function to run callback_fn() on a series of samples in the ring buffer. callback_fn() has the following signature:
long callback_fn(struct bpf_dynptr *dynptr, void *context);
Samples are provided to the callback in the form of struct bpf_dynptr *'s, which the program can read using BPF helper functions for querying struct bpf_dynptr's.
In order to support bpf_ringbuf_drain(), a new PTR_TO_DYNPTR register type is added to the verifier to reflect a dynptr that was allocated by a helper function and passed to a BPF program. Unlike PTR_TO_STACK dynptrs which are allocated on the stack by a BPF program, PTR_TO_DYNPTR dynptrs need not use reference tracking, as the BPF helper is trusted to properly free the dynptr before returning. The verifier currently only supports PTR_TO_DYNPTR registers that are also DYNPTR_TYPE_LOCAL.
Note that while the corresponding user-space libbpf logic will be added in a subsequent patch, this patch does contain an implementation of the .map_poll() callback for BPF_MAP_TYPE_USER_RINGBUF maps. This .map_poll() callback guarantees that an epoll-waiting user-space producer will receive at least one event notification whenever at least one sample is drained in an invocation of bpf_user_ringbuf_drain(), provided that the function is not invoked with the BPF_RB_NO_WAKEUP flag. If the BPF_RB_FORCE_WAKEUP flag is provided, a wakeup notification is sent even if no sample was drained.
--- include/linux/bpf.h | 11 +- include/uapi/linux/bpf.h | 38 +++++++ kernel/bpf/helpers.c | 2 + kernel/bpf/ringbuf.c | 181 ++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 61 ++++++++++- tools/include/uapi/linux/bpf.h | 38 +++++++ 6 files changed, 320 insertions(+), 11 deletions(-)
include/linux/bpf.h +9 -2
@@ -451,7 +451,7 @@ enum bpf_type_flag {/* DYNPTR points to memory local to the bpf program. */ DYNPTR_TYPE_LOCAL =BIT (8 +BPF_BASE_TYPE_BITS ), -/* DYNPTR points to a ringbuf record. */ +/* DYNPTR points to a kernel-produced ringbuf record. */ DYNPTR_TYPE_RINGBUF = BIT(9 + BPF_BASE_TYPE_BITS),/* Size is known at compile time. */
@@ -656,6 +656,7 @@ enum bpf_reg_type {PTR_TO_MEM ,/* reg points to valid memory region */ PTR_TO_BUF ,/* reg points to a read/write buffer */ PTR_TO_FUNC ,/* reg points to a bpf program function */ +PTR_TO_DYNPTR ,/* reg points to a dynptr */ __BPF_REG_TYPE_MAX,/* Extended reg_types. */
@@ -1394,6 +1395,11 @@ struct bpf_array {#define BPF_MAP_CAN_READ BIT(0)#define BPF_MAP_CAN_WRITE BIT(1) +/* Maximum number of user-producer ring buffer samples that can be drained in +* a call to bpf_user_ringbuf_drain(). +*/ +#define BPF_MAX_USER_RINGBUF_SAMPLES (128 * 1024) +static inline u32 bpf_map_flags_to_cap (struct bpf_map *map) {u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG |BPF_F_WRONLY_PROG );
@@ -2495,6 +2501,7 @@ extern const struct bpf_func_proto bpf_loop_proto;extern const struct bpf_func_proto bpf_copy_from_user_task_proto;extern const struct bpf_func_proto bpf_set_retval_proto;extern const struct bpf_func_proto bpf_get_retval_proto; +extern const struct bpf_func_proto bpf_user_ringbuf_drain_proto;const struct bpf_func_proto *tracing_prog_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog);
@@ -2639,7 +2646,7 @@ enum bpf_dynptr_type {BPF_DYNPTR_TYPE_INVALID ,/* Points to memory that is local to the bpf program */ BPF_DYNPTR_TYPE_LOCAL , -/* Underlying data is a ringbuf record */ +/* Underlying data is a kernel-produced ringbuf record */ BPF_DYNPTR_TYPE_RINGBUF, };
include/uapi/linux/bpf.h +38 -0
@@ -5388,6 +5388,43 @@ union bpf_attr {* Return * Current *ktime*. * +* long bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx, u64 flags) +* Description +* Drain samples from the specified user ring buffer, and invoke +* the provided callback for each such sample: +* +* long (\*callback_fn)(struct bpf_dynptr \*dynptr, void \*ctx); +* +* If **callback_fn** returns 0, the helper will continue to try +* and drain the next sample, up to a maximum of +* BPF_MAX_USER_RINGBUF_SAMPLES samples. If the return value is 1, +* the helper will skip the rest of the samples and return. Other +* return values are not used now, and will be rejected by the +* verifier. +* Return +* The number of drained samples if no error was encountered while +* draining samples, or 0 if no samples were present in the ring +* buffer. If a user-space producer was epoll-waiting on this map, +* and at least one sample was drained, they will receive an event +* notification notifying them of available space in the ring +* buffer. If the BPF_RB_NO_WAKEUP flag is passed to this +* function, no wakeup notification will be sent. If the +* BPF_RB_FORCE_WAKEUP flag is passed, a wakeup notification will +* be sent even if no sample was drained. +* +* On failure, the returned value is one of the following: +* +* **-EBUSY** if the ring buffer is contended, and another calling +* context was concurrently draining the ring buffer. +* +* **-EINVAL** if user-space is not properly tracking the ring +* buffer due to the producer position not being aligned to 8 +* bytes, a sample not being aligned to 8 bytes, or the producer +* position not matching the advertised length of a sample. +* +* **-E2BIG** if user-space has tried to publish a sample which is +* larger than the size of the ring buffer, or which cannot fit +* within a struct bpf_dynptr. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -5599,6 +5636,7 @@ union bpf_attr {FN (tcp_raw_check_syncookie_ipv4), \FN (tcp_raw_check_syncookie_ipv6), \FN (ktime_get_tai_ns), \ +FN (user_ringbuf_drain), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper
kernel/bpf/helpers.c +2 -0
@@ -1627,6 +1627,8 @@ bpf_base_func_proto(enum bpf_func_id func_id)return &bpf_dynptr_write_proto;case BPF_FUNC_dynptr_data:return &bpf_dynptr_data_proto; +case BPF_FUNC_user_ringbuf_drain: +return &bpf_user_ringbuf_drain_proto;default :break ; }
kernel/bpf/ringbuf.c +176 -5
232 lines not on this page · raw message
kernel/bpf/verifier.c +57 -4
119 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +38 -0
@@ -5388,6 +5388,43 @@ union bpf_attr {* Return * Current *ktime*. * +* long bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx, u64 flags) +* Description +* Drain samples from the specified user ring buffer, and invoke +* the provided callback for each such sample: +* +* long (\*callback_fn)(struct bpf_dynptr \*dynptr, void \*ctx); +* +* If **callback_fn** returns 0, the helper will continue to try +* and drain the next sample, up to a maximum of +* BPF_MAX_USER_RINGBUF_SAMPLES samples. If the return value is 1, +* the helper will skip the rest of the samples and return. Other +* return values are not used now, and will be rejected by the +* verifier. +* Return +* The number of drained samples if no error was encountered while +* draining samples, or 0 if no samples were present in the ring +* buffer. If a user-space producer was epoll-waiting on this map, +* and at least one sample was drained, they will receive an event +* notification notifying them of available space in the ring +* buffer. If the BPF_RB_NO_WAKEUP flag is passed to this +* function, no wakeup notification will be sent. If the +* BPF_RB_FORCE_WAKEUP flag is passed, a wakeup notification will +* be sent even if no sample was drained. +* +* On failure, the returned value is one of the following: +* +* **-EBUSY** if the ring buffer is contended, and another calling +* context was concurrently draining the ring buffer. +* +* **-EINVAL** if user-space is not properly tracking the ring +* buffer due to the producer position not being aligned to 8 +* bytes, a sample not being aligned to 8 bytes, or the producer +* position not matching the advertised length of a sample. +* +* **-E2BIG** if user-space has tried to publish a sample which is +* larger than the size of the ring buffer, or which cannot fit +* within a struct bpf_dynptr. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -5599,6 +5636,7 @@ union bpf_attr {FN (tcp_raw_check_syncookie_ipv4), \FN (tcp_raw_check_syncookie_ipv6), \FN (ktime_get_tai_ns), \ +FN (user_ringbuf_drain), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper
Signature · 1 line
2.37.3
Re: [PATCH v6 2/4] bpf: Add bpf_user_ringbuf_drain() helper
Quoted 48 lines · David Vernet
In a prior change, we added a new
BPF_MAP_TYPE_USER_RINGBUFmap type which will allow user-space applications to publish messages to a ring buffer that is consumed by a BPF program in kernel-space. In order for this map-type to be useful, it will require a BPF helper function that BPF programs can invoke to drain samples from the ring buffer, and invoke callbacks on those samples. This change adds that capability via a new BPF helper function:bpf_user_ringbuf_drain(struct
bpf_map*map, void*callback_fn, void *ctx,u64 flags)BPF programs may invoke this function to run
callback_fn()on a series of samples in the ring buffer.callback_fn()has the following signature:long
callback_fn(struct bpf_dynptr *dynptr, void *context);Samples are provided to the callback in the form of struct
bpf_dynptr*'s, which the program can read using BPF helper functions for querying struct bpf_dynptr's.In order to support
bpf_ringbuf_drain(), a newPTR_TO_DYNPTRregister type is added to the verifier to reflect a dynptr that was allocated by a helper function and passed to a BPF program. UnlikePTR_TO_STACKdynptrs which are allocated on the stack by a BPF program,PTR_TO_DYNPTRdynptrs need not use reference tracking, as the BPF helper is trusted to properly free the dynptr before returning. The verifier currently only supportsPTR_TO_DYNPTRregisters that are alsoDYNPTR_TYPE_LOCAL.Note that while the corresponding user-space libbpf logic will be added in a subsequent patch, this patch does contain an implementation of the .map_poll() callback for
BPF_MAP_TYPE_USER_RINGBUFmaps. This .map_poll() callback guarantees that an epoll-waiting user-space producer will receive at least one event notification whenever at least one sample is drained in an invocation ofbpf_user_ringbuf_drain(), provided that the function is not invoked with theBPF_RB_NO_WAKEUPflag. If theBPF_RB_FORCE_WAKEUPflag is provided, a wakeup notification is sent even if no sample was drained.Signed-off-by David Vernet <[email protected]>--- include/linux/bpf.h | 11 +- include/uapi/linux/bpf.h | 38 +++++++ kernel/bpf/helpers.c | 2 + kernel/bpf/ringbuf.c | 181 ++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 61 ++++++++++- tools/include/uapi/linux/bpf.h | 38 +++++++ 6 files changed, 320 insertions(+), 11 deletions(-)
[...]
Quoted 15 lines
#define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \include/uapi/linux/bpf.h +1
-0@@ -5599,6 +5636,7 @@ union bpf_attr {FN (tcp_raw_check_syncookie_ipv4), \FN (tcp_raw_check_syncookie_ipv6), \FN (ktime_get_tai_ns), \ +FN (user_ringbuf_drain), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper kernel/bpf/helpers.c +0
-0
kernel/bpf/helpers.c bpf_base_func_proto(enum bpf_func_id func_id) · line 1631
Quoted 5 lines
kernel/bpf/helpers.c +2
-0@@ -1627,6 +1627,8 @@ bpf_base_func_proto(enum bpf_func_id func_id)return &bpf_dynptr_write_proto;case BPF_FUNC_dynptr_data:return &bpf_dynptr_data_proto; +case BPF_FUNC_user_ringbuf_drain: +return &bpf_user_ringbuf_drain_proto;
In light of [0], where we now allow dynptr only with CAP_BPF, I've moved this lower behind CAP_BPF check while applying. Thanks!
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Quoted 3 lines
kernel/bpf/helpers.c +0
-0default :break ; }
[...]
PATCH v6 3/4 bpf: Add libbpf logic for user-space ring buffer
In reply to David VernetNow that all of the logic is in place in the kernel to support user-space produced ring buffers, we can add the user-space logic to libbpf. This patch therefore adds the following public symbols to libbpf:
struct user_ring_buffer *
user_ring_buffer__new(int map_fd,
const struct user_ring_buffer_opts *opts);
void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size);
void *user_ring_buffer__reserve_blocking(struct user_ring_buffer *rb,
__u32 size, int timeout_ms);void user_ring_buffer__submit(struct user_ring_buffer *rb, void *sample); void user_ring_buffer__discard(struct user_ring_buffer *rb, void user_ring_buffer__free(struct user_ring_buffer *rb);
A user-space producer must first create a struct user_ring_buffer * object with user_ring_buffer__new(), and can then reserve samples in the ring buffer using one of the following two symbols:
void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size); void *user_ring_buffer__reserve_blocking(struct user_ring_buffer *rb,
__u32 size, int timeout_ms);
With user_ring_buffer__reserve(), a pointer to a 'size' region of the ring buffer will be returned if sufficient space is available in the buffer. user_ring_buffer__reserve_blocking() provides similar semantics, but will block for up to 'timeout_ms' in epoll_wait if there is insufficient space in the buffer. This function has the guarantee from the kernel that it will receive at least one event-notification per invocation to bpf_ringbuf_drain(), provided that at least one sample is drained, and the BPF program did not pass the BPF_RB_NO_WAKEUP flag to bpf_ringbuf_drain().
Once a sample is reserved, it must either be committed to the ring buffer with user_ring_buffer__submit(), or discarded with user_ring_buffer__discard().
--- tools/lib/bpf/libbpf.c | 10 +- tools/lib/bpf/libbpf.h | 107 +++++++++++++ tools/lib/bpf/libbpf.map | 10 ++ tools/lib/bpf/libbpf_probes.c | 1 + tools/lib/bpf/libbpf_version.h | 2 +- tools/lib/bpf/ringbuf.c | 271 +++++++++++++++++++++++++++++++++ 6 files changed, 398 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +8 -2
@@ -2373,6 +2373,12 @@ static size_t adjust_ringbuf_sz(size_t sz)return sz; } +static bool map_is_ringbuf (const struct bpf_map *map) +{ +return map->def.type ==BPF_MAP_TYPE_RINGBUF || + map->def.type ==BPF_MAP_TYPE_USER_RINGBUF ; +} +static void fill_map_from_def (struct bpf_map *map,const struct btf_map_def *def) { map->def.type = def->map_type;
@@ -2387,7 +2393,7 @@ static void fill_map_from_def(struct bpf_map *map, const struct btf_map_def *def map->btf_value_type_id = def->value_type_id;/* auto-adjust BPF ringbuf map max_entries to be a multiple of page size */ -if (map->def.type ==BPF_MAP_TYPE_RINGBUF ) +if (map_is_ringbuf (map)) map->def.max_entries =adjust_ringbuf_sz (map->def.max_entries);if (def->parts &MAP_DEF_MAP_TYPE )
@@ -4370,7 +4376,7 @@ int bpf_map__set_max_entries(struct bpf_map *map, __u32 max_entries) map->def.max_entries = max_entries;/* auto-adjust BPF ringbuf map max_entries to be a multiple of page size */ -if (map->def.type ==BPF_MAP_TYPE_RINGBUF ) +if (map_is_ringbuf (map)) map->def.max_entries =adjust_ringbuf_sz (map->def.max_entries);return 0 ;
tools/lib/bpf/libbpf.h +107 -0
119 lines not on this page · raw message
tools/lib/bpf/libbpf.map +10 -0
@@ -368,3 +368,13 @@ LIBBPF_1.0.0 { libbpf_bpf_prog_type_str; perf_buffer__buffer; }; + +LIBBPF_1.1.0 { + global: + user_ring_buffer__discard; + user_ring_buffer__free; + user_ring_buffer__new; + user_ring_buffer__reserve; + user_ring_buffer__reserve_blocking; + user_ring_buffer__submit; +} LIBBPF_1.0.0;
tools/lib/bpf/libbpf_probes.c +1 -0
@@ -231,6 +231,7 @@ static int probe_map_create(enum bpf_map_type map_type)return btf_fd;break ;case BPF_MAP_TYPE_RINGBUF : +case BPF_MAP_TYPE_USER_RINGBUF : key_size =0 ; value_size =0 ; max_entries =4096 ;
tools/lib/bpf/libbpf_version.h +1 -1
@@ -4,6 +4,6 @@#define __LIBBPF_VERSION_H#define LIBBPF_MAJOR_VERSION 1 -#define LIBBPF_MINOR_VERSION 0 +#define LIBBPF_MINOR_VERSION 1#endif /* __LIBBPF_VERSION_H */
tools/lib/bpf/ringbuf.c +271 -0
286 lines not on this page · raw message
Signature · 1 line
2.37.3
PATCH v6 4/4 selftests/bpf: Add selftests validating the user ringbuf
In reply to David VernetThis change includes selftests that validate the expected behavior and APIs of the new BPF_MAP_TYPE_USER_RINGBUF map type.
--- tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/user_ringbuf.c | 754 ++++++++++++++++++ .../selftests/bpf/progs/test_user_ringbuf.h | 35 + .../selftests/bpf/progs/user_ringbuf_fail.c | 177 ++++ .../bpf/progs/user_ringbuf_success.c | 218 +++++ 5 files changed, 1185 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/user_ringbuf.c create mode 100644 tools/testing/selftests/bpf/progs/test_user_ringbuf.h create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_fail.c create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_success.c
tools/testing/selftests/bpf/DENYLIST.s390x +1 -0
@@ -71,3 +71,4 @@ cb_refs # expected error message unexpected err cgroup_hierarchical_stats # JIT does not support calling kernel function (kfunc) htab_update # failed to attach: ERROR: strerror_r(-524)=22 (trampoline) tracing_struct # failed to auto-attach: -524 (trampoline) +user_ringbuf # failed to find kernel BTF type ID of '__s390x_sys_prctl': -3 (?)
tools/testing/selftests/bpf/prog_tests/user_ringbuf.c new file +754 -0
754 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_user_ringbuf.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ + +#ifndef _TEST_USER_RINGBUF_H +#define _TEST_USER_RINGBUF_H + +#define TEST_OP_64 4 +#define TEST_OP_32 2 + +enum test_msg_op { +TEST_MSG_OP_INC64 , +TEST_MSG_OP_INC32 , +TEST_MSG_OP_MUL64 , +TEST_MSG_OP_MUL32 , + +// Must come last. +TEST_MSG_OP_NUM_OPS , +}; + +struct test_msg { +enum test_msg_op msg_op; +union { +__s64 operand_64; +__s32 operand_32; + }; +}; + +struct sample { +int pid; +int seq; +long value; +char comm[16 ]; +}; + +#endif /* _TEST_USER_RINGBUF_H */
tools/testing/selftests/bpf/progs/user_ringbuf_fail.c new file +177 -0
177 lines not on this page · raw message
tools/testing/selftests/bpf/progs/user_ringbuf_success.c new file +218 -0
218 lines not on this page · raw message
Signature · 1 line
2.37.3
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Quoted 8 lines · On Mon, 19 Sep 2022 19:00:56 -0500 you wrote:
This patch set defines a new map type,
BPF_MAP_TYPE_USER_RINGBUF, which provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer. Along with the new map type, a helper function calledbpf_user_ringbuf_drain()is added which allows a BPF program to specify a callback with the following signature, to which samples are posted by the helper:[...]
Here is the summary with links:
- [v6,1/4] bpf: Define new BPF_MAP_TYPE_USER_RINGBUF map type
https://git.kernel.org/bpf/bpf-next/c/583c1f420173 - [v6,2/4] bpf: Add bpf_user_ringbuf_drain() helper https://git.kernel.org/bpf/bpf-next/c/205715673844 - [v6,3/4] bpf: Add libbpf logic for user-space ring buffer https://git.kernel.org/bpf/bpf-next/c/b66ccae01f1d - [v6,4/4] selftests/bpf: Add selftests validating the user ringbuf https://git.kernel.org/bpf/bpf-next/c/e5a9df51c746
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html