bpf: Add user-space-publisher ring buffer map type
Subsystems (7)
v5 cover bpf: Add user-space-publisher ring buffer map type
This patch set defines a new map type, BPF_MAP_TYPE_USER_RINGBUF, which provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer. Along with the new map type, a helper function called bpf_user_ringbuf_drain() is added which allows a BPF program to specify a callback with the following signature, to which samples are posted by the helper:
void (struct bpf_dynptr *dynptr, void *context);
The program can then use the bpf_dynptr_read() or bpf_dynptr_data() helper functions to safely read the sample from the dynptr. There are currently no helpers available to determine the size of the sample, but one could easily be added if required.
On the user-space side, libbpf has been updated to export a new 'struct ring_buffer_user' type, along with the following symbols:
struct ring_buffer_user *
ring_buffer_user__new(int map_fd,
const struct ring_buffer_user_opts *opts); void ring_buffer_user__free(struct ring_buffer_user *rb); void *ring_buffer_user__reserve(struct ring_buffer_user *rb, uint32_t size); void *ring_buffer_user__poll(struct ring_buffer_user *rb, uint32_t size, int timeout_ms);
void ring_buffer_user__discard(struct ring_buffer_user *rb, void *sample); void ring_buffer_user__submit(struct ring_buffer_user *rb, void *sample);
These symbols are exported for inclusion in libbpf version 1.0.0.
--
v4 -> v5:
- DENYLISTed the user-ringbuf test suite on s390x. We have a number of functions in the progs/user_ringbuf_success.c prog that user-space fires by invoking a syscall. Not all of these syscalls are available on s390x. If and when we add the ability to kick the kernel from user-space, or if we end up using iterators for that per Hao's suggestion, we could re-enable this test suite on s390x.
- Fixed a few more places that needed ringbuffer -> ring buffer. v3 -> v4:
- Update BPF_MAX_USER_RINGBUF_SAMPLES to not specify a bit, and instead just specify a number of samples. (Andrii)
- Update "ringbuffer" in comments and commit summaries to say "ring buffer". (Andrii)
- Return -E2BIG from bpf_user_ringbuf_drain() both when a sample can't fit into the ring buffer, and when it can't fit into a dynptr. (Andrii)
- Don't loop over samples in __bpf_user_ringbuf_peek() if a sample was discarded. Instead, return -EAGAIN so the caller can deal with it. Also updated the caller to detect -EAGAIN and skip over it when iterating. (Andrii)
- Removed the heuristic for notifying user-space when a sample is drained, causing the ring buffer to no longer be full. This may be useful in the future, but is being removed now because it's strictly a heuristic.
- Re-add BPF_RB_FORCE_WAKEUP flag to bpf_user_ringbuf_drain(). (Andrii)
- Remove helper_allocated_dynptr tracker from verifier. (Andrii)
- Add libbpf function header comments to tools/lib/bpf/libbpf.h, so that they will be included in rendered libbpf docs. (Andrii)
- Add symbols to a new LIBBPF_1.1.0 section in linker version script, rather than including them in LIBBPF_1.0.0. (Andrii)
- Remove libbpf_err() calls from static libbpf functions. (Andrii)
- Check user_ring_buffer_opts instead of ring_buffer_opts in user_ring_buffer__new(). (Andrii)
- Avoid an extra if in the hot path in user_ringbuf__commit(). (Andrii)
- Use ENOSPC rather than ENODATA if no space is available in the ring buffer. (Andrii)
- Don't round sample size in header to 8, but still round size that is reserved and written to 8, and validate positions are multiples of 8 (Andrii).
- Use nanoseconds for most calculations in user_ring_buffer__reserve_blocking(). (Andrii)
- Don't use CHECK() in testcases, instead use ASSERT_*. (Andrii)
- Use SEC("?raw_tp") instead of SEC("?raw_tp/sys_nanosleep") in negative test. (Andrii)
- Move test_user_ringbuf.h header to live next to BPF program instead of a directory up from both it and the user-space test program. (Andrii)
- Update bpftool help message / docs to also include user_ringbuf. v2 -> v3:
- Lots of formatting fixes, such as keeping things on one line if they fit within 100 characters, and removing some extraneous newlines. Applies to all diffs in the patch-set. (Andrii)
- Renamed ring_buffer_user__* symbols to user_ring_buffer__*. (Andrii)
- Added a missing smb_mb__before_atomic() in __bpf_user_ringbuf_sample_release(). (Hao)
- Restructure how and when notification events are sent from the kernel to the user-space producers via the .map_poll() callback for the BPF_MAP_TYPE_USER_RINGBUF map. Before, we only sent a notification when the ringbuffer was fully drained. Now, we guarantee user-space that we'll send an event at least once per bpf_user_ringbuf_drain(), as long as at least one sample was drained, and BPF_RB_NO_WAKEUP was not passed. As a heuristic, we also send a notification event any time a sample being drained causes the ringbuffer to no longer be full. (Andrii)
- Continuing on the above point, updated user_ring_buffer__reserve_blocking() to loop around epoll_wait() until a sufficiently large sample is found. (Andrii)
- Communicate BPF_RINGBUF_BUSY_BIT and BPF_RINGBUF_DISCARD_BIT in sample headers. The ringbuffer implementation still only supports single-producer semantics, but we can now add synchronization support in user_ring_buffer__reserve(), and will automatically get multi-producer semantics. (Andrii)
- Updated some commit summaries, specifically adding more details where warranted. (Andrii)
- Improved function documentation for bpf_user_ringbuf_drain(), more clearly explaining all function arguments and return types, as well as the semantics for waking up user-space producers.
- Add function header comments for user_ring_buffer__reserve{_blocking}(). (Andrii)
- Rounding-up all samples to 8-bytes in the user-space producer, and enforcing that all samples are properly aligned in the kernel. (Andrii)
- Added testcases that verify that bpf_user_ringbuf_drain() properly validates samples, and returns error conditions if any invalid samples are encountered. (Andrii)
- Move atomic_t busy field out of the consumer page, and into the struct bpf_ringbuf. (Andrii)
- Split ringbuf_map_{mmap, poll}_{kern, user}() into separate implementations. (Andrii)
- Don't silently consume errors in bpf_user_ringbuf_drain(). (Andrii)
- Remove magic number of samples (4096) from bpf_user_ringbuf_drain(), and instead use BPF_MAX_USER_RINGBUF_SAMPLES macro, which allows 128k samples. (Andrii)
- Remove MEM_ALLOC modifier from PTR_TO_DYNPTR register in verifier, and instead rely solely on the register being PTR_TO_DYNPTR. (Andrii)
- Move freeing of atomic_t busy bit to before we invoke irq_work_queue() in __bpf_user_ringbuf_sample_release(). (Andrii)
- Only check for BPF_RB_NO_WAKEUP flag in bpf_ringbuf_drain().
- Remove libbpf function names from kernel smp_{load, store}* comments in the kernel. (Andrii)
- Don't use double-underscore naming convention in libbpf functions. (Andrii)
- Use proper __u32 and __u64 for types where we need to guarantee their size. (Andrii)
v1 -> v2:
- Following Joanne landing 883743422ced ("bpf: Fix ref_obj_id for dynptr data slices in verifier") [0], removed [PATCH 1/5] bpf: Clear callee saved regs after updating REG0 [1]. (Joanne)
- Following the above adjustment, updated check_helper_call() to not store a reference for bpf_dynptr_data() if the register containing the dynptr is of type MEM_ALLOC. (Joanne)
- Fixed casting issue pointed out by kernel test robot by adding a missing (uintptr_t) cast. (lkp)
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
David Vernet (4):
bpf: Define new BPF_MAP_TYPE_USER_RINGBUF map type
bpf: Add bpf_user_ringbuf_drain() helper
bpf: Add libbpf logic for user-space ring buffer selftests/bpf: Add selftests validating the user ringbuf
include/linux/bpf.h | 11 +- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 37 + kernel/bpf/helpers.c | 2 + kernel/bpf/ringbuf.c | 272 ++++++- kernel/bpf/verifier.c | 73 +- tools/include/uapi/linux/bpf.h | 37 + tools/lib/bpf/libbpf.c | 11 +- tools/lib/bpf/libbpf.h | 21 + tools/lib/bpf/libbpf.map | 6 + tools/lib/bpf/libbpf_probes.c | 1 + tools/lib/bpf/ringbuf.c | 327 ++++++++ .../selftests/bpf/prog_tests/user_ringbuf.c | 715 ++++++++++++++++++ .../selftests/bpf/progs/user_ringbuf_fail.c | 177 +++++ .../bpf/progs/user_ringbuf_success.c | 220 ++++++ .../testing/selftests/bpf/test_user_ringbuf.h | 35 + 16 files changed, 1924 insertions(+), 22 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/user_ringbuf.c create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_fail.c create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_success.c create mode 100644 tools/testing/selftests/bpf/test_user_ringbuf.h
Signature · 1 line
2.37.1
PATCH v5 1/4 bpf: Define new BPF_MAP_TYPE_USER_RINGBUF map type
We want to support a ringbuf map type where samples are published from user-space, to be consumed by BPF programs. BPF currently supports a kernel -> user-space circular ring buffer via the BPF_MAP_TYPE_RINGBUF map type. We'll need to define a new map type for user-space -> kernel, as none of the helpers exported for BPF_MAP_TYPE_RINGBUF will apply to a user-space producer ring buffer, and we'll want to add one or more helper functions that would not apply for a kernel-producer ring buffer.
This patch therefore adds a new BPF_MAP_TYPE_USER_RINGBUF map type definition. The map type is useless in its current form, as there is no way to access or use it for anything until we one or more BPF helpers. A follow-on patch will therefore add a new helper function that allows BPF programs to run callbacks on samples that are published to the ring buffer.
--- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 1 + kernel/bpf/ringbuf.c | 62 +++++++++++++++++-- kernel/bpf/verifier.c | 3 + .../bpf/bpftool/Documentation/bpftool-map.rst | 2 +- tools/bpf/bpftool/map.c | 2 +- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/libbpf.c | 1 + 8 files changed, 65 insertions(+), 8 deletions(-)
include/linux/bpf_types.h +1 -0
@@ -126,6 +126,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops)#endif BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) +BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)BPF_LINK_TYPE (BPF_LINK_TYPE_TRACING , tracing)
include/uapi/linux/bpf.h +1 -0
@@ -928,6 +928,7 @@ enum bpf_map_type {BPF_MAP_TYPE_INODE_STORAGE ,BPF_MAP_TYPE_TASK_STORAGE ,BPF_MAP_TYPE_BLOOM_FILTER , + BPF_MAP_TYPE_USER_RINGBUF, };/* Note that tracing related programs such as
kernel/bpf/ringbuf.c +56 -6
92 lines not on this page · raw message
kernel/bpf/verifier.c +3 -0
@@ -6206,6 +6206,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, func_id != BPF_FUNC_ringbuf_discard_dynptr) goto error;break ; +case BPF_MAP_TYPE_USER_RINGBUF : + goto error;case BPF_MAP_TYPE_STACK_TRACE :if (func_id != BPF_FUNC_get_stackid) goto error;
@@ -12705,6 +12707,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, }break ;case BPF_MAP_TYPE_RINGBUF : +case BPF_MAP_TYPE_USER_RINGBUF :case BPF_MAP_TYPE_INODE_STORAGE :case BPF_MAP_TYPE_SK_STORAGE :case BPF_MAP_TYPE_TASK_STORAGE :
tools/bpf/bpftool/Documentation/bpftool-map.rst +1 -1
@@ -55,7 +55,7 @@ MAP COMMANDS | | **devmap** | **devmap_hash** | **sockmap** | **cpumap** | **xskmap** | **sockhash** | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage** | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage** -| | **task_storage** | **bloom_filter** } +| | **task_storage** | **bloom_filter** | **user_ringbuf** } DESCRIPTION ===========
tools/bpf/bpftool/map.c +1 -1
@@ -1459,7 +1459,7 @@ static int do_help(int argc, char **argv)" devmap | devmap_hash | sockmap | cpumap | xskmap | sockhash |\n" " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n" " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n" -" task_storage | bloom_filter }\n" +" task_storage | bloom_filter | user_ringbuf }\n" " " HELP_SPEC_OPTIONS " |\n" " {-f|--bpffs} | {-n|--nomount} }\n" "",
tools/include/uapi/linux/bpf.h +1 -0
@@ -928,6 +928,7 @@ enum bpf_map_type {BPF_MAP_TYPE_INODE_STORAGE ,BPF_MAP_TYPE_TASK_STORAGE ,BPF_MAP_TYPE_BLOOM_FILTER , + BPF_MAP_TYPE_USER_RINGBUF, };/* Note that tracing related programs such as
tools/lib/bpf/libbpf.c +1 -0
@@ -163,6 +163,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_INODE_STORAGE] = "inode_storage", [BPF_MAP_TYPE_TASK_STORAGE] = "task_storage", [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter", + [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", };static const char *const prog_type_name[] = {
Signature · 1 line
2.37.1
PATCH v5 2/4 bpf: Add bpf_user_ringbuf_drain() helper
In reply to David VernetIn a prior change, we added a new BPF_MAP_TYPE_USER_RINGBUF map type which will allow user-space applications to publish messages to a ring buffer that is consumed by a BPF program in kernel-space. In order for this map-type to be useful, it will require a BPF helper function that BPF programs can invoke to drain samples from the ring buffer, and invoke callbacks on those samples. This change adds that capability via a new BPF helper function:
bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx,
u64 flags)
BPF programs may invoke this function to run callback_fn() on a series of samples in the ring buffer. callback_fn() has the following signature:
long callback_fn(struct bpf_dynptr *dynptr, void *context);
Samples are provided to the callback in the form of struct bpf_dynptr *'s, which the program can read using BPF helper functions for querying struct bpf_dynptr's.
In order to support bpf_ringbuf_drain(), a new PTR_TO_DYNPTR register type is added to the verifier to reflect a dynptr that was allocated by a helper function and passed to a BPF program. Unlike PTR_TO_STACK dynptrs which are allocated on the stack by a BPF program, PTR_TO_DYNPTR dynptrs need not use reference tracking, as the BPF helper is trusted to properly free the dynptr before returning. The verifier currently only supports PTR_TO_DYNPTR registers that are also DYNPTR_TYPE_LOCAL.
Note that while the corresponding user-space libbpf logic will be added in a subsequent patch, this patch does contain an implementation of the .map_poll() callback for BPF_MAP_TYPE_USER_RINGBUF maps. This .map_poll() callback guarantees that an epoll-waiting user-space producer will receive at least one event notification whenever at least one sample is drained in an invocation of bpf_user_ringbuf_drain(), provided that the function is not invoked with the BPF_RB_NO_WAKEUP flag.
Sending an event notification for every sample is not an option, as it could cause the system to hang due to invoking irq_work_queue() in too-frequent succession. So as to try and optimize for the common case, however, bpf_user_ringbuf_drain() will also send an event notification whenever a sample being drained causes the ring buffer to no longer be full. This heuristic may not help some user-space producers, as a producer can publish samples of varying size, and there may not be enough space in the ring buffer after the first sample is drained which causes it to no longer be full. In this case, the producer may have to wait until bpf_ringbuf_drain() returns to receive an event notification.
--- include/linux/bpf.h | 11 +- include/uapi/linux/bpf.h | 36 ++++++ kernel/bpf/helpers.c | 2 + kernel/bpf/ringbuf.c | 199 ++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 61 +++++++++- tools/include/uapi/linux/bpf.h | 36 ++++++ 6 files changed, 334 insertions(+), 11 deletions(-)
include/linux/bpf.h +9 -2
@@ -402,7 +402,7 @@ enum bpf_type_flag {/* DYNPTR points to memory local to the bpf program. */ DYNPTR_TYPE_LOCAL =BIT (8 +BPF_BASE_TYPE_BITS ), -/* DYNPTR points to a ringbuf record. */ +/* DYNPTR points to a kernel-produced ringbuf record. */ DYNPTR_TYPE_RINGBUF = BIT(9 + BPF_BASE_TYPE_BITS),/* Size is known at compile time. */
@@ -607,6 +607,7 @@ enum bpf_reg_type {PTR_TO_MEM ,/* reg points to valid memory region */ PTR_TO_BUF ,/* reg points to a read/write buffer */ PTR_TO_FUNC ,/* reg points to a bpf program function */ +PTR_TO_DYNPTR ,/* reg points to a dynptr */ __BPF_REG_TYPE_MAX,/* Extended reg_types. */
@@ -1334,6 +1335,11 @@ struct bpf_array {#define BPF_MAP_CAN_READ BIT(0)#define BPF_MAP_CAN_WRITE BIT(1) +/* Maximum number of user-producer ring buffer samples that can be drained in +* a call to bpf_user_ringbuf_drain(). +*/ +#define BPF_MAX_USER_RINGBUF_SAMPLES (128 * 1024) +static inline u32 bpf_map_flags_to_cap (struct bpf_map *map) {u32 access_flags = map->map_flags & (BPF_F_RDONLY_PROG |BPF_F_WRONLY_PROG );
@@ -2433,6 +2439,7 @@ extern const struct bpf_func_proto bpf_loop_proto;extern const struct bpf_func_proto bpf_copy_from_user_task_proto;extern const struct bpf_func_proto bpf_set_retval_proto;extern const struct bpf_func_proto bpf_get_retval_proto; +extern const struct bpf_func_proto bpf_user_ringbuf_drain_proto;const struct bpf_func_proto *tracing_prog_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog);
@@ -2577,7 +2584,7 @@ enum bpf_dynptr_type {BPF_DYNPTR_TYPE_INVALID ,/* Points to memory that is local to the bpf program */ BPF_DYNPTR_TYPE_LOCAL , -/* Underlying data is a ringbuf record */ +/* Underlying data is a kernel-produced ringbuf record */ BPF_DYNPTR_TYPE_RINGBUF, };
include/uapi/linux/bpf.h +36 -0
@@ -5387,6 +5387,41 @@ union bpf_attr {* Return * Current *ktime*. * +* long bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx, u64 flags) +* Description +* Drain samples from the specified user ring buffer, and invoke +* the provided callback for each such sample: +* +* long (\*callback_fn)(struct bpf_dynptr \*dynptr, void \*ctx); +* +* If **callback_fn** returns 0, the helper will continue to try +* and drain the next sample, up to a maximum of +* BPF_MAX_USER_RINGBUF_SAMPLES samples. If the return value is 1, +* the helper will skip the rest of the samples and return. Other +* return values are not used now, and will be rejected by the +* verifier. +* Return +* The number of drained samples if no error was encountered while +* draining samples. If a user-space producer was epoll-waiting on +* this map, and at least one sample was drained, they will +* receive an event notification notifying them of available space +* in the ring buffer. If the BPF_RB_NO_WAKEUP flag is passed to +* this function, no wakeup notification will be sent. If there +* are no samples in the ring buffer, 0 is returned. +* +* On failure, the returned value is one of the following: +* +* **-EBUSY** if the ring buffer is contended, and another calling +* context was concurrently draining the ring buffer. +* +* **-EINVAL** if user-space is not properly tracking the ring +* buffer due to the producer position not being aligned to 8 +* bytes, a sample not being aligned to 8 bytes, or the producer +* position not matching the advertised length of a sample. +* +* **-E2BIG** if user-space has tried to publish a sample which is +* larger than the size of the ring buffer, or which cannot fit +* within a struct bpf_dynptr. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -5598,6 +5633,7 @@ union bpf_attr {FN (tcp_raw_check_syncookie_ipv4), \FN (tcp_raw_check_syncookie_ipv6), \FN (ktime_get_tai_ns), \ +FN (user_ringbuf_drain), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper
kernel/bpf/helpers.c +2 -0
@@ -1627,6 +1627,8 @@ bpf_base_func_proto(enum bpf_func_id func_id)return &bpf_dynptr_write_proto;case BPF_FUNC_dynptr_data:return &bpf_dynptr_data_proto; +case BPF_FUNC_user_ringbuf_drain: +return &bpf_user_ringbuf_drain_proto;default :break ; }
kernel/bpf/ringbuf.c +194 -5
250 lines not on this page · raw message
kernel/bpf/verifier.c +57 -4
119 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +36 -0
@@ -5387,6 +5387,41 @@ union bpf_attr {* Return * Current *ktime*. * +* long bpf_user_ringbuf_drain(struct bpf_map *map, void *callback_fn, void *ctx, u64 flags) +* Description +* Drain samples from the specified user ring buffer, and invoke +* the provided callback for each such sample: +* +* long (\*callback_fn)(struct bpf_dynptr \*dynptr, void \*ctx); +* +* If **callback_fn** returns 0, the helper will continue to try +* and drain the next sample, up to a maximum of +* BPF_MAX_USER_RINGBUF_SAMPLES samples. If the return value is 1, +* the helper will skip the rest of the samples and return. Other +* return values are not used now, and will be rejected by the +* verifier. +* Return +* The number of drained samples if no error was encountered while +* draining samples. If a user-space producer was epoll-waiting on +* this map, and at least one sample was drained, they will +* receive an event notification notifying them of available space +* in the ring buffer. If the BPF_RB_NO_WAKEUP flag is passed to +* this function, no wakeup notification will be sent. If there +* are no samples in the ring buffer, 0 is returned. +* +* On failure, the returned value is one of the following: +* +* **-EBUSY** if the ring buffer is contended, and another calling +* context was concurrently draining the ring buffer. +* +* **-EINVAL** if user-space is not properly tracking the ring +* buffer due to the producer position not being aligned to 8 +* bytes, a sample not being aligned to 8 bytes, or the producer +* position not matching the advertised length of a sample. +* +* **-E2BIG** if user-space has tried to publish a sample which is +* larger than the size of the ring buffer, or which cannot fit +* within a struct bpf_dynptr. */ #define __BPF_FUNC_MAPPER (FN ) \ FN(unspec), \
@@ -5598,6 +5633,7 @@ union bpf_attr {FN (tcp_raw_check_syncookie_ipv4), \FN (tcp_raw_check_syncookie_ipv6), \FN (ktime_get_tai_ns), \ +FN (user_ringbuf_drain), \/* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper
Signature · 1 line
2.37.1
Re: [PATCH v5 2/4] bpf: Add bpf_user_ringbuf_drain() helper
Quoted 37 lines · David Vernet
In a prior change, we added a new
BPF_MAP_TYPE_USER_RINGBUFmap type which will allow user-space applications to publish messages to a ring buffer that is consumed by a BPF program in kernel-space. In order for this map-type to be useful, it will require a BPF helper function that BPF programs can invoke to drain samples from the ring buffer, and invoke callbacks on those samples. This change adds that capability via a new BPF helper function:bpf_user_ringbuf_drain(struct
bpf_map*map, void*callback_fn, void *ctx,u64 flags)BPF programs may invoke this function to run
callback_fn()on a series of samples in the ring buffer.callback_fn()has the following signature:long
callback_fn(struct bpf_dynptr *dynptr, void *context);Samples are provided to the callback in the form of struct
bpf_dynptr*'s, which the program can read using BPF helper functions for querying struct bpf_dynptr's.In order to support
bpf_ringbuf_drain(), a newPTR_TO_DYNPTRregister type is added to the verifier to reflect a dynptr that was allocated by a helper function and passed to a BPF program. UnlikePTR_TO_STACKdynptrs which are allocated on the stack by a BPF program,PTR_TO_DYNPTRdynptrs need not use reference tracking, as the BPF helper is trusted to properly free the dynptr before returning. The verifier currently only supportsPTR_TO_DYNPTRregisters that are alsoDYNPTR_TYPE_LOCAL.Note that while the corresponding user-space libbpf logic will be added in a subsequent patch, this patch does contain an implementation of the .map_poll() callback for
BPF_MAP_TYPE_USER_RINGBUFmaps. This .map_poll() callback guarantees that an epoll-waiting user-space producer will receive at least one event notification whenever at least one sample is drained in an invocation ofbpf_user_ringbuf_drain(), provided that the function is not invoked with theBPF_RB_NO_WAKEUPflag.
Quoted 10 lines
Sending an event notification for every sample is not an option, as it could cause the system to hang due to invoking
irq_work_queue()in too-frequent succession. So as to try and optimize for the common case, however,bpf_user_ringbuf_drain()will also send an event notification whenever a sample being drained causes the ring buffer to no longer be full. This heuristic may not help some user-space producers, as a producer can publish samples of varying size, and there may not be enough space in the ring buffer after the first sample is drained which causes it to no longer be full. In this case, the producer may have to wait untilbpf_ringbuf_drain()returns to receive an event notification.
most of this paragraph should be removed now?
I mentioned few things I didn't see last time (or that were added), but overall looks good to me.
Quoted 11 lines
Signed-off-by David Vernet <[email protected]>--- include/linux/bpf.h | 11 +- include/uapi/linux/bpf.h | 36 ++++++ kernel/bpf/helpers.c | 2 + kernel/bpf/ringbuf.c | 199 ++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 61 +++++++++- tools/include/uapi/linux/bpf.h | 36 ++++++ 6 files changed, 334 insertions(+), 11 deletions(-)
[...]
Quoted 16 lines
kernel/bpf/ringbuf.c +7
-2-static __poll_t ringbuf_map_poll (struct bpf_map *map,struct file *filp, -struct poll_table_struct *pts) +static u32 ringbuf_total_data_sz (const struct bpf_ringbuf *rb) +{ +return rb->mask +1 ; +} + +static __poll_t ringbuf_map_poll_kern (struct bpf_map *map,struct file *filp, +struct poll_table_struct *pts) {struct bpf_ringbuf_map *rb_map;@@ -301,13 +323,26 @@ static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp,return 0 ; }
kernel/bpf/ringbuf.c static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp, · line 335
Quoted 10 lines
kernel/bpf/ringbuf.c +10
-0+static __poll_t ringbuf_map_poll_user (struct bpf_map *map,struct file *filp, +struct poll_table_struct *pts) +{ +struct bpf_ringbuf_map *rb_map; + + rb_map = container_of(map, struct bpf_ringbuf_map, map); +poll_wait (filp, &rb_map->rb->waitq, pts); + +if (ringbuf_avail_data_sz (rb_map->rb) <ringbuf_total_data_sz (rb_map->rb)) +return EPOLLOUT |EPOLLWRNORM ;
nit: extra space after return
Quoted 56 lines
kernel/bpf/ringbuf.c +19
-3+return 0 ; +} + BTF_ID_LIST_SINGLE(ringbuf_map_btf_ids, struct, bpf_ringbuf_map)const struct bpf_map_ops ringbuf_map_ops = { .map_meta_equal = bpf_map_meta_equal, .map_alloc = ringbuf_map_alloc, .map_free = ringbuf_map_free, .map_mmap = ringbuf_map_mmap_kern, - .map_poll = ringbuf_map_poll, + .map_poll = ringbuf_map_poll_kern, .map_lookup_elem = ringbuf_map_lookup_elem, .map_update_elem = ringbuf_map_update_elem, .map_delete_elem = ringbuf_map_delete_elem,@@ -321,6 +356,7 @@ const struct bpf_map_ops user_ringbuf_map_ops = { .map_alloc = ringbuf_map_alloc, .map_free = ringbuf_map_free, .map_mmap = ringbuf_map_mmap_user, + .map_poll = ringbuf_map_poll_user, .map_lookup_elem = ringbuf_map_lookup_elem, .map_update_elem = ringbuf_map_update_elem, .map_delete_elem = ringbuf_map_delete_elem,@@ -362,7 +398,7 @@ static void *__bpf_ringbuf_reserve(struct bpf_ringbuf *rb, u64 size)return NULL ; len =round_up (size +BPF_RINGBUF_HDR_SZ ,8 ); -if (len > rb->mask +1 ) +if (len >ringbuf_total_data_sz (rb))return NULL ; cons_pos =smp_load_acquire (&rb->consumer_pos);@@ -509,7 +545,7 @@ BPF_CALL_2(bpf_ringbuf_query, struct bpf_map *, map, u64, flags)case BPF_RB_AVAIL_DATA :return ringbuf_avail_data_sz (rb);case BPF_RB_RING_SIZE : -return rb->mask +1 ; +return ringbuf_total_data_sz (rb);case BPF_RB_CONS_POS :return smp_load_acquire (&rb->consumer_pos);case BPF_RB_PROD_POS :@@ -603,3 +639,156 @@ const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { .arg1_type = ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_RINGBUF | OBJ_RELEASE, .arg2_type = ARG_ANYTHING, }; + +static int __bpf_user_ringbuf_peek (struct bpf_ringbuf *rb,void **sample,u32 *size) +{ +int err, busy =0 ; +u32 hdr_len, sample_len, total_len, flags, *hdr; +u64 cons_pos, prod_pos; + +/* If another consumer is already consuming a sample, wait for them to finish. */ +if (!atomic_try_cmpxchg (&rb->busy, &busy,1 )) +return -EBUSY ; + +/* Synchronizes with smp_store_release() in user-space producer. */
kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 663
Quoted 10 lines
kernel/bpf/ringbuf.c +10
-0+ prod_pos =smp_load_acquire (&rb->producer_pos); +if (prod_pos %8 ) { + err = -EINVAL ; + goto err_unlock; + } + +/* Synchronizes with smp_store_release() in __bpf_user_ringbuf_sample_release() */ + cons_pos =smp_load_acquire (&rb->consumer_pos); +if (cons_pos >= prod_pos) { + err = -ENOSPC ;
funny, this one actually feels like -ENODATA (there is no data to peek at, right?). It's different from ENOSPC, as we are not trying to enqueue anything. I probably missed it before?
Quoted 66 lines
kernel/bpf/ringbuf.c +66
-0+ goto err_unlock; + } + + hdr = (u32 *)((uintptr_t )rb->data + (uintptr_t )(cons_pos & rb->mask)); +/* Synchronizes with smp_store_release() in user-space producer. */ + hdr_len =smp_load_acquire (hdr); + flags = hdr_len & (BPF_RINGBUF_BUSY_BIT |BPF_RINGBUF_DISCARD_BIT ); + sample_len = hdr_len & ~flags; + total_len =round_up (sample_len +BPF_RINGBUF_HDR_SZ ,8 ); + +/* The sample must fit within the region advertised by the producer position. */ +if (total_len > prod_pos - cons_pos) { + err = -EINVAL ; + goto err_unlock; + } + +/* The sample must fit within the data region of the ring buffer. */ +if (total_len >ringbuf_total_data_sz (rb)) { + err = -E2BIG ; + goto err_unlock; + } + +/* The sample must fit into a struct bpf_dynptr. */ + err =bpf_dynptr_check_size (sample_len); +if (err) { + err = -E2BIG ; + goto err_unlock; + } + +if (flags &BPF_RINGBUF_DISCARD_BIT ) { +/* If the discard bit is set, the sample should be skipped. +* +* Update the consumer pos, and return -EAGAIN so the caller +* knows to skip this sample and try to read the next one. +*/ +smp_store_release (&rb->consumer_pos, cons_pos + total_len); + err = -EAGAIN ; + goto err_unlock; + } + +if (flags &BPF_RINGBUF_BUSY_BIT ) { + err = -ENOSPC ; + goto err_unlock; + } + + *sample = (void *)((uintptr_t )rb->data + + (uintptr_t )((cons_pos +BPF_RINGBUF_HDR_SZ ) & rb->mask)); + *size = sample_len; +return 0 ; + +err_unlock: +atomic_set (&rb->busy,0 ); +return err; +} + +static void __bpf_user_ringbuf_sample_release (struct bpf_ringbuf *rb,size_t size,u64 flags) +{ +u64 consumer_pos; +u32 rounded_size =round_up (size +BPF_RINGBUF_HDR_SZ ,8 ); + +/* Using smp_load_acquire() is unnecessary here, as the busy-bit +* prevents another task from writing to consumer_pos after it was read +* by this task with smp_load_acquire() in __bpf_user_ringbuf_peek(). +*/ + consumer_pos = rb->consumer_pos; +/* Synchronizes with smp_load_acquire() in user-space producer. */
kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 739
Quoted 10 lines
kernel/bpf/ringbuf.c +10
-0+smp_store_release (&rb->consumer_pos, consumer_pos + rounded_size); + +/* Prevent the clearing of the busy-bit from being reordered before the +* storing of the updated rb->consumer_pos value. +*/ +smp_mb__before_atomic (); +atomic_set (&rb->busy,0 ); + +if (flags &BPF_RB_FORCE_WAKEUP ) +irq_work_queue (&rb->work);
I think this part is new, you decided to define that FORCE_WAKEUP sends wakeup after every single consumed sample? I have no strong opinion on this, tbh, just wonder if it wasn't enough to do it once after drain?
Quoted 11 lines
kernel/bpf/ringbuf.c +11
-0+} + +BPF_CALL_4(bpf_user_ringbuf_drain, struct bpf_map *, map, + void *, callback_fn, void *, callback_ctx, u64, flags) +{ +struct bpf_ringbuf *rb; +long samples, discarded_samples =0 , ret =0 ; +bpf_callback_t callback = (bpf_callback_t )callback_fn; +u64 wakeup_flags =BPF_RB_NO_WAKEUP |BPF_RB_FORCE_WAKEUP ; + +if (unlikely (flags & ~wakeup_flags))
kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 760
Quoted 10 lines
kernel/bpf/ringbuf.c +10
-0+return -EINVAL ; + + rb = container_of(map, struct bpf_ringbuf_map, map)->rb; +for (samples =0 ; samples <BPF_MAX_USER_RINGBUF_SAMPLES && ret ==0 ; samples++) { +int err; +u32 size; +void *sample; +struct bpf_dynptr_kern dynptr; + + err =__bpf_user_ringbuf_peek (rb, &sample, &size);
so I also just realized that ringbuf_peek will keep setting/resetting busy flag, and in like all the practical case it's a completely useless work as we don't intend to have competing consumers, right? So maybe move busy bit handling into drain itself and document that peek expect busy taken care of?
This should be noticeable faster when there are multiple records consumed in one drain.
Quoted 12 lines
kernel/bpf/ringbuf.c +12
-0+if (err) { +if (err == -ENOSPC ) { +break ; + }else if (err == -EAGAIN ) { + discarded_samples++; +continue ; + }else { + ret = err; + goto schedule_work_return; + } + } +
[...]
Quoted 14 lines
static int
prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx){ struct bpf_verifier_state *state = env->cur_state;kernel/bpf/verifier.c +2
-1@@ -7343,12 +7386,18 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insncase BPF_FUNC_dynptr_data:for (i =0 ; i <MAX_BPF_FUNC_REG_ARGS ; i++) {if (arg_type_is_dynptr (fn->arg_type[i])) { +struct bpf_reg_state *reg = ®s[BPF_REG_1 + i]; +if (meta.ref_obj_id) {verbose (env,"verifier internal error: meta.ref_obj_id already set\n" );return -EFAULT ; } -/* Find the id of the dynptr we're tracking the reference of */
kernel/bpf/verifier.c static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · line 7403
Quoted 10 lines
kernel/bpf/verifier.c +6
-1- meta.ref_obj_id =stack_slot_get_id (env, ®s[BPF_REG_1 + i]); + +if (base_type (reg->type) !=PTR_TO_DYNPTR ) +/* Find the id of the dynptr we're +* tracking the reference of +*/ + meta.ref_obj_id =stack_slot_get_id (env, reg);break ; } }
Joanne, I'd appreciate it if you could take a look at dynptr bits and ack if it looks good to you, thanks!
Quoted 11 lines
kernel/bpf/verifier.c +4
-0@@ -7357,6 +7406,10 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnreturn -EFAULT ; }break ; +case BPF_FUNC_user_ringbuf_drain: + err =__check_func_call (env, insn, insn_idx_p, meta.subprogno, + set_user_ringbuf_callback_state); +break ; }if (err)
[...]
Re: [PATCH v5 2/4] bpf: Add bpf_user_ringbuf_drain() helper
Quoted 49 lines · Andrii Nakryiko
Quoted 37 lines · David Vernet
In a prior change, we added a new
BPF_MAP_TYPE_USER_RINGBUFmap type which will allow user-space applications to publish messages to a ring buffer that is consumed by a BPF program in kernel-space. In order for this map-type to be useful, it will require a BPF helper function that BPF programs can invoke to drain samples from the ring buffer, and invoke callbacks on those samples. This change adds that capability via a new BPF helper function:bpf_user_ringbuf_drain(struct
bpf_map*map, void*callback_fn, void *ctx,u64 flags)BPF programs may invoke this function to run
callback_fn()on a series of samples in the ring buffer.callback_fn()has the following signature:long
callback_fn(struct bpf_dynptr *dynptr, void *context);Samples are provided to the callback in the form of struct
bpf_dynptr*'s, which the program can read using BPF helper functions for querying struct bpf_dynptr's.In order to support
bpf_ringbuf_drain(), a newPTR_TO_DYNPTRregister type is added to the verifier to reflect a dynptr that was allocated by a helper function and passed to a BPF program. UnlikePTR_TO_STACKdynptrs which are allocated on the stack by a BPF program,PTR_TO_DYNPTRdynptrs need not use reference tracking, as the BPF helper is trusted to properly free the dynptr before returning. The verifier currently only supportsPTR_TO_DYNPTRregisters that are alsoDYNPTR_TYPE_LOCAL.Note that while the corresponding user-space libbpf logic will be added in a subsequent patch, this patch does contain an implementation of the .map_poll() callback for
BPF_MAP_TYPE_USER_RINGBUFmaps. This .map_poll() callback guarantees that an epoll-waiting user-space producer will receive at least one event notification whenever at least one sample is drained in an invocation ofbpf_user_ringbuf_drain(), provided that the function is not invoked with theBPF_RB_NO_WAKEUPflag.Quoted 10 lines
Sending an event notification for every sample is not an option, as it could cause the system to hang due to invoking
irq_work_queue()in too-frequent succession. So as to try and optimize for the common case, however,bpf_user_ringbuf_drain()will also send an event notification whenever a sample being drained causes the ring buffer to no longer be full. This heuristic may not help some user-space producers, as a producer can publish samples of varying size, and there may not be enough space in the ring buffer after the first sample is drained which causes it to no longer be full. In this case, the producer may have to wait untilbpf_ringbuf_drain()returns to receive an event notification.
Quoted 1 line
most of this paragraph should be removed now?
Sorry, good catch. Will remove in v6.
Quoted 2 lines
I mentioned few things I didn't see last time (or that were added), but overall looks good to me.
Hmm, let me know if you remember what this is. After reading through this, I can't think of anything else that should be added in this commit summary.
[...]
Quoted 4 lines
Quoted 4 lines
kernel/bpf/ringbuf.c +4
-0+static __poll_t ringbuf_map_poll_user (struct bpf_map *map,struct file *filp, +struct poll_table_struct *pts) +{ +struct bpf_ringbuf_map *rb_map;
Quoted 8 lines
kernel/bpf/ringbuf.c static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp, · line 335
Quoted 6 lines
kernel/bpf/ringbuf.c +6
-0+ + rb_map = container_of(map, struct bpf_ringbuf_map, map); +poll_wait (filp, &rb_map->rb->waitq, pts); + +if (ringbuf_avail_data_sz (rb_map->rb) <ringbuf_total_data_sz (rb_map->rb)) +return EPOLLOUT |EPOLLWRNORM ;nit: extra space after return
Ack.
[...]
Quoted 17 lines
Quoted 17 lines
kernel/bpf/ringbuf.c +17
-0+static int __bpf_user_ringbuf_peek (struct bpf_ringbuf *rb,void **sample,u32 *size) +{ +int err, busy =0 ; +u32 hdr_len, sample_len, total_len, flags, *hdr; +u64 cons_pos, prod_pos; + +/* If another consumer is already consuming a sample, wait for them to finish. */ +if (!atomic_try_cmpxchg (&rb->busy, &busy,1 )) +return -EBUSY ; + +/* Synchronizes with smp_store_release() in user-space producer. */ + prod_pos =smp_load_acquire (&rb->producer_pos); +if (prod_pos %8 ) { + err = -EINVAL ; + goto err_unlock; + } +
Quoted 8 lines
kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 663
Quoted 4 lines
kernel/bpf/ringbuf.c +4
-0+/* Synchronizes with smp_store_release() in __bpf_user_ringbuf_sample_release() */ + cons_pos =smp_load_acquire (&rb->consumer_pos); +if (cons_pos >= prod_pos) { + err = -ENOSPC ;funny, this one actually feels like
-ENODATA(there is no data to peek at, right?). It's different from ENOSPC, as we are not trying to enqueue anything. I probably missed it before?
Ah, yeah, that makes sense. I think the value should be ENODATA for a BPF program trying to read samples when the ring buffer is empty, and ENOSPC for a user space program trying to post a sample to a ring buffer that doesn't have enough room. I'll update this to be the case in v6.
[...]
Quoted 18 lines
Quoted 18 lines
kernel/bpf/ringbuf.c +18
-0+static void __bpf_user_ringbuf_sample_release (struct bpf_ringbuf *rb,size_t size,u64 flags) +{ +u64 consumer_pos; +u32 rounded_size =round_up (size +BPF_RINGBUF_HDR_SZ ,8 ); + +/* Using smp_load_acquire() is unnecessary here, as the busy-bit +* prevents another task from writing to consumer_pos after it was read +* by this task with smp_load_acquire() in __bpf_user_ringbuf_peek(). +*/ + consumer_pos = rb->consumer_pos; +/* Synchronizes with smp_load_acquire() in user-space producer. */ +smp_store_release (&rb->consumer_pos, consumer_pos + rounded_size); + +/* Prevent the clearing of the busy-bit from being reordered before the +* storing of the updated rb->consumer_pos value. +*/ +smp_mb__before_atomic (); +atomic_set (&rb->busy,0 );
Quoted 8 lines
kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 739
Quoted 3 lines
kernel/bpf/ringbuf.c +3
-0+ +if (flags &BPF_RB_FORCE_WAKEUP ) +irq_work_queue (&rb->work);I think this part is new, you decided to define that
FORCE_WAKEUPsends wakeup after every single consumed sample? I have no strong opinion on this, tbh, just wonder if it wasn't enough to do it once after drain?
I didn't have a strong reason for doing this other than that I think it more closely matches the behavior for BPF_MAP_TYPE_RINGBUF (which invokes irq_work_queue() after every call to bpf_ringbuf_commit() if BPF_RB_FORCE_WAKEUP is passed). Let's just match that behavior unless we have a good reason not to? I think that will be more intuitive for users.
Quoted 28 lines
Quoted 11 lines
kernel/bpf/ringbuf.c +11
-0+} + +BPF_CALL_4(bpf_user_ringbuf_drain, struct bpf_map *, map, + void *, callback_fn, void *, callback_ctx, u64, flags) +{ +struct bpf_ringbuf *rb; +long samples, discarded_samples =0 , ret =0 ; +bpf_callback_t callback = (bpf_callback_t )callback_fn; +u64 wakeup_flags =BPF_RB_NO_WAKEUP |BPF_RB_FORCE_WAKEUP ; + +if (unlikely (flags & ~wakeup_flags))kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 760
Quoted 10 lines
kernel/bpf/ringbuf.c +10
-0+return -EINVAL ; + + rb = container_of(map, struct bpf_ringbuf_map, map)->rb; +for (samples =0 ; samples <BPF_MAX_USER_RINGBUF_SAMPLES && ret ==0 ; samples++) { +int err; +u32 size; +void *sample; +struct bpf_dynptr_kern dynptr; + + err =__bpf_user_ringbuf_peek (rb, &sample, &size);so I also just realized that
ringbuf_peekwill keep setting/resetting busy flag, and in like all the practical case it's a completely useless work as we don't intend to have competing consumers, right? So maybe move busy bit handling into drain itself and document that peek expect busy taken care of?
Quoted 2 lines
This should be noticeable faster when there are multiple records consumed in one drain.
Great idea, I'll do this in v6.
Re: [PATCH v5 2/4] bpf: Add bpf_user_ringbuf_drain() helper
Quoted 27 lines · David Vernet
Quoted 18 lines
Quoted 18 lines
kernel/bpf/ringbuf.c +18
-0+static void __bpf_user_ringbuf_sample_release (struct bpf_ringbuf *rb,size_t size,u64 flags) +{ +u64 consumer_pos; +u32 rounded_size =round_up (size +BPF_RINGBUF_HDR_SZ ,8 ); + +/* Using smp_load_acquire() is unnecessary here, as the busy-bit +* prevents another task from writing to consumer_pos after it was read +* by this task with smp_load_acquire() in __bpf_user_ringbuf_peek(). +*/ + consumer_pos = rb->consumer_pos; +/* Synchronizes with smp_load_acquire() in user-space producer. */ +smp_store_release (&rb->consumer_pos, consumer_pos + rounded_size); + +/* Prevent the clearing of the busy-bit from being reordered before the +* storing of the updated rb->consumer_pos value. +*/ +smp_mb__before_atomic (); +atomic_set (&rb->busy,0 );Quoted 8 lines
kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 739
Quoted 3 lines
kernel/bpf/ringbuf.c +3
-0+ +if (flags &BPF_RB_FORCE_WAKEUP ) +irq_work_queue (&rb->work);I think this part is new, you decided to define that
FORCE_WAKEUPsends wakeup after every single consumed sample? I have no strong opinion on this, tbh, just wonder if it wasn't enough to do it once after drain?
Quoted 5 lines
I didn't have a strong reason for doing this other than that I think it more closely matches the behavior for
BPF_MAP_TYPE_RINGBUF(which invokesirq_work_queue()after every call tobpf_ringbuf_commit()ifBPF_RB_FORCE_WAKEUPis passed). Let's just match that behavior unless we have a good reason not to? I think that will be more intuitive for users.
Hmm, something else to consider is that if we move the busy-bit setting into bpf_user_ringbuf_drain() per your suggestion below, the critical section is now the the whole sample drain loop. That's of course _not_ the case for BPF_MAP_TYPE_RINGBUF, which just holds the spinlock while reserving the sample. It seems excessive to invoke irq_work_queue() while the busy bit is held, so I think we should just have the behavior be to only have BPF_RB_FORCE_WAKEUP imply that a wakeup will always be sent, even if no sample was drained.
Let me know if you disagree, but for now I'll work on spinning up a v6 that only issues the forced wakeup event once after drain.
Quoted 28 lines
Quoted 28 lines
Quoted 11 lines
kernel/bpf/ringbuf.c +11
-0+} + +BPF_CALL_4(bpf_user_ringbuf_drain, struct bpf_map *, map, + void *, callback_fn, void *, callback_ctx, u64, flags) +{ +struct bpf_ringbuf *rb; +long samples, discarded_samples =0 , ret =0 ; +bpf_callback_t callback = (bpf_callback_t )callback_fn; +u64 wakeup_flags =BPF_RB_NO_WAKEUP |BPF_RB_FORCE_WAKEUP ; + +if (unlikely (flags & ~wakeup_flags))kernel/bpf/ringbuf.c const struct bpf_func_proto bpf_ringbuf_discard_dynptr_proto = { · line 760
Quoted 10 lines
kernel/bpf/ringbuf.c +10
-0+return -EINVAL ; + + rb = container_of(map, struct bpf_ringbuf_map, map)->rb; +for (samples =0 ; samples <BPF_MAX_USER_RINGBUF_SAMPLES && ret ==0 ; samples++) { +int err; +u32 size; +void *sample; +struct bpf_dynptr_kern dynptr; + + err =__bpf_user_ringbuf_peek (rb, &sample, &size);so I also just realized that
ringbuf_peekwill keep setting/resetting busy flag, and in like all the practical case it's a completely useless work as we don't intend to have competing consumers, right? So maybe move busy bit handling into drain itself and document that peek expect busy taken care of?
Quoted 4 lines
Quoted 2 lines
This should be noticeable faster when there are multiple records consumed in one drain.
Great idea, I'll do this in v6.
Thanks,
David
PATCH v5 3/4 bpf: Add libbpf logic for user-space ring buffer
In reply to David VernetNow that all of the logic is in place in the kernel to support user-space produced ring buffers, we can add the user-space logic to libbpf. This patch therefore adds the following public symbols to libbpf:
struct user_ring_buffer *
user_ring_buffer__new(int map_fd,
const struct user_ring_buffer_opts *opts);
void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size);
void *user_ring_buffer__reserve_blocking(struct user_ring_buffer *rb,
__u32 size, int timeout_ms);void user_ring_buffer__submit(struct user_ring_buffer *rb, void *sample); void user_ring_buffer__discard(struct user_ring_buffer *rb, void user_ring_buffer__free(struct user_ring_buffer *rb);
A user-space producer must first create a struct user_ring_buffer * object with user_ring_buffer__new(), and can then reserve samples in the ring buffer using one of the following two symbols:
void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size); void *user_ring_buffer__reserve_blocking(struct user_ring_buffer *rb,
__u32 size, int timeout_ms);
With user_ring_buffer__reserve(), a pointer to a 'size' region of the ring buffer will be returned if sufficient space is available in the buffer. user_ring_buffer__reserve_blocking() provides similar semantics, but will block for up to 'timeout_ms' in epoll_wait if there is insufficient space in the buffer. This function has the guarantee from the kernel that it will receive at least one event-notification per invocation to bpf_ringbuf_drain(), provided that at least one sample is drained, and the BPF program did not pass the BPF_RB_NO_WAKEUP flag to bpf_ringbuf_drain().
Once a sample is reserved, it must either be committed to the ring buffer with user_ring_buffer__submit(), or discarded with user_ring_buffer__discard().
--- tools/lib/bpf/libbpf.c | 10 +- tools/lib/bpf/libbpf.h | 105 +++++++++++++ tools/lib/bpf/libbpf.map | 10 ++ tools/lib/bpf/libbpf_probes.c | 1 + tools/lib/bpf/libbpf_version.h | 2 +- tools/lib/bpf/ringbuf.c | 270 +++++++++++++++++++++++++++++++++ 6 files changed, 395 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +8 -2
@@ -2373,6 +2373,12 @@ static size_t adjust_ringbuf_sz(size_t sz)return sz; } +static bool map_is_ringbuf (const struct bpf_map *map) +{ +return map->def.type ==BPF_MAP_TYPE_RINGBUF || + map->def.type ==BPF_MAP_TYPE_USER_RINGBUF ; +} +static void fill_map_from_def (struct bpf_map *map,const struct btf_map_def *def) { map->def.type = def->map_type;
@@ -2387,7 +2393,7 @@ static void fill_map_from_def(struct bpf_map *map, const struct btf_map_def *def map->btf_value_type_id = def->value_type_id;/* auto-adjust BPF ringbuf map max_entries to be a multiple of page size */ -if (map->def.type ==BPF_MAP_TYPE_RINGBUF ) +if (map_is_ringbuf (map)) map->def.max_entries =adjust_ringbuf_sz (map->def.max_entries);if (def->parts &MAP_DEF_MAP_TYPE )
@@ -4370,7 +4376,7 @@ int bpf_map__set_max_entries(struct bpf_map *map, __u32 max_entries) map->def.max_entries = max_entries;/* auto-adjust BPF ringbuf map max_entries to be a multiple of page size */ -if (map->def.type ==BPF_MAP_TYPE_RINGBUF ) +if (map_is_ringbuf (map)) map->def.max_entries =adjust_ringbuf_sz (map->def.max_entries);return 0 ;
tools/lib/bpf/libbpf.h +105 -0
117 lines not on this page · raw message
tools/lib/bpf/libbpf.map +10 -0
@@ -368,3 +368,13 @@ LIBBPF_1.0.0 { libbpf_bpf_prog_type_str; perf_buffer__buffer; }; + +LIBBPF_1.1.0 { + global: + user_ring_buffer__discard; + user_ring_buffer__free; + user_ring_buffer__new; + user_ring_buffer__reserve; + user_ring_buffer__reserve_blocking; + user_ring_buffer__submit; +} LIBBPF_1.0.0;
tools/lib/bpf/libbpf_probes.c +1 -0
@@ -231,6 +231,7 @@ static int probe_map_create(enum bpf_map_type map_type)return btf_fd;break ;case BPF_MAP_TYPE_RINGBUF : +case BPF_MAP_TYPE_USER_RINGBUF : key_size =0 ; value_size =0 ; max_entries =4096 ;
tools/lib/bpf/libbpf_version.h +1 -1
@@ -4,6 +4,6 @@#define __LIBBPF_VERSION_H#define LIBBPF_MAJOR_VERSION 1 -#define LIBBPF_MINOR_VERSION 0 +#define LIBBPF_MINOR_VERSION 1#endif /* __LIBBPF_VERSION_H */
tools/lib/bpf/ringbuf.c +270 -0
285 lines not on this page · raw message
Signature · 1 line
2.37.1
Re: [PATCH v5 3/4] bpf: Add libbpf logic for user-space ring buffer
Quoted 37 lines · David Vernet
Now that all of the logic is in place in the kernel to support user-space produced ring buffers, we can add the user-space logic to libbpf. This patch therefore adds the following public symbols to libbpf:
struct
user_ring_buffer*
user_ring_buffer__new(intmap_fd,const struct user_ring_buffer_opts *opts); void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size); void *user_ring_buffer__reserve_blocking(struct user_ring_buffer *rb, __u32 size, int timeout_ms);void
user_ring_buffer__submit(struct user_ring_buffer *rb, void *sample); void user_ring_buffer__discard(structuser_ring_buffer*rb, voiduser_ring_buffer__free(struct user_ring_buffer *rb);A user-space producer must first create a struct
user_ring_buffer* object withuser_ring_buffer__new(), and can then reserve samples in the ring buffer using one of the following two symbols:void *user_ring_buffer__reserve(struct
user_ring_buffer*rb,__u32size); void *user_ring_buffer__reserve_blocking(structuser_ring_buffer*rb,__u32 size, int timeout_ms);With
user_ring_buffer__reserve(), a pointer to a 'size' region of the ring buffer will be returned if sufficient space is available in the buffer.user_ring_buffer__reserve_blocking()provides similar semantics, but will block for up to 'timeout_ms' inepoll_waitif there is insufficient space in the buffer. This function has the guarantee from the kernel that it will receive at least one event-notification per invocation tobpf_ringbuf_drain(), provided that at least one sample is drained, and the BPF program did not pass theBPF_RB_NO_WAKEUPflag tobpf_ringbuf_drain().Once a sample is reserved, it must either be committed to the ring buffer with
user_ring_buffer__submit(), or discarded withuser_ring_buffer__discard().Signed-off-by David Vernet <[email protected]>
Quoted 9 lines
--- tools/lib/bpf/libbpf.c | 10 +- tools/lib/bpf/libbpf.h | 105 +++++++++++++ tools/lib/bpf/libbpf.map | 10 ++ tools/lib/bpf/libbpf_probes.c | 1 + tools/lib/bpf/libbpf_version.h | 2 +- tools/lib/bpf/ringbuf.c | 270 +++++++++++++++++++++++++++++++++ 6 files changed, 395 insertions(+), 3 deletions(-)
Thanks for adding nice doc comments! See below for few minor nits, but ns_elapsed issue I think is pretty bad and should be fixed.
Quoted 12 lines
tools/lib/bpf/libbpf.h +1
-0@@ -1011,6 +1011,7 @@ LIBBPF_API int bpf_tc_query(const struct bpf_tc_hook *hook,/* Ring buffer APIs */ struct ring_buffer ; +struct user_ring_buffer ;typedef int (*ring_buffer_sample_fn )(void *ctx,void *data,size_t size);@@ -1030,6 +1031,110 @@ LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb); LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);
tools/lib/bpf/libbpf.h LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); · line 1043
Quoted 10 lines
tools/lib/bpf/libbpf.h +10
-0+struct user_ring_buffer_opts { +size_t sz;/* size of this struct, for forward/backward compatibility */ +}; + +#define user_ring_buffer_opts__last_field sz + +/* @brief **user_ring_buffer__new()** creates a new instance of a user ring +* buffer. +* +* @param map_fd A file descriptor to a BPF_MAP_TYPE_RINGBUF map.
typo: USER_RINGBUF
Quoted 10 lines
tools/lib/bpf/libbpf.h +10
-0+* @param opts Options for how the ring buffer should be created. +* @return A user ring buffer on success; NULL and errno being set on a +* failure. +*/ +LIBBPF_API struct user_ring_buffer * +user_ring_buffer__new (int map_fd,const struct user_ring_buffer_opts *opts); + +/* @brief **user_ring_buffer__reserve()** reserves a pointer to a sample in the +* user ring buffer. +* @param rb A pointer to a user ring buffer.
tools/lib/bpf/libbpf.h LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); · line 1063
Quoted 10 lines
tools/lib/bpf/libbpf.h +10
-0+* @param size The size of the sample, in bytes. +* @return A pointer to a reserved region of the user ring buffer; NULL, and +* errno being set if a sample could not be reserved. +* +* This function is *not* thread safe, and callers must synchronize accessing +* this function if there are multiple producers. If a size is requested that +* is larger than the size of the entire ring buffer, errno will be set to +* E2BIG and NULL is returned. If the ring buffer could accommodate the size, +* but currently does not have enough space, errno is set to ENOSPC and NULL is +* returned.
we might want to mention that returned pointer is 8-byte aligned
Quoted 7 lines
tools/lib/bpf/libbpf.h +7
-0+* +* After initializing the sample, callers must invoke +* **user_ring_buffer__submit()** to post the sample to the kernel. Otherwise, +* the sample must be freed with **user_ring_buffer__discard()**. +*/ +LIBBPF_API void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size); +
[...]
tools/lib/bpf/ringbuf.c int ring_buffer__epoll_fd(const struct ring_buffer *rb) · line 443
Quoted 12 lines
tools/lib/bpf/ringbuf.c +12
-0+ err =user_ringbuf_map (rb, map_fd); +if (err) + goto err_out; + +return rb; + +err_out: +user_ring_buffer__free (rb); +return errno = -err,NULL ; +} + +static void user_ringbuf__commit (struct user_ring_buffer *rb,void *sample,bool discard)
small nit if you are going to resubmit: we stopped using double underscore naming for internal static functions, so this should be called user_ringbuf_commit
Quoted 5 lines
tools/lib/bpf/ringbuf.c +5
-0+{ +__u32 new_len; +struct ringbuf_hdr *hdr; +uintptr_t hdr_offset; +
[...]
Quoted 19 lines
tools/lib/bpf/ringbuf.c +19
-0+void *user_ring_buffer__reserve_blocking (struct user_ring_buffer *rb,__u32 size,int timeout_ms) +{ +int err; +struct timespec start; +__u64 ns_per_ms =1000000 , ns_elapsed =0 , timeout_ns; + +if (timeout_ms <0 && timeout_ms !=-1 ) +return errno =EINVAL ,NULL ; + +if (timeout_ms !=-1 ) { + err =clock_gettime (CLOCK_MONOTONIC , &start); +if (err) +return NULL ; + } + + timeout_ns = timeout_ms * ns_per_ms; + do { + __u64 ns_remaining = timeout_ns - ns_elapsed; + int cnt, ms_remaining;
tools/lib/bpf/ringbuf.c int ring_buffer__epoll_fd(const struct ring_buffer *rb) · line 547
Quoted 10 lines
tools/lib/bpf/ringbuf.c +10
-0+void *sample; +struct timespec curr; + + sample =user_ring_buffer__reserve (rb, size); +if (sample) +return sample; +else if (errno !=ENOSPC ) +return NULL ; + + ms_remaining = timeout_ms ==-1 ?-1 : ns_remaining / ns_per_ms;
ok, so you've special-cased timeout_ms == -1 but still didn't do max(0, ns_remaining). Can you prove that ns_elapsed will never be bigger than timeout_ns due to various delays in waking up this thread? If not, let's please have max(0) otherwise we can accidentally epoll_wait(-1).
Quoted 10 lines
tools/lib/bpf/ringbuf.c +10
-0+/* The kernel guarantees at least one event notification +* delivery whenever at least one sample is drained from the +* ring buffer in an invocation to bpf_ringbuf_drain(). Other +* additional events may be delivered at any time, but only one +* event is guaranteed per bpf_ringbuf_drain() invocation, +* provided that a sample is drained, and the BPF program did +* not pass BPF_RB_NO_WAKEUP to bpf_ringbuf_drain(). +*/ + cnt =epoll_wait (rb->epoll_fd, &rb->event,1 , ms_remaining); +if (cnt <0 )
tools/lib/bpf/ringbuf.c int ring_buffer__epoll_fd(const struct ring_buffer *rb) · line 567
Quoted 10 lines
tools/lib/bpf/ringbuf.c +10
-0+return NULL ; + +if (timeout_ms ==-1 ) +continue ; + + err =clock_gettime (CLOCK_MONOTONIC , &curr); +if (err) +return NULL ; + + ns_elapsed =ns_elapsed_timespec (&start, &curr);
nit: if you move re-calculation of ms_remaining and ns_remaining to here, I think overall loop logic will be even more straightforwad. You can initialize ms_remaining to -1 if timeout_ms < 0 and never recalculate it, right? Note that you can also do ns_elapsed conversion to ms right here and then keep everything else in ms (so no need for timeout_ns, ns_remaining, etc).
Quoted 8 lines
tools/lib/bpf/ringbuf.c +5
-1+ }while (ns_elapsed <= timeout_ns); + + errno =ENOSPC ; +return NULL ; +} --2.37.1
Re: [PATCH v5 3/4] bpf: Add libbpf logic for user-space ring buffer
Quoted 12 lines · Andrii Nakryiko
Quoted 12 lines
tools/lib/bpf/libbpf.h +1
-0@@ -1011,6 +1011,7 @@ LIBBPF_API int bpf_tc_query(const struct bpf_tc_hook *hook,/* Ring buffer APIs */ struct ring_buffer ; +struct user_ring_buffer ;typedef int (*ring_buffer_sample_fn )(void *ctx,void *data,size_t size);@@ -1030,6 +1031,110 @@ LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb); LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);
Quoted 12 lines
tools/lib/bpf/libbpf.h LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); · line 1043
Quoted 10 lines
tools/lib/bpf/libbpf.h +10
-0+struct user_ring_buffer_opts { +size_t sz;/* size of this struct, for forward/backward compatibility */ +}; + +#define user_ring_buffer_opts__last_field sz + +/* @brief **user_ring_buffer__new()** creates a new instance of a user ring +* buffer. +* +* @param map_fd A file descriptor to a BPF_MAP_TYPE_RINGBUF map. typo:
USER_RINGBUF
Good catch.
Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/libbpf.h +14
-0+* @param opts Options for how the ring buffer should be created. +* @return A user ring buffer on success; NULL and errno being set on a +* failure. +*/ +LIBBPF_API struct user_ring_buffer * +user_ring_buffer__new (int map_fd,const struct user_ring_buffer_opts *opts); + +/* @brief **user_ring_buffer__reserve()** reserves a pointer to a sample in the +* user ring buffer. +* @param rb A pointer to a user ring buffer. +* @param size The size of the sample, in bytes. +* @return A pointer to a reserved region of the user ring buffer; NULL, and +* errno being set if a sample could not be reserved. +*
Quoted 8 lines
tools/lib/bpf/libbpf.h LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); · line 1063
Quoted 6 lines
tools/lib/bpf/libbpf.h +6
-0+* This function is *not* thread safe, and callers must synchronize accessing +* this function if there are multiple producers. If a size is requested that +* is larger than the size of the entire ring buffer, errno will be set to +* E2BIG and NULL is returned. If the ring buffer could accommodate the size, +* but currently does not have enough space, errno is set to ENOSPC and NULL is +* returned. we might want to mention that returned pointer is 8-byte aligned
Will do.
Quoted 10 lines
Quoted 7 lines
tools/lib/bpf/libbpf.h +7
-0+* +* After initializing the sample, callers must invoke +* **user_ring_buffer__submit()** to post the sample to the kernel. Otherwise, +* the sample must be freed with **user_ring_buffer__discard()**. +*/ +LIBBPF_API void *user_ring_buffer__reserve(struct user_ring_buffer *rb, __u32 size); +[...]
Quoted 16 lines
tools/lib/bpf/ringbuf.c int ring_buffer__epoll_fd(const struct ring_buffer *rb) · line 443
Quoted 12 lines
tools/lib/bpf/ringbuf.c +12
-0+ err =user_ringbuf_map (rb, map_fd); +if (err) + goto err_out; + +return rb; + +err_out: +user_ring_buffer__free (rb); +return errno = -err,NULL ; +} + +static void user_ringbuf__commit (struct user_ring_buffer *rb,void *sample,bool discard)small nit if you are going to resubmit: we stopped using double underscore naming for internal static functions, so this should be called
user_ringbuf_commit
Will do.
Quoted 35 lines
Quoted 5 lines
tools/lib/bpf/ringbuf.c +5
-0+{ +__u32 new_len; +struct ringbuf_hdr *hdr; +uintptr_t hdr_offset; +[...]
Quoted 27 lines
tools/lib/bpf/ringbuf.c +27
-0+void *user_ring_buffer__reserve_blocking (struct user_ring_buffer *rb,__u32 size,int timeout_ms) +{ +int err; +struct timespec start; +__u64 ns_per_ms =1000000 , ns_elapsed =0 , timeout_ns; + +if (timeout_ms <0 && timeout_ms !=-1 ) +return errno =EINVAL ,NULL ; + +if (timeout_ms !=-1 ) { + err =clock_gettime (CLOCK_MONOTONIC , &start); +if (err) +return NULL ; + } + + timeout_ns = timeout_ms * ns_per_ms; + do { + __u64 ns_remaining = timeout_ns - ns_elapsed; + int cnt, ms_remaining; + void *sample; + struct timespec curr; + + sample = user_ring_buffer__reserve(rb, size); + if (sample) + return sample; + else if (errno != ENOSPC) + return NULL;
Quoted 8 lines
tools/lib/bpf/ringbuf.c int ring_buffer__epoll_fd(const struct ring_buffer *rb) · line 547
Quoted 2 lines
tools/lib/bpf/ringbuf.c +2
-0+ + ms_remaining = timeout_ms ==-1 ?-1 : ns_remaining / ns_per_ms;ok, so you've special-cased
timeout_ms== -1 but still didn't domax(0, ns_remaining). Can you prove thatns_elapsedwill never be bigger thantimeout_nsdue to various delays in waking up this thread? If not, let's please have max(0) otherwise we can accidentallyepoll_wait(-1).
Yes you're right, this was an oversight. Thanks for catching this!
Quoted 19 lines
Quoted 19 lines
tools/lib/bpf/ringbuf.c +19
-0+/* The kernel guarantees at least one event notification +* delivery whenever at least one sample is drained from the +* ring buffer in an invocation to bpf_ringbuf_drain(). Other +* additional events may be delivered at any time, but only one +* event is guaranteed per bpf_ringbuf_drain() invocation, +* provided that a sample is drained, and the BPF program did +* not pass BPF_RB_NO_WAKEUP to bpf_ringbuf_drain(). +*/ + cnt =epoll_wait (rb->epoll_fd, &rb->event,1 , ms_remaining); +if (cnt <0 ) +return NULL ; + +if (timeout_ms ==-1 ) +continue ; + + err =clock_gettime (CLOCK_MONOTONIC , &curr); +if (err) +return NULL ; +
Quoted 8 lines
Quoted 1 line
+
ns_elapsed=ns_elapsed_timespec(&start, &curr);nit: if you move re-calculation of
ms_remainingandns_remainingto here, I think overall loop logic will be even more straightforwad. You can initializems_remainingto -1 iftimeout_ms< 0 and never recalculate it, right? Note that you can also dons_elapsedconversion to ms right here and then keep everything else in ms (so no need fortimeout_ns,ns_remaining, etc).
Sounds good, let me give this a shot in v6.
Thanks for another detailed review!
Re: [PATCH v5 3/4] bpf: Add libbpf logic for user-space ring buffer
On Mon, Sep 19, 2022 at 03:22:09PM -0500, David Vernet wrote:
[...]
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
tools/lib/bpf/ringbuf.c +12
-0+ timeout_ns = timeout_ms * ns_per_ms; + do { + __u64 ns_remaining = timeout_ns - ns_elapsed; + int cnt, ms_remaining; + void *sample; + struct timespec curr; + + sample = user_ring_buffer__reserve(rb, size); + if (sample) + return sample; + else if (errno != ENOSPC) + return NULL;Quoted 8 lines
tools/lib/bpf/ringbuf.c int ring_buffer__epoll_fd(const struct ring_buffer *rb) · line 547
Quoted 2 lines
tools/lib/bpf/ringbuf.c +2
-0+ + ms_remaining = timeout_ms ==-1 ?-1 : ns_remaining / ns_per_ms;ok, so you've special-cased
timeout_ms== -1 but still didn't domax(0, ns_remaining). Can you prove thatns_elapsedwill never be bigger thantimeout_nsdue to various delays in waking up this thread? If not, let's please have max(0) otherwise we can accidentallyepoll_wait(-1).
Quoted 1 line
Yes you're right, this was an oversight. Thanks for catching this!
Wait, actually, this can't happen because of the while check at the end of the loop:
while (ns_elapsed <= timeout_ns)
So I don't think the max is necessary to prevent underflowing, but I do think we need to have one more attempt to invoke user_ring_buffer__reserve() at the end of the function to account for wakeup delays after epoll_wait() returns. Otherwise, we might think we've timed out when there's actually a sample left in the buffer. I also still think your suggestion below for cleaning up makes sense, so I'll still add it in v6, but I think I can leave off the max() call.
Quoted 32 lines
Quoted 20 lines
Quoted 20 lines
tools/lib/bpf/ringbuf.c +19
-0+/* The kernel guarantees at least one event notification +* delivery whenever at least one sample is drained from the +* ring buffer in an invocation to bpf_ringbuf_drain(). Other +* additional events may be delivered at any time, but only one +* event is guaranteed per bpf_ringbuf_drain() invocation, +* provided that a sample is drained, and the BPF program did +* not pass BPF_RB_NO_WAKEUP to bpf_ringbuf_drain(). +*/ + cnt =epoll_wait (rb->epoll_fd, &rb->event,1 , ms_remaining); +if (cnt <0 ) +return NULL ; + +if (timeout_ms ==-1 ) +continue ; + + err =clock_gettime (CLOCK_MONOTONIC , &curr); +if (err) +return NULL ; +Quoted 8 lines
Quoted 1 line
+
ns_elapsed=ns_elapsed_timespec(&start, &curr);nit: if you move re-calculation of
ms_remainingandns_remainingto here, I think overall loop logic will be even more straightforwad. You can initializems_remainingto -1 iftimeout_ms< 0 and never recalculate it, right? Note that you can also dons_elapsedconversion to ms right here and then keep everything else in ms (so no need fortimeout_ns,ns_remaining, etc).Sounds good, let me give this a shot in v6.
Thanks for another detailed review!
PATCH v5 4/4 selftests/bpf: Add selftests validating the user ringbuf
In reply to David VernetThis change includes selftests that validate the expected behavior and APIs of the new BPF_MAP_TYPE_USER_RINGBUF map type.
--- tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/user_ringbuf.c | 754 ++++++++++++++++++ .../selftests/bpf/progs/test_user_ringbuf.h | 35 + .../selftests/bpf/progs/user_ringbuf_fail.c | 177 ++++ .../bpf/progs/user_ringbuf_success.c | 218 +++++ 5 files changed, 1185 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/user_ringbuf.c create mode 100644 tools/testing/selftests/bpf/progs/test_user_ringbuf.h create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_fail.c create mode 100644 tools/testing/selftests/bpf/progs/user_ringbuf_success.c
tools/testing/selftests/bpf/DENYLIST.s390x +1 -0
@@ -69,3 +69,4 @@ setget_sockopt # attach unexpected error: -524 cb_refs # expected error message unexpected error: -524 (trampoline) cgroup_hierarchical_stats # JIT does not support calling kernel function (kfunc) htab_update # failed to attach: ERROR: strerror_r(-524)=22 (trampoline) +user_ringbuf # failed to find kernel BTF type ID of '__s390x_sys_prctl': -3 (?)
tools/testing/selftests/bpf/prog_tests/user_ringbuf.c new file +754 -0
754 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_user_ringbuf.h new file +35 -0
@@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ + +#ifndef _TEST_USER_RINGBUF_H +#define _TEST_USER_RINGBUF_H + +#define TEST_OP_64 4 +#define TEST_OP_32 2 + +enum test_msg_op { +TEST_MSG_OP_INC64 , +TEST_MSG_OP_INC32 , +TEST_MSG_OP_MUL64 , +TEST_MSG_OP_MUL32 , + +// Must come last. +TEST_MSG_OP_NUM_OPS , +}; + +struct test_msg { +enum test_msg_op msg_op; +union { +__s64 operand_64; +__s32 operand_32; + }; +}; + +struct sample { +int pid; +int seq; +long value; +char comm[16 ]; +}; + +#endif /* _TEST_USER_RINGBUF_H */
tools/testing/selftests/bpf/progs/user_ringbuf_fail.c new file +177 -0
177 lines not on this page · raw message
tools/testing/selftests/bpf/progs/user_ringbuf_success.c new file +218 -0
218 lines not on this page · raw message
Signature · 1 line
2.37.1