libbpf: Fix ring buffer consumption
Subsystems (4)
cover libbpf: Fix ring buffer consumption
Fix several correctness issues in libbpf's ring buffer consumer.
A zero record bound currently consumes one record. A NULL callback is accepted during manager construction but crashes when callback-based consumption reaches the ring. Position counters stop consumption after wrapping because they are compared by magnitude.
The consumer can also miss a readiness notification after publishing its position and checking for new data without a full StoreLoad barrier. Use compiler atomics and add the missing barrier, including when retrying a busy record after publishing earlier records.
Callback traversal does not follow the overwrite position maintained by BPF_F_RB_OVERWRITE maps. Reject callback consumption of those maps, as discussed here:
https://lore.kernel.org/bpf/CAEf4Bzaq5drHWChXoRBnrmkb6reAsSVj8r=uByFSup31FMA7hw@mail.gmail.com/
Andrew Werner found the position-wrap and missed-wakeup failures while implementing Aya's ring buffer reader. Aya's original implementation contains the equality reasoning and edge-triggered regression test:
https://github.com/aya-rs/aya/commit/e2cf734490bc188bcedb1eac92d23d81123e42cd
Aya later corrected the consumer ordering with the same explicit fence:
https://github.com/aya-rs/aya/commit/7277a57ea8cdb74918d3096a4b22b6d814481973
---
Tamir Duberstein (6):
libbpf: ringbuf: Honor zero consume bounds
libbpf: ringbuf: Prevent NULL callback crash
libbpf: ringbuf: Handle position counter wrap
libbpf: ringbuf: Use compiler atomics
libbpf: ringbuf: Prevent missed wakeups
libbpf: ringbuf: Reject overwrite callback use
tools/lib/bpf/libbpf.h | 34 +++- tools/lib/bpf/ringbuf.c | 84 +++++++-- tools/testing/selftests/bpf/prog_tests/ringbuf.c | 229 +++++++++++++++++++++++ 3 files changed, 323 insertions(+), 24 deletions(-)
---
base-commit: e7ae89a0c97ce2b68b0983cd01eda67cf373517d
change-id: 20260613-bpf-ringbuf-fixes-e9a8b3c6125b
Best regards,
Signature · 1 line
Tamir Duberstein <[email protected]>
PATCH 1/6 libbpf: ringbuf: Honor zero consume bounds
ringbuf_process_ring() checks the record bound only after advancing the consumer position and invoking the callback. A zero bound therefore consumes the first available record.
Return before reading the ring positions when the bound is zero so ring_buffer__consume_n() and ring__consume_n() leave all records queued.
--- tools/lib/bpf/ringbuf.c | 3 +++ tools/testing/selftests/bpf/prog_tests/ringbuf.c | 13 +++++++++++++ 2 files changed, 16 insertions(+)
tools/lib/bpf/ringbuf.c +3 -0
@@ -240,6 +240,9 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)bool got_new_data;void *sample; +if (n ==0 ) +return 0 ; + cons_pos =smp_load_acquire (r->consumer_pos); do { got_new_data = false;
tools/testing/selftests/bpf/prog_tests/ringbuf.c +13 -0
@@ -404,6 +404,7 @@ static int process_n_sample(void *ctx, void *data, size_t len)static void ringbuf_n_subtest (void ) {struct test_ringbuf_n_lskel *skel_n; +struct ring *ring;int err, i; skel_n =test_ringbuf_n_lskel__open ();
@@ -431,6 +432,18 @@ static void ringbuf_n_subtest(void)for (i =0 ; i <N_TOT_SAMPLES ; i++)syscall (__NR_getpgid); + ring =ring_buffer__ring (ringbuf,0 ); +if (!ASSERT_OK_PTR (ring,"ring_buffer__ring" )) + goto cleanup_ringbuf; + + err =ring_buffer__consume_n (ringbuf,0 ); +if (!ASSERT_EQ (err,0 ,"ringbuf_consume_zero" )) + goto cleanup_ringbuf; + + err =ring__consume_n (ring,0 ); +if (!ASSERT_EQ (err,0 ,"ring_consume_zero" )) + goto cleanup_ringbuf; +/* Consume all samples from the ring buffer in batches of N_SAMPLES */ for (i =0 ; i <N_TOT_SAMPLES ; i += err) { err =ring_buffer__consume_n (ringbuf,N_SAMPLES );
Signature · 1 line
2.55.0.rc0.96.gc050c23164
PATCH 2/6 libbpf: ringbuf: Prevent NULL callback crash
In reply to Tamir Dubersteinring_buffer__new() and ring_buffer__add() allow a NULL sample callback. When callback-based consumption reaches such a ring, it calls through the NULL function pointer and crashes.
Validate every ring in a manager before polling or consuming. Return -EINVAL without consuming records from an earlier valid ring or waiting for an event. Perform the same check before honoring a zero record bound so invalid callback consumption consistently reports the error.
--- tools/lib/bpf/libbpf.h | 11 ++- tools/lib/bpf/ringbuf.c | 41 +++++++++-- tools/testing/selftests/bpf/prog_tests/ringbuf.c | 93 ++++++++++++++++++++++++ 3 files changed, 134 insertions(+), 11 deletions(-)
tools/lib/bpf/libbpf.h +5 -6
@@ -1526,18 +1526,17 @@ LIBBPF_API int ring__map_fd(const struct ring *r);* * @param r A ringbuffer object. * @return The number of records consumed (or INT_MAX, whichever is less), or -* a negative number if any of the callbacks return an error. +* a negative error code on failure. */ LIBBPF_API int ring__consume(struct ring *r);/** -* @brief **ring__consume_n()** consumes up to a requested amount of items from -* a ringbuffer without event polling. +* @brief **ring__consume_n()** consumes up to a requested number of records +* from a ringbuffer without event polling. * * @param r A ringbuffer object. -* @param n Maximum amount of items to consume. -* @return The number of items consumed, or a negative number if any of the -* callbacks return an error. +* @param n Maximum number of records to consume. +* @return The number of records consumed, or a negative error code on failure. */ LIBBPF_API int ring__consume_n(struct ring *r, size_t n);
tools/lib/bpf/ringbuf.c +36 -5
89 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/ringbuf.c +93 -0
105 lines not on this page · raw message
Signature · 1 line
2.55.0.rc0.96.gc050c23164
PATCH 3/6 libbpf: ringbuf: Handle position counter wrap
In reply to Tamir DubersteinRing buffer positions are unsigned long counters and can wrap on 32-bit systems. ringbuf_process_ring() stops consuming when producer_pos wraps below consumer_pos because it compares the counters by magnitude.
Compare the positions for equality instead. The producer cannot move logically behind the consumer in a non-overwrite ring.
--- tools/lib/bpf/ringbuf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/ringbuf.c +1 -1
@@ -268,7 +268,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) do { got_new_data = false; prod_pos = smp_load_acquire(r->producer_pos); - while (cons_pos < prod_pos) { + while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); len = smp_load_acquire(len_ptr);
Signature · 1 line
2.55.0.rc0.96.gc050c23164
PATCH 4/6 libbpf: ringbuf: Use compiler atomics
In reply to Tamir DubersteinConsumer-side ring buffer code uses architecture-specific smp_* helpers for shared memory accesses.
Use compiler atomics instead. They provide equivalent acquire and release ordering through a portable userspace interface and allow the next commit to use compiler fences in the wakeup protocol without mixing atomic interfaces.
--- tools/lib/bpf/ringbuf.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
tools/lib/bpf/ringbuf.c +11 -9
@@ -264,13 +264,13 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)if (n ==0 )return 0 ; - cons_pos =smp_load_acquire (r->consumer_pos); + cons_pos =__atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); do { got_new_data = false; - prod_pos = smp_load_acquire(r->producer_pos); + prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE); while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); - len = smp_load_acquire(len_ptr); + len = __atomic_load_n(len_ptr, __ATOMIC_ACQUIRE); /* sample not committed yet, bail out for now */ if (len & BPF_RINGBUF_BUSY_BIT)
@@ -284,14 +284,16 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) err = r->sample_cb (r->ctx, sample, len);if (err <0 ) {/* update consumer pos and bail out */ -smp_store_release (r->consumer_pos, - cons_pos); +__atomic_store_n (r->consumer_pos, + cons_pos, + __ATOMIC_RELEASE);return err; } cnt++; } -smp_store_release (r->consumer_pos, cons_pos); +__atomic_store_n (r->consumer_pos, cons_pos, + __ATOMIC_RELEASE);if (cnt >= n) goto done;
@@ -406,8 +408,8 @@ struct ring *ring_buffer__ring(struct ring_buffer *rb, unsigned int idx)unsigned long ring__consumer_pos (const struct ring *r) { -/* Synchronizes with smp_store_release() in ringbuf_process_ring(). */ -return smp_load_acquire (r->consumer_pos); +/* Synchronizes with the release store in ringbuf_process_ring(). */ +return __atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); }unsigned long ring__producer_pos (const struct ring *r)
@@ -415,7 +417,7 @@ unsigned long ring__producer_pos(const struct ring *r)/* Synchronizes with smp_store_release() in __bpf_ringbuf_reserve() in * the kernel. */ -return smp_load_acquire (r->producer_pos); +return __atomic_load_n (r->producer_pos, __ATOMIC_ACQUIRE); }size_t ring__avail_data_size (const struct ring *r)
Signature · 1 line
2.55.0.rc0.96.gc050c23164
PATCH 5/6 libbpf: ringbuf: Prevent missed wakeups
In reply to Tamir DubersteinAfter consuming the last visible record, ringbuf_process_ring() publishes the consumer position and checks the producer position. These operations lack a full StoreLoad barrier. A producer can therefore commit a new record but read the old consumer position while the consumer reads the old producer position. The producer sends no notification and the consumer waits despite a queued record.
Insert a full barrier before checking for new data, ensuring that either the consumer observes the producer update or the producer observes the consumer update and sends a notification. Apply the same handshake when a busy record follows records whose consumer position was published.
Add an edge-triggered epoll test with a concurrent producer. Without the barrier, a missed notification leaves the producer dropping records from a full ring while the consumer times out. Document that bounded consumers and callbacks that terminate consumption must drain before waiting again.
--- tools/lib/bpf/libbpf.h | 22 +++++++ tools/lib/bpf/ringbuf.c | 14 +++- tools/testing/selftests/bpf/prog_tests/ringbuf.c | 84 ++++++++++++++++++++++++ 3 files changed, 117 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.h +22 -0
@@ -1439,6 +1439,10 @@ struct ring_buffer;struct ring ;struct user_ring_buffer ; +/* A negative return stops consumption; non-negative values continue. Stopping +* can leave records queued without a new readiness notification. Before +* waiting for readiness again, consume until no records remain. +*/ typedef int (*ring_buffer_sample_fn )(void *ctx,void *data,size_t size);struct ring_buffer_opts {
@@ -1455,6 +1459,20 @@ LIBBPF_API int ring_buffer__add(struct ring_buffer *rb, int map_fd, ring_buffer_sample_fn sample_cb, void *ctx); LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb); + +/** +* @brief **ring_buffer__consume_n()** consumes up to a requested number of +* records from a ring buffer manager without event polling. +* +* @param rb A ring buffer manager object. +* @param n Maximum number of records to consume. +* @return The number of records consumed, or a negative error code on failure. +* +* Reaching the requested bound does not establish that every ring is empty. +* Records can remain queued without a new readiness notification. Before +* waiting on ring_buffer__epoll_fd(), call ring_buffer__consume() until it +* returns 0. +*/ LIBBPF_API int ring_buffer__consume_n(struct ring_buffer *rb, size_t n); LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);
@@ -1537,6 +1555,10 @@ LIBBPF_API int ring__consume(struct ring *r);* @param r A ringbuffer object. * @param n Maximum number of records to consume. * @return The number of records consumed, or a negative error code on failure. +* +* Reaching the requested bound does not establish that the ring is empty. +* Records can remain queued without a new readiness notification. Before +* waiting on ring__map_fd(), call ring__consume() until it returns 0. */ LIBBPF_API int ring__consume_n(struct ring *r, size_t n);
tools/lib/bpf/ringbuf.c +11 -3
@@ -255,7 +255,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)/* 64-bit to avoid overflow in case of extreme application behavior */ int64_t cnt =0 ;unsigned long cons_pos, prod_pos; -bool got_new_data; +bool got_new_data, needs_wakeup = false;void *sample; err =ringbuf_validate (r);
@@ -267,14 +267,21 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) cons_pos =__atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); do { got_new_data = false; + if (needs_wakeup) { + /* Ensure either this sees a new record or its producer sees + * the updated consumer position and sends a notification. + */ + __atomic_thread_fence(__ATOMIC_SEQ_CST); + needs_wakeup = false; + } prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE); while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); len = __atomic_load_n(len_ptr, __ATOMIC_ACQUIRE); - /* sample not committed yet, bail out for now */ + /* Retry a busy record once after publishing prior records. */ if (len & BPF_RINGBUF_BUSY_BIT) - goto done; + break; got_new_data = true; cons_pos += roundup_len(len);
@@ -294,6 +301,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)__atomic_store_n (r->consumer_pos, cons_pos, __ATOMIC_RELEASE); + needs_wakeup = true;if (cnt >= n) goto done;
tools/testing/selftests/bpf/prog_tests/ringbuf.c +84 -0
96 lines not on this page · raw message
Signature · 1 line
2.55.0.rc0.96.gc050c23164
PATCH 6/6 libbpf: ringbuf: Reject overwrite callback use
In reply to Tamir DubersteinBPF_F_RB_OVERWRITE can advance overwrite_pos past consumer_pos. Callback traversal does not read overwrite_pos, so after the producer laps the consumer it can treat overwritten data as a record header.
An earlier proposal[0] copied the readable window before invoking callbacks. Review concluded that callbacks are a poor fit because copying penalizes zero-copy users and the API cannot report skipped records.
Record the map flag and reject callback consumption with -EOPNOTSUPP.
--- tools/lib/bpf/libbpf.h | 1 + tools/lib/bpf/ringbuf.c | 4 +++ tools/testing/selftests/bpf/prog_tests/ringbuf.c | 39 ++++++++++++++++++++++++ 3 files changed, 44 insertions(+)
tools/lib/bpf/libbpf.h +1 -0
@@ -1439,6 +1439,7 @@ struct ring_buffer;struct ring ;struct user_ring_buffer ; +/* Callback-based consumption is unsupported for BPF_F_RB_OVERWRITE maps. */ /* A negative return stops consumption; non-negative values continue. Stopping * can leave records queued without a new readiness notification. Before * waiting for readiness again, consume until no records remain.
tools/lib/bpf/ringbuf.c +4 -0
@@ -30,6 +30,7 @@ struct ring {unsigned long *producer_pos;unsigned long mask;int map_fd; +bool overwrite; };struct ring_buffer {
@@ -118,6 +119,7 @@ int ring_buffer__add(struct ring_buffer *rb, int map_fd, r->sample_cb = sample_cb; r->ctx = ctx; r->mask = info.max_entries -1 ; + r->overwrite = info.map_flags &BPF_F_RB_OVERWRITE ;/* Map writable consumer page */ tmp =mmap (NULL , rb->page_size,PROT_READ |PROT_WRITE ,MAP_SHARED , map_fd,0 );
@@ -233,6 +235,8 @@ static inline int roundup_len(__u32 len)static int ringbuf_validate (const struct ring *r) { +if (r->overwrite) +return -EOPNOTSUPP ;return r->sample_cb ?0 : -EINVAL ; }
tools/testing/selftests/bpf/prog_tests/ringbuf.c +39 -0
@@ -684,6 +684,43 @@ static void ringbuf_map_key_subtest(void)test_ringbuf_map_key_lskel__destroy (skel_map_key); } +static void ringbuf_overwrite_callback_subtest (void ) +{ + LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_RB_OVERWRITE); +struct ring_buffer *ringbuf; +struct ring *ring; +int map_fd, err; + + map_fd =bpf_map_create (BPF_MAP_TYPE_RINGBUF ,NULL ,0 ,0 ,getpagesize (), + &opts); +if (!ASSERT_OK_FD (map_fd,"bpf_map_create" )) +return ; + + ringbuf =ring_buffer__new (map_fd, process_noop_sample,NULL ,NULL ); +if (!ASSERT_OK_PTR (ringbuf,"ring_buffer__new" )) + goto cleanup_fd; + + ring =ring_buffer__ring (ringbuf,0 ); +if (!ASSERT_OK_PTR (ring,"ring_buffer__ring" )) + goto cleanup_ringbuf; + + err =ring_buffer__consume_n (ringbuf,0 ); +ASSERT_EQ (err, -EOPNOTSUPP ,"ringbuf_consume_zero" ); + err =ring_buffer__consume (ringbuf); +ASSERT_EQ (err, -EOPNOTSUPP ,"ringbuf_consume" ); + err =ring_buffer__poll (ringbuf,0 ); +ASSERT_EQ (err, -EOPNOTSUPP ,"ringbuf_poll" ); + err =ring__consume_n (ring,0 ); +ASSERT_EQ (err, -EOPNOTSUPP ,"ring_consume_zero" ); + err =ring__consume (ring); +ASSERT_EQ (err, -EOPNOTSUPP ,"ring_consume" ); + +cleanup_ringbuf: +ring_buffer__free (ringbuf); +cleanup_fd: +close (map_fd); +} +static void ringbuf_overwrite_mode_subtest (void ) {unsigned long size, len1, len2, len3, len4, len5;
@@ -760,6 +797,8 @@ void test_ringbuf(void)ringbuf_map_key_subtest ();if (test__start_subtest ("ringbuf_write" ))ringbuf_write_subtest (); +if (test__start_subtest ("ringbuf_overwrite_callback" )) +ringbuf_overwrite_callback_subtest ();if (test__start_subtest ("ringbuf_overwrite_mode" ))ringbuf_overwrite_mode_subtest (); }
Signature · 1 line
2.55.0.rc0.96.gc050c23164
[email protected]
AI reviewRe: [PATCH bpf 5/6] libbpf: ringbuf: Prevent missed wakeups
In reply to Tamir DubersteinThank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Early returns in `ringbuf_process_ring()` bypass the newly added StoreLoad memory barrier, leaving…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Early returns in ringbuf_process_ring() bypass the newly added StoreLoad memory barrier, leaving bounded consumers vulnerable to missed wakeups.
- [Low] Multi-line comments introduced by the patch violate the BPF subsystem comment style.
--
commit 1b4f252cfd5df3b60a00d140737f0f9bc3b4ad87
Author: Tamir Duberstein <[email protected]>
Subject: libbpf: ringbuf: Prevent missed wakeups
This commit addresses missed wakeups in the BPF ring buffer by adding a full StoreLoad barrier before checking for new data. It ensures either the consumer observes the producer's update, or the producer observes the consumer's update and sends a notification. It also updates documentation and adds a selftest.
tools/lib/bpf/libbpf.h struct ring_buffer; · line 1445
Quoted 12 lines
tools/lib/bpf/libbpf.h +4
-0@@ -1439,6 +1439,10 @@ struct ring_buffer;struct ring ;struct user_ring_buffer ; +/* A negative return stops consumption; non-negative values continue. Stopping +* can leave records queued without a new readiness notification. Before +* waiting for readiness again, consume until no records remain. +*/
[Severity: Low]
This isn't a bug, but does this multi-line comment follow the BPF subsystem style? The guidelines prefer the opening /* to be on its own line.
[ ... ]
Quoted 4 lines
tools/lib/bpf/ringbuf.c +0
-0
[ ... ]
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 273
Quoted 8 lines
tools/lib/bpf/ringbuf.c +4
-0@@ -267,14 +267,21 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) cons_pos =__atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); do { got_new_data = false; + if (needs_wakeup) { + /* Ensure either this sees a new record or its producer sees + * the updated consumer position and sends a notification. + */
[Severity: Low]
This isn't a bug, but should this multi-line comment have the opening /* on its own line to match the BPF subsystem comment style?
Quoted 5 lines
tools/lib/bpf/ringbuf.c +3
-0+__atomic_thread_fence (__ATOMIC_SEQ_CST); + needs_wakeup = false; + } prod_pos =__atomic_load_n (r->producer_pos, __ATOMIC_ACQUIRE);while (cons_pos != prod_pos) {
[ ... ]
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 307
Quoted 8 lines
tools/lib/bpf/ringbuf.c +1
-0@@ -294,6 +301,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)__atomic_store_n (r->consumer_pos, cons_pos, __ATOMIC_RELEASE); + needs_wakeup = true;if (cnt >= n) goto done;
[Severity: High]
If this loop terminates early by reaching the bound (cnt >= n) or due to a negative callback return, consumer_pos is updated and the function returns, discarding the local needs_wakeup variable.
When ring_buffer__consume() is called again to drain the remaining records before sleeping, it will initialize needs_wakeup = false. This causes the __ATOMIC_SEQ_CST fence to be skipped prior to loading producer_pos.
Could this allow the CPU to delay the visibility of the consumer_pos store while reading a stale producer_pos, resulting in the consumer incorrectly sleeping and the concurrent producer skipping the wakeup notification?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
[email protected]
AI reviewRe: [PATCH bpf 4/6] libbpf: ringbuf: Use compiler atomics
In reply to Tamir DubersteinThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] 32-bit monitoring applications using the BPF ring buffer on a 64-bit kernel will permanently stall after processing 4GB of events due to an unsigned long pointer type mismatch for producer/consumer positions. --
commit b302f5c853774170c0ab9e37b85abcc4d647ac3b
Author: Tamir Duberstein <[email protected]>
libbpf: ringbuf: Use compiler atomics
This commit replaces architecture-specific smp_* helpers with portable compiler atomics for shared memory accesses in the consumer-side ring buffer code.
Quoted 4 lines
tools/lib/bpf/ringbuf.c +0
-0
[ ... ]
Quoted 7 lines
tools/lib/bpf/ringbuf.c +1
-2@@ -284,14 +284,16 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) err = r->sample_cb (r->ctx, sample, len);if (err <0 ) {/* update consumer pos and bail out */ -smp_store_release (r->consumer_pos, - cons_pos); + __atomic_store_n(r->consumer_pos,
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 296
Quoted 10 lines
tools/lib/bpf/ringbuf.c +4
-1+ cons_pos, + __ATOMIC_RELEASE);return err; } cnt++; } -smp_store_release (r->consumer_pos, cons_pos); +__atomic_store_n (r->consumer_pos, cons_pos, + __ATOMIC_RELEASE);
[Severity: High]
This isn't a bug introduced by this patch, but does this write to consumer_pos safely handle a 32-bit userspace application running on a 64-bit kernel?
In the kernel, struct bpf_ringbuf defines consumer_pos and producer_pos as unsigned long, which are 64 bits. However, in libbpf's struct ring, consumer_pos is mapped as an unsigned long pointer, which is only 32 bits in a 32-bit userspace environment.
When the 32-bit user process updates consumer_pos via __atomic_store_n(), it only overwrites the lower 32 bits of the 64-bit value on little-endian systems. Once the true consumer position exceeds 4GB, the userspace 32-bit value wraps to 0. Writing 0 to the lower 32 bits while the upper 32 bits remain 0 causes the kernel to see consumer_pos drop to 0, while the 64-bit producer_pos continues climbing past 4GB.
This causes the kernel's bpf_ringbuf_has_space() to see a delta larger than the ring buffer mask, incorrectly treating the buffer as completely full and permanently dropping all subsequent events for that buffer.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
[email protected]
AI reviewRe: [PATCH bpf 3/6] libbpf: ringbuf: Handle position counter wrap
In reply to Tamir DubersteinThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Pre-existing issues:
- [High] __bpf_ringbuf_reserve in the kernel permanently fails allocations after 32-bit position wrap-around.
- [High] Kernel-side __bpf_user_ringbuf_peek incorrectly handles 32-bit wrap-around, leading to permanent stalls and bypassing bounds checks.
- [High] 32-bit userspace running on a 64-bit kernel permanently breaks BPF ring buffers after 4GB of data is processed due to pointer size mismatches. --
commit ce5e8d811804f9ff598195a7f394512f77c373b0
Author: Tamir Duberstein <[email protected]>
libbpf: ringbuf: Handle position counter wrap
Ring buffer positions are unsigned long counters and can wrap on 32-bit systems. ringbuf_process_ring() stops consuming when producer_pos wraps below consumer_pos because it compares the counters by magnitude.
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 274
Quoted 13 lines
tools/lib/bpf/ringbuf.c +1
-1@@ -268,7 +268,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) do { got_new_data = false; prod_pos = smp_load_acquire(r->producer_pos); - while (cons_pos < prod_pos) { + while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); len = smp_load_acquire(len_ptr);
[Severity: High]
This is a pre-existing issue, but does this fix fully resolve the behavior for 32-bit userspace applications running on 64-bit kernels?
Looking further down in ringbuf_process_ring(), userspace updates the consumer position:
__atomic_store_n(r->consumer_pos, cons_pos, __ATOMIC_RELEASE);
On a 32-bit architecture, this writes 4 bytes. However, the 64-bit kernel defines consumer_pos as an 8-byte unsigned long. When the 32-bit userspace counter wraps at 4GB, it writes 0 to the lower 4 bytes while leaving the upper 4 bytes unchanged.
Could this cause the kernel's bpf_ringbuf_has_space() to perceive an instant drop in the consumer position, making it evaluate the ring buffer as permanently full?
[Severity: High]
This is a pre-existing issue, but does the kernel side have a similar wrap-around issue in __bpf_ringbuf_reserve()?
kernel/bpf/ringbuf.c:__bpf_ringbuf_reserve() {
...
while (pend_pos < prod_pos) {
...
}
...
}Since pend_pos and prod_pos are unsigned longs, on 32-bit kernels, does this magnitude comparison fail when producer_pos wraps around to a small value? It appears pend_pos would freeze at its pre-wrap value, causing bpf_ringbuf_has_space() to permanently return false and deny future allocations.
[Severity: High]
This is also a pre-existing issue, but does __bpf_user_ringbuf_peek() have a related wrap-around bug on 32-bit systems?
kernel/bpf/ringbuf.c:__bpf_user_ringbuf_peek() {
u64 cons_pos, prod_pos;
...
prod_pos = smp_load_acquire(&rb->producer_pos);
...
cons_pos = smp_load_acquire(&rb->consumer_pos);
if (cons_pos >= prod_pos)
...
}Because rb->producer_pos and rb->consumer_pos are 32-bit unsigned longs on 32-bit systems, are they zero-extended to 64-bit u64 variables? If producer_pos wraps around while consumer_pos is still large, does the 64-bit promotion cause the magnitude comparison to evaluate to true, permanently stalling consumption by returning -ENODATA?
Additionally, would the 64-bit subtraction (prod_pos - cons_pos) yield a massive value that defeats the bounds check?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH bpf 1/6] libbpf: ringbuf: Honor zero consume bounds
In reply to Tamir DubersteinQuoted 10 lines · Tamir Duberstein
ringbuf_process_ring()checks the record bound only after advancing the consumer position and invoking the callback. A zero bound therefore consumes the first available record.Return before reading the ring positions when the bound is zero so
ring_buffer__consume_n()andring__consume_n()leave all records queued.Fixes 4d22ea94ea33 ("libbpf: Add ring__consume_n / ring_buffer__consume_n")Assisted-by Codex:gpt-5.5Signed-off-by Tamir Duberstein <[email protected]>
Quoted 50 lines
--- tools/lib/bpf/ringbuf.c | 3 +++ tools/testing/selftests/bpf/prog_tests/ringbuf.c | 13 +++++++++++++ 2 files changed, 16 insertions(+)tools/lib/bpf/ringbuf.c +3
-0@@ -240,6 +240,9 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)bool got_new_data;void *sample; +if (n ==0 ) +return 0 ; + cons_pos =smp_load_acquire (r->consumer_pos); do { got_new_data = false;tools/testing/selftests/bpf/prog_tests/ringbuf.c +13
-0@@ -404,6 +404,7 @@ static int process_n_sample(void *ctx, void *data, size_t len)static void ringbuf_n_subtest (void ) {struct test_ringbuf_n_lskel *skel_n; +struct ring *ring;int err, i; skel_n =test_ringbuf_n_lskel__open ();@@ -431,6 +432,18 @@ static void ringbuf_n_subtest(void)for (i =0 ; i <N_TOT_SAMPLES ; i++)syscall (__NR_getpgid); + ring =ring_buffer__ring (ringbuf,0 ); +if (!ASSERT_OK_PTR (ring,"ring_buffer__ring" )) + goto cleanup_ringbuf; + + err =ring_buffer__consume_n (ringbuf,0 ); +if (!ASSERT_EQ (err,0 ,"ringbuf_consume_zero" )) + goto cleanup_ringbuf; + + err =ring__consume_n (ring,0 ); +if (!ASSERT_EQ (err,0 ,"ring_consume_zero" )) + goto cleanup_ringbuf; +/* Consume all samples from the ring buffer in batches of N_SAMPLES */ for (i =0 ; i <N_TOT_SAMPLES ; i += err) { err =ring_buffer__consume_n (ringbuf,N_SAMPLES );
Emil Tsalapatis
reviewerRe: [PATCH bpf 2/6] libbpf: ringbuf: Prevent NULL callback crash
In reply to Tamir DubersteinQuoted 9 lines · Tamir Duberstein
ring_buffer__new()andring_buffer__add()allow a NULL sample callback. When callback-based consumption reaches such a ring, it calls through the NULL function pointer and crashes.Validate every ring in a manager before polling or consuming. Return
-EINVALwithout consuming records from an earlier valid ring or waiting for an event. Perform the same check before honoring a zero record bound so invalid callback consumption consistently reports the error.
Can we just prevent a ring from being added with a NULL sample_cb? What's the use for permitting it? Even if we don't, rechecking the callbacks every single time we consume the ringbuf seems overkill.
Quoted 246 lines
Fixes bf99c936f947 ("libbpf: Add BPF ring buffer support")Assisted-by Codex:gpt-5.5Signed-off-by Tamir Duberstein <[email protected]>--- tools/lib/bpf/libbpf.h | 11 ++- tools/lib/bpf/ringbuf.c | 41 +++++++++-- tools/testing/selftests/bpf/prog_tests/ringbuf.c | 93 ++++++++++++++++++++++++ 3 files changed, 134 insertions(+), 11 deletions(-)tools/lib/bpf/libbpf.h +5
-6@@ -1526,18 +1526,17 @@ LIBBPF_API int ring__map_fd(const struct ring *r);* * @param r A ringbuffer object. * @return The number of records consumed (or INT_MAX, whichever is less), or -* a negative number if any of the callbacks return an error. +* a negative error code on failure. */ LIBBPF_API int ring__consume(struct ring *r);/** -* @brief **ring__consume_n()** consumes up to a requested amount of items from -* a ringbuffer without event polling. +* @brief **ring__consume_n()** consumes up to a requested number of records +* from a ringbuffer without event polling. * * @param r A ringbuffer object. -* @param n Maximum amount of items to consume. -* @return The number of items consumed, or a negative number if any of the -* callbacks return an error. +* @param n Maximum number of records to consume. +* @return The number of records consumed, or a negative error code on failure. */ LIBBPF_API int ring__consume_n(struct ring *r, size_t n);tools/lib/bpf/ringbuf.c +36
-5@@ -231,6 +231,24 @@ static inline int roundup_len(__u32 len)return (len +7 ) /8 *8 ; } +static int ringbuf_validate (const struct ring *r) +{ +return r->sample_cb ?0 : -EINVAL ; +} + +static int ringbuf_validate_callbacks (const struct ring_buffer *rb) +{ +int i, err; + +for (i =0 ; i < rb->ring_cnt; i++) { + err =ringbuf_validate (rb->rings[i]); +if (err) +return err; + } + +return 0 ; +} +static int64_t ringbuf_process_ring (struct ring *r,size_t n) {int *len_ptr, len, err;@@ -240,6 +258,9 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)bool got_new_data;void *sample; + err =ringbuf_validate (r); +if (err) +return err;if (n ==0 )return 0 ;@@ -284,14 +305,17 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)* records. * * Returns number of records consumed across all registered ring buffers (or -* n, whichever is less), or negative number if any of the callbacks return -* error. +* n, whichever is less), or a negative error code on failure. */ int ring_buffer__consume_n (struct ring_buffer *rb,size_t n) {int64_t err, res =0 ;int i; + err =ringbuf_validate_callbacks (rb); +if (err) +return libbpf_err (err); +for (i =0 ; i < rb->ring_cnt; i++) {struct ring *ring = rb->rings[i];@@ -309,14 +333,17 @@ int ring_buffer__consume_n(struct ring_buffer *rb, size_t n)/* Consume available ring buffer(s) data without event polling. * Returns number of records consumed across all registered ring buffers (or -* INT_MAX, whichever is less), or negative number if any of the callbacks -* return error. +* INT_MAX, whichever is less), or a negative error code on failure. */ int ring_buffer__consume (struct ring_buffer *rb) {int64_t err, res =0 ;int i; + err =ringbuf_validate_callbacks (rb); +if (err) +return libbpf_err (err); +for (i =0 ; i < rb->ring_cnt; i++) {struct ring *ring = rb->rings[i];@@ -334,13 +361,17 @@ int ring_buffer__consume(struct ring_buffer *rb)/* Poll for available data and consume records, if any are available. * Returns number of records consumed (or INT_MAX, whichever is less), or -* negative number, if any of the registered callbacks returned error. +* a negative error code on failure. */ int ring_buffer__poll (struct ring_buffer *rb,int timeout_ms) {int i, cnt;int64_t err, res =0 ; + err =ringbuf_validate_callbacks (rb); +if (err) +return libbpf_err (err); + cnt =epoll_wait (rb->epoll_fd, rb->events, rb->ring_cnt, timeout_ms);if (cnt <0 )return libbpf_err (-errno);tools/testing/selftests/bpf/prog_tests/ringbuf.c +93
-0@@ -401,6 +401,97 @@ static int process_n_sample(void *ctx, void *data, size_t len)return 0 ; } +static int process_noop_sample (void *ctx,void *data,size_t len) +{ +return 0 ; +} + +static void ringbuf_null_cb_subtest (void ) +{ +struct test_ringbuf_n_lskel *skel_n; +struct ring_buffer *ringbuf =NULL ; +struct ring *ring; +unsigned long consumer_pos; +int no_cb_map_fd =-1 ; +int err; + + skel_n =test_ringbuf_n_lskel__open (); +if (!ASSERT_OK_PTR (skel_n,"test_ringbuf_n_lskel__open" )) +return ; + + skel_n->maps.ringbuf.max_entries =getpagesize (); + skel_n->bss->pid =getpid (); + skel_n->bss->value =SAMPLE_VALUE ; + + err =test_ringbuf_n_lskel__load (skel_n); +if (!ASSERT_OK (err,"test_ringbuf_n_lskel__load" )) + goto cleanup; + + err =test_ringbuf_n_lskel__attach (skel_n); +if (!ASSERT_OK (err,"test_ringbuf_n_lskel__attach" )) + goto cleanup; + +syscall (__NR_getpgid); + + no_cb_map_fd =bpf_map_create (BPF_MAP_TYPE_RINGBUF ,NULL ,0 ,0 , +getpagesize (),NULL ); +if (!ASSERT_OK_FD (no_cb_map_fd,"bpf_map_create" )) + goto cleanup; + +/* Manager APIs must validate all rings before consuming any of them. */ + ringbuf =ring_buffer__new (skel_n->maps.ringbuf.map_fd, + process_noop_sample,NULL ,NULL ); +if (!ASSERT_OK_PTR (ringbuf,"ring_buffer__new" )) + goto cleanup_fd; + + ring =ring_buffer__ring (ringbuf,0 ); +if (!ASSERT_OK_PTR (ring,"ring_buffer__ring" )) + goto cleanup_ringbuf; + + err =ring_buffer__add (ringbuf, no_cb_map_fd,NULL ,NULL ); +if (!ASSERT_OK (err,"ring_buffer__add_no_cb" )) + goto cleanup_ringbuf; + + consumer_pos =ring__consumer_pos (ring); +ASSERT_GT (ring__producer_pos (ring), consumer_pos, +"producer_pos_mixed_cb" ); + + err =ring_buffer__consume_n (ringbuf,0 ); +ASSERT_EQ (err, -EINVAL ,"ringbuf_consume_zero_mixed_cb" ); + err =ring_buffer__consume (ringbuf); +ASSERT_EQ (err, -EINVAL ,"ringbuf_consume_mixed_cb" ); + err =ring_buffer__poll (ringbuf,0 ); +ASSERT_EQ (err, -EINVAL ,"ringbuf_poll_mixed_cb" ); +ASSERT_EQ (ring__consumer_pos (ring), consumer_pos, +"consumer_pos_mixed_cb" ); + +ring_buffer__free (ringbuf); + ringbuf = +ring_buffer__new (skel_n->maps.ringbuf.map_fd,NULL ,NULL ,NULL ); +if (!ASSERT_OK_PTR (ringbuf,"ring_buffer__new_no_cb" )) + goto cleanup_fd; + + ring =ring_buffer__ring (ringbuf,0 ); +if (!ASSERT_OK_PTR (ring,"ring_buffer__ring_no_cb" )) + goto cleanup_ringbuf; + consumer_pos =ring__consumer_pos (ring); + + err =ring_buffer__consume_n (ringbuf,0 ); +ASSERT_EQ (err, -EINVAL ,"ringbuf_consume_zero_no_cb" ); + err =ring__consume_n (ring,0 ); +ASSERT_EQ (err, -EINVAL ,"ring_consume_zero_no_cb" ); + err =ring__consume (ring); +ASSERT_EQ (err, -EINVAL ,"ring_consume_no_cb" ); +ASSERT_EQ (ring__consumer_pos (ring), consumer_pos,"consumer_pos_no_cb" ); + +cleanup_ringbuf: +ring_buffer__free (ringbuf); +cleanup_fd: +close (no_cb_map_fd); +cleanup: +test_ringbuf_n_lskel__destroy (skel_n); +} +static void ringbuf_n_subtest (void ) {struct test_ringbuf_n_lskel *skel_n;@@ -579,6 +670,8 @@ void test_ringbuf(void)ringbuf_subtest ();if (test__start_subtest ("ringbuf_n" ))ringbuf_n_subtest (); +if (test__start_subtest ("ringbuf_null_cb" )) +ringbuf_null_cb_subtest ();if (test__start_subtest ("ringbuf_map_key" ))ringbuf_map_key_subtest ();if (test__start_subtest ("ringbuf_write" ))
Emil Tsalapatis
reviewerRe: [PATCH bpf 3/6] libbpf: ringbuf: Handle position counter wrap
In reply to [email protected]Quoted 31 lines · sashiko-bot
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Pre-existing issues:
- [High]__bpf_ringbuf_reservein the kernel permanently fails allocations after 32-bit position wrap-around.
- [High] Kernel-side__bpf_user_ringbuf_peekincorrectly handles 32-bit wrap-around, leading to permanent stalls and bypassing bounds checks.
- [High] 32-bit userspace running on a 64-bit kernel permanently breaks BPF ring buffers after 4GB of data is processed due to pointer size mismatches. --commit
ce5e8d811804f9ff598195a7f394512f77c373b0
Author: Tamir Duberstein <[email protected]>libbpf: ringbuf: Handle position counter wrap
Ring buffer positions are unsigned long counters and can wrap on 32-bit systems.
ringbuf_process_ring()stops consuming whenproducer_poswraps belowconsumer_posbecause it compares the counters by magnitude.Quoted 13 lines
tools/lib/bpf/ringbuf.c +1
-1@@ -268,7 +268,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) do { got_new_data = false; prod_pos = smp_load_acquire(r->producer_pos); - while (cons_pos < prod_pos) { + while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); len = smp_load_acquire(len_ptr);
<SNIP>
Quoted 13 lines
[Severity: High]
This is a pre-existing issue, but does the kernel side have a similar wrap-around issue in__bpf_ringbuf_reserve()?kernel/bpf/ringbuf.c:__bpf_ringbuf_reserve() { ... while (pend_pos < prod_pos) { ... } ... }
Quoted 5 lines
Since
pend_posandprod_posare unsigned longs, on 32-bit kernels, does this magnitude comparison fail whenproducer_poswraps around to a small value? It appearspend_poswould freeze at its pre-wrap value, causingbpf_ringbuf_has_space()to permanently return false and deny future allocations.
This seems like a fair point, if we're going to modify one let's also modify the other. Wrt the "32-bit binaries on 64-bit" Sashiko warnings they're a bridge too far, we have to draw the line somewhere.
Can we also add a comment as to why we're using != in both locations?
<SNIP>
Re: [PATCH bpf 4/6] libbpf: ringbuf: Use compiler atomics
In reply to Tamir DubersteinQuoted 10 lines · Tamir Duberstein
Consumer-side ring buffer code uses architecture-specific smp_* helpers for shared memory accesses.
Use compiler atomics instead. They provide equivalent acquire and release ordering through a portable userspace interface and allow the next commit to use compiler fences in the wakeup protocol without mixing atomic interfaces.
Assisted-by Codex:gpt-5.5Signed-off-by Tamir Duberstein <[email protected]>
Quoted 65 lines
--- tools/lib/bpf/ringbuf.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)tools/lib/bpf/ringbuf.c +11
-9@@ -264,13 +264,13 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)if (n ==0 )return 0 ; - cons_pos =smp_load_acquire (r->consumer_pos); + cons_pos =__atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); do { got_new_data = false; - prod_pos = smp_load_acquire(r->producer_pos); + prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE); while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); - len = smp_load_acquire(len_ptr); + len = __atomic_load_n(len_ptr, __ATOMIC_ACQUIRE); /* sample not committed yet, bail out for now */ if (len & BPF_RINGBUF_BUSY_BIT)@@ -284,14 +284,16 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) err = r->sample_cb (r->ctx, sample, len);if (err <0 ) {/* update consumer pos and bail out */ -smp_store_release (r->consumer_pos, - cons_pos); +__atomic_store_n (r->consumer_pos, + cons_pos, + __ATOMIC_RELEASE);return err; } cnt++; } -smp_store_release (r->consumer_pos, cons_pos); +__atomic_store_n (r->consumer_pos, cons_pos, + __ATOMIC_RELEASE);if (cnt >= n) goto done;@@ -406,8 +408,8 @@ struct ring *ring_buffer__ring(struct ring_buffer *rb, unsigned int idx)unsigned long ring__consumer_pos (const struct ring *r) { -/* Synchronizes with smp_store_release() in ringbuf_process_ring(). */ -return smp_load_acquire (r->consumer_pos); +/* Synchronizes with the release store in ringbuf_process_ring(). */ +return __atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); }unsigned long ring__producer_pos (const struct ring *r)@@ -415,7 +417,7 @@ unsigned long ring__producer_pos(const struct ring *r)/* Synchronizes with smp_store_release() in __bpf_ringbuf_reserve() in * the kernel. */ -return smp_load_acquire (r->producer_pos); +return __atomic_load_n (r->producer_pos, __ATOMIC_ACQUIRE); }size_t ring__avail_data_size (const struct ring *r)
Emil Tsalapatis
reviewerRe: [PATCH bpf 5/6] libbpf: ringbuf: Prevent missed wakeups
In reply to Tamir DubersteinQuoted 33 lines · Tamir Duberstein
After consuming the last visible record,
ringbuf_process_ring()publishes the consumer position and checks the producer position. These operations lack a full StoreLoad barrier. A producer can therefore commit a new record but read the old consumer position while the consumer reads the old producer position. The producer sends no notification and the consumer waits despite a queued record.Insert a full barrier before checking for new data, ensuring that either the consumer observes the producer update or the producer observes the consumer update and sends a notification. Apply the same handshake when a busy record follows records whose consumer position was published.
Add an edge-triggered epoll test with a concurrent producer. Without the barrier, a missed notification leaves the producer dropping records from a full ring while the consumer times out. Document that bounded consumers and callbacks that terminate consumption must drain before waiting again.
Fixes bf99c936f947 ("libbpf: Add BPF ring buffer support")Reported-by Andrew Werner <[email protected]>Assisted-by Codex:gpt-5.5Signed-off-by Tamir Duberstein <[email protected]>--- tools/lib/bpf/libbpf.h | 22 +++++++ tools/lib/bpf/ringbuf.c | 14 +++- tools/testing/selftests/bpf/prog_tests/ringbuf.c | 84 ++++++++++++++++++++++++ 3 files changed, 117 insertions(+), 3 deletions(-)tools/lib/bpf/libbpf.h +0
-0
tools/lib/bpf/libbpf.h struct ring_buffer; · line 1442
Quoted 4 lines
tools/lib/bpf/libbpf.h +1
-0@@ -1439,6 +1439,10 @@ struct ring_buffer;struct ring ;struct user_ring_buffer ; +/* A negative return stops consumption; non-negative values continue. Stopping
Sashiko is right on this, let's use regular kernel style for new comments.
Quoted 43 lines
tools/lib/bpf/libbpf.h +21
-0+* can leave records queued without a new readiness notification. Before +* waiting for readiness again, consume until no records remain. +*/ typedef int (*ring_buffer_sample_fn )(void *ctx,void *data,size_t size);struct ring_buffer_opts {@@ -1455,6 +1459,20 @@ LIBBPF_API int ring_buffer__add(struct ring_buffer *rb, int map_fd, ring_buffer_sample_fn sample_cb, void *ctx); LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms); LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb); + +/** +* @brief **ring_buffer__consume_n()** consumes up to a requested number of +* records from a ring buffer manager without event polling. +* +* @param rb A ring buffer manager object. +* @param n Maximum number of records to consume. +* @return The number of records consumed, or a negative error code on failure. +* +* Reaching the requested bound does not establish that every ring is empty. +* Records can remain queued without a new readiness notification. Before +* waiting on ring_buffer__epoll_fd(), call ring_buffer__consume() until it +* returns 0. +*/ LIBBPF_API int ring_buffer__consume_n(struct ring_buffer *rb, size_t n); LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);@@ -1537,6 +1555,10 @@ LIBBPF_API int ring__consume(struct ring *r);* @param r A ringbuffer object. * @param n Maximum number of records to consume. * @return The number of records consumed, or a negative error code on failure. +* +* Reaching the requested bound does not establish that the ring is empty. +* Records can remain queued without a new readiness notification. Before +* waiting on ring__map_fd(), call ring__consume() until it returns 0. */ LIBBPF_API int ring__consume_n(struct ring *r, size_t n);tools/lib/bpf/ringbuf.c +0
-0
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 258
Quoted 5 lines
tools/lib/bpf/ringbuf.c +1
-1@@ -255,7 +255,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)/* 64-bit to avoid overflow in case of extreme application behavior */ int64_t cnt =0 ;unsigned long cons_pos, prod_pos; -bool got_new_data; +bool got_new_data, needs_wakeup = false;
The extra new variable feels overly complicated. I think this is becuase the new code is going off of the got_new_data pattern, which imo is also unnecessary. We should just remove got_new data and just do:
while (true) {
prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE);
if (cons_pos == prod_pos)
break;
while (cons_pos != prod_pos) {
...
}
}Quoted 4 lines
tools/lib/bpf/ringbuf.c +0
-0void *sample; err =ringbuf_validate (r);
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 276
Quoted 10 lines
tools/lib/bpf/ringbuf.c +7
-0@@ -267,14 +267,21 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n) cons_pos =__atomic_load_n (r->consumer_pos, __ATOMIC_ACQUIRE); do { got_new_data = false; + if (needs_wakeup) { + /* Ensure either this sees a new record or its producer sees + * the updated consumer position and sends a notification. + */ + __atomic_thread_fence(__ATOMIC_SEQ_CST); + needs_wakeup = false; + }
Now on this, I think to address Sashiko's warning instead of the if above...
Quoted 14 lines
tools/lib/bpf/ringbuf.c +2
-2prod_pos =__atomic_load_n (r->producer_pos, __ATOMIC_ACQUIRE);while (cons_pos != prod_pos) { len_ptr = r->data + (cons_pos & r->mask); len =__atomic_load_n (len_ptr, __ATOMIC_ACQUIRE); -/* sample not committed yet, bail out for now */ +/* Retry a busy record once after publishing prior records. */ if (len &BPF_RINGBUF_BUSY_BIT ) - goto done; +break ; got_new_data = true; cons_pos +=roundup_len (len);
tools/lib/bpf/ringbuf.c static int64_t ringbuf_process_ring(struct ring *r, size_t n) · line 307
Quoted 7 lines
tools/lib/bpf/ringbuf.c +1
-0@@ -294,6 +301,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)__atomic_store_n (r->consumer_pos, cons_pos, __ATOMIC_RELEASE); + needs_wakeup = true;if (cnt >= n) goto done;
We just add an smp_mb() here. And since AFAICT smb_mb is available here, we can also avoid the move from smp_* to __atomic in the previous patch.
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/ringbuf.c +1
-0@@ -492,6 +492,88 @@ static void ringbuf_null_cb_subtest(void)test_ringbuf_n_lskel__destroy (skel_n); } +#define N_WAKEUP_SAMPLES 20000
tools/testing/selftests/bpf/prog_tests/ringbuf.c static void ringbuf_null_cb_subtest(void) · line 505
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/ringbuf.c +10
-0+ +struct wakeup_ctx { +bool stop; +}; + +static void *wakeup_producer (void *arg) +{ +struct wakeup_ctx *ctx = arg; + +while (!__atomic_load_n (&ctx->stop, __ATOMIC_RELAXED))
Why __ATOMIC_RELAXED here? Maybe just declare the variable volatile and do regular reads?
Quoted 48 lines
tools/testing/selftests/bpf/prog_tests/ringbuf.c +48
-0+syscall (__NR_getpgid); +return NULL ; +} + +static void ringbuf_wakeup_subtest (void ) +{ +struct test_ringbuf_n_lskel *skel_n; +struct ring_buffer *ringbuf =NULL ; +struct epoll_event event = { + .events =EPOLLIN |EPOLLET , + }; +struct wakeup_ctx ctx = {}; +pthread_t producer; +int epoll_fd =-1 ; +int err, total =0 ; + + skel_n =test_ringbuf_n_lskel__open (); +if (!ASSERT_OK_PTR (skel_n,"test_ringbuf_n_lskel__open" )) +return ; + + skel_n->maps.ringbuf.max_entries =getpagesize (); + skel_n->bss->pid =getpid (); + skel_n->bss->value =SAMPLE_VALUE ; + + err =test_ringbuf_n_lskel__load (skel_n); +if (!ASSERT_OK (err,"test_ringbuf_n_lskel__load" )) + goto cleanup; + + err =test_ringbuf_n_lskel__attach (skel_n); +if (!ASSERT_OK (err,"test_ringbuf_n_lskel__attach" )) + goto cleanup; + + ringbuf =ring_buffer__new (skel_n->maps.ringbuf.map_fd, + process_noop_sample,NULL ,NULL ); +if (!ASSERT_OK_PTR (ringbuf,"ring_buffer__new" )) + goto cleanup; + + epoll_fd =epoll_create1 (EPOLL_CLOEXEC ); +if (!ASSERT_OK_FD (epoll_fd,"epoll_create1" )) + goto cleanup_ringbuf; + + err =epoll_ctl (epoll_fd,EPOLL_CTL_ADD , skel_n->maps.ringbuf.map_fd, + &event); +if (!ASSERT_OK (err,"epoll_ctl" )) + goto cleanup_epoll; + + err =pthread_create (&producer,NULL , wakeup_producer, &ctx); +if (!ASSERT_OK (err,"pthread_create" ))
tools/testing/selftests/bpf/prog_tests/ringbuf.c static void ringbuf_null_cb_subtest(void) · line 563
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/ringbuf.c +10
-0+ goto cleanup_epoll; + +while (total <N_WAKEUP_SAMPLES ) { + err =epoll_wait (epoll_fd, &event,1 ,1000 ); +if (!ASSERT_EQ (err,1 ,"epoll_wait" )) + goto cleanup_thread; +while ((err =ring_buffer__consume (ringbuf)) >0 ) + total += err; +if (!ASSERT_OK (err,"ring_buffer__consume" )) + goto cleanup_thread;
Label is unnecessary, just call break;
Quoted 17 lines
tools/testing/selftests/bpf/prog_tests/ringbuf.c +13
-0+ } + +cleanup_thread: +__atomic_store_n (&ctx.stop, true, __ATOMIC_RELAXED); +pthread_join (producer,NULL ); +cleanup_epoll: +close (epoll_fd); +cleanup_ringbuf: +ring_buffer__free (ringbuf); +cleanup: +test_ringbuf_n_lskel__destroy (skel_n); +} +static void ringbuf_n_subtest (void ) {struct test_ringbuf_n_lskel *skel_n;
tools/testing/selftests/bpf/prog_tests/ringbuf.c void test_ringbuf(void) · line 761
Quoted 8 lines
tools/testing/selftests/bpf/prog_tests/ringbuf.c +2
-0@@ -672,6 +754,8 @@ void test_ringbuf(void)ringbuf_n_subtest ();if (test__start_subtest ("ringbuf_null_cb" ))ringbuf_null_cb_subtest (); +if (test__start_subtest ("ringbuf_wakeup" )) +ringbuf_wakeup_subtest ();if (test__start_subtest ("ringbuf_map_key" ))ringbuf_map_key_subtest ();if (test__start_subtest ("ringbuf_write" ))
pw-bot: cr
Emil Tsalapatis
reviewerRe: [PATCH bpf 6/6] libbpf: ringbuf: Reject overwrite callback use
In reply to Tamir DubersteinQuoted 54 lines · Tamir Duberstein
BPF_F_RB_OVERWRITEcan advanceoverwrite_pospastconsumer_pos. Callback traversal does not readoverwrite_pos, so after the producer laps the consumer it can treat overwritten data as a record header.An earlier proposal[0] copied the readable window before invoking callbacks. Review concluded that callbacks are a poor fit because copying penalizes zero-copy users and the API cannot report skipped records.
Record the map flag and reject callback consumption with
-EOPNOTSUPP.Link https://lore.kernel.org/bpf/CAEf4Bzaq5drHWChXoRBnrmkb6reAsSVj8r=uByFSup31FMA7hw@mail.gmail.com/ [0] open hereFixes feeaf1346f80 ("bpf: Add overwrite mode for BPF ring buffer")Assisted-by Codex:gpt-5.5Signed-off-by Tamir Duberstein <[email protected]>--- tools/lib/bpf/libbpf.h | 1 + tools/lib/bpf/ringbuf.c | 4 +++ tools/testing/selftests/bpf/prog_tests/ringbuf.c | 39 ++++++++++++++++++++++++ 3 files changed, 44 insertions(+)tools/lib/bpf/libbpf.h +1
-0@@ -1439,6 +1439,7 @@ struct ring_buffer;struct ring ;struct user_ring_buffer ; +/* Callback-based consumption is unsupported for BPF_F_RB_OVERWRITE maps. */ /* A negative return stops consumption; non-negative values continue. Stopping * can leave records queued without a new readiness notification. Before * waiting for readiness again, consume until no records remain. tools/lib/bpf/ringbuf.c +2
-0@@ -30,6 +30,7 @@ struct ring {unsigned long *producer_pos;unsigned long mask;int map_fd; +bool overwrite; };struct ring_buffer {@@ -118,6 +119,7 @@ int ring_buffer__add(struct ring_buffer *rb, int map_fd, r->sample_cb = sample_cb; r->ctx = ctx; r->mask = info.max_entries -1 ; + r->overwrite = info.map_flags &BPF_F_RB_OVERWRITE ;/* Map writable consumer page */ tmp =mmap (NULL , rb->page_size,PROT_READ |PROT_WRITE ,MAP_SHARED , map_fd,0 );
tools/lib/bpf/ringbuf.c static inline int roundup_len(__u32 len) · line 239
Quoted 5 lines
tools/lib/bpf/ringbuf.c +2
-0@@ -233,6 +235,8 @@ static inline int roundup_len(__u32 len)static int ringbuf_validate (const struct ring *r) { +if (r->overwrite) +return -EOPNOTSUPP ;
Nit; unlikely(r->overwrite)?
The check makes sense, we can add it whether we keep ringbuf_validate() or not.
Quoted 60 lines
tools/lib/bpf/ringbuf.c +0
-0return r->sample_cb ?0 : -EINVAL ; }tools/testing/selftests/bpf/prog_tests/ringbuf.c +39
-0@@ -684,6 +684,43 @@ static void ringbuf_map_key_subtest(void)test_ringbuf_map_key_lskel__destroy (skel_map_key); } +static void ringbuf_overwrite_callback_subtest (void ) +{ + LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_RB_OVERWRITE); +struct ring_buffer *ringbuf; +struct ring *ring; +int map_fd, err; + + map_fd =bpf_map_create (BPF_MAP_TYPE_RINGBUF ,NULL ,0 ,0 ,getpagesize (), + &opts); +if (!ASSERT_OK_FD (map_fd,"bpf_map_create" )) +return ; + + ringbuf =ring_buffer__new (map_fd, process_noop_sample,NULL ,NULL ); +if (!ASSERT_OK_PTR (ringbuf,"ring_buffer__new" )) + goto cleanup_fd; + + ring =ring_buffer__ring (ringbuf,0 ); +if (!ASSERT_OK_PTR (ring,"ring_buffer__ring" )) + goto cleanup_ringbuf; + + err =ring_buffer__consume_n (ringbuf,0 ); +ASSERT_EQ (err, -EOPNOTSUPP ,"ringbuf_consume_zero" ); + err =ring_buffer__consume (ringbuf); +ASSERT_EQ (err, -EOPNOTSUPP ,"ringbuf_consume" ); + err =ring_buffer__poll (ringbuf,0 ); +ASSERT_EQ (err, -EOPNOTSUPP ,"ringbuf_poll" ); + err =ring__consume_n (ring,0 ); +ASSERT_EQ (err, -EOPNOTSUPP ,"ring_consume_zero" ); + err =ring__consume (ring); +ASSERT_EQ (err, -EOPNOTSUPP ,"ring_consume" ); + +cleanup_ringbuf: +ring_buffer__free (ringbuf); +cleanup_fd: +close (map_fd); +} +static void ringbuf_overwrite_mode_subtest (void ) {unsigned long size, len1, len2, len3, len4, len5;@@ -760,6 +797,8 @@ void test_ringbuf(void)ringbuf_map_key_subtest ();if (test__start_subtest ("ringbuf_write" ))ringbuf_write_subtest (); +if (test__start_subtest ("ringbuf_overwrite_callback" )) +ringbuf_overwrite_callback_subtest ();if (test__start_subtest ("ringbuf_overwrite_mode" ))ringbuf_overwrite_mode_subtest (); }
Re: [PATCH bpf 2/6] libbpf: ringbuf: Prevent NULL callback crash
In reply to Emil TsalapatisOn Tue, Jun 16, 2026 at 08:44:13PM -0400, Emil Tsalapatis wrote:
[...]
Quoted 2 lines
Can we just prevent a ring from being added with a NULL
sample_cb? What's the use for permitting it?
I kept NULL callbacks valid because I plan to add a caller-owned iterator that does not use callbacks. Rejecting NULL during construction would prevent callers from creating iterator-only rings.
Quoted 2 lines
Even if we don't, rechecking the callbacks every single time we consume the ringbuf seems overkill.
Your unlikely() suggestion on patch 6 applies here too, so I used it in v2.
Re: [PATCH bpf 3/6] libbpf: ringbuf: Handle position counter wrap
In reply to Emil TsalapatisOn Tue, Jun 16, 2026 at 09:19:43PM -0400, Emil Tsalapatis wrote:
[...]
Quoted 3 lines
This seems like a fair point, if we're going to modify one let's also modify the other. Wrt the "32-bit binaries on 64-bit" Sashiko warnings they're a bridge too far, we have to draw the line somewhere.
Yep, my clanker also flagged the 32-bit-on-64-bit problem -- I agree it doesn't make sense to tackle here. The other two now have their own patches in v2.
Quoted 1 line
Can we also add a comment as to why we're using != in both locations?
Done in v2.
Re: [PATCH bpf 5/6] libbpf: ringbuf: Prevent missed wakeups
In reply to Emil TsalapatisOn Thu, Jun 18, 2026 at 02:41:54AM -0400, Emil Tsalapatis wrote:
[...]
Quoted 2 lines
Sashiko is right on this, let's use regular kernel style for new comments.
Done in v2.
[...]
Quoted 5 lines
The extra new variable feels overly complicated. I think this is becuase the new code is going off of the
got_new_datapattern, which imo is also unnecessary. We should just removegot_newdata and just do:
Quoted 9 lines
while (true) {
prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE);
if (cons_pos == prod_pos)
break;
while (cons_pos != prod_pos) {
...
}
}V2 removes needs_wakeup and leaves one fence after each batch that advances the consumer position. I kept got_new_data because the proposed unconditional outer loop would retry the same busy record indefinitely. The revised loop retries a busy record once after publishing progress and returns if the record is still busy.
[...]
Quoted 3 lines
We just add an
smp_mb()here. And since AFAICTsmb_mbis available here, we can also avoid the move from smp_* to__atomicin the previous patch.
I kept the compiler atomics because smp_mb() is not a portable full barrier in the tools headers. Only x86, arm64, and RISC-V define it directly; the generic fallback can reduce mb() to a compiler barrier. A single __atomic_thread_fence() after each batch orders the preceding consumer-position stores without a full barrier after every record.
[...]
Quoted 2 lines
Why
__ATOMIC_RELAXEDhere? Maybe just declare the variable volatile and do regular reads?
I kept the stop flag as a relaxed atomic because volatile would leave a C data race; the flag needs atomicity but no ordering.
[...]
Quoted 1 line
Label is unnecessary, just call break;
Done in v2.
Re: [PATCH bpf 6/6] libbpf: ringbuf: Reject overwrite callback use
In reply to Emil TsalapatisOn Thu, Jun 18, 2026 at 02:52:58AM -0400, Emil Tsalapatis wrote:
[...]
Quoted 4 lines
Nit;
unlikely(r->overwrite)?The check makes sense, we can add it whether we keep
ringbuf_validate()or not.
Done in v2.