Replace CONFIG_DMABUF_SYSFS_STATS with BPF
Subsystems (6)
v7 cover Replace CONFIG_DMABUF_SYSFS_STATS with BPF
Until CONFIG_DMABUF_SYSFS_STATS was added [1] it was only possible to perform per-buffer accounting with debugfs which is not suitable for production environments. Eventually we discovered the overhead with per-buffer sysfs file creation/removal was significantly impacting allocation and free times, and exacerbated kernfs lock contention. [2] dma_buf_stats_setup() is responsible for 39% of single-page buffer creation duration, or 74% of single-page dma_buf_export() duration when stressing dmabuf allocations and frees.
I prototyped a change from per-buffer to per-exporter statistics with a RCU protected list of exporter allocations that accommodates most (but not all) of our use-cases and avoids almost all of the sysfs overhead. While that adds less overhead than per-buffer sysfs, and less even than the maintenance of the dmabuf debugfs_list, it's still *additional* overhead on top of the debugfs_list and doesn't give us per-buffer info.
This series uses the existing dmabuf debugfs_list to implement a BPF dmabuf iterator, which adds no overhead to buffer allocation/free and provides per-buffer info. The list has been moved outside of CONFIG_DEBUG_FS scope so that it is always populated. The BPF program loaded by userspace that extracts per-buffer information gets to define its own interface which avoids the lack of ABI stability with debugfs.
This will allow us to replace our use of CONFIG_DMABUF_SYSFS_STATS, and the plan is to remove it from the kernel after the next longterm stable release.
[1] https://lore.kernel.org/linux-media/[email protected]
[2] https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]
v1 -> v2:
Make the DMA buffer list independent of CONFIG_DEBUG_FS per Christian
König
Add CONFIG_DMA_SHARED_BUFFER check to kernel/bpf/Makefile per kernel
test robot
Use BTF_ID_LIST_SINGLE instead of BTF_ID_LIST_GLOBAL_SINGLE per Song Liu Fixup comment style, mixing code/declarations, and use ASSERT_OK_FD in
selftest per Song Liu
Add BPF_ITER_RESCHED feature to bpf_dmabuf_reg_info per Alexei
Starovoitov
Add open-coded iterator and selftest per Alexei Starovoitov
Add a second test buffer from the system dmabuf heap to selftests Use the BPF program we'll use in production for selftest per Alexei
Starovoitov
https://r.android.com/c/platform/system/bpfprogs/+/3616123/2/dmabufIter.c
https://r.android.com/c/platform/system/memory/libmeminfo/+/3614259/1/libdmabufinfo/dmabuf_bpf_stats.cpp
v2: https://lore.kernel.org/all/[email protected]
v2 -> v3:
Rebase onto bpf-next/master
Move get_next_dmabuf() into drivers/dma-buf/dma-buf.c, along with the
new get_first_dmabuf(). This avoids having to expose the dmabuf list and mutex to the rest of the kernel, and keeps the dmabuf mutex operations near each other in the same file. (Christian König)
Add Christian's RB to dma-buf: Rename debugfs symbols
Drop RFC: dma-buf: Remove DMA-BUF statistics
v3: https://lore.kernel.org/all/[email protected]
v3 -> v4:
Fix selftest BPF program comment style (not kdoc) per Alexei Starovoitov Fix dma-buf.c kdoc comment style per Alexei Starovoitov
Rename get_first_dmabuf / get_next_dmabuf to dma_buf_iter_begin /
dma_buf_iter_next per Christian König
Add Christian's RB to bpf: Add dmabuf iterator
v4: https://lore.kernel.org/all/[email protected]
v4 -> v5:
Add Christian's Acks to all patches
Add Song Liu's Acks
Move BTF_ID_LIST_SINGLE and DEFINE_BPF_ITER_FUNC closer to usage per
Song Liu
Fix open-coded iterator comment style per Song Liu
Move iterator termination check to its own subtest per Song Liu Rework selftest buffer creation per Song Liu
Fix spacing in sanitize_string per BPF CI
v5: https://lore.kernel.org/all/[email protected]
v5 -> v6:
Song Liu:
Init test buffer FDs to -1
Zero-init udmabuf_create for future proofing
Bail early for iterator fd/FILE creation failure
Dereference char ptr to check for NUL in sanitize_string()
Move map insertion from create_test_buffers() to test_dmabuf_iter() Add ACK to selftests/bpf: Add test for open coded dmabuf_iter
v6: https://lore.kernel.org/all/[email protected]
v6 -> v7:
Zero uninitialized name bytes following the end of name strings per
s390x BPF CI
Reorder sanitize_string bounds checks per Song Liu
Add Song's Ack to: selftests/bpf: Add test for dmabuf_iter
Rebase onto bpf-next/master per BPF CI
T.J. Mercier (5):
dma-buf: Rename debugfs symbols
bpf: Add dmabuf iterator
bpf: Add open coded dmabuf iterator selftests/bpf: Add test for dmabuf_iter selftests/bpf: Add test for open coded dmabuf_iter
drivers/dma-buf/dma-buf.c | 98 ++++-- include/linux/dma-buf.h | 4 +- kernel/bpf/Makefile | 3 + kernel/bpf/dmabuf_iter.c | 150 +++++++++ kernel/bpf/helpers.c | 5 + .../testing/selftests/bpf/bpf_experimental.h | 5 + tools/testing/selftests/bpf/config | 3 + .../selftests/bpf/prog_tests/dmabuf_iter.c | 285 ++++++++++++++++++ .../testing/selftests/bpf/progs/dmabuf_iter.c | 101 +++++++ 9 files changed, 632 insertions(+), 22 deletions(-) create mode 100644 kernel/bpf/dmabuf_iter.c create mode 100644 tools/testing/selftests/bpf/prog_tests/dmabuf_iter.c create mode 100644 tools/testing/selftests/bpf/progs/dmabuf_iter.c
base-commit: 6888a036cfc3d617d0843ecc9bd8504e91fb9de6
Signature · 1 line
2.49.0.1151.ga128411c76-goog
PATCH v7 1/5 dma-buf: Rename debugfs symbols
Rename the debugfs list and mutex so it's clear they are now usable without the need for CONFIG_DEBUG_FS. The list will always be populated to support the creation of a BPF iterator for dmabufs.
--- drivers/dma-buf/dma-buf.c | 40 +++++++++++++++------------------------ include/linux/dma-buf.h | 2 -- 2 files changed, 15 insertions(+), 27 deletions(-)
drivers/dma-buf/dma-buf.c +15 -25
@@ -35,35 +35,25 @@static inline int is_dma_buf_file (struct file *); -#if IS_ENABLED (CONFIG_DEBUG_FS ) -static DEFINE_MUTEX (debugfs_list_mutex); -static LIST_HEAD (debugfs_list); +static DEFINE_MUTEX (dmabuf_list_mutex); +static LIST_HEAD (dmabuf_list); -static void __dma_buf_debugfs_list_add (struct dma_buf *dmabuf) +static void __dma_buf_list_add (struct dma_buf *dmabuf) { -mutex_lock (&debugfs_list_mutex); -list_add (&dmabuf->list_node, &debugfs_list); -mutex_unlock (&debugfs_list_mutex); +mutex_lock (&dmabuf_list_mutex); +list_add (&dmabuf->list_node, &dmabuf_list); +mutex_unlock (&dmabuf_list_mutex); } -static void __dma_buf_debugfs_list_del (struct dma_buf *dmabuf) +static void __dma_buf_list_del (struct dma_buf *dmabuf) {if (!dmabuf)return ; -mutex_lock (&debugfs_list_mutex); +mutex_lock (&dmabuf_list_mutex);list_del (&dmabuf->list_node); -mutex_unlock (&debugfs_list_mutex); +mutex_unlock (&dmabuf_list_mutex); } -#else -static void __dma_buf_debugfs_list_add (struct dma_buf *dmabuf) -{ -} - -static void __dma_buf_debugfs_list_del (struct dma_buf *dmabuf) -{ -} -#endif static char *dmabuffs_dname (struct dentry *dentry,char *buffer,int buflen) {
@@ -115,7 +105,7 @@ static int dma_buf_file_release(struct inode *inode, struct file *file)if (!is_dma_buf_file (file))return -EINVAL ; -__dma_buf_debugfs_list_del (file->private_data); +__dma_buf_list_del (file->private_data);return 0 ; }
@@ -689,7 +679,7 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info) file->f_path.dentry->d_fsdata = dmabuf; dmabuf->file = file; -__dma_buf_debugfs_list_add (dmabuf); +__dma_buf_list_add (dmabuf);return dmabuf;
@@ -1630,7 +1620,7 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)size_t size =0 ;int ret; - ret =mutex_lock_interruptible (&debugfs_list_mutex); + ret =mutex_lock_interruptible (&dmabuf_list_mutex);if (ret)return ret;
@@ -1639,7 +1629,7 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)seq_printf (s,"%-8s\t%-8s\t%-8s\t%-8s\texp_name\t%-8s\tname\n" ,"size" ,"flags" ,"mode" ,"count" ,"ino" ); -list_for_each_entry (buf_obj, &debugfs_list, list_node) { +list_for_each_entry (buf_obj, &dmabuf_list, list_node) { ret =dma_resv_lock_interruptible (buf_obj->resv,NULL );if (ret)
@@ -1676,11 +1666,11 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)seq_printf (s,"\nTotal %d objects, %zu bytes\n" , count, size); -mutex_unlock (&debugfs_list_mutex); +mutex_unlock (&dmabuf_list_mutex);return 0 ; error_unlock: -mutex_unlock (&debugfs_list_mutex); +mutex_unlock (&dmabuf_list_mutex);return ret; }
include/linux/dma-buf.h +0 -2
@@ -370,10 +370,8 @@ struct dma_buf {*/ struct module *owner; -#if IS_ENABLED (CONFIG_DEBUG_FS )/** @list_node: node for dma_buf accounting and debugging. */ struct list_head list_node; -#endif /** @priv: exporter specific private data for this buffer object. */ void *priv;
Signature · 1 line
2.49.0.1151.ga128411c76-goog
PATCH v7 2/5 bpf: Add dmabuf iterator
In reply to T.J. MercierThe dmabuf iterator traverses the list of all DMA buffers.
DMA buffers are refcounted through their associated struct file. A reference is taken on each buffer as the list is iterated to ensure each buffer persists for the duration of the bpf program execution without holding the list mutex.
--- drivers/dma-buf/dma-buf.c | 68 +++++++++++++++++++++++++ include/linux/dma-buf.h | 2 + kernel/bpf/Makefile | 3 ++ kernel/bpf/dmabuf_iter.c | 102 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 175 insertions(+) create mode 100644 kernel/bpf/dmabuf_iter.c
drivers/dma-buf/dma-buf.c +68 -0
@@ -19,7 +19,9 @@#include <linux/anon_inodes.h> #include <linux/export.h> #include <linux/debugfs.h> +#include <linux/list.h> #include <linux/module.h> +#include <linux/mutex.h> #include <linux/seq_file.h> #include <linux/sync_file.h> #include <linux/poll.h>
@@ -55,6 +57,72 @@ static void __dma_buf_list_del(struct dma_buf *dmabuf)mutex_unlock (&dmabuf_list_mutex); } +/** +* dma_buf_iter_begin - begin iteration through global list of all DMA buffers +* +* Returns the first buffer in the global list of DMA-bufs that's not in the +* process of being destroyed. Increments that buffer's reference count to +* prevent buffer destruction. Callers must release the reference, either by +* continuing iteration with dma_buf_iter_next(), or with dma_buf_put(). +* +* Return: +* * First buffer from global list, with refcount elevated +* * NULL if no active buffers are present +*/ +struct dma_buf *dma_buf_iter_begin (void ) +{ +struct dma_buf *ret =NULL , *dmabuf; + +/* +* The list mutex does not protect a dmabuf's refcount, so it can be +* zeroed while we are iterating. We cannot call get_dma_buf() since the +* caller may not already own a reference to the buffer. +*/ +mutex_lock (&dmabuf_list_mutex); +list_for_each_entry (dmabuf, &dmabuf_list, list_node) { +if (file_ref_get (&dmabuf->file->f_ref)) { + ret = dmabuf; +break ; + } + } +mutex_unlock (&dmabuf_list_mutex); +return ret; +} + +/** +* dma_buf_iter_next - continue iteration through global list of all DMA buffers +* @dmabuf: [in] pointer to dma_buf +* +* Decrements the reference count on the provided buffer. Returns the next +* buffer from the remainder of the global list of DMA-bufs with its reference +* count incremented. Callers must release the reference, either by continuing +* iteration with dma_buf_iter_next(), or with dma_buf_put(). +* +* Return: +* * Next buffer from global list, with refcount elevated +* * NULL if no additional active buffers are present +*/ +struct dma_buf *dma_buf_iter_next (struct dma_buf *dmabuf) +{ +struct dma_buf *ret =NULL ; + +/* +* The list mutex does not protect a dmabuf's refcount, so it can be +* zeroed while we are iterating. We cannot call get_dma_buf() since the +* caller may not already own a reference to the buffer. +*/ +mutex_lock (&dmabuf_list_mutex); +dma_buf_put (dmabuf); +list_for_each_entry_continue (dmabuf, &dmabuf_list, list_node) { +if (file_ref_get (&dmabuf->file->f_ref)) { + ret = dmabuf; +break ; + } + } +mutex_unlock (&dmabuf_list_mutex); +return ret; +} +static char *dmabuffs_dname (struct dentry *dentry,char *buffer,int buflen) {struct dma_buf *dmabuf;
include/linux/dma-buf.h +2 -0
@@ -634,4 +634,6 @@ int dma_buf_vmap(struct dma_buf *dmabuf, struct iosys_map *map);void dma_buf_vunmap (struct dma_buf *dmabuf,struct iosys_map *map);int dma_buf_vmap_unlocked (struct dma_buf *dmabuf,struct iosys_map *map);void dma_buf_vunmap_unlocked (struct dma_buf *dmabuf,struct iosys_map *map); +struct dma_buf *dma_buf_iter_begin (void ); +struct dma_buf *dma_buf_iter_next (struct dma_buf *dmbuf);#endif /* __DMA_BUF_H__ */
kernel/bpf/Makefile +3 -0
@@ -53,6 +53,9 @@ obj-$(CONFIG_BPF_SYSCALL) += relo_core.o obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o obj-$(CONFIG_BPF_SYSCALL) += btf_relocate.o obj-$(CONFIG_BPF_SYSCALL) += kmem_cache_iter.o +ifeq ($(CONFIG_DMA_SHARED_BUFFER),y) +obj-$(CONFIG_BPF_SYSCALL) += dmabuf_iter.o +endif CFLAGS_REMOVE_percpu_freelist.o = $(CC_FLAGS_FTRACE) CFLAGS_REMOVE_bpf_lru_list.o = $(CC_FLAGS_FTRACE)
kernel/bpf/dmabuf_iter.c new file +102 -0
102 lines not on this page · raw message
Signature · 1 line
2.49.0.1151.ga128411c76-goog
PATCH v7 3/5 bpf: Add open coded dmabuf iterator
In reply to T.J. MercierThis open coded iterator allows for more flexibility when creating BPF programs. It can support output in formats other than text. With an open coded iterator, a single BPF program can traverse multiple kernel data structures (now including dmabufs), allowing for more efficient analysis of kernel data compared to multiple reads from procfs, sysfs, or multiple traditional BPF iterator invocations.
--- kernel/bpf/dmabuf_iter.c | 48 ++++++++++++++++++++++++++++++++++++++++ kernel/bpf/helpers.c | 5 +++++ 2 files changed, 53 insertions(+)
kernel/bpf/dmabuf_iter.c +48 -0
@@ -100,3 +100,51 @@ static int __init dmabuf_iter_init(void) }late_initcall (dmabuf_iter_init); + +struct bpf_iter_dmabuf { +/* +* opaque iterator state; having __u64 here allows to preserve correct +* alignment requirements in vmlinux.h, generated from BTF +*/ +__u64 __opaque[1 ]; +}__aligned(8) ; + +/* Non-opaque version of bpf_iter_dmabuf */ +struct bpf_iter_dmabuf_kern { +struct dma_buf *dmabuf; +}__aligned(8) ; + +__bpf_kfunc_start_defs (); + +__bpf_kfunc int bpf_iter_dmabuf_new(struct bpf_iter_dmabuf *it) +{ +struct bpf_iter_dmabuf_kern *kit = (void *)it; + +BUILD_BUG_ON (sizeof (*kit) >sizeof (*it)); + BUILD_BUG_ON(__alignof__(*kit) != __alignof__(*it)); + + kit->dmabuf =NULL ; +return 0 ; +} + +__bpf_kfunc struct dma_buf *bpf_iter_dmabuf_next(struct bpf_iter_dmabuf *it) +{ +struct bpf_iter_dmabuf_kern *kit = (void *)it; + +if (kit->dmabuf) + kit->dmabuf =dma_buf_iter_next (kit->dmabuf); +else + kit->dmabuf =dma_buf_iter_begin (); + +return kit->dmabuf; +} + +__bpf_kfunc void bpf_iter_dmabuf_destroy(struct bpf_iter_dmabuf *it) +{ +struct bpf_iter_dmabuf_kern *kit = (void *)it; + +if (kit->dmabuf) +dma_buf_put (kit->dmabuf); +} + +__bpf_kfunc_end_defs ();
kernel/bpf/helpers.c +5 -0
@@ -3386,6 +3386,11 @@ BTF_ID_FLAGS(func, bpf_copy_from_user_dynptr, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_copy_from_user_str_dynptr, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_copy_from_user_task_dynptr, KF_SLEEPABLE | KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_copy_from_user_task_str_dynptr, KF_SLEEPABLE | KF_TRUSTED_ARGS) +#ifdef CONFIG_DMA_SHARED_BUFFER +BTF_ID_FLAGS(func, bpf_iter_dmabuf_new, KF_ITER_NEW | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_iter_dmabuf_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_iter_dmabuf_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) +#endif BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
Signature · 1 line
2.49.0.1151.ga128411c76-goog
PATCH v7 4/5 selftests/bpf: Add test for dmabuf_iter
In reply to T.J. MercierThis test creates a udmabuf, and a dmabuf from the system dmabuf heap, and uses a BPF program that prints dmabuf metadata with the new dmabuf_iter to verify they can be found.
--- tools/testing/selftests/bpf/config | 3 + .../selftests/bpf/prog_tests/dmabuf_iter.c | 244 ++++++++++++++++++ .../testing/selftests/bpf/progs/dmabuf_iter.c | 53 ++++ 3 files changed, 300 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/dmabuf_iter.c create mode 100644 tools/testing/selftests/bpf/progs/dmabuf_iter.c
tools/testing/selftests/bpf/config +3 -0
@@ -22,6 +22,8 @@ CONFIG_CRYPTO_AES=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_BTF=y CONFIG_DEBUG_INFO_DWARF4=y +CONFIG_DMABUF_HEAPS=y +CONFIG_DMABUF_HEAPS_SYSTEM=y CONFIG_DUMMY=y CONFIG_DYNAMIC_FTRACE=y CONFIG_FPROBE=y
@@ -106,6 +108,7 @@ CONFIG_SECURITY=y CONFIG_SECURITYFS=y CONFIG_SYN_COOKIES=y CONFIG_TEST_BPF=m +CONFIG_UDMABUF=y CONFIG_USERFAULTFD=y CONFIG_VSOCKETS=y CONFIG_VXLAN=y
tools/testing/selftests/bpf/prog_tests/dmabuf_iter.c new file +244 -0
244 lines not on this page · raw message
tools/testing/selftests/bpf/progs/dmabuf_iter.c new file +53 -0
53 lines not on this page · raw message
Signature · 1 line
2.49.0.1151.ga128411c76-goog
PATCH v7 5/5 selftests/bpf: Add test for open coded dmabuf_iter
In reply to T.J. MercierUse the same test buffers as the traditional iterator and a new BPF map to verify the test buffers can be found with the open coded dmabuf iterator.
--- .../testing/selftests/bpf/bpf_experimental.h | 5 ++ .../selftests/bpf/prog_tests/dmabuf_iter.c | 41 ++++++++++++++++ .../testing/selftests/bpf/progs/dmabuf_iter.c | 48 +++++++++++++++++++ 3 files changed, 94 insertions(+)
tools/testing/selftests/bpf/bpf_experimental.h +5 -0
@@ -591,4 +591,9 @@ extern int bpf_iter_kmem_cache_new(struct bpf_iter_kmem_cache *it) __weak __ksymextern struct kmem_cache *bpf_iter_kmem_cache_next (struct bpf_iter_kmem_cache *it)__weak __ksym; extern void bpf_iter_kmem_cache_destroy(struct bpf_iter_kmem_cache *it) __weak __ksym; +struct bpf_iter_dmabuf ; +extern int bpf_iter_dmabuf_new(struct bpf_iter_dmabuf *it) __weak __ksym; +extern struct dma_buf *bpf_iter_dmabuf_next (struct bpf_iter_dmabuf *it)__weak __ksym; +extern void bpf_iter_dmabuf_destroy(struct bpf_iter_dmabuf *it) __weak __ksym; +#endif
tools/testing/selftests/bpf/prog_tests/dmabuf_iter.c +41 -0
63 lines not on this page · raw message
tools/testing/selftests/bpf/progs/dmabuf_iter.c +48 -0
57 lines not on this page · raw message
Signature · 1 line
2.49.0.1151.ga128411c76-goog
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 10 lines · On Thu, 22 May 2025 23:04:24 +0000 you wrote:
Until
CONFIG_DMABUF_SYSFS_STATSwas added [1] it was only possible to perform per-buffer accounting with debugfs which is not suitable for production environments. Eventually we discovered the overhead with per-buffer sysfs file creation/removal was significantly impacting allocation and free times, and exacerbated kernfs lock contention. [2]dma_buf_stats_setup()is responsible for 39% of single-page buffer creation duration, or 74% of single-pagedma_buf_export()duration when stressing dmabuf allocations and frees.[...]
Here is the summary with links:
- [bpf-next,v7,1/5] dma-buf: Rename debugfs symbols
https://git.kernel.org/bpf/bpf-next/c/89f9dba365e1 - [bpf-next,v7,2/5] bpf: Add dmabuf iterator https://git.kernel.org/bpf/bpf-next/c/76ea95534995 - [bpf-next,v7,3/5] bpf: Add open coded dmabuf iterator https://git.kernel.org/bpf/bpf-next/c/6eab7ac7c5ee - [bpf-next,v7,4/5] selftests/bpf: Add test for dmabuf_iter https://git.kernel.org/bpf/bpf-next/c/ae5d2c59ecd7 - [bpf-next,v7,5/5] selftests/bpf: Add test for open coded dmabuf_iter https://git.kernel.org/bpf/bpf-next/c/7594dcb71ff8
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html