*** Support BPF traversal of wakeup sources ***
Subsystems (5)
cover *** Support BPF traversal of wakeup sources ***
This patchset adds requisite kfuncs for BPF programs to safely traverse wakeup_sources, and puts a config flag around the sysfs interface.
Currently, a traversal of wakeup sources require going through /sys/class/wakeup/* or /d/wakeup_sources/*. The repeated syscalls to query sysfs is inefficient, as there can be hundreds of wakeup_sources, with each wakeup source also having multiple attributes. debugfs is unstable and insecure.
Adding kfuncs to lock/unlock wakeup sources allows BPF program to safely traverse the wakeup sources list. The head address of wakeup_sources can safely be resolved through BPF helper functions or variable attributes.
On a quiescent Pixel 6 traversing 150 wakeup_sources, I am seeing ~34x speedup (sampled 75 times in table below). For a device under load, the speedup is greater.
+-------+----+----------+----------+ | | n | AVG (ms) | STD (ms) | +-------+----+----------+----------+ | sysfs | 75 | 44.9 | 12.6 | +-------+----+----------+----------+ | BPF | 75 | 1.3 | 0.7 | +-------+----+----------+----------+
On the memory side, between kernfs, dentry, and kmalloc, each wakeup source removed from sysfs saves at least 10kB.
The initial attempts for BPF traversal of wakeup_sources was with BPF iterators [1]. However, BPF already allows for traversing of a simple list with bpf_for(), and this current patchset has the added benefit of being ~2-3x more performant than BPF iterators.
[1]: https://lore.kernel.org/all/[email protected]/
Samuel Wu (2):
PM: wakeup: Add kfuncs to lock/unlock wakeup_sources
PM: Add config flag to gate sysfs wakeup_sources
drivers/base/power/Makefile | 3 +- drivers/base/power/power.h | 14 +++++++++ drivers/base/power/wakeup.c | 63 +++++++++++++++++++++++++++++++++++-- kernel/power/Kconfig | 13 ++++++++ 4 files changed, 90 insertions(+), 3 deletions(-)
Signature · 1 line
2.53.0.959.g497ff81fa9-goog
PATCH 1/2 PM: wakeup: Add kfuncs to lock/unlock wakeup_sources
Add kfuncs to lock/unlock for safe traversal of wakeup sources.
Currently, a traversal of wakeup sources require going through /sys/class/wakeup/* or /d/wakeup_sources/*. The repeated syscalls to query sysfs is inefficient, as there can be hundreds of wakeup_sources, with each wakeup source also having multiple attributes. debugfs is unstable and insecure.
Adding kfuncs to lock/unlock wakeup sources allows BPF program to safely traverse the wakeup sources list. A new structure, bpf_ws_lock, acts as an opaque wrapper for the SRCU index. The head address of wakeup_sources can be safely resolved through BPF helper functions or variable attributes.
Doing the traversal in BPF is significantly more performant, and has an output in a format that the user specifies; this solves all the drawbacks of current interfaces.
--- drivers/base/power/wakeup.c | 63 +++++++++++++++++++++++++++++++++++-- 1 file changed, 61 insertions(+), 2 deletions(-)
drivers/base/power/wakeup.c +61 -2
@@ -1168,11 +1168,70 @@ static const struct file_operations wakeup_sources_stats_fops = { .release = seq_release_private, }; -static int __init wakeup_sources_debugfs_init (void ) +#ifdef CONFIG_BPF_SYSCALL +#include <linux/btf.h> + +struct bpf_ws_lock { }; + +__bpf_kfunc_start_defs (); + +/** +* bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources +* +* The underlying SRCU lock returns an integer index. However, the BPF verifier +* requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired +* resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque +* structure pointer (struct bpf_ws_lock *) to satisfy the verifier while +* safely encoding the integer index within the pointer address itself. +* +* Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL). +*/ +__bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void) +{ +return (struct bpf_ws_lock *)(long )(wakeup_sources_read_lock () +1 ); +} + +/** +* bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources +* @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock() +* +* The BPF verifier guarantees that @lock is a valid, unreleased pointer from +* the acquire function. We decode the pointer back into the integer SRCU index +* by subtracting 1 and release the lock. +*/ +__bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock) +{ +wakeup_sources_read_unlock ((int )(long )lock -1 ); +} + +__bpf_kfunc_end_defs (); + +BTF_KFUNCS_START (wakeup_source_kfunc_ids) +BTF_ID_FLAGS (func ,bpf_wakeup_sources_read_lock ,KF_ACQUIRE ) +BTF_ID_FLAGS (func, bpf_wakeup_sources_read_unlock,KF_RELEASE ) +BTF_KFUNCS_END (wakeup_source_kfunc_ids) + +static const struct btf_kfunc_id_set wakeup_source_kfunc_set = { + .owner =THIS_MODULE , + .set = &wakeup_source_kfunc_ids, +}; + +static void __init wakeup_sources_bpf_init (void ) +{ +if (register_btf_kfunc_id_set (BPF_PROG_TYPE_SYSCALL , &wakeup_source_kfunc_set)) +pm_pr_dbg ("Wakeup: failed to register BTF kfuncs\n" ); +} +#else +static inline void wakeup_sources_bpf_init (void ) {} +#endif /* CONFIG_BPF_SYSCALL */ + +static int __init wakeup_sources_init (void ) {debugfs_create_file ("wakeup_sources" ,0444 ,NULL ,NULL , &wakeup_sources_stats_fops); +wakeup_sources_bpf_init (); +return 0 ; } -postcore_initcall (wakeup_sources_debugfs_init); +postcore_initcall (wakeup_sources_init);
Signature · 1 line
2.53.0.959.g497ff81fa9-goog
kernel test robot
build botRe: [PATCH v1 1/2] PM: wakeup: Add kfuncs to lock/unlock wakeup_sources
kernel test robot noticed the following build warnings:
Hi Samuel,
kernel test robot noticed the following build warnings:
[auto build test WARNING on rafael-pm/linux-next]
[also build test WARNING on rafael-pm/bleeding-edge linus/master v7.0-rc5 next-20260320] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Samuel-Wu/PM-wakeup-Add-kfuncs-to-lock-unlock-wakeup_sources/20260323-064540 base: https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git linux-next patch link: https://lore.kernel.org/r/20260320160055.4114055-2-wusamuel%40google.com patch subject: [PATCH v1 1/2] PM: wakeup: Add kfuncs to lock/unlock wakeup_sources config: x86_64-randconfig-121-20260323 (https://download.01.org/0day-ci/archive/20260323/[email protected]/config) compiler: gcc-12 (Debian 12.4.0-5) 12.4.0 sparse: v0.6.5-rc1 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260323/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
>> drivers/base/power/wakeup.c:1189:32: sparse: sparse: symbol 'bpf_wakeup_sources_read_lock' was not declared. Should it be static? >> drivers/base/power/wakeup.c:1202:18: sparse: sparse: symbol 'bpf_wakeup_sources_read_unlock' was not declared. Should it be static?
vim +/bpf_wakeup_sources_read_lock +1189 drivers/base/power/wakeup.c
1177
1178 /**
1179 * bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources
1180 *
1181 * The underlying SRCU lock returns an integer index. However, the BPF verifier
1182 * requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired
1183 * resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque
1184 * structure pointer (struct bpf_ws_lock *) to satisfy the verifier while
1185 * safely encoding the integer index within the pointer address itself.
1186 *
1187 * Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL).
1188 */
> 1189 __bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void)
1190 {
1191 return (struct bpf_ws_lock *)(long)(wakeup_sources_read_lock() + 1);
1192 }
1193
1194 /**
1195 * bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources
1196 * @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock()
1197 *
1198 * The BPF verifier guarantees that @lock is a valid, unreleased pointer from
1199 * the acquire function. We decode the pointer back into the integer SRCU index
1200 * by subtracting 1 and release the lock.
1201 */
> 1202 __bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock) 1203 {
1204 wakeup_sources_read_unlock((int)(long)lock - 1);
1205 }
1206
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 2/2 PM: Add config flag to gate sysfs wakeup_sources
In reply to Samuel WuAdd a config flag that gates the creation of /sys/class/wakeup_sources/*. This has the benefit of eliminating the work needed to create the nodes and corresponding attributes; and between kernfs, dentry, and malloc, there are some memory savings depending on the number of wakeup sources.
--- drivers/base/power/Makefile | 3 ++- drivers/base/power/power.h | 14 ++++++++++++++ kernel/power/Kconfig | 13 +++++++++++++ 3 files changed, 29 insertions(+), 1 deletion(-)
drivers/base/power/Makefile +2 -1
@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_PM) += sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o -obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o wakeup_stats.o +obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o +obj-$(CONFIG_PM_WAKEUP_STATS_SYSFS) += wakeup_stats.o obj-$(CONFIG_PM_TRACE_RTC) += trace.o obj-$(CONFIG_HAVE_CLK) += clock_ops.o obj-$(CONFIG_PM_QOS_KUNIT_TEST) += qos-test.o
drivers/base/power/power.h +14 -0
@@ -125,11 +125,25 @@ static inline bool device_pm_initialized(struct device *dev) }/* drivers/base/power/wakeup_stats.c */ +#ifdef CONFIG_PM_WAKEUP_STATS_SYSFS extern int wakeup_source_sysfs_add (struct device *parent,struct wakeup_source *ws);extern void wakeup_source_sysfs_remove (struct wakeup_source *ws);extern int pm_wakeup_source_sysfs_add (struct device *parent); +#else /* !CONFIG_PM_WAKEUP_STATS_SYSFS */ +static inline int wakeup_source_sysfs_add (struct device *parent, +struct wakeup_source *ws) +{ +return 0 ; +} +static inline void wakeup_source_sysfs_remove (struct wakeup_source *ws) {} + +static inline int pm_wakeup_source_sysfs_add (struct device *parent) +{ +return 0 ; +} +#endif /* !CONFIG_PM_WAKEUP_STATS_SYSFS */ #else /* !CONFIG_PM_SLEEP */
kernel/power/Kconfig +13 -0
@@ -202,6 +202,19 @@ config PM_WAKELOCKS_GC depends on PM_WAKELOCKS default y +config PM_WAKEUP_STATS_SYSFS + bool "Sysfs wakeup statistics" + depends on PM_SLEEP + default y + help + Enable this for wakeup statistics in sysfs under /sys/class/wakeup/ + + Disabling this option eliminates the work of creating the wakeup + sources and each of their attributes in sysfs. Depending on the + number of wakeup sources, this can also have a non-negligible memory + impact. Regardless of this config option's value, wakeup statistics + are still available via debugfs and BPF. + config PM_QOS_CPU_SYSTEM_WAKEUP bool "User space interface for CPU system wakeup QoS" depends on CPU_IDLE
Signature · 1 line
2.53.0.959.g497ff81fa9-goog
Rafael J. Wysocki
maintainerRe: [PATCH v1 2/2] PM: Add config flag to gate sysfs wakeup_sources
Quoted 67 lines · Samuel Wu
Add a config flag that gates the creation of
/sys/class/wakeup_sources/*. This has the benefit of eliminating the work needed to create the nodes and corresponding attributes; and between kernfs, dentry, and malloc, there are some memory savings depending on the number of wakeup sources.Signed-off-by Samuel Wu <[email protected]>--- drivers/base/power/Makefile | 3 ++- drivers/base/power/power.h | 14 ++++++++++++++ kernel/power/Kconfig | 13 +++++++++++++ 3 files changed, 29 insertions(+), 1 deletion(-)drivers/base/power/Makefile +2
-1@@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_PM) += sysfs.o generic_ops.o common.o qos.o runtime.o wakeirq.o -obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o wakeup_stats.o +obj-$(CONFIG_PM_SLEEP) += main.o wakeup.o +obj-$(CONFIG_PM_WAKEUP_STATS_SYSFS) += wakeup_stats.o obj-$(CONFIG_PM_TRACE_RTC) += trace.o obj-$(CONFIG_HAVE_CLK) += clock_ops.o obj-$(CONFIG_PM_QOS_KUNIT_TEST) += qos-test.odrivers/base/power/power.h +14
-0@@ -125,11 +125,25 @@ static inline bool device_pm_initialized(struct device *dev) }/* drivers/base/power/wakeup_stats.c */ +#ifdef CONFIG_PM_WAKEUP_STATS_SYSFS extern int wakeup_source_sysfs_add (struct device *parent,struct wakeup_source *ws);extern void wakeup_source_sysfs_remove (struct wakeup_source *ws);extern int pm_wakeup_source_sysfs_add (struct device *parent); +#else /* !CONFIG_PM_WAKEUP_STATS_SYSFS */ +static inline int wakeup_source_sysfs_add (struct device *parent, +struct wakeup_source *ws) +{ +return 0 ; +} +static inline void wakeup_source_sysfs_remove (struct wakeup_source *ws) {} + +static inline int pm_wakeup_source_sysfs_add (struct device *parent) +{ +return 0 ; +} +#endif /* !CONFIG_PM_WAKEUP_STATS_SYSFS */ #else /* !CONFIG_PM_SLEEP */ kernel/power/Kconfig +2
-0@@ -202,6 +202,19 @@ config PM_WAKELOCKS_GC depends on PM_WAKELOCKS default y +config PM_WAKEUP_STATS_SYSFS + bool "Sysfs wakeup statistics"
kernel/power/Kconfig config PM_WAKELOCKS_GC · line 216
Quoted 10 lines
kernel/power/Kconfig +10
-0+ depends on PM_SLEEP + default y + help + Enable this for wakeup statistics in sysfs under /sys/class/wakeup/ + + Disabling this option eliminates the work of creating the wakeup + sources and each of their attributes in sysfs. Depending on the + number of wakeup sources, this can also have a non-negligible memory + impact. Regardless of this config option's value, wakeup statistics + are still available via debugfs and BPF.
Well, except that otherwise it may be hard to figure out which device the given wakeup source belongs to, so I'd rather not do that.
Quoted 5 lines
kernel/power/Kconfig +1
-1+ config PM_QOS_CPU_SYSTEM_WAKEUP bool "User space interface for CPU system wakeup QoS" depends on CPU_IDLE --
Re: [PATCH v1 2/2] PM: Add config flag to gate sysfs wakeup_sources
Quoted 1 line · Rafael J. Wysocki
[ ... ]
Quoted 17 lines
kernel/power/Kconfig config PM_WAKELOCKS_GC · line 216
Quoted 13 lines
kernel/power/Kconfig +12
-0+config PM_WAKEUP_STATS_SYSFS + bool "Sysfs wakeup statistics" + depends on PM_SLEEP + default y + help + Enable this for wakeup statistics in sysfs under /sys/class/wakeup/ + + Disabling this option eliminates the work of creating the wakeup + sources and each of their attributes in sysfs. Depending on the + number of wakeup sources, this can also have a non-negligible memory + impact. Regardless of this config option's value, wakeup statistics + are still available via debugfs and BPF.Well, except that otherwise it may be hard to figure out which device the given wakeup source belongs to, so I'd rather not do that.
Since the config flag defaults to yes, this wouldn't affect current users right? And then the idea is that other users who don't need this sysfs node can turn the flag off to save some cycles and memory.
[ ... ]
Kumar Kartikeya Dwivedi
maintainerQuoted 25 lines · Samuel Wu
This patchset adds requisite kfuncs for BPF programs to safely traverse
wakeup_sources, and puts a config flag around the sysfs interface.Currently, a traversal of wakeup sources require going through
/sys/class/wakeup/*or /d/wakeup_sources/*. The repeated syscalls to query sysfs is inefficient, as there can be hundreds ofwakeup_sources, with each wakeup source also having multiple attributes. debugfs is unstable and insecure.Adding kfuncs to lock/unlock wakeup sources allows BPF program to safely traverse the wakeup sources list. The head address of
wakeup_sourcescan safely be resolved through BPF helper functions or variable attributes.On a quiescent Pixel 6 traversing 150
wakeup_sources, I am seeing ~34x speedup (sampled 75 times in table below). For a device under load, the speedup is greater.+-------+----+----------+----------+ | | n | AVG (ms) | STD (ms) | +-------+----+----------+----------+ | sysfs | 75 | 44.9 | 12.6 | +-------+----+----------+----------+ | BPF | 75 | 1.3 | 0.7 | +-------+----+----------+----------+
Quoted 7 lines
On the memory side, between kernfs, dentry, and kmalloc, each wakeup source removed from sysfs saves at least 10kB.
The initial attempts for BPF traversal of
wakeup_sourceswas with BPF iterators [1]. However, BPF already allows for traversing of a simple list withbpf_for(), and this current patchset has the added benefit of being ~2-3x more performant than BPF iterators.
See, it ended up being faster ;-).
That said, I didn't understand why you dropped the test. We still need unit tests that ensure the verifier causes the bpf_ws_lock to be released before exit, and an example program (like the one you benchmarked with) to demonstrate usage. Both are mandatory. See various *_fail.c files in progs/ and RUN_TESTS() macro for running negative tests easily. I doubt we will have any wakeup_sources in the CI environment, but the example has value nonetheless.
pw-bot: cr
Quoted 16 lines
[1]: https://lore.kernel.org/all/[email protected]/
Samuel Wu (2):
PM: wakeup: Add kfuncs to lock/unlockwakeup_sources
PM: Add config flag to gate sysfswakeup_sourcesdrivers/base/power/Makefile | 3 +- drivers/base/power/power.h | 14 +++++++++ drivers/base/power/wakeup.c | 63 +++++++++++++++++++++++++++++++++++-- kernel/power/Kconfig | 13 ++++++++ 4 files changed, 90 insertions(+), 3 deletions(-)--
2.53.0.959.g497ff81fa9-goog
Quoted 1 line · Kumar Kartikeya Dwivedi
[ ... ]
Quoted 7 lines
Quoted 5 lines
The initial attempts for BPF traversal of
wakeup_sourceswas with BPF iterators [1]. However, BPF already allows for traversing of a simple list withbpf_for(), and this current patchset has the added benefit of being ~2-3x more performant than BPF iterators.See, it ended up being faster ;-).
Yup, thanks for the suggestion!
Quoted 11 lines
That said, I didn't understand why you dropped the test. We still need unit tests that ensure the verifier causes the
bpf_ws_lockto be released before exit, and an example program (like the one you benchmarked with) to demonstrate usage. Both are mandatory. See various *_fail.c files in progs/ andRUN_TESTS()macro for running negative tests easily. I doubt we will have anywakeup_sourcesin the CI environment, but the example has value nonetheless.pw-bot: cr
Sounds good, in v2 I can add a test for the lock/unlock kfunc. And thanks for the thorough feedback, let me take a look at the *_fail.c files and test macros.