Statsfs: a new ram-based file sytem for Linux kernel statistics
v2 cover Statsfs: a new ram-based file sytem for Linux kernel statistics
There is currently no common way for Linux kernel subsystems to expose statistics to userspace shared throughout the Linux kernel; subsystems have to take care of gathering and displaying statistics by themselves, for example in the form of files in debugfs. For example KVM has its own code section that takes care of this in virt/kvm/kvm_main.c, where it sets up debugfs handlers for displaying values and aggregating them from various subfolders to obtain information about the system state (i.e. displaying the total number of exits, calculated by summing all exits of all cpus of all running virtual machines).
Allowing each section of the kernel to do so has two disadvantages. First, it will introduce redundant code. Second, debugfs is anyway not the right place for statistics (for example it is affected by lockdown)
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
The file system is mounted on /sys/kernel/stats and would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].
Statsfs offers a generic and stable API, allowing any kind of directory/file organization and supporting multiple kind of aggregations (not only sum, but also average, max, min and count_zero) and data types (all unsigned and signed types plus boolean). The implementation, which is a generalization of KVM’s debugfs statistics code, takes care of gathering and displaying information at run time; users only need to specify the values to be included in each source.
Statsfs would also be a different mountpoint from debugfs, and would not suffer from limited access due to the security lock down patches. Its main function is to display each statistics as a file in the desired folder hierarchy defined through the API. Statsfs files can be read, and possibly cleared if their file mode allows it.
Statsfs has two main components: the public API defined by include/linux/statsfs.h, and the virtual file system which should end up in /sys/kernel/stats.
The API has two main elements, values and sources. Kernel subsystems like KVM can use the API to create a source, add child sources/values/aggregates and register it to the root source (that on the virtual fs would be /sys/kernel/statsfs).
Sources are created via statsfs_source_create(), and each source becomes a directory in the file system. Sources form a parent-child relationship; root sources are added to the file system via statsfs_source_register(). Every other source is added to or removed from a parent through the statsfs_source_add_subordinate and statsfs_source_remote_subordinate APIs. Once a source is created and added to the tree (via add_subordinate), it will be used to compute aggregate values in the parent source.
Values represent quantites that are gathered by the statsfs user. Examples of values include the number of vm exits of a given kind, the amount of memory used by some data structure, the length of the longest hash table chain, or anything like that. Values are defined with the statsfs_source_add_values function. Each value is defined by a struct statsfs_value; the same statsfs_value can be added to many different sources. A value can be considered "simple" if it fetches data from a user-provided location, or "aggregate" if it groups all values in the subordinates sources that include the same statsfs_value.
For more information, please consult the kerneldoc documentation in patch 2 and the sample uses in the kunit tests and in KVM.
This series of patches is based on my previous series "libfs: group and simplify linux fs code" and the single patch sent to kvm "kvm_host: unify VM_STAT and VCPU_STAT definitions in a single place". The former simplifies code duplicated in debugfs and tracefs (from which statsfs is based on), the latter groups all macros definition for statistics in kvm in a single common file shared by all architectures.
Patch 1 adds a new refcount and kref destructor wrappers that take a semaphore, as those are used later by statsfs. Patch 2 introduces the statsfs API, patch 3 provides extensive tests that can also be used as example on how to use the API and patch 4 adds the file system support. Finally, patch 5 provides a real-life example of statsfs usage in KVM.
v1->v2 remove unnecessary list_foreach_safe loops, fix wrong indentation, change statsfs in stats_fs
Emanuele Giuseppe Esposito (5):
refcount, kref: add dec-and-test wrappers for rw_semaphores stats_fs API: create, add and remove stats_fs sources and values
kunit: tests for stats_fs API
stats_fs fs: virtual fs to show stats to the end-user
kvm_main: replace debugfs with stats_fs
MAINTAINERS | 7 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 6 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 -- arch/x86/kvm/stats_fs.c | 56 ++ arch/x86/kvm/x86.c | 6 +- fs/Kconfig | 12 + fs/Makefile | 1 + fs/stats_fs/Makefile | 6 + fs/stats_fs/inode.c | 337 ++++++++++ fs/stats_fs/internal.h | 35 + fs/stats_fs/stats_fs-tests.c | 1088 +++++++++++++++++++++++++++++++ fs/stats_fs/stats_fs.c | 773 ++++++++++++++++++++++ include/linux/kref.h | 11 + include/linux/kvm_host.h | 39 +- include/linux/refcount.h | 2 + include/linux/stats_fs.h | 304 +++++++++ include/uapi/linux/magic.h | 1 + lib/refcount.c | 32 + tools/lib/api/fs/fs.c | 21 + virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 314 ++------- 32 files changed, 2772 insertions(+), 382 deletions(-) delete mode 100644 arch/x86/kvm/debugfs.c create mode 100644 arch/x86/kvm/stats_fs.c create mode 100644 fs/stats_fs/Makefile create mode 100644 fs/stats_fs/inode.c create mode 100644 fs/stats_fs/internal.h create mode 100644 fs/stats_fs/stats_fs-tests.c create mode 100644 fs/stats_fs/stats_fs.c create mode 100644 include/linux/stats_fs.h
Signature · 1 line
2.25.2
PATCH v2 1/5 refcount, kref: add dec-and-test wrappers for rw_semaphores
Similar to the existing functions that take a mutex or spinlock if and only if a reference count is decremented to zero, these new function take an rwsem for writing just before the refcount reaches 0 (and call a user-provided function in the case of kref_put_rwsem).
These will be used for stats_fs_source data structures, which are protected by an rw_semaphore to allow concurrent sysfs reads.
--- include/linux/kref.h | 11 +++++++++++ include/linux/refcount.h | 2 ++ lib/refcount.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+)
include/linux/kref.h +11 -0
@@ -79,6 +79,17 @@ static inline int kref_put_mutex(struct kref *kref,return 0 ; } +static inline int kref_put_rwsem (struct kref *kref, +void (*release)(struct kref *kref), +struct rw_semaphore *rwsem) +{ +if (refcount_dec_and_down_write (&kref->refcount, rwsem)) { +release (kref); +return 1 ; + } +return 0 ; +} +static inline int kref_put_lock (struct kref *kref,void (*release)(struct kref *kref),spinlock_t *lock)
include/linux/refcount.h +2 -0
@@ -99,6 +99,7 @@#include <linux/spinlock_types.h> struct mutex ; +struct rw_semaphore ;/** * struct refcount_t - variant of atomic_t specialized for reference counts
@@ -313,6 +314,7 @@ static inline void refcount_dec(refcount_t *r)extern __must_check bool refcount_dec_if_one (refcount_t *r);extern __must_check bool refcount_dec_not_one (refcount_t *r);extern __must_check bool refcount_dec_and_mutex_lock (refcount_t *r,struct mutex *lock); +extern __must_check bool refcount_dec_and_down_write (refcount_t *r,struct rw_semaphore *rwsem);extern __must_check bool refcount_dec_and_lock (refcount_t *r,spinlock_t *lock); extern __must_check bool refcount_dec_and_lock_irqsave(refcount_t *r, spinlock_t *lock,
lib/refcount.c +32 -0
@@ -4,6 +4,7 @@*/ #include <linux/mutex.h> +#include <linux/rwsem.h> #include <linux/refcount.h> #include <linux/spinlock.h> #include <linux/bug.h>
@@ -94,6 +95,37 @@ bool refcount_dec_not_one(refcount_t *r) }EXPORT_SYMBOL (refcount_dec_not_one); +/** +* refcount_dec_and_down_write - return holding rwsem for writing if able to decrement +* refcount to 0 +* @r: the refcount +* @lock: the mutex to be locked +* +* Similar to atomic_dec_and_mutex_lock(), it will WARN on underflow and fail +* to decrement when saturated at REFCOUNT_SATURATED. +* +* Provides release memory ordering, such that prior loads and stores are done +* before, and provides a control dependency such that free() must come after. +* See the comment on top. +* +* Return: true and hold rwsem for writing if able to decrement refcount to 0, false +* otherwise +*/ +bool refcount_dec_and_down_write (refcount_t *r,struct rw_semaphore *lock) +{ +if (refcount_dec_not_one (r)) +return false; + +down_write (lock); +if (!refcount_dec_and_test (r)) { +up_write (lock); +return false; + } + +return true; +} +EXPORT_SYMBOL (refcount_dec_and_down_write); +/** * refcount_dec_and_mutex_lock - return holding mutex if able to decrement * refcount to 0
Signature · 1 line
2.25.2
PATCH v2 2/5 stats_fs API: create, add and remove stats_fs sources and values
In reply to Emanuele Giuseppe EspositoIntroduction to the stats_fs API, that allows to easily create, add and remove stats_fs sources and values. The API allows to easily building the statistics directory tree to automatically gather them for the linux kernel. The main functionalities are: create a source, add child sources/values/aggregates, register it to the root source (that on the virtual fs would be /sys/kernel/statsfs), ad perform a search for a value/aggregate.
This allows creating any kind of source tree, making it more flexible also to future readjustments.
The API representation is only logical and will be backed up by a virtual file system in patch 4.
Its usage will be shared between the stats_fs file system and the end-users like kvm, the former calling it when it needs to display and clear statistics, the latter to add values and sources.
--- MAINTAINERS | 7 + fs/Kconfig | 6 + fs/Makefile | 1 + fs/stats_fs/Makefile | 4 + fs/stats_fs/internal.h | 20 ++ fs/stats_fs/stats_fs.c | 610 +++++++++++++++++++++++++++++++++++++++ include/linux/stats_fs.h | 289 +++++++++++++++++++ 7 files changed, 937 insertions(+) create mode 100644 fs/stats_fs/Makefile create mode 100644 fs/stats_fs/internal.h create mode 100644 fs/stats_fs/stats_fs.c create mode 100644 include/linux/stats_fs.h
MAINTAINERS +7 -0
@@ -5229,6 +5229,13 @@ F: include/linux/debugfs.h F: include/linux/kobj* F: lib/kobj* +STATS_FS +M: Paolo Bonzini <[email protected]> +R: Emanuele Giuseppe Esposito <[email protected]> +S: Supported +F: include/linux/stats_fs.h +F: fs/stats_fs + DRIVERS FOR ADAPTIVE VOLTAGE SCALING (AVS) M: Kevin Hilman <[email protected]> M: Nishanth Menon <[email protected]>
fs/Kconfig +6 -0
@@ -328,4 +328,10 @@ source "fs/unicode/Kconfig" config IO_WQ bool +config STATS_FS + bool "Statistics Filesystem" + help + stats_fs is a virtual file system that provides counters and + other statistics about the running kernel. + endmenu
fs/Makefile +1 -0
@@ -125,6 +125,7 @@ obj-$(CONFIG_BEFS_FS) += befs/ obj-$(CONFIG_HOSTFS) += hostfs/ obj-$(CONFIG_CACHEFILES) += cachefiles/ obj-$(CONFIG_DEBUG_FS) += debugfs/ +obj-$(CONFIG_STATS_FS) += stats_fs/ obj-$(CONFIG_TRACING) += tracefs/ obj-$(CONFIG_OCFS2_FS) += ocfs2/ obj-$(CONFIG_BTRFS_FS) += btrfs/
fs/stats_fs/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only +stats_fs-objs := stats_fs.o + +obj-$(CONFIG_STATS_FS) += stats_fs.o
fs/stats_fs/internal.h new file +20 -0
@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATS_FS_INTERNAL_H_ +#define _STATS_FS_INTERNAL_H_ + +#include <linux/list.h> +#include <linux/kref.h> +#include <linux/rwsem.h> +#include <linux/stats_fs.h> + +/* values, grouped by base */ +struct stats_fs_value_source { +void *base_addr; +bool files_created; +struct stats_fs_value *values; +struct list_head list_element; +}; + +int stats_fs_val_get_mode (struct stats_fs_value *val); + +#endif /* _STATS_FS_INTERNAL_H_ */
fs/stats_fs/stats_fs.c new file +608 -0
608 lines not on this page · raw message
include/linux/stats_fs.h new file +289 -0
289 lines not on this page · raw message
Signature · 1 line
2.25.2
Re: [PATCH v2 2/5] stats_fs API: create, add and remove stats_fs sources and values
Quoted 5 lines · Emanuele Giuseppe Esposito
fs/Kconfig +0
-0
fs/Kconfig source "fs/unicode/Kconfig" · line 337
Quoted 10 lines
fs/Kconfig +6
-0@@ -328,4 +328,10 @@ source "fs/unicode/Kconfig" config IO_WQ bool +config STATS_FS + bool "Statistics Filesystem" + help + stats_fs is a virtual file system that provides counters and + other statistics about the running kernel. + endmenu
Hi,
This kconfig entry should be under (inside) "Pseudo filesystems", i.e., between 'menu "Pseudo filesystems"' and its corresponding "endmenu".
Thanks.
Signature · 1 line
~Randy
PATCH v2 3/5 kunit: tests for stats_fs API
In reply to Emanuele Giuseppe EspositoAdd kunit tests to extensively test the stats_fs API functionality.
In order to run them, the kernel .config must set CONFIG_KUNIT=y and a new .kunitconfig file must be created with CONFIG_STATS_FS=y and CONFIG_STATS_FS_TEST=y
Tests can be then started by running the following command from the root directory of the linux kernel source tree:./tools/testing/kunit/kunit.py run --timeout0 --jobs=nproc --all
--- fs/Kconfig | 6 + fs/stats_fs/Makefile | 2 + fs/stats_fs/stats_fs-tests.c | 1088 ++++++++++++++++++++++++++++++++++ 3 files changed, 1096 insertions(+) create mode 100644 fs/stats_fs/stats_fs-tests.c
fs/Kconfig +6 -0
@@ -334,4 +334,10 @@ config STATS_FS stats_fs is a virtual file system that provides counters and other statistics about the running kernel. +config STATS_FS_TEST + bool "Tests for stats_fs" + depends on STATS_FS && KUNIT + help + tests for the stats_fs API. + endmenu
fs/stats_fs/Makefile +2 -0
@@ -1,4 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only stats_fs-objs := stats_fs.o +stats_fs-tests-objs := stats_fs-tests.o obj-$(CONFIG_STATS_FS) += stats_fs.o +obj-$(CONFIG_STATS_FS_TEST) += stats_fs-tests.o
fs/stats_fs/stats_fs-tests.c new file +1088 -0
1,088 lines not on this page · raw message
Signature · 1 line
2.25.2
PATCH v2 4/5 stats_fs fs: virtual fs to show stats to the end-user
In reply to Emanuele Giuseppe EspositoAdd virtual fs that maps stats_fs sources with directories, and values (simple or aggregates) to files.
Every time a file is read/cleared, the fs internally invokes the stats_fs API to get/set the requested value.
fs/stats_fs/inode.c is pretty much similar to what is done in fs/debugfs/inode.c, with the exception that the API is only composed by stats_fs_create_file, stats_fs_create_dir and stats_fs_remove.
--- fs/stats_fs/Makefile | 2 +- fs/stats_fs/inode.c | 337 +++++++++++++++++++++++++++++++++++++ fs/stats_fs/internal.h | 15 ++ fs/stats_fs/stats_fs.c | 163 ++++++++++++++++++ include/linux/stats_fs.h | 15 ++ include/uapi/linux/magic.h | 1 + tools/lib/api/fs/fs.c | 21 +++ 7 files changed, 553 insertions(+), 1 deletion(-) create mode 100644 fs/stats_fs/inode.c
fs/stats_fs/Makefile +1 -1
@@ -1,5 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only -stats_fs-objs := stats_fs.o +stats_fs-objs := inode.o stats_fs.o stats_fs-tests-objs := stats_fs-tests.o obj-$(CONFIG_STATS_FS) += stats_fs.o
fs/stats_fs/inode.c new file +337 -0
337 lines not on this page · raw message
fs/stats_fs/internal.h +15 -0
@@ -15,6 +15,21 @@ struct stats_fs_value_source { struct list_head list_element; }; +struct stats_fs_data_inode { +struct stats_fs_source *src; +struct stats_fs_value *val; +}; + +extern const struct file_operations stats_fs_ops; + +struct dentry *stats_fs_create_file (struct stats_fs_value *val, +struct stats_fs_source *src); + +struct dentry *stats_fs_create_dir (const char *name,struct dentry *parent); + +void stats_fs_remove (struct dentry *dentry); +#define stats_fs_remove_recursive stats_fs_remove +int stats_fs_val_get_mode (struct stats_fs_value *val);#endif /* _STATS_FS_INTERNAL_H_ */
fs/stats_fs/stats_fs.c +160 -0
206 lines not on this page · raw message
include/linux/stats_fs.h +15 -0
@@ -87,6 +87,18 @@ struct stats_fs_source {*/ struct stats_fs_source *stats_fs_source_create (const char *fmt, ...); +/** +* stats_fs_source_register - register a source in the stats_fs filesystem +* @source: a pointer to the source that will be registered +* +* Add the given folder as direct child of /sys/kernel/statsfs. +* It also starts to recursively search its own child and create all folders +* and files if they weren't already. All subsequent add_subordinate calls +* on the same source that is used in this function will create corresponding +* files and directories. +*/ +void stats_fs_source_register (struct stats_fs_source *source); +/** * stats_fs_source_add_values - adds values to the given source * @source: a pointer to the source that will receive the values
@@ -235,6 +247,9 @@ static inline struct stats_fs_source *stats_fs_source_create(const char *fmt,return ERR_PTR (-ENODEV ); } +static inline void stats_fs_source_register (struct stats_fs_source *source) +{ } +static inline int stats_fs_source_add_values (struct stats_fs_source *source,struct stats_fs_value *val,void *base_ptr)
include/uapi/linux/magic.h +1 -0
@@ -10,6 +10,7 @@#define CRAMFS_MAGIC 0x28cd3d45/* some random number */ #define CRAMFS_MAGIC_WEND 0x453dcd28/* magic number with the wrong endianess */ #define DEBUGFS_MAGIC 0x64626720 +#define STATSFS_MAGIC 0x73746174#define SECURITYFS_MAGIC 0x73636673#define SELINUX_MAGIC 0xf97cff8c#define SMACK_MAGIC 0x43415d53/* "SMAC" */
tools/lib/api/fs/fs.c +21 -0
@@ -35,6 +35,10 @@#define TRACEFS_MAGIC 0x74726163#endif +#ifndef STATSFS_MAGIC +#define STATSFS_MAGIC 0x73746174 +#endif +#ifndef HUGETLBFS_MAGIC #define HUGETLBFS_MAGIC 0x958458f6#endif
@@ -76,6 +80,16 @@ static const char * const tracefs__known_mountpoints[] = { 0, }; +#ifndef STATSFS_DEFAULT_PATH +#define STATSFS_DEFAULT_PATH "/sys/kernel/statsfs" +#endif + +static const char *const statsfs__known_mountpoints[] = { +STATSFS_DEFAULT_PATH , +"/statsfs" , +0 , +}; +static const char *const hugetlbfs__known_mountpoints[] = {0 , };
@@ -100,6 +114,7 @@ enum {FS__TRACEFS =3 ,FS__HUGETLBFS =4 ,FS__BPF_FS =5 , + FS__STATSFS = 6, };#ifndef TRACEFS_MAGIC
@@ -127,6 +142,11 @@ static struct fs fs__entries[] = { .mounts = tracefs__known_mountpoints, .magic = TRACEFS_MAGIC, }, + [FS__STATSFS] = { + .name = "statsfs", + .mounts = statsfs__known_mountpoints, + .magic = STATSFS_MAGIC, + }, [FS__HUGETLBFS] = { .name = "hugetlbfs", .mounts = hugetlbfs__known_mountpoints,
@@ -297,6 +317,7 @@ FS(sysfs, FS__SYSFS);FS (procfs,FS__PROCFS );FS (debugfs,FS__DEBUGFS );FS (tracefs,FS__TRACEFS ); +FS (statsfs,FS__STATSFS );FS (hugetlbfs,FS__HUGETLBFS );FS (bpf_fs,FS__BPF_FS );
Signature · 1 line
2.25.2
PATCH v2 5/5 kvm_main: replace debugfs with stats_fs
In reply to Emanuele Giuseppe EspositoUse stats_fs API instead of debugfs to create sources and add values.
This also requires to change all architecture files to replace the old debugfs_entries with stats_fs_vcpu_entries and statsfs_vm_entries.
The files/folders name and organization is kept unchanged, and a symlink in sys/kernel/debugfs/kvm is left for backward compatibility.
--- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 6 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 ------- arch/x86/kvm/stats_fs.c | 56 ++++++ arch/x86/kvm/x86.c | 6 +- include/linux/kvm_host.h | 39 +--- virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 314 ++++---------------------------- 18 files changed, 142 insertions(+), 382 deletions(-) delete mode 100644 arch/x86/kvm/debugfs.c create mode 100644 arch/x86/kvm/stats_fs.c
arch/arm64/kvm/Kconfig +1 -0
@@ -23,6 +23,7 @@ config KVM depends on OF # for TASKSTATS/TASK_DELAY_ACCT: depends on NET && MULTIUSER + imply STATS_FS select MMU_NOTIFIER select PREEMPT_NOTIFIERS select HAVE_KVM_CPU_RELAX_INTERCEPT
arch/arm64/kvm/guest.c +1 -1
@@ -29,7 +29,7 @@#include "trace.h" -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("halt_successful_poll" , halt_successful_poll),VCPU_STAT ("halt_attempted_poll" , halt_attempted_poll),VCPU_STAT ("halt_poll_invalid" , halt_poll_invalid),
arch/mips/kvm/Kconfig +1 -0
@@ -19,6 +19,7 @@ config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM depends on MIPS_FP_SUPPORT + imply STATS_FS select EXPORT_UASM select PREEMPT_NOTIFIERS select KVM_GENERIC_DIRTYLOG_READ_PROTECT
arch/mips/kvm/mips.c +1 -1
@@ -39,7 +39,7 @@#define VECTORSPACING 0x100/* for EI/VI mode */ #endif -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("wait" , wait_exits),VCPU_STAT ("cache" , cache_exits),VCPU_STAT ("signal" , signal_exits),
arch/powerpc/kvm/Kconfig +1 -0
@@ -19,6 +19,7 @@ if VIRTUALIZATION config KVM bool + imply STATS_FS select PREEMPT_NOTIFIERS select HAVE_KVM_EVENTFD select HAVE_KVM_VCPU_ASYNC_IOCTL
arch/powerpc/kvm/book3s.c +5 -1
@@ -38,7 +38,7 @@/* #define EXIT_DEBUG */ -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("exits" , sum_exits),VCPU_STAT ("mmio" , mmio_exits),VCPU_STAT ("sig" , signal_exits),
@@ -66,6 +66,10 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("pthru_all" , pthru_all),VCPU_STAT ("pthru_host" , pthru_host),VCPU_STAT ("pthru_bad_aff" , pthru_bad_aff), + { NULL } +}; + +struct stats_fs_value stats_fs_vm_entries[] = { VM_STAT("largepages_2M", num_2M_pages, .mode = 0444), VM_STAT("largepages_1G", num_1G_pages, .mode = 0444), { NULL }
arch/powerpc/kvm/booke.c +6 -2
@@ -35,7 +35,12 @@unsigned long kvmppc_booke_handlers; -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vm_entries[] = { +VM_STAT ("remote_tlb_flush" , remote_tlb_flush), + {NULL } +}; + +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("mmio" , mmio_exits),VCPU_STAT ("sig" , signal_exits),VCPU_STAT ("itlb_r" , itlb_real_miss_exits),
@@ -54,7 +59,6 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("halt_wakeup" , halt_wakeup),VCPU_STAT ("doorbell" , dbell_exits),VCPU_STAT ("guest doorbell" , gdbell_exits), -VM_STAT ("remote_tlb_flush" , remote_tlb_flush), { NULL } };
arch/s390/kvm/Kconfig +1 -0
@@ -20,6 +20,7 @@ config KVM def_tristate y prompt "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM + imply STATS_FS select PREEMPT_NOTIFIERS select HAVE_KVM_CPU_RELAX_INTERCEPT select HAVE_KVM_VCPU_ASYNC_IOCTL
arch/s390/kvm/kvm-s390.c +10 -6
@@ -57,7 +57,16 @@#define VCPU_IRQS_MAX_BUF (sizeof(struct kvm_s390_irq) * \ (KVM_MAX_VCPUS + LOCAL_IRQS)) -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vm_entries[] = { +VM_STAT ("inject_float_mchk" , inject_float_mchk), +VM_STAT ("inject_io" , inject_io), +VM_STAT ("inject_pfault_done" , inject_pfault_done), +VM_STAT ("inject_service_signal" , inject_service_signal), +VM_STAT ("inject_virtio" , inject_virtio), + {NULL } +}; + +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("userspace_handled" , exit_userspace),VCPU_STAT ("exit_null" , exit_null),VCPU_STAT ("exit_validity" , exit_validity),
@@ -95,18 +104,13 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("inject_ckc" , inject_ckc),VCPU_STAT ("inject_cputm" , inject_cputm),VCPU_STAT ("inject_external_call" , inject_external_call), -VM_STAT ("inject_float_mchk" , inject_float_mchk),VCPU_STAT ("inject_emergency_signal" , inject_emergency_signal), -VM_STAT ("inject_io" , inject_io),VCPU_STAT ("inject_mchk" , inject_mchk), -VM_STAT ("inject_pfault_done" , inject_pfault_done),VCPU_STAT ("inject_program" , inject_program),VCPU_STAT ("inject_restart" , inject_restart), -VM_STAT ("inject_service_signal" , inject_service_signal),VCPU_STAT ("inject_set_prefix" , inject_set_prefix),VCPU_STAT ("inject_stop_signal" , inject_stop_signal),VCPU_STAT ("inject_pfault_init" , inject_pfault_init), -VM_STAT ("inject_virtio" , inject_virtio),VCPU_STAT ("instruction_epsw" , instruction_epsw),VCPU_STAT ("instruction_gs" , instruction_gs), VCPU_STAT("instruction_io_other", instruction_io_other),
arch/x86/include/asm/kvm_host.h +1 -1
@@ -35,7 +35,7 @@#include <asm/kvm_vcpu_regs.h> #include <asm/hyperv-tlfs.h> -#define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_VCPU_STATS_FS#define KVM_MAX_VCPUS 288#define KVM_SOFT_MAX_VCPUS 240
arch/x86/kvm/Kconfig +1 -0
@@ -25,6 +25,7 @@ config KVM # for TASKSTATS/TASK_DELAY_ACCT: depends on NET && MULTIUSER depends on X86_LOCAL_APIC + imply STATS_FS select PREEMPT_NOTIFIERS select MMU_NOTIFIER select HAVE_KVM_IRQCHIP
arch/x86/kvm/Makefile +1 -1
@@ -11,7 +11,7 @@ kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \ - hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o + hyperv.o stats_fs.o mmu/mmu.o mmu/page_track.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o
arch/x86/kvm/debugfs.c deleted +0 -64
64 lines not on this page · raw message
arch/x86/kvm/stats_fs.c new file +56 -0
56 lines not on this page · raw message
arch/x86/kvm/x86.c +5 -1
@@ -190,7 +190,7 @@ static u64 __read_mostly host_xss;u64 __read_mostly supported_xss;EXPORT_SYMBOL_GPL (supported_xss); -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("pf_fixed" , pf_fixed),VCPU_STAT ("pf_guest" , pf_guest),VCPU_STAT ("tlb_flush" , tlb_flush),
@@ -217,6 +217,10 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("nmi_injections" , nmi_injections),VCPU_STAT ("req_event" , req_event),VCPU_STAT ("l1d_flush" , l1d_flush), + { NULL } +}; + +struct stats_fs_value stats_fs_vm_entries[] = {VM_STAT ("mmu_shadow_zapped" , mmu_shadow_zapped),VM_STAT ("mmu_pte_write" , mmu_pte_write), VM_STAT("mmu_pte_updated", mmu_pte_updated),
include/linux/kvm_host.h +10 -29
@@ -27,6 +27,7 @@#include <linux/refcount.h> #include <linux/nospec.h> #include <asm/signal.h> +#include <linux/stats_fs.h> #include <linux/kvm.h> #include <linux/kvm_para.h>
@@ -318,7 +319,7 @@ struct kvm_vcpu {bool preempted;bool ready;struct kvm_vcpu_arch arch; -struct dentry *debugfs_dentry; +struct stats_fs_source *stats_fs_src; };static inline int kvm_vcpu_exiting_guest_mode (struct kvm_vcpu *vcpu)
@@ -498,8 +499,7 @@ struct kvm {long tlbs_dirty;struct list_head devices;u64 manual_dirty_log_protect; -struct dentry *debugfs_dentry; -struct kvm_stat_data **debugfs_stat_data; +struct stats_fs_source *stats_fs_src;struct srcu_struct srcu;struct srcu_struct irq_srcu;pid_t userspace_pid;
@@ -880,8 +880,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu);void kvm_arch_vcpu_postcreate (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_destroy (struct kvm_vcpu *vcpu); -#ifdef __KVM_HAVE_ARCH_VCPU_DEBUGFS -void kvm_arch_create_vcpu_debugfs (struct kvm_vcpu *vcpu); +#ifdef __KVM_HAVE_ARCH_VCPU_STATS_FS +void kvm_arch_create_vcpu_stats_fs (struct kvm_vcpu *vcpu);#endif int kvm_arch_hardware_enable (void );
@@ -1110,33 +1110,14 @@ static inline bool kvm_is_error_gpa(struct kvm *kvm, gpa_t gpa)return kvm_is_error_hva (hva); } -enum kvm_stat_kind { -KVM_STAT_VM , -KVM_STAT_VCPU , -}; - -struct kvm_stat_data { -struct kvm *kvm; -struct kvm_stats_debugfs_item *dbgfs_item; -}; - -struct kvm_stats_debugfs_item { -const char *name; -int offset; -enum kvm_stat_kind kind; -int mode; -}; - -#define KVM_DBGFS_GET_MODE (dbgfs_item) \ - ((dbgfs_item)->mode ? (dbgfs_item)->mode : 0644) -#define VM_STAT (n, x, ...) \ - { n, offsetof(struct kvm, stat.x), KVM_STAT_VM, ## __VA_ARGS__ } + { n, offsetof(struct kvm, stat.x), STATS_FS_U64, STATS_FS_SUM, ## __VA_ARGS__ }#define VCPU_STAT (n, x, ...) \ - { n, offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU, ## __VA_ARGS__ } + { n, offsetof(struct kvm_vcpu, stat.x), STATS_FS_U64, STATS_FS_SUM, ## __VA_ARGS__ } -extern struct kvm_stats_debugfs_item debugfs_entries[]; -extern struct dentry *kvm_debugfs_dir; +extern struct stats_fs_value stats_fs_vcpu_entries[]; +extern struct stats_fs_value stats_fs_vm_entries[]; +extern struct stats_fs_source *kvm_stats_fs_dir;#if defined(CONFIG_MMU_NOTIFIER ) && defined(KVM_ARCH_WANT_MMU_NOTIFIER )static inline int mmu_notifier_retry (struct kvm *kvm,unsigned long mmu_seq)
virt/kvm/arm/arm.c +1 -1
@@ -140,7 +140,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)return ret; } -int kvm_arch_create_vcpu_debugfs (struct kvm_vcpu *vcpu) +int kvm_arch_create_vcpu_stats_fs (struct kvm_vcpu *vcpu) {return 0 ; }
virt/kvm/kvm_main.c +40 -274
436 lines not on this page · raw message
Signature · 1 line
2.25.2
Quoted 10 lines · Emanuele Giuseppe Esposito
There is currently no common way for Linux kernel subsystems to expose statistics to userspace shared throughout the Linux kernel; subsystems have to take care of gathering and displaying statistics by themselves, for example in the form of files in debugfs. For example KVM has its own code section that takes care of this in
virt/kvm/kvm_main.c, where it sets up debugfs handlers for displaying values and aggregating them from various subfolders to obtain information about the system state (i.e. displaying the total number of exits, calculated by summing all exits of all cpus of all running virtual machines).
Quoted 8 lines
Allowing each section of the kernel to do so has two disadvantages. First, it will introduce redundant code. Second, debugfs is anyway not the right place for statistics (for example it is affected by lockdown)
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
This is exciting, we have been looking in the same area recently. Adding Jonathan Adams <[email protected]>.
In your diffstat, one thing I notice that is omitted: an update to Documentation/* :) Any chance of getting some proposed Documentation/ updates with structure of the fs, the per subsystem breakdown, and best practices for managing the stats from the kernel level?
Quoted 34 lines
The file system is mounted on
/sys/kernel/statsand would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].Statsfs offers a generic and stable API, allowing any kind of directory/file organization and supporting multiple kind of aggregations (not only sum, but also average, max, min and
count_zero) and data types (all unsigned and signed types plus boolean). The implementation, which is a generalization of KVM’s debugfs statistics code, takes care of gathering and displaying information at run time; users only need to specify the values to be included in each source.Statsfs would also be a different mountpoint from debugfs, and would not suffer from limited access due to the security lock down patches. Its main function is to display each statistics as a file in the desired folder hierarchy defined through the API. Statsfs files can be read, and possibly cleared if their file mode allows it.
Statsfs has two main components: the public API defined by
include/linux/statsfs.h, and the virtual file system which should end up in/sys/kernel/stats.The API has two main elements, values and sources. Kernel subsystems like KVM can use the API to create a source, add child sources/values/aggregates and register it to the root source (that on the virtual fs would be
/sys/kernel/statsfs).Sources are created via
statsfs_source_create(), and each source becomes a directory in the file system. Sources form a parent-child relationship; root sources are added to the file system viastatsfs_source_register(). Every other source is added to or removed from a parent through thestatsfs_source_add_subordinateandstatsfs_source_remote_subordinateAPIs. Once a source is created and added to the tree (viaadd_subordinate), it will be used to compute aggregate values in the parent source.
Quoted 10 lines
Values represent quantites that are gathered by the statsfs user. Examples of values include the number of vm exits of a given kind, the amount of memory used by some data structure, the length of the longest hash table chain, or anything like that. Values are defined with the
statsfs_source_add_valuesfunction. Each value is defined by a structstatsfs_value; the samestatsfs_valuecan be added to many different sources. A value can be considered "simple" if it fetches data from a user-provided location, or "aggregate" if it groups all values in the subordinates sources that include the samestatsfs_value.
This seems like it could have a lot of overhead if we wanted to periodically track the totality of subsystem stats as a form of telemetry gathering from userspace. To collect telemetry for 1,000 different stats, do we need to issue lseek()+read() syscalls for each of them individually (or, worse, open()+read()+close())?
Any thoughts on how that can be optimized? A couple of ideas:
- an interface that allows gathering of all stats for a particular interface through a single file that would likely be encoded in binary and the responsibility of userspace to disseminate, or
- an interface that extends beyond this proposal and allows the reader to specify which stats they are interested in collecting and then the kernel will only provide these stats in a well formed structure and also be binary encoded.
We've found that the one-file-per-stat method is pretty much a show stopper from the performance view and we always must execute at least two syscalls to obtain a single stat.
Since this is becoming a generic API (good!!), maybe we can discuss possible ways to optimize gathering of stats in mass?
Quoted 76 lines
For more information, please consult the kerneldoc documentation in patch 2 and the sample uses in the kunit tests and in KVM.
This series of patches is based on my previous series "libfs: group and simplify linux fs code" and the single patch sent to kvm "
kvm_host: unifyVM_STATandVCPU_STATdefinitions in a single place". The former simplifies code duplicated in debugfs and tracefs (from which statsfs is based on), the latter groups all macros definition for statistics in kvm in a single common file shared by all architectures.Patch 1 adds a new refcount and kref destructor wrappers that take a semaphore, as those are used later by statsfs. Patch 2 introduces the statsfs API, patch 3 provides extensive tests that can also be used as example on how to use the API and patch 4 adds the file system support. Finally, patch 5 provides a real-life example of statsfs usage in KVM.
Signed-off-by Emanuele Giuseppe Esposito <[email protected]>
v1->v2remove unnecessarylist_foreach_safeloops, fix wrong indentation, change statsfs instats_fsEmanuele Giuseppe Esposito (5):
refcount, kref: add dec-and-test wrappers forrw_semaphoresstats_fsAPI: create, add and removestats_fssources and values
kunit: tests forstats_fsAPI
stats_fsfs: virtual fs to show stats to the end-user
kvm_main: replace debugfs withstats_fsMAINTAINERS | 7 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 6 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 -- arch/x86/kvm/stats_fs.c | 56 ++ arch/x86/kvm/x86.c | 6 +- fs/Kconfig | 12 + fs/Makefile | 1 + fs/stats_fs/Makefile | 6 + fs/stats_fs/inode.c | 337 ++++++++++ fs/stats_fs/internal.h | 35 + fs/stats_fs/stats_fs-tests.c | 1088 +++++++++++++++++++++++++++++++ fs/stats_fs/stats_fs.c | 773 ++++++++++++++++++++++ include/linux/kref.h | 11 + include/linux/kvm_host.h | 39 +- include/linux/refcount.h | 2 + include/linux/stats_fs.h | 304 +++++++++ include/uapi/linux/magic.h | 1 + lib/refcount.c | 32 + tools/lib/api/fs/fs.c | 21 + virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 314 ++------- 32 files changed, 2772 insertions(+), 382 deletions(-) delete mode 100644 arch/x86/kvm/debugfs.c create mode 100644 arch/x86/kvm/stats_fs.c create mode 100644 fs/stats_fs/Makefile create mode 100644 fs/stats_fs/inode.c create mode 100644 fs/stats_fs/internal.h create mode 100644 fs/stats_fs/stats_fs-tests.c create mode 100644 fs/stats_fs/stats_fs.c create mode 100644 include/linux/stats_fs.hSignature · 1 line
2.25.2
Quoted 8 lines · David Rientjes
Quoted 5 lines · Emanuele Giuseppe Esposito
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
Quoted 7 lines
This is exciting, we have been looking in the same area recently. Adding Jonathan Adams <[email protected]>.
In your diffstat, one thing I notice that is omitted: an update to
Documentation/*:) Any chance of getting some proposedDocumentation/updates with structure of the fs, the per subsystem breakdown, and best practices for managing the stats from the kernel level?
Yes, I will write some documentation. Thank you for the suggestion.
Quoted 20 lines
Quoted 11 lines
Values represent quantites that are gathered by the statsfs user. Examples of values include the number of vm exits of a given kind, the amount of memory used by some data structure, the length of the longest hash table chain, or anything like that. Values are defined with the
statsfs_source_add_valuesfunction. Each value is defined by a structstatsfs_value; the samestatsfs_valuecan be added to many different sources. A value can be considered "simple" if it fetches data from a user-provided location, or "aggregate" if it groups all values in the subordinates sources that include the samestatsfs_value.This seems like it could have a lot of overhead if we wanted to periodically track the totality of subsystem stats as a form of telemetry gathering from userspace. To collect telemetry for 1,000 different stats, do we need to issue lseek()+read() syscalls for each of them individually (or, worse, open()+read()+close())?
Any thoughts on how that can be optimized? A couple of ideas:
Quoted 8 lines
- an interface that allows gathering of all stats for a particular interface through a single file that would likely be encoded in binary and the responsibility of userspace to disseminate, or
- an interface that extends beyond this proposal and allows the reader to specify which stats they are interested in collecting and then the kernel will only provide these stats in a well formed structure and also be binary encoded.
Are you thinking of another file, containing all the stats for the directory in binary format?
Quoted 6 lines
We've found that the one-file-per-stat method is pretty much a show stopper from the performance view and we always must execute at least two syscalls to obtain a single stat.
Since this is becoming a generic API (good!!), maybe we can discuss possible ways to optimize gathering of stats in mass?
Sure, the idea of a binary format was considered from the beginning in [1], and it can be done either together with the current filesystem, or as a replacement via different mount options.
Thank you,
Emanuele
Quoted 61 lines
Quoted 61 lines
Signed-off-by Emanuele Giuseppe Esposito <[email protected]>
v1->v2remove unnecessarylist_foreach_safeloops, fix wrong indentation, change statsfs instats_fsEmanuele Giuseppe Esposito (5):
refcount, kref: add dec-and-test wrappers forrw_semaphoresstats_fsAPI: create, add and removestats_fssources and values
kunit: tests forstats_fsAPI
stats_fsfs: virtual fs to show stats to the end-user
kvm_main: replace debugfs withstats_fsMAINTAINERS | 7 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 6 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 -- arch/x86/kvm/stats_fs.c | 56 ++ arch/x86/kvm/x86.c | 6 +- fs/Kconfig | 12 + fs/Makefile | 1 + fs/stats_fs/Makefile | 6 + fs/stats_fs/inode.c | 337 ++++++++++ fs/stats_fs/internal.h | 35 + fs/stats_fs/stats_fs-tests.c | 1088 +++++++++++++++++++++++++++++++ fs/stats_fs/stats_fs.c | 773 ++++++++++++++++++++++ include/linux/kref.h | 11 + include/linux/kvm_host.h | 39 +- include/linux/refcount.h | 2 + include/linux/stats_fs.h | 304 +++++++++ include/uapi/linux/magic.h | 1 + lib/refcount.c | 32 + tools/lib/api/fs/fs.c | 21 + virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 314 ++------- 32 files changed, 2772 insertions(+), 382 deletions(-)delete mode 100644
arch/x86/kvm/debugfs.c
create mode 100644arch/x86/kvm/stats_fs.c
create mode 100644fs/stats_fs/Makefile
create mode 100644fs/stats_fs/inode.c
create mode 100644fs/stats_fs/internal.h
create mode 100644fs/stats_fs/stats_fs-tests.c
create mode 100644fs/stats_fs/stats_fs.c
create mode 100644include/linux/stats_fs.hSignature · 1 line
2.25.2
Quoted 10 lines · Emanuele Giuseppe Esposito
Quoted 2 lines · David Rientjes
Since this is becoming a generic API (good!!), maybe we can discuss possible ways to optimize gathering of stats in mass?
Sure, the idea of a binary format was considered from the beginning in [1], and it can be done either together with the current filesystem, or as a replacement via different mount options.
ASCII stats are not scalable. A binary format is definitely the way to go.
Paolo Bonzini
maintainerQuoted 7 lines · Jim Mattson
Quoted 5 lines
Quoted 2 lines
Since this is becoming a generic API (good!!), maybe we can discuss possible ways to optimize gathering of stats in mass?
Sure, the idea of a binary format was considered from the beginning in [1], and it can be done either together with the current filesystem, or as a replacement via different mount options.
ASCII stats are not scalable. A binary format is definitely the way to go.
I am totally in favor of having a binary format, but it should be introduced as a separate series on top of this one---and preferably by someone who has already put some thought into the problem (which Emanuele and I have not, beyond ensuring that the statsfs concept and API is flexible enough).
ASCII stats are necessary for quick userspace consumption and for backwards compatibility with KVM debugfs (which is not an ABI, but it's damn useful and should not be dropped without providing something as handy), so this is what this series starts from.
Paolo
Quoted 6 lines · Paolo Bonzini
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Since this is becoming a generic API (good!!), maybe we can discuss possible ways to optimize gathering of stats in mass?
Sure, the idea of a binary format was considered from the beginning in [1], and it can be done either together with the current filesystem, or as a replacement via different mount options.
Quoted 8 lines
Quoted 1 line
ASCII stats are not scalable. A binary format is definitely the way to go.
I am totally in favor of having a binary format, but it should be introduced as a separate series on top of this one---and preferably by someone who has already put some thought into the problem (which Emanuele and I have not, beyond ensuring that the statsfs concept and API is flexible enough).
The concern is that once this series is merged then /sys/kernel/stats could be considered an ABI and there would be a reasonable expectation that it will remain stable, in so far as the stats that userspace is interested in are stable and not obsoleted.
So is this a suggestion that the binary format becomes complementary to statsfs and provide a means for getting all stats from a single subsystem, or that this series gets converted to such a format before it is merged?
Quoted 5 lines
ASCII stats are necessary for quick userspace consumption and for backwards compatibility with KVM debugfs (which is not an ABI, but it's damn useful and should not be dropped without providing something as handy), so this is what this series starts from.
Paolo Bonzini
maintainerQuoted 6 lines · David Rientjes
Quoted 6 lines
I am totally in favor of having a binary format, but it should be introduced as a separate series on top of this one---and preferably by someone who has already put some thought into the problem (which Emanuele and I have not, beyond ensuring that the statsfs concept and API is flexible enough).
Quoted 8 lines
The concern is that once this series is merged then
/sys/kernel/statscould be considered an ABI and there would be a reasonable expectation that it will remain stable, in so far as the stats that userspace is interested in are stable and not obsoleted.So is this a suggestion that the binary format becomes complementary to statsfs and provide a means for getting all stats from a single subsystem, or that this series gets converted to such a format before it is merged?
The binary format should be complementary. The ASCII format should indeed be considered stable even though individual statistics would come and go. It may make sense to allow disabling ASCII files via mount and/or Kconfig options; but either way, the binary format can and should be added on top.
I have not put any thought into what the binary format would look like and what its features would be. For example these are but the first questions that come to mind:
* would it be possible to read/clear an arbitrary statistic with pread/pwrite, or do you have to read all of them?
* if userspace wants to read the schema just once and then read the statistics many times, how is it informed of schema changes?
* and of course the details of how the schema (names of stat and subsources) is encoded and what details it should include about the values (e.g. type or just signedness).
Another possibility is to query stats via BPF. This could be a third way to access the stats, or it could be alternative to a binary format.
Paolo
Adding Stefan Raspl, who has done a lot of kvm_stat work in the past.
Quoted 41 lines · Paolo Bonzini
Quoted 6 lines · David Rientjes
Quoted 6 lines
I am totally in favor of having a binary format, but it should be introduced as a separate series on top of this one---and preferably by someone who has already put some thought into the problem (which Emanuele and I have not, beyond ensuring that the statsfs concept and API is flexible enough).
Quoted 8 lines
The concern is that once this series is merged then
/sys/kernel/statscould be considered an ABI and there would be a reasonable expectation that it will remain stable, in so far as the stats that userspace is interested in are stable and not obsoleted.So is this a suggestion that the binary format becomes complementary to statsfs and provide a means for getting all stats from a single subsystem, or that this series gets converted to such a format before it is merged?
The binary format should be complementary. The ASCII format should indeed be considered stable even though individual statistics would come and go. It may make sense to allow disabling ASCII files via mount and/or
Kconfigoptions; but either way, the binary format can and should be added on top.I have not put any thought into what the binary format would look like and what its features would be. For example these are but the first questions that come to mind:
* would it be possible to read/clear an arbitrary statistic with pread/pwrite, or do you have to read all of them?
* if userspace wants to read the schema just once and then read the statistics many times, how is it informed of schema changes?
* and of course the details of how the schema (names of stat and subsources) is encoded and what details it should include about the values (e.g. type or just signedness).
Another possibility is to query stats via BPF. This could be a third way to access the stats, or it could be alternative to a binary format.
Paolo
Quoted 83 lines · David Rientjes
Quoted 10 lines · Emanuele Giuseppe Esposito
There is currently no common way for Linux kernel subsystems to expose statistics to userspace shared throughout the Linux kernel; subsystems have to take care of gathering and displaying statistics by themselves, for example in the form of files in debugfs. For example KVM has its own code section that takes care of this in
virt/kvm/kvm_main.c, where it sets up debugfs handlers for displaying values and aggregating them from various subfolders to obtain information about the system state (i.e. displaying the total number of exits, calculated by summing all exits of all cpus of all running virtual machines).Quoted 8 lines
Allowing each section of the kernel to do so has two disadvantages. First, it will introduce redundant code. Second, debugfs is anyway not the right place for statistics (for example it is affected by lockdown)
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
This is exciting, we have been looking in the same area recently. Adding Jonathan Adams <[email protected]>.
In your diffstat, one thing I notice that is omitted: an update to
Documentation/*:) Any chance of getting some proposedDocumentation/updates with structure of the fs, the per subsystem breakdown, and best practices for managing the stats from the kernel level?Quoted 34 lines
The file system is mounted on
/sys/kernel/statsand would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].Statsfs offers a generic and stable API, allowing any kind of directory/file organization and supporting multiple kind of aggregations (not only sum, but also average, max, min and
count_zero) and data types (all unsigned and signed types plus boolean). The implementation, which is a generalization of KVM’s debugfs statistics code, takes care of gathering and displaying information at run time; users only need to specify the values to be included in each source.Statsfs would also be a different mountpoint from debugfs, and would not suffer from limited access due to the security lock down patches. Its main function is to display each statistics as a file in the desired folder hierarchy defined through the API. Statsfs files can be read, and possibly cleared if their file mode allows it.
Statsfs has two main components: the public API defined by
include/linux/statsfs.h, and the virtual file system which should end up in/sys/kernel/stats.The API has two main elements, values and sources. Kernel subsystems like KVM can use the API to create a source, add child sources/values/aggregates and register it to the root source (that on the virtual fs would be
/sys/kernel/statsfs).Sources are created via
statsfs_source_create(), and each source becomes a directory in the file system. Sources form a parent-child relationship; root sources are added to the file system viastatsfs_source_register(). Every other source is added to or removed from a parent through thestatsfs_source_add_subordinateandstatsfs_source_remote_subordinateAPIs. Once a source is created and added to the tree (viaadd_subordinate), it will be used to compute aggregate values in the parent source.Quoted 10 lines
Values represent quantites that are gathered by the statsfs user. Examples of values include the number of vm exits of a given kind, the amount of memory used by some data structure, the length of the longest hash table chain, or anything like that. Values are defined with the
statsfs_source_add_valuesfunction. Each value is defined by a structstatsfs_value; the samestatsfs_valuecan be added to many different sources. A value can be considered "simple" if it fetches data from a user-provided location, or "aggregate" if it groups all values in the subordinates sources that include the samestatsfs_value.This seems like it could have a lot of overhead if we wanted to periodically track the totality of subsystem stats as a form of telemetry gathering from userspace. To collect telemetry for 1,000 different stats, do we need to issue lseek()+read() syscalls for each of them individually (or, worse, open()+read()+close())?
Any thoughts on how that can be optimized? A couple of ideas:
Quoted 8 lines
- an interface that allows gathering of all stats for a particular interface through a single file that would likely be encoded in binary and the responsibility of userspace to disseminate, or
- an interface that extends beyond this proposal and allows the reader to specify which stats they are interested in collecting and then the kernel will only provide these stats in a well formed structure and also be binary encoded.
Something akin to how ftrace allows you specify the list of functions in /sys/kernel/debug/tracing/set_ftrace_filter would make this a lot easier to use than the one-file-per-stat interface.
That would be useful, e.g. in capturing correlated stats periodically e.g. scheduler, power and thermal stats
Quoted 83 lines
We've found that the one-file-per-stat method is pretty much a show stopper from the performance view and we always must execute at least two syscalls to obtain a single stat.
Since this is becoming a generic API (good!!), maybe we can discuss possible ways to optimize gathering of stats in mass?
Quoted 76 lines
For more information, please consult the kerneldoc documentation in patch 2 and the sample uses in the kunit tests and in KVM.
This series of patches is based on my previous series "libfs: group and simplify linux fs code" and the single patch sent to kvm "
kvm_host: unifyVM_STATandVCPU_STATdefinitions in a single place". The former simplifies code duplicated in debugfs and tracefs (from which statsfs is based on), the latter groups all macros definition for statistics in kvm in a single common file shared by all architectures.Patch 1 adds a new refcount and kref destructor wrappers that take a semaphore, as those are used later by statsfs. Patch 2 introduces the statsfs API, patch 3 provides extensive tests that can also be used as example on how to use the API and patch 4 adds the file system support. Finally, patch 5 provides a real-life example of statsfs usage in KVM.
Signed-off-by Emanuele Giuseppe Esposito <[email protected]>
v1->v2remove unnecessarylist_foreach_safeloops, fix wrong indentation, change statsfs instats_fsEmanuele Giuseppe Esposito (5):
refcount, kref: add dec-and-test wrappers forrw_semaphoresstats_fsAPI: create, add and removestats_fssources and values
kunit: tests forstats_fsAPI
stats_fsfs: virtual fs to show stats to the end-user
kvm_main: replace debugfs withstats_fsMAINTAINERS | 7 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 6 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 -- arch/x86/kvm/stats_fs.c | 56 ++ arch/x86/kvm/x86.c | 6 +- fs/Kconfig | 12 + fs/Makefile | 1 + fs/stats_fs/Makefile | 6 + fs/stats_fs/inode.c | 337 ++++++++++ fs/stats_fs/internal.h | 35 + fs/stats_fs/stats_fs-tests.c | 1088 +++++++++++++++++++++++++++++++ fs/stats_fs/stats_fs.c | 773 ++++++++++++++++++++++ include/linux/kref.h | 11 + include/linux/kvm_host.h | 39 +- include/linux/refcount.h | 2 + include/linux/stats_fs.h | 304 +++++++++ include/uapi/linux/magic.h | 1 + lib/refcount.c | 32 + tools/lib/api/fs/fs.c | 21 + virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 314 ++------- 32 files changed, 2772 insertions(+), 382 deletions(-) delete mode 100644 arch/x86/kvm/debugfs.c create mode 100644 arch/x86/kvm/stats_fs.c create mode 100644 fs/stats_fs/Makefile create mode 100644 fs/stats_fs/inode.c create mode 100644 fs/stats_fs/internal.h create mode 100644 fs/stats_fs/stats_fs-tests.c create mode 100644 fs/stats_fs/stats_fs.c create mode 100644 include/linux/stats_fs.h--
2.25.2
On Mon, May 4, 2020 at 4:05 AM Emanuele Giuseppe Esposito <[email protected]> wrote:
...
Quoted 14 lines
Statsfs offers a generic and stable API, allowing any kind of directory/file organization and supporting multiple kind of aggregations (not only sum, but also average, max, min and
count_zero) and data types (all unsigned and signed types plus boolean). The implementation, which is a generalization of KVM’s debugfs statistics code, takes care of gathering and displaying information at run time; users only need to specify the values to be included in each source.Statsfs would also be a different mountpoint from debugfs, and would not suffer from limited access due to the security lock down patches. Its main function is to display each statistics as a file in the desired folder hierarchy defined through the API. Statsfs files can be read, and possibly cleared if their file mode allows it.
Quoted 3 lines
Statsfs has two main components: the public API defined by
include/linux/statsfs.h, and the virtual file system which should end up in/sys/kernel/stats.
This is good work. As David Rientjes mentioned, I'm currently investigating a similar project, based on a google-internal debugfs-based FS we call "metricfs". It's
designed in a slightly different fashion than statsfs here is, and the statistics exported are
mostly fed into our OpenTelemetry-like system. We're motivated by wanting an upstreamed solution, so that we can upstream the metrics we create that are of general interest, and lower the overall rebasing burden for our tree.
Some feedback on your design as proposed:
- the 8/16/32/64 signed/unsigned integers seems like a wart, and the
built-in support to grab any offset from a structure doesn't seem like much of an advantage. A simpler interface would be to just support an "integer" (possibly signed/unsigned) type, which is always 64-bit, and allow the caller to provide a function pointer to retrieve the value, with one or two void *s cbargs. Then the framework could provide an offset-based callback (or callbacks) similar to the existing functionality, and a similar one for per-CPU based statistics. A second "clear" callback could be optionally provided to allow for statistics to be cleared, as in your current proposal.
- A callback-style interface also allows for a lot more flexibility
in sourcing values, and doesn't lock your callers into one way of storing them. You would, of course, have to be clear about locking rules etc. for the callbacks.
- Beyond the statistic's type, one *very* useful piece of metadata
for telemetry tools is knowing whether a given statistic is "cumulative" (an unsigned counter which is only ever increased), as opposed to a floating value (like "amount of memory used").
I agree with the folks asking for a binary interface to read statistics, but I also agree that it can be added on later. I'm more concerned with getting the statistics model and capabilities right from the beginning, because those are harder to adjust later.
Would you be open to collaborating on the statsfs design? As background for this discussion, here are some details of how our metricfs implementation approaches statistics:
1. Each metricfs metric can have one or two string or integer "keys". If these exist, they expand the metric from a single value into a multi-dimensional table. For example, we use this to report a hash table we keep of functions calling "WARN()", in a 'warnings' statistic:
% cat .../warnings/valuesx86_pmu_stop 1
%
Indicates that the x86_pmu_stop() function has had a WARN() fire once since the system was booted. If multiple functions have fired WARN()s, they are listed in this table with their own counts. [1] We also use these to report per-CPU counters on a CPU-by-CPU basis:
% cat .../irq_x86/NMI/values
0 42
1 18
... one line per cpu
%
2. We also export some metadata about each statistic. For example, the metadata for the NMI counter above looks like:
% cat .../NMI/annotations
DESCRIPTION Non-maskable\ interrupts
CUMULATIVE
% cat .../NMI/fields
cpu value
int int
%
(Describing the statistic, marking it as "cumulative", and saying the fields are "cpu" and "value", both ints). The metadata doesn't change much, so having separate files allows the user-space agent to read them once and then the values multiple times.
3. We have a (very few) statistics where the value itself is a string, usually for device statuses.
For our use cases, we generally don't both output a statistic and it's aggregation from the kernel; either we sum up things in the kernel (e.g. over a bunch of per-cpu or per-memcg counters) and only have the result statistic, or we expect user-space to sum up the data if it's interested. The tabular form makes it pretty easy to do so (i.e. you can use awk(1) to sum all of the per-cpu NMI counters). We don't generally reset statistics, except as a side effect of removing a device.
Thanks again for the patchset, and for pointing out that KVM also needs statistics sent out; it's great that there is interest in this.
Cheers,
- jonathan
P.S. I also have a couple (non-critical) high-level notes:
* It's not clear what tree your patches are against, or their
dependencies; I was able to get them to apply to linux-next master with a little massaging, but then they failed to compile because they're built on top of your "libfs: group and simplify linux fs code" patch series you sent out in late april. Including a git link or at least a baseline tree and a list of the patch series you rely upon is helpful for anyone wanting to try out your changes.
* The main reason I was trying to try out your patches was to get a
sense of the set of directories and things the KVM example generates; while it's apparently the same as the existing KVM debugfs tree, it's useful to know how this ends up looking on a real system, and I'm not familiar with the KVM stats. Since this patch is intended slightly more broadly than just KVM, it might have been useful to include sample output for those not familiar with how things are today.
[1] We also use this to export various network/storage statistics
on a per-device basis. e.g. network bytes received counts:
% cat .../rx_bytes/values
lo 501360681
eth0 1457631256
...
%
Paolo Bonzini
maintainer[Answering for Emanuele because he's not available until Monday]
Quoted 9 lines · Jonathan Adams
This is good work. As David Rientjes mentioned, I'm currently investigating a similar project, based on a google-internal debugfs-based FS we call "metricfs". It's
designed in a slightly different fashion than statsfs here is, and the statistics exported are
mostly fed into our OpenTelemetry-like system. We're motivated by wanting an upstreamed solution, so that we can upstream the metrics we create that are of general interest, and lower the overall rebasing burden for our tree.
Cool. We included a public reading API exactly so that there could be other "frontends". I was mostly thinking of BPF as an in-tree user, but your metricfs could definitely use the reading API.
Quoted 9 lines
- the 8/16/32/64 signed/unsigned integers seems like a wart, and the
built-in support to grab any offset from a structure doesn't seem like
much of an advantage. A simpler interface would be to just support an> "integer" (possibly signed/unsigned) type, which is always 64-bit, and allow the caller to provide a function pointer to retrieve the value,
with one or two void *s cbargs. Then the framework could provide an
offset-based callback (or callbacks) similar to the existing
functionality, and a similar one for per-CPU based statistics. A
second "clear" callback could be optionally provided to allow for
statistics to be cleared, as in your current proposal.
Ok, so basically splitting get_simple_value into many separate callbacks. The callbacks would be in a struct like
struct stats_fs_type {
uint64_t (*get)(struct stats_fs_value *, void *);
void (*clear)(struct stats_fs_value *, void *);
bool signed;
}static uint64_t stats_fs_get_u8(struct stats_fs_value *val, void *base)
{
return *((uint8_t *)(base + (uintptr_t)val->arg);
}static void stats_fs_clear_u8(struct stats_fs_value *val, void *base)
{
*((uint8_t *)(base + (uintptr_t)val->arg) = 0;
}struct stats_fs_type stats_fs_type_u8 = {
stats_fs_get_u8,
stats_fs_clear_u8,
false
};and custom types can be defined using "&(struct stats_fs_type) {...}".
Quoted 4 lines
- Beyond the statistic's type, one *very* useful piece of metadata
for telemetry tools is knowing whether a given statistic is "cumulative" (an unsigned counter which is only ever increased), as opposed to a floating value (like "amount of memory used").
Good idea. Also, clearing does not make sense for a floating value, so we can use cumulative/floating to get a default for the mode: KVM statistics for example are mostly cumulative and mode 644, except a few that are floating and those are all mode 444. Therefore it makes sense to add cumulative/floating even before outputting it as metadata.
Quoted 3 lines
I'm more
concerned with getting the statistics model and capabilities right from the beginning, because those are harder to adjust later.
Agreed.
Quoted 15 lines
1. Each metricfs metric can have one or two string or integer "keys". If these exist, they expand the metric from a single value into a multi-dimensional table. For example, we use this to report a hash table we keep of functions calling "
WARN()", in a 'warnings' statistic:% cat .../warnings/values
x86_pmu_stop1
%Indicates that the
x86_pmu_stop()function has had aWARN()fire once since the system was booted. If multiple functions have fired WARN()s, they are listed in this table with their own counts. [1] We also use these to report per-CPU counters on a CPU-by-CPU basis:
Quoted 7 lines
% cat .../irq_x86/NMI/values
0 42
1 18
... one line per cpu
% cat .../rx_bytes/values
lo 501360681
eth0 1457631256
These seem like two different things.
The percpu and per-interface values are best represented as subordinate sources, one per CPU and one per interface. For interfaces I would just use a separate directory, but it doesn't really make sense for CPUs. So if we can cater for it in the model, it's better. For example:
- add a new argument to statsfs_create_source and statsfs_create_values that makes it not create directories and files respectively.
- add a new "aggregate function" STATS_FS_LIST that directs the parent to build a table of all the simple values below it
We can also add a helper statsfs_add_values_percpu that creates a new source for each CPU, I think.
The warnings one instead is a real hash table. It should be possible to implement it as some kind of customized aggregation, that is implemented in the client instead of coming from subordinate sources. The presentation can then just use STATS_FS_LIST. I don't see anything in the design that is a blocker.
Quoted 10 lines
2. We also export some metadata about each statistic. For example, the metadata for the NMI counter above looks like:
% cat .../NMI/annotations
DESCRIPTION Non-maskable\ interrupts
CUMULATIVE
% cat .../NMI/fields
cpu value
int int
%
Good idea. I would prefer per-directory dot-named files for this. For example a hypothetical statsfs version of /proc/interrupts could be like this:
$ cat /sys/kernel/stats/interrupts/.schema 0 // Name CUMULATIVE // Flags int:int // Type(s) IR-IO-APIC 2-edge timer // Description ... LOC CUMULATIVE int:int Local timer interrupts ... $ cat /sys/kernel/stats/interrupts/LOC 0 4286815 1 4151572 2 4199361 3 4229248
Quoted 2 lines
3. We have a (very few) statistics where the value itself is a string, usually for device statuses.
Maybe in addition to CUMULATIVE and FLOATING we can have ENUM properties, and a table to convert those enums to strings. Aggregation could also be used to make a histogram out of enums in subordinate sources, e.g.
$ cat /sys/kernel/stats/kvm/637-1/vcpu_state running 12 uninitialized 0 halted 4
So in general I'd say the sources/values model holds up. We certainly want to:
- switch immediately to callbacks instead of the type constants (so that core statsfs code only does signed/unsigned)
- add a field to distinguish cumulative and floating properties (and use it to determine the default file mode)
- add a new argument to statsfs_create_source and statsfs_create_values that makes it not create directories and files respectively
- add a new API to look for a statsfs_value recursively in all the subordinate sources, and pass the source/value pair to a callback function; and reimplement recursive aggregation and clear in terms of this function.
Quoted 6 lines
For our use cases, we generally don't both output a statistic and it's aggregation from the kernel; either we sum up things in the kernel (e.g. over a bunch of per-cpu or per-memcg counters) and only have the result statistic, or we expect user-space to sum up the data if it's interested. The tabular form makes it pretty easy to do so (i.e. you can use awk(1) to sum all of the per-cpu NMI counters).
Yep, the above "not create a dentry" flag would handle the case where you sum things up in the kernel because the more fine grained counters would be overwhelming.
Paolo
Quoted 9 lines · Paolo Bonzini
So in general I'd say the sources/values model holds up. We certainly want to:
- switch immediately to callbacks instead of the type constants (so that core statsfs code only does signed/unsigned)
- add a field to distinguish cumulative and floating properties (and use it to determine the default file mode)
Quoted 7 lines
- add a new argument to
statsfs_create_sourceandstatsfs_create_valuesthat makes it not create directories and files respectively- add a new API to look for a
statsfs_valuerecursively in all the subordinate sources, and pass the source/value pair to a callback function; and reimplement recursive aggregation and clear in terms of this function.
Ok I will apply this, thank you for all the suggestions. I will post the v3 patchset in the next few weeks.
In the meanwhile, I wrote the documentation you asked (even though it's going to change in v3), you can find it here:
https://github.com/esposem/linux/commit/dfa92f270f1aed73d5f3b7f12640b2a1635c711f
Thank you,
Emanuele
Quoted 36 lines · Paolo Bonzini
[Answering for Emanuele because he's not available until Monday]
Quoted 9 lines · Jonathan Adams
This is good work. As David Rientjes mentioned, I'm currently investigating a similar project, based on a google-internal debugfs-based FS we call "metricfs". It's
designed in a slightly different fashion than statsfs here is, and the statistics exported are
mostly fed into our OpenTelemetry-like system. We're motivated by wanting an upstreamed solution, so that we can upstream the metrics we create that are of general interest, and lower the overall rebasing burden for our tree.Cool. We included a public reading API exactly so that there could be other "frontends". I was mostly thinking of BPF as an in-tree user, but your metricfs could definitely use the reading API.
Quoted 9 lines
- the 8/16/32/64 signed/unsigned integers seems like a wart, and the
built-in support to grab any offset from a structure doesn't seem like
much of an advantage. A simpler interface would be to just support an> "integer" (possibly signed/unsigned) type, which is always 64-bit, and allow the caller to provide a function pointer to retrieve the value,
with one or two void *s cbargs. Then the framework could provide an
offset-based callback (or callbacks) similar to the existing
functionality, and a similar one for per-CPU based statistics. A
second "clear" callback could be optionally provided to allow for
statistics to be cleared, as in your current proposal.Ok, so basically splitting
get_simple_valueinto many separate callbacks. The callbacks would be in a struct likestruct stats_fs_type { uint64_t (*get)(struct stats_fs_value *, void *); void (*clear)(struct stats_fs_value *, void *); bool signed; }
...
Quoted 7 lines
struct stats_fs_type stats_fs_type_u8 = { stats_fs_get_u8, stats_fs_clear_u8, false };and custom types can be defined using "&(struct
stats_fs_type) {...}".
That makes sense.
Quoted 40 lines
Quoted 4 lines
- Beyond the statistic's type, one *very* useful piece of metadata
for telemetry tools is knowing whether a given statistic is "cumulative" (an unsigned counter which is only ever increased), as opposed to a floating value (like "amount of memory used").Good idea. Also, clearing does not make sense for a floating value, so we can use cumulative/floating to get a default for the mode: KVM statistics for example are mostly cumulative and mode 644, except a few that are floating and those are all mode 444. Therefore it makes sense to add cumulative/floating even before outputting it as metadata.
Quoted 3 lines
I'm more
concerned with getting the statistics model and capabilities right from the beginning, because those are harder to adjust later.Agreed.
Quoted 15 lines
1. Each metricfs metric can have one or two string or integer "keys". If these exist, they expand the metric from a single value into a multi-dimensional table. For example, we use this to report a hash table we keep of functions calling "
WARN()", in a 'warnings' statistic:% cat .../warnings/values
x86_pmu_stop1
%Indicates that the
x86_pmu_stop()function has had aWARN()fire once since the system was booted. If multiple functions have fired WARN()s, they are listed in this table with their own counts. [1] We also use these to report per-CPU counters on a CPU-by-CPU basis:Quoted 7 lines
% cat .../irq_x86/NMI/values
0 42
1 18
... one line per cpu
% cat .../rx_bytes/values
lo 501360681
eth0 1457631256
Quoted 1 line
These seem like two different things.
I see your point; I agree that there are two different things here.
Quoted 5 lines
The percpu and per-interface values are best represented as subordinate sources, one per CPU and one per interface. For interfaces I would just use a separate directory, but it doesn't really make sense for CPUs. So if we can cater for it in the model, it's better. For example:
Quoted 8 lines
- add a new argument to
statsfs_create_sourceandstatsfs_create_valuesthat makes it not create directories and files respectively.- add a new "aggregate function"
STATS_FS_LISTthat directs the parent to build a table of all the simple values below itWe can also add a helper
statsfs_add_values_percputhat creates a new source for each CPU, I think.
I think I'd characterize this slightly differently; we have a set of statistics which are essentially "in parallel":
- a variety of statistics, N CPUs they're available for, or
- a variety of statistics, N interfaces they're available for.
- a variety of statistics, N kvm object they're available for.
Recreating a parallel hierarchy of statistics any time we add/subtract a CPU or interface seems like a lot of overhead. Perhaps a better model would
be some sort of "parameter enumn" (naming is hard; parameter set?), so when a CPU/network interface/etc is added you'd add its ID to the "CPUs" we know about, and at removal time you'd take it out; it would have an associated cbarg for the value getting callback.
Does that make sense as a design?
I'm working on characterizing all of our metricfs usage; I'll see if this looks like it mostly covers our usecases.
Quoted 5 lines
The warnings one instead is a real hash table. It should be possible to implement it as some kind of customized aggregation, that is implemented in the client instead of coming from subordinate sources. The presentation can then just use
STATS_FS_LIST. I don't see anything in the design that is a blocker.
Yes; though if it's low-enough overhead, you could imagine having a dynamically-updated parameter enum based on the hash table.
Quoted 40 lines
Quoted 10 lines
2. We also export some metadata about each statistic. For example, the metadata for the NMI counter above looks like:
% cat .../NMI/annotations
DESCRIPTION Non-maskable\ interrupts
CUMULATIVE
% cat .../NMI/fields
cpu value
int int
%Good idea. I would prefer per-directory dot-named files for this. For example a hypothetical statsfs version of
/proc/interruptscould be like this:$ cat /sys/kernel/stats/interrupts/.schema 0 // Name CUMULATIVE // Flags int:int // Type(s) IR-IO-APIC 2-edge timer // Description ... LOC CUMULATIVE int:int Local timer interrupts ... $ cat /sys/kernel/stats/interrupts/LOC 0 4286815 1 4151572 2 4199361 3 4229248Quoted 2 lines
3. We have a (very few) statistics where the value itself is a string, usually for device statuses.
Maybe in addition to CUMULATIVE and FLOATING we can have ENUM properties, and a table to convert those enums to strings. Aggregation could also be used to make a histogram out of enums in subordinate sources, e.g.
Quoted 4 lines
$ cat /sys/kernel/stats/kvm/637-1/vcpu_state running 12 uninitialized 0 halted 4
That's along similar lines to the parameter enums, yeah.
Quoted 8 lines
So in general I'd say the sources/values model holds up. We certainly want to:
- switch immediately to callbacks instead of the type constants (so that core statsfs code only does signed/unsigned)
- add a field to distinguish cumulative and floating properties (and use it to determine the default file mode)
Yup, these make sense.
Quoted 7 lines
- add a new argument to
statsfs_create_sourceandstatsfs_create_valuesthat makes it not create directories and files respectively- add a new API to look for a
statsfs_valuerecursively in all the subordinate sources, and pass the source/value pair to a callback function; and reimplement recursive aggregation and clear in terms of this function.
This is where I think a little iteration on the "parameter enums" should happen before jumping into implementation.
Quoted 10 lines
Quoted 6 lines
For our use cases, we generally don't both output a statistic and it's aggregation from the kernel; either we sum up things in the kernel (e.g. over a bunch of per-cpu or per-memcg counters) and only have the result statistic, or we expect user-space to sum up the data if it's interested. The tabular form makes it pretty easy to do so (i.e. you can use awk(1) to sum all of the per-cpu NMI counters).
Yep, the above "not create a dentry" flag would handle the case where you sum things up in the kernel because the more fine grained counters would be overwhelming.
nodnod; or the callback could handle the sum itself.
Thanks,
- jonathan
Paolo Bonzini
maintainerHi Jonathan, I think the remaining sticky point is this one:
Quoted 15 lines · Jonathan Adams
I think I'd characterize this slightly differently; we have a set of statistics which are essentially "in parallel":
- a variety of statistics, N CPUs they're available for, or
- a variety of statistics, N interfaces they're available for.
- a variety of statistics, N kvm object they're available for.Recreating a parallel hierarchy of statistics any time we add/subtract a CPU or interface seems like a lot of overhead. Perhaps a better model would be some sort of "parameter enumn" (naming is hard; parameter set?), so when a CPU/network interface/etc is added you'd add its ID to the "CPUs" we know about, and at removal time you'd take it out; it would have an associated cbarg for the value getting callback.
Quoted 5 lines
Quoted 3 lines
Yep, the above "not create a dentry" flag would handle the case where you sum things up in the kernel because the more fine grained counters would be overwhelming.
nodnod; or the callback could handle the sum itself.
In general for statsfs we took a more explicit approach where each addend in a sum is a separate stats_fs_source. In this version of the patches it's also a directory, but we'll take your feedback and add both the ability to hide directories (first) and to list values (second).
So, in the cases of interfaces and KVM objects I would prefer to keep each addend separate.
For CPUs that however would be pretty bad. Many subsystems might accumulate stats percpu for performance reason, which would then be exposed as the sum (usually). So yeah, native handling of percpu values makes sense. I think it should fit naturally into the same custom aggregation framework as hash table keys, we'll see if there's any devil in the details.
Core kernel stats such as /proc/interrupts or /proc/stat are the exception here, since individual per-CPU values can be vital for debugging. For those, creating a source per stat, possibly on-the-fly at hotplug/hot-unplug time because NR_CPUS can be huge, would still be my preferred way to do it.
Thanks,
Paolo
Quoted 2 lines · Paolo Bonzini
Hi Jonathan, I think the remaining sticky point is this one:
Apologies it took a couple days for me to respond; I wanted to finish evaluating our current usage to make sure I had a full picture; I'll summarize our state at the bottom.
Quoted 22 lines · Jonathan Adams
Quoted 15 lines
I think I'd characterize this slightly differently; we have a set of statistics which are essentially "in parallel":
- a variety of statistics, N CPUs they're available for, or
- a variety of statistics, N interfaces they're available for.
- a variety of statistics, N kvm object they're available for.Recreating a parallel hierarchy of statistics any time we add/subtract a CPU or interface seems like a lot of overhead. Perhaps a better model would be some sort of "parameter enumn" (naming is hard; parameter set?), so when a CPU/network interface/etc is added you'd add its ID to the "CPUs" we know about, and at removal time you'd take it out; it would have an associated cbarg for the value getting callback.
Quoted 5 lines
Quoted 3 lines
Yep, the above "not create a dentry" flag would handle the case where you sum things up in the kernel because the more fine grained counters would be overwhelming.
nodnod; or the callback could handle the sum itself.
Quoted 7 lines
In general for statsfs we took a more explicit approach where each addend in a sum is a separate
stats_fs_source. In this version of the patches it's also a directory, but we'll take your feedback and add both the ability to hide directories (first) and to list values (second).So, in the cases of interfaces and KVM objects I would prefer to keep each addend separate.
This just feels like a lot of churn just to add a statistic or object; in your model, every time a KVM or VCPU is created, you create the N statistics, leading to N*M total objects. As I was imagining it, you'd have:
A 'parameter enum' which maps names to object pointers and A set of statistics which map a statfs path to {callback, cbarg,
zero or more parameter enums}
So adding a new KVM VCPU would just be "add an object to the KVM's VCPU parameter enum", and removing it would be the opposite, and a
couple callbacks could handle basically all of the stats. The only
tricky part would be making sure the parameter enum value create/destroy and the callback calls are coordinated correctly.
If you wanted stats for a particular VCPU, we could mark the overall directory as "include subdirs for VCPU parameter", and you'd automatically get one directory per VCPU, with the same set of stats in it, constrained to the single VCPU. I could also imagine having an ".agg_sum/{stata,statb,...}" to report using the aggregations you have, or a mode to say "stats in this directory are sums over the following VCPU parameter".
Quoted 7 lines
For CPUs that however would be pretty bad. Many subsystems might accumulate stats percpu for performance reason, which would then be exposed as the sum (usually). So yeah, native handling of percpu values makes sense. I think it should fit naturally into the same custom aggregation framework as hash table keys, we'll see if there's any devil in the details.
Quoted 5 lines
Core kernel stats such as
/proc/interruptsor/proc/statare the exception here, since individual per-CPU values can be vital for debugging. For those, creating a source per stat, possibly on-the-fly at hotplug/hot-unplug time becauseNR_CPUScan be huge, would still be my preferred way to do it.
Our metricfs has basically two modes: report all per-CPU values (for the IPI counts etc; you pass a callback which takes a 'int cpu' argument) or a callback that sums over CPUs and reports the full value. It also seems hard to have any subsystem with a per-CPU stat having to install a hotplug callback to add/remove statistics.
In my model, a "CPU" parameter enum which is automatically kept up-to-date is probably sufficient for the "report all per-CPU values".
Does this make sense to you? I realize that this is a significant change to the model y'all are starting with; I'm willing to do the work to flesh it out.
Thanks for your time,
- Jonathan
P.S. Here's a summary of the types of statistics we use in metricfs in google, to give a little context:
- integer values (single value per stat, source also a single value); a couple of these are boolean values exported as '0' or '1'.
- per-CPU integer values, reported as a <cpuid, value> table
- per-CPU integer values, summed and reported as an aggregate
- single-value values, keys related to objects:
- many per-device (disk, network, etc) integer stats
- some per-device string data (version strings, UUIDs, and
occasional statuses.)
- a few histograms (usually counts by duration ranges)
- the "function name" to count for the WARN statistic I mentioned.
- A single statistic with two keys (for livepatch statistics; the value is the livepatch status as a string)
Most of the stats with keys are "complete" (every key has a value), but there are several examples of statistics where only some of the possible keys have values, or (e.g. for networking statistics) only the keys visible to the reading process (e.g. in its namespaces) are included.
Paolo Bonzini
maintainerQuoted 11 lines · Jonathan Adams
Quoted 7 lines
In general for statsfs we took a more explicit approach where each addend in a sum is a separate
stats_fs_source. In this version of the patches it's also a directory, but we'll take your feedback and add both the ability to hide directories (first) and to list values (second).So, in the cases of interfaces and KVM objects I would prefer to keep each addend separate.
This just feels like a lot of churn just to add a statistic or object; in your model, every time a KVM or VCPU is created, you create the N statistics, leading to N*M total objects.
While it's N*M files, only O(M) statsfs API calls are needed to create them. Whether you have O(N*M) total kmalloc-ed objects or O(M) is an implementation detail.
Having O(N*M) API calls would be a non-started, I agree - especially once you start thinking of more efficient publishing mechanisms that unlike files are also O(M).
Quoted 13 lines
Quoted 7 lines
For CPUs that however would be pretty bad. Many subsystems might accumulate stats percpu for performance reason, which would then be exposed as the sum (usually). So yeah, native handling of percpu values makes sense. I think it should fit naturally into the same custom aggregation framework as hash table keys, we'll see if there's any devil in the details.
Quoted 5 lines
Core kernel stats such as
/proc/interruptsor/proc/statare the exception here, since individual per-CPU values can be vital for debugging. For those, creating a source per stat, possibly on-the-fly at hotplug/hot-unplug time becauseNR_CPUScan be huge, would still be my preferred way to do it.
Quoted 5 lines
Our metricfs has basically two modes: report all per-CPU values (for the IPI counts etc; you pass a callback which takes a 'int cpu' argument) or a callback that sums over CPUs and reports the full value. It also seems hard to have any subsystem with a per-CPU stat having to install a hotplug callback to add/remove statistics.
Yes, this is also why I think percpu values should have some kind of native handling. Reporting per-CPU values individually is the exception.
Quoted 2 lines
In my model, a "CPU" parameter enum which is automatically kept up-to-date is probably sufficient for the "report all per-CPU values".
Yes (or a separate CPU source in my model).
Paolo
Quoted 30 lines
Does this make sense to you? I realize that this is a significant change to the model y'all are starting with; I'm willing to do the work to flesh it out.
Thanks for your time,
- Jonathan
P.S. Here's a summary of the types of statistics we use in metricfs in google, to give a little context:- integer values (single value per stat, source also a single value); a couple of these are boolean values exported as '0' or '1'.
- per-CPU integer values, reported as a <cpuid, value> table
- per-CPU integer values, summed and reported as an aggregate
- single-value values, keys related to objects:
- many per-device (disk, network, etc) integer stats
- some per-device string data (version strings, UUIDs, and
occasional statuses.)
- a few histograms (usually counts by duration ranges)
- the "function name" to count for the WARN statistic I mentioned.
- A single statistic with two keys (for livepatch statistics; the value is the livepatch status as a string)Most of the stats with keys are "complete" (every key has a value), but there are several examples of statistics where only some of the possible keys have values, or (e.g. for networking statistics) only the keys visible to the reading process (e.g. in its namespaces) are included.