Statsfs: a new ram-based file system for Linux kernel statistics
Subsystems (16)
v3 cover Statsfs: a new ram-based file system for Linux kernel statistics
There is currently no common way for Linux kernel subsystems to expose statistics to userspace shared throughout the Linux kernel; subsystems have to take care of gathering and displaying statistics by themselves, for example in the form of files in debugfs. For example KVM has its own code section that takes care of this in virt/kvm/kvm_main.c, where it sets up debugfs handlers for displaying values and aggregating them from various subfolders to obtain information about the system state (i.e. displaying the total number of exits, calculated by summing all exits of all cpus of all running virtual machines).
Allowing each section of the kernel to do so has two disadvantages. First, it will introduce redundant code. Second, debugfs is anyway not the right place for statistics (for example it is affected by lockdown)
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
The file system is mounted on /sys/kernel/stats and would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].
Statsfs offers a generic and stable API, allowing any kind of directory/file organization and supporting multiple kind of aggregations (not only sum, but also average, max, min and count_zero) and data types (boolean, unsigned/signed and custom types). The implementation, which is a generalization of KVM’s debugfs statistics code, takes care of gathering and displaying information at run time; users only need to specify the values to be included in each source.
Statsfs would also be a different mountpoint from debugfs, and would not suffer from limited access due to the security lock down patches. Its main function is to display each statistics as a file in the desired folder hierarchy defined through the API. Statsfs files can be read, and possibly cleared if their file mode allows it.
Statsfs has two main components: the public API defined by include/linux/statsfs.h, and the virtual file system which should end up in /sys/kernel/stats.
The API has two main elements, values and sources. Kernel subsystems like KVM can use the API to create a source, add child sources/values/aggregates and register it to the root source (that on the virtual fs would be /sys/kernel/statsfs).
Sources are created via statsfs_source_create(), and each source becomes a directory in the file system. Sources form a parent-child relationship; root sources are added to the file system via statsfs_source_register(). Every other source is added to or removed from a parent through the statsfs_source_add_subordinate and statsfs_source_remote_subordinate APIs. Once a source is created and added to the tree (via add_subordinate), it will be used to compute aggregate values in the parent source. A source can optionally be hidden from the filesystem
but still considered in the aggregation operations if the corresponding flag is set during initialization.
Values represent quantites that are gathered by the statsfs user. Examples of values include the number of vm exits of a given kind, the amount of memory used by some data structure, the length of the longest hash table chain, or anything like that. Values are defined with the statsfs_source_add_values function. Each value is defined by a struct statsfs_value; the same statsfs_value can be added to many different sources. A value can be considered "simple" if it fetches data from a user-provided location, or "aggregate" if it groups all values in the subordinates sources that include the same statsfs_value.
Each value has a stats_fs_type pointer in order to allow the user to provide custom get and clear functions. The library, however, also exports default stats_fs_type structs for the standard types (all unsigned and signed types plus boolean).
A value can also provide a show function, that takes care
of displaying the value in a custom string format. This can be especially useful when displaying enums.
For more information, please consult the kerneldoc documentation in patch 2 and the sample uses in the kunit tests, KVM and networking.
This series of patches is based on my previous series "libfs: group and simplify linux fs code" and the single patch sent to kvm "kvm_host: unify VM_STAT and VCPU_STAT definitions in a single place". The former simplifies code duplicated in debugfs and tracefs (from which statsfs is based on), the latter groups all macros definition for statistics in kvm in a single common file shared by all architectures.
Patch 1 adds a new refcount and kref destructor wrappers that take a semaphore, as those are used later by statsfs. Patch 2 introduces the statsfs API, patch 3 provides extensive tests that can also be used as example on how to use the API and patch 4 adds the file system support. Finally, patch 5 provides a real-life example of statsfs usage in KVM, with patch 6 providing a concrete example of the show function and patch 7 another real-life example in the networking subsystem.
v2 -> v3 move kconfig entry in the pseudo filesystem menu, add documentation, get/clear function for value types, show function, floating/cumulative and hidden flags. Also added the netstat example
Emanuele Giuseppe Esposito (7):
stats_fs API: create, add and remove stats_fs sources and values documentation for stats_fs
kunit: tests for stats_fs API
stats_fs fs: virtual fs to show stats to the end-user
kvm_main: replace debugfs with stats_fs
[not for merge] kvm: example of stats_fs_value show function [not for merge] netstats: example use of stats_fs API
Documentation/filesystems/index.rst | 1 + Documentation/filesystems/stats_fs.rst | 222 +++++ MAINTAINERS | 7 + arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 12 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 -- arch/x86/kvm/stats_fs.c | 114 +++ arch/x86/kvm/x86.c | 11 +- fs/Kconfig | 20 + fs/Makefile | 1 + fs/stats_fs/Makefile | 7 + fs/stats_fs/inode.c | 461 ++++++++++ fs/stats_fs/internal.h | 34 + fs/stats_fs/stats_fs-tests.c | 1097 ++++++++++++++++++++++++ fs/stats_fs/stats_fs.c | 642 ++++++++++++++ fs/stats_fs/stub.c | 13 + include/linux/kvm_host.h | 45 +- include/linux/netdevice.h | 2 + include/linux/stats_fs.h | 381 ++++++++ include/uapi/linux/magic.h | 1 + net/Kconfig | 1 + net/core/dev.c | 68 ++ tools/lib/api/fs/fs.c | 21 + virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 317 +------ 35 files changed, 3193 insertions(+), 388 deletions(-) create mode 100644 Documentation/filesystems/stats_fs.rst delete mode 100644 arch/x86/kvm/debugfs.c create mode 100644 arch/x86/kvm/stats_fs.c create mode 100644 fs/stats_fs/Makefile create mode 100644 fs/stats_fs/inode.c create mode 100644 fs/stats_fs/internal.h create mode 100644 fs/stats_fs/stats_fs-tests.c create mode 100644 fs/stats_fs/stats_fs.c create mode 100644 fs/stats_fs/stub.c create mode 100644 include/linux/stats_fs.h
Signature · 1 line
2.25.4
PATCH v3 1/7 stats_fs API: create, add and remove stats_fs sources and values
Introduction to the stats_fs API, that allows to easily create, add and remove stats_fs sources and values. The API allows to easily building the statistics directory tree to automatically gather them for the linux kernel. The main functionalities are: create a source, add child sources/values/aggregates, register it to the root source (that on the virtual fs would be /sys/kernel/stats), ad perform a search for a value/aggregate.
Each source and value has an optional flag parameter:
in a value, it represent whether the statistic is cumulative or floating, in a source whether it should be visible from the filesystem or not. Defaults are respectively cumulative and visible.
Both flags fields are represented as an uint32_t to offer portability for future flags.
Each value also takes a struct stats_fs_type pointer that defines get and clear function for that stat, allowing custom
types handling. The API also provides default get and clear types for the supported standard types (stats_fs_type_*).
The API representation is only logical and will be backed up by a virtual file system in patch 4.
Its usage will be shared between the stats_fs file system and the end-users like kvm, the former calling it when it needs to display and clear statistics, the latter to add values and sources.
--- MAINTAINERS | 7 + fs/Kconfig | 14 + fs/Makefile | 1 + fs/stats_fs/Makefile | 5 + fs/stats_fs/internal.h | 19 ++ fs/stats_fs/stats_fs.c | 552 +++++++++++++++++++++++++++++++++++++++ fs/stats_fs/stub.c | 13 + include/linux/stats_fs.h | 363 +++++++++++++++++++++++++ 8 files changed, 974 insertions(+) create mode 100644 fs/stats_fs/Makefile create mode 100644 fs/stats_fs/internal.h create mode 100644 fs/stats_fs/stats_fs.c create mode 100644 fs/stats_fs/stub.c create mode 100644 include/linux/stats_fs.h
MAINTAINERS +7 -0
@@ -5229,6 +5229,13 @@ F: include/linux/debugfs.h F: include/linux/kobj* F: lib/kobj* +STATS_FS +M: Paolo Bonzini <[email protected]> +R: Emanuele Giuseppe Esposito <[email protected]> +S: Supported +F: include/linux/stats_fs.h +F: fs/stats_fs + DRIVERS FOR ADAPTIVE VOLTAGE SCALING (AVS) M: Kevin Hilman <[email protected]> M: Nishanth Menon <[email protected]>
fs/Kconfig +14 -0
@@ -221,6 +221,20 @@ config MEMFD_CREATE config ARCH_HAS_GIGANTIC_PAGE bool +config STATS_FS + bool "Statistics Filesystem" + help + stats_fs is a virtual file system that provides counters and + other statistics about the running kernel. + +config STATS_FS_API + bool + imply STATS_FS + +config STATS_FS_STUB + bool + default y if STATS_FS_API && !STATS_FS + source "fs/configfs/Kconfig" source "fs/efivarfs/Kconfig"
fs/Makefile +1 -0
@@ -125,6 +125,7 @@ obj-$(CONFIG_BEFS_FS) += befs/ obj-$(CONFIG_HOSTFS) += hostfs/ obj-$(CONFIG_CACHEFILES) += cachefiles/ obj-$(CONFIG_DEBUG_FS) += debugfs/ +obj-$(CONFIG_STATS_FS) += stats_fs/ obj-$(CONFIG_TRACING) += tracefs/ obj-$(CONFIG_OCFS2_FS) += ocfs2/ obj-$(CONFIG_BTRFS_FS) += btrfs/
fs/stats_fs/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0-only +stats_fs-objs := stats_fs.o + +obj-$(CONFIG_STATS_FS) += stats_fs.o +obj-$(CONFIG_STATS_FS_STUB) += stub.o
fs/stats_fs/internal.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATS_FS_INTERNAL_H_ +#define _STATS_FS_INTERNAL_H_ + +#include <linux/list.h> +#include <linux/kref.h> +#include <linux/rwsem.h> +#include <linux/stats_fs.h> + +/* values, grouped by base */ +struct stats_fs_value_source { +void *base_addr; +bool files_created; +uint32_t common_flags; +struct stats_fs_value *values; +struct list_head list_element; +}; + +#endif /* _STATS_FS_INTERNAL_H_ */
fs/stats_fs/stats_fs.c new file +550 -0
550 lines not on this page · raw message
fs/stats_fs/stub.c new file +13 -0
@@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/module.h> +#include <linux/errno.h> +#include <linux/file.h> +#include <linux/fs.h> +#include <linux/slab.h> +#include <linux/rwsem.h> +#include <linux/list.h> +#include <linux/kref.h> +#include <linux/limits.h> +#include <linux/stats_fs.h> + +const struct stats_fs_type stats_fs_type_stub;
include/linux/stats_fs.h new file +363 -0
363 lines not on this page · raw message
Signature · 1 line
2.25.4
PATCH v3 2/7 documentation for stats_fs
In reply to Emanuele Giuseppe EspositoHtml docs for a complete documentation of the stats_fs API, filesystem and usage.
--- Documentation/filesystems/index.rst | 1 + Documentation/filesystems/stats_fs.rst | 222 +++++++++++++++++++++++++ 2 files changed, 223 insertions(+) create mode 100644 Documentation/filesystems/stats_fs.rst
Documentation/filesystems/index.rst +1 -0
@@ -89,6 +89,7 @@ Documentation for filesystem implementations. relay romfs squashfs + stats_fs sysfs sysv-fs tmpfs
Documentation/filesystems/stats_fs.rst new file +221 -0
222 lines not on this page · raw message
Signature · 1 line
2.25.4
Randy Dunlap
reviewerRe: [PATCH v3 2/7] documentation for stats_fs
Hi--
Here are a few comments for you.
Quoted 16 lines · Emanuele Giuseppe Esposito
Html docs for a complete documentation of the
stats_fsAPI, filesystem and usage.Signed-off-by Emanuele Giuseppe Esposito <[email protected]>--- Documentation/filesystems/index.rst | 1 + Documentation/filesystems/stats_fs.rst | 222 +++++++++++++++++++++++++ 2 files changed, 223 insertions(+) create mode 100644 Documentation/filesystems/stats_fs.rstDocumentation/filesystems/stats_fs.rst new file +0
-0
Documentation/filesystems/stats_fs.rst line 5
Quoted 5 lines
Documentation/filesystems/stats_fs.rst new file +5
-0@@ -0,0 +1,222 @@ +==== +Stats_FS +==== + +Stats_fs is a synthetic ram-based virtual filesystem that takes care of
RAM-based
Quoted 4 lines
Documentation/filesystems/stats_fs.rst new file +4
-0+gathering and displaying statistics for the Linux kernel subsystems. + +The motivation for stats_fs comes from the fact that there is no common +way for Linux kernel subsystems to expose statistics to userspace shared
Documentation/filesystems/stats_fs.rst line 19
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+throughout the Linux kernel; subsystems have to take care of gathering and +displaying statistics by themselves, for example in the form of files in +debugfs. + +Allowing each subsystem of the kernel to do so has two disadvantages. +First, it will introduce redundant code. Second, debugfs is anyway not the +right place for statistics (for example it is affected by lockdown). + +Stats_fs offers a generic and stable API, allowing any kind of +directory/file organization and supporting multiple kind of aggregations
kinds
Documentation/filesystems/stats_fs.rst line 27
Quoted 8 lines
Documentation/filesystems/stats_fs.rst new file +8
-0+(not only sum, but also average, max, min and count_zero) and data types +(boolean, all unsigned/signed and custom types). The implementation takes +care of gathering and displaying information at run time; users only need +to specify the values to be included in each source. Optionally, users can +also provide a display function for each value, that will take care of +displaying the provided value in a custom format. + +Its main function is to display each statistics as a file in the desired
statistic
Quoted 16 lines
Documentation/filesystems/stats_fs.rst new file +16
-0+folder hierarchy defined through the API. Stats_fs files can be read, and +possibly cleared if their file mode allows it. + +Stats_fs is typically mounted with a command like:: + + mount -t stats_fs stats_fs /sys/kernel/stats_fs + +(Or an equivalent /etc/fstab line). + +Stats_fs has two main components: the public API defined by +include/linux/stats_fs.h, and the virtual file system in +/sys/kernel/stats. + +The API has two main elements, values and sources. Kernel +subsystems will create a source, add child +sources/values/aggregates and register it to the root source (that on the
Documentation/filesystems/stats_fs.rst line 53
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+virtual fs would be /sys/kernel/stats). + +The stats_fs API is defined in ``<linux/stats_fs.h>``. + + Sources + Sources are created via ``stats_fs_source_create()``, and each + source becomes a directory in the file system. Sources form a + parent-child relationship; root sources are added to the file + system via ``stats_fs_source_register()``. Therefore each Linux + subsystem will add its own entry to the root, filesystem similar
root filesystem
Documentation/filesystems/stats_fs.rst line 65
Quoted 12 lines
Documentation/filesystems/stats_fs.rst new file +12
-0+ to what it is done in debugfs. Every other source is added to or + removed from a parent through the + ``stats_fs_source_add_subordinate()`` and + ``stats_fs_source_remove_subordinate()`` APIs. Once a source is + created and added to the tree (via add_subordinate), it will be + used to compute aggregate values in the parent source. A source + can optionally be hidden from the filesystem but still considered + in the aggregation operations if the corresponding flag is set + during initialization. + + Values + Values represent quantites that are gathered by the stats_fs user.
quantities
Documentation/filesystems/stats_fs.rst line 66
Quoted 1 line
Documentation/filesystems/stats_fs.rst new file +1
-0+ Examples of values include the number of vm exits of a given kind,
VM
Documentation/filesystems/stats_fs.rst line 77
Quoted 11 lines
Documentation/filesystems/stats_fs.rst new file +11
-0+ the amount of memory used by some data structure, the length of + the longest hash table chain, or anything like that. Values are + defined with the stats_fs_source_add_values function. Each value + is defined by a ``struct stats_fs_value``; the same + ``stats_fs_value`` can be added to many different sources. A value + can be considered "simple" if it fetches data from a user-provided + location, or "aggregate" if it groups all values in the + subordinate sources that include the same ``stats_fs_value``. + Values by default are considered to be cumulative, meaning the + value they represent never decreases, but can also be defined as + floating if they exibith a different behavior. The main difference
exhibit
Quoted 45 lines
Documentation/filesystems/stats_fs.rst new file +45
-0+ between these two is reflected into the file permission, since a + floating value file does not allow the user to clear it. Each + value has a ``stats_fs_type`` pointer in order to allow the user + to provide custom get and clear functions. The library, however, + also exports default ``stats_fs_type`` structs for the standard + types (all unsigned and signed types plus boolean). A value can + also provide a show function that takes care of displaying the + value in a custom string format. This can be especially useful + when displaying enums. + +Because stats_fs is a different mountpoint than debugfs, it is not affected +by security lockdown. + +Using Stats_fs +======== + +Define a value:: + + struct statistics{ + uint64_t exit; + ... + }; + + struct kvm { + ... + struct statistics stat; + }; + + struct stats_fs_value kvm_stats[] = { + { "exit_vm", offsetof(struct kvm, stat.exit), &stats_fs_type_u64, + STATS_FS_SUM }, + { NULL } + }; + +The same ``struct stats_fs_value`` is used for both simple and aggregate +values, though the type and offset are only used for simple values. +Aggregates merge all values that use the same ``struct stats_fs_value``. + +Create the parent source:: + + struct stats_fs_source parent_source = stats_fs_source_create(0, "parent"); + +Register it (files and folders +will only be visible after this function is called):: +
Documentation/filesystems/stats_fs.rst line 132
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+ stats_fs_source_register(parent_source); + +Create and add a child:: + + struct stats_fs_source child_source = stats_fs_source_create(STATS_FS_HIDDEN, "child"); + + stats_fs_source_add_subordinate(parent_source, child_source); + +The STATS_FS_HIDDEN attribute won't affect the aggregation, it will only +block the creation of the files.
Why does HIDDEN block the creation of files? instead of their visibility?
Quoted 30 lines
Documentation/filesystems/stats_fs.rst new file +30
-0+ +Add values to parent and child (also here order doesn't matter):: + + struct kvm *base_ptr = kmalloc(..., sizeof(struct kvm)); + ... + stats_fs_source_add_values(child_source, kvm_stats, base_ptr, 0); + stats_fs_source_add_values(parent_source, kvm_stats, NULL, STATS_FS_HIDDEN); + +``child_source`` will be a simple value, since it has a non-NULL base +pointer, while ``parent_source`` will be an aggregate. During the adding +phase, also values can optionally be marked as hidden, so that the folder +and other values can be still shown. + +Of course the same ``struct stats_fs_value`` array can be also passed with a +different base pointer, to represent the same value but in another instance +of the kvm struct. + +Search: + +Fetch a value from the child source, returning the value +pointed by ``(uint64_t *) base_ptr + kvm_stats[0].offset``:: + + uint64_t ret_child, ret_parent; + + stats_fs_source_get_value(child_source, &kvm_stats[0], &ret_child); + +Fetch an aggregate value, searching all subsources of ``parent_source`` for +the specified ``struct stats_fs_value``:: + + stats_fs_source_get_value(parent_source, &kvm_stats[0], &ret_parent);
Documentation/filesystems/stats_fs.rst line 172
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+ + assert(ret_child = ret_parent); // check expected result + +To make it more interesting, add another child:: + + struct stats_fs_source child_source2 = stats_fs_source_create(0, "child2"); + + stats_fs_source_add_subordinate(parent_source, child_source2); + // now the structure is parent -> child1 + // -> child2
Is that the same as parent -> child1 -> child2
? It could almost be read as
parent -> child1
parent -> child2Whichever it is, can you make it more explicit, please?
Quoted 23 lines
Documentation/filesystems/stats_fs.rst new file +23
-0+ + struct kvm *other_base_ptr = kmalloc(..., sizeof(struct kvm)); + ... + stats_fs_source_add_values(child_source2, kvm_stats, other_base_ptr, 0); + +Note that other_base_ptr points to another instance of kvm, so the struct +stats_fs_value is the same but the address at which they point is not. + +Now get the aggregate value:: + + uint64_t ret_child, ret_child2, ret_parent; + + stats_fs_source_get_value(child_source, &kvm_stats[0], &ret_child); + stats_fs_source_get_value(parent_source, &kvm_stats[0], &ret_parent); + stats_fs_source_get_value(child_source2, &kvm_stats[0], &ret_child2); + + assert((ret_child + ret_child2) = ret_parent); + +Cleanup:: + + stats_fs_source_remove_subordinate(parent_source, child_source); + stats_fs_source_revoke(child_source); + stats_fs_source_put(child_source);
Documentation/filesystems/stats_fs.rst line 205
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+ + stats_fs_source_remove_subordinate(parent_source, child_source2); + stats_fs_source_revoke(child_source2); + stats_fs_source_put(child_source2); + + stats_fs_source_put(parent_source); + kfree(other_base_ptr); + kfree(base_ptr); + +Calling stats_fs_source_revoke is very important, because it will ensure
stats_fs_source_revoke()
Documentation/filesystems/stats_fs.rst line 211
Quoted 6 lines
Documentation/filesystems/stats_fs.rst new file +6
-0+that stats_fs will not access the data that were passed to +stats_fs_source_add_value for this source. + +Because open files increase the reference count for a stats_fs_source, the +source can end up living longer than the data that provides the values for +the source. Calling stats_fs_source_revoke just before the backing data
stats_fs_source_revoke()
Documentation/filesystems/stats_fs.rst line 221
Quoted 11 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+is freed avoids accesses to freed data structures. The sources will return +0. + +This is not needed for the parent_source, since it just contains +aggregates that would be 0 anyways if no matching child value exist. + +API Documentation +========> + +.. kernel-doc:: include/linux/stats_fs.h + :export: fs/stats_fs/*.c
Please fix that. ^^^^^
Thanks for the documentation.
Signature · 1 line
~Randy
Re: [PATCH v3 2/7] documentation for stats_fs
Hi,
Quoted 5 lines
Documentation/filesystems/stats_fs.rst line 132
Quoted 3 lines
Documentation/filesystems/stats_fs.rst new file +3
-0+ +The STATS_FS_HIDDEN attribute won't affect the aggregation, it will only +block the creation of the files.Why does HIDDEN block the creation of files? instead of their visibility?
The file itself is used to allow the user to view the content of a value. In order to make it hidden, the framework just doesn't create the file.
The structure is still present and considered in statsfs, however.
Hidden in this case means not visible at all thus not created, not the hidden file concept of dotted files (".filename")
Quoted 44 lines
Quoted 31 lines
Documentation/filesystems/stats_fs.rst new file +30
-0+ +Add values to parent and child (also here order doesn't matter):: + + struct kvm *base_ptr = kmalloc(..., sizeof(struct kvm)); + ... + stats_fs_source_add_values(child_source, kvm_stats, base_ptr, 0); + stats_fs_source_add_values(parent_source, kvm_stats, NULL, STATS_FS_HIDDEN); + +``child_source`` will be a simple value, since it has a non-NULL base +pointer, while ``parent_source`` will be an aggregate. During the adding +phase, also values can optionally be marked as hidden, so that the folder +and other values can be still shown. + +Of course the same ``struct stats_fs_value`` array can be also passed with a +different base pointer, to represent the same value but in another instance +of the kvm struct. + +Search: + +Fetch a value from the child source, returning the value +pointed by ``(uint64_t *) base_ptr + kvm_stats[0].offset``:: + + uint64_t ret_child, ret_parent; + + stats_fs_source_get_value(child_source, &kvm_stats[0], &ret_child); + +Fetch an aggregate value, searching all subsources of ``parent_source`` for +the specified ``struct stats_fs_value``:: + + stats_fs_source_get_value(parent_source, &kvm_stats[0], &ret_parent);Documentation/filesystems/stats_fs.rst line 172
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+ + assert(ret_child = ret_parent); // check expected result + +To make it more interesting, add another child:: + + struct stats_fs_source child_source2 = stats_fs_source_create(0, "child2"); + + stats_fs_source_add_subordinate(parent_source, child_source2); + // now the structure is parent -> child1 + // -> child2Is that the same as parent -> child1 -> child2
? It could almost be read as
Quoted 2 lines
parent -> child1
parent -> child2No the example in the documentation shows the relationship parent -> child1 and
parent -> child2.
It's not the same as
parent -> child1 -> child2.
In order to do the latter, one would need to do:
stats_fs_source_add_subordinate(parent_source, child_source1); stats_fs_source_add_subordinate(child_source1, child_source2);
Hope that this clarifies it.
Quoted 64 lines
Whichever it is, can you make it more explicit, please?
Quoted 23 lines
Documentation/filesystems/stats_fs.rst new file +23
-0+ + struct kvm *other_base_ptr = kmalloc(..., sizeof(struct kvm)); + ... + stats_fs_source_add_values(child_source2, kvm_stats, other_base_ptr, 0); + +Note that other_base_ptr points to another instance of kvm, so the struct +stats_fs_value is the same but the address at which they point is not. + +Now get the aggregate value:: + + uint64_t ret_child, ret_child2, ret_parent; + + stats_fs_source_get_value(child_source, &kvm_stats[0], &ret_child); + stats_fs_source_get_value(parent_source, &kvm_stats[0], &ret_parent); + stats_fs_source_get_value(child_source2, &kvm_stats[0], &ret_child2); + + assert((ret_child + ret_child2) = ret_parent); + +Cleanup:: + + stats_fs_source_remove_subordinate(parent_source, child_source); + stats_fs_source_revoke(child_source); + stats_fs_source_put(child_source);Documentation/filesystems/stats_fs.rst line 205
Quoted 10 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+ + stats_fs_source_remove_subordinate(parent_source, child_source2); + stats_fs_source_revoke(child_source2); + stats_fs_source_put(child_source2); + + stats_fs_source_put(parent_source); + kfree(other_base_ptr); + kfree(base_ptr); + +Calling stats_fs_source_revoke is very important, because it will ensure
stats_fs_source_revoke()Documentation/filesystems/stats_fs.rst line 211
Quoted 6 lines
Documentation/filesystems/stats_fs.rst new file +6
-0+that stats_fs will not access the data that were passed to +stats_fs_source_add_value for this source. + +Because open files increase the reference count for a stats_fs_source, the +source can end up living longer than the data that provides the values for +the source. Calling stats_fs_source_revoke just before the backing data
stats_fs_source_revoke()Documentation/filesystems/stats_fs.rst line 221
Quoted 11 lines
Documentation/filesystems/stats_fs.rst new file +10
-0+is freed avoids accesses to freed data structures. The sources will return +0. + +This is not needed for the parent_source, since it just contains +aggregates that would be 0 anyways if no matching child value exist. + +API Documentation +========>> + +.. kernel-doc:: include/linux/stats_fs.h + :export: fs/stats_fs/*.cPlease fix that. ^^^^^
Quoted 2 lines
Thanks for the documentation.
Thank you for the feedback,
Emanuele
PATCH v3 3/7 kunit: tests for stats_fs API
In reply to Emanuele Giuseppe EspositoAdd kunit tests to extensively test the stats_fs API functionality.
In order to run them, the kernel .config must set CONFIG_KUNIT=y and a new .kunitconfig file must be created with CONFIG_STATS_FS=y and CONFIG_STATS_FS_TEST=y
Tests can be then started by running the following command from the root directory of the linux kernel source tree:./tools/testing/kunit/kunit.py run --timeout0 --jobs=nproc --all
--- fs/Kconfig | 6 + fs/stats_fs/Makefile | 2 + fs/stats_fs/stats_fs-tests.c | 1097 ++++++++++++++++++++++++++++++++++ 3 files changed, 1105 insertions(+) create mode 100644 fs/stats_fs/stats_fs-tests.c
fs/Kconfig +6 -0
@@ -227,6 +227,12 @@ config STATS_FS stats_fs is a virtual file system that provides counters and other statistics about the running kernel. +config STATS_FS_TEST + bool "Tests for stats_fs" + depends on STATS_FS && KUNIT + help + tests for the stats_fs API. + config STATS_FS_API bool imply STATS_FS
fs/stats_fs/Makefile +2 -0
@@ -1,5 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only stats_fs-objs := stats_fs.o +stats_fs-tests-objs := stats_fs-tests.o obj-$(CONFIG_STATS_FS) += stats_fs.o obj-$(CONFIG_STATS_FS_STUB) += stub.o +obj-$(CONFIG_STATS_FS_TEST) += stats_fs-tests.o
fs/stats_fs/stats_fs-tests.c new file +1097 -0
1,097 lines not on this page · raw message
Signature · 1 line
2.25.4
Re: [PATCH v3 3/7] kunit: tests for stats_fs API
Quoted 2 lines · Emanuele Giuseppe Esposito
Add kunit tests to extensively test the
stats_fsAPI functionality.
I've added in the kunit-related folks.
Quoted 4 lines
In order to run them, the kernel .config must set
CONFIG_KUNIT=yand a new .kunitconfig file must be created withCONFIG_STATS_FS=yandCONFIG_STATS_FS_TEST=y
It looks like CONFIG_STATS_FS is built-in, but it exports much of the functionality you are testing. However could the tests also be built as a module (i.e. make CONFIG_STATS_FS_TEST a tristate variable)? To test this you'd need to specify CONFIG_KUNIT=m and CONFIG_STATS_FS_TEST=m, and testing would simply be a case of "modprobe"ing the stats fs module and collecting results in /sys/kernel/debug/kunit/<module_name> (rather than running kunit.py). Are you relying on unexported internals in the the tests that would prevent building them as a module?
Thanks!
Alan
Re: [PATCH v3 3/7] kunit: tests for stats_fs API
Quoted 6 lines
Quoted 5 lines
In order to run them, the kernel .config must set
CONFIG_KUNIT=yand a new .kunitconfig file must be created withCONFIG_STATS_FS=yandCONFIG_STATS_FS_TEST=y
Quoted 10 lines
It looks like
CONFIG_STATS_FSis built-in, but it exports much of the functionality you are testing. However could the tests also be built as a module (i.e. makeCONFIG_STATS_FS_TESTa tristate variable)? To test this you'd need to specifyCONFIG_KUNIT=mandCONFIG_STATS_FS_TEST=m, and testing would simply be a case of "modprobe"ing the stats fs module and collecting results in /sys/kernel/debug/kunit/<module_name> (rather than runningkunit.py). Are you relying on unexported internals in the the tests that would prevent building them as a module?
I haven't checked it yet, but tests should work as separate module. I will look into it, thanks.
Emanuele
PATCH v3 4/7 stats_fs fs: virtual fs to show stats to the end-user
In reply to Emanuele Giuseppe EspositoAdd virtual fs that maps stats_fs sources with directories, and values (simple or aggregates) to files.
Every time a file is read/cleared, the fs internally invokes the stats_fs API to get/set the requested value.
Also introduce the optional show function in each value, that allows to customize how the value is displayed inside a file. This could be especially useful with enums.
fs/stats_fs/inode.cis pretty much similar to what is done in fs/debugfs/inode.c, with the exception that the API is only composed by stats_fs_create_file, stats_fs_create_dir and stats_fs_remove.
--- fs/stats_fs/Makefile | 2 +- fs/stats_fs/inode.c | 461 +++++++++++++++++++++++++++++++++++++ fs/stats_fs/internal.h | 15 ++ fs/stats_fs/stats_fs.c | 92 +++++++- include/linux/stats_fs.h | 18 ++ include/uapi/linux/magic.h | 1 + tools/lib/api/fs/fs.c | 21 ++ 7 files changed, 608 insertions(+), 2 deletions(-) create mode 100644 fs/stats_fs/inode.c
fs/stats_fs/Makefile +1 -1
@@ -1,5 +1,5 @@ # SPDX-License-Identifier: GPL-2.0-only -stats_fs-objs := stats_fs.o +stats_fs-objs := inode.o stats_fs.o stats_fs-tests-objs := stats_fs-tests.o obj-$(CONFIG_STATS_FS) += stats_fs.o
fs/stats_fs/inode.c new file +459 -0
459 lines not on this page · raw message
fs/stats_fs/internal.h +15 -0
@@ -16,4 +16,19 @@ struct stats_fs_value_source { struct list_head list_element; }; +struct stats_fs_data_inode { +struct stats_fs_source *src; +struct stats_fs_value *val; +}; + +struct dentry *stats_fs_create_file (struct stats_fs_value *val, +struct stats_fs_source *src); + +struct dentry *stats_fs_create_dir (const char *name,struct dentry *parent); + +void stats_fs_remove (struct dentry *dentry); +#define stats_fs_remove_recursive stats_fs_remove + +int is_val_signed (struct stats_fs_value *val); +#endif /* _STATS_FS_INTERNAL_H_ */
fs/stats_fs/stats_fs.c +90 -1
131 lines not on this page · raw message
include/linux/stats_fs.h +18 -0
@@ -52,6 +52,9 @@ struct stats_fs_value {enum stat_aggr aggr_kind;uint32_t value_flag; + +/* optional show function */ +char *(*show)(uint64_t ); };struct stats_fs_source {
@@ -144,6 +147,18 @@ extern const struct stats_fs_type stats_fs_type_bool;struct stats_fs_source *stats_fs_source_create (uint32_t flags,const char *fmt, ...); +/** +* stats_fs_source_register - register a source in the stats_fs filesystem +* @source: a pointer to the source that will be registered +* +* Add the given folder as direct child of /sys/kernel/stats. +* It also starts to recursively search its own child and create all folders +* and files if they weren't already. All subsequent add_subordinate calls +* on the same source that is used in this function will create corresponding +* files and directories. +*/ +void stats_fs_source_register (struct stats_fs_source *source); +/** * stats_fs_source_add_values - adds values to the given source * @source: a pointer to the source that will receive the values
@@ -307,6 +322,9 @@ static inline struct stats_fs_source *stats_fs_source_create(uint32_t flags,return ERR_PTR (-ENODEV ); } +static inline void stats_fs_source_register (struct stats_fs_source *source) +{ } +static inline int stats_fs_source_add_values (struct stats_fs_source *source,struct stats_fs_value *val,void *base_ptr,uint32_t flags)
include/uapi/linux/magic.h +1 -0
@@ -10,6 +10,7 @@#define CRAMFS_MAGIC 0x28cd3d45/* some random number */ #define CRAMFS_MAGIC_WEND 0x453dcd28/* magic number with the wrong endianess */ #define DEBUGFS_MAGIC 0x64626720 +#define STATSFS_MAGIC 0x73746174#define SECURITYFS_MAGIC 0x73636673#define SELINUX_MAGIC 0xf97cff8c#define SMACK_MAGIC 0x43415d53/* "SMAC" */
tools/lib/api/fs/fs.c +21 -0
@@ -35,6 +35,10 @@#define TRACEFS_MAGIC 0x74726163#endif +#ifndef STATSFS_MAGIC +#define STATSFS_MAGIC 0x73746174 +#endif +#ifndef HUGETLBFS_MAGIC #define HUGETLBFS_MAGIC 0x958458f6#endif
@@ -76,6 +80,16 @@ static const char * const tracefs__known_mountpoints[] = { 0, }; +#ifndef STATSFS_DEFAULT_PATH +#define STATSFS_DEFAULT_PATH "/sys/kernel/stats" +#endif + +static const char *const statsfs__known_mountpoints[] = { +STATSFS_DEFAULT_PATH , +"/stats" , +0 , +}; +static const char *const hugetlbfs__known_mountpoints[] = {0 , };
@@ -100,6 +114,7 @@ enum {FS__TRACEFS =3 ,FS__HUGETLBFS =4 ,FS__BPF_FS =5 , + FS__STATSFS = 6, };#ifndef TRACEFS_MAGIC
@@ -127,6 +142,11 @@ static struct fs fs__entries[] = { .mounts = tracefs__known_mountpoints, .magic = TRACEFS_MAGIC, }, + [FS__STATSFS] = { + .name = "statsfs", + .mounts = statsfs__known_mountpoints, + .magic = STATSFS_MAGIC, + }, [FS__HUGETLBFS] = { .name = "hugetlbfs", .mounts = hugetlbfs__known_mountpoints,
@@ -297,6 +317,7 @@ FS(sysfs, FS__SYSFS);FS (procfs,FS__PROCFS );FS (debugfs,FS__DEBUGFS );FS (tracefs,FS__TRACEFS ); +FS (statsfs,FS__STATSFS );FS (hugetlbfs,FS__HUGETLBFS );FS (bpf_fs,FS__BPF_FS );
Signature · 1 line
2.25.4
PATCH v3 5/7 kvm_main: replace debugfs with stats_fs
In reply to Emanuele Giuseppe EspositoUse stats_fs API instead of debugfs to create sources and add values.
This also requires to change all architecture files to replace the old debugfs_entries with stats_fs_vcpu_entries and statsfs_vm_entries.
The files/folders name and organization is kept unchanged, and a symlink in sys/kernel/debugfs/kvm is left for backward compatibility.
--- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/guest.c | 2 +- arch/mips/kvm/Kconfig | 1 + arch/mips/kvm/mips.c | 2 +- arch/powerpc/kvm/Kconfig | 1 + arch/powerpc/kvm/book3s.c | 12 +- arch/powerpc/kvm/booke.c | 8 +- arch/s390/kvm/Kconfig | 1 + arch/s390/kvm/kvm-s390.c | 16 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/debugfs.c | 64 ------- arch/x86/kvm/stats_fs.c | 60 ++++++ arch/x86/kvm/x86.c | 11 +- include/linux/kvm_host.h | 45 ++--- virt/kvm/arm/arm.c | 2 +- virt/kvm/kvm_main.c | 318 +++++--------------------------- 18 files changed, 161 insertions(+), 388 deletions(-) delete mode 100644 arch/x86/kvm/debugfs.c create mode 100644 arch/x86/kvm/stats_fs.c
arch/arm64/kvm/Kconfig +1 -0
@@ -23,6 +23,7 @@ config KVM depends on OF # for TASKSTATS/TASK_DELAY_ACCT: depends on NET && MULTIUSER + select STATS_FS_API select MMU_NOTIFIER select PREEMPT_NOTIFIERS select HAVE_KVM_CPU_RELAX_INTERCEPT
arch/arm64/kvm/guest.c +1 -1
@@ -29,7 +29,7 @@#include "trace.h" -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("halt_successful_poll" , halt_successful_poll),VCPU_STAT ("halt_attempted_poll" , halt_attempted_poll),VCPU_STAT ("halt_poll_invalid" , halt_poll_invalid),
arch/mips/kvm/Kconfig +1 -0
@@ -19,6 +19,7 @@ config KVM tristate "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM depends on MIPS_FP_SUPPORT + select STATS_FS_API select EXPORT_UASM select PREEMPT_NOTIFIERS select KVM_GENERIC_DIRTYLOG_READ_PROTECT
arch/mips/kvm/mips.c +1 -1
@@ -39,7 +39,7 @@#define VECTORSPACING 0x100/* for EI/VI mode */ #endif -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("wait" , wait_exits),VCPU_STAT ("cache" , cache_exits),VCPU_STAT ("signal" , signal_exits),
arch/powerpc/kvm/Kconfig +1 -0
@@ -19,6 +19,7 @@ if VIRTUALIZATION config KVM bool + select STATS_FS_API select PREEMPT_NOTIFIERS select HAVE_KVM_EVENTFD select HAVE_KVM_VCPU_ASYNC_IOCTL
arch/powerpc/kvm/book3s.c +9 -3
@@ -38,7 +38,7 @@/* #define EXIT_DEBUG */ -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("exits" , sum_exits),VCPU_STAT ("mmio" , mmio_exits),VCPU_STAT ("sig" , signal_exits),
@@ -66,8 +66,14 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("pthru_all" , pthru_all),VCPU_STAT ("pthru_host" , pthru_host),VCPU_STAT ("pthru_bad_aff" , pthru_bad_aff), - VM_STAT("largepages_2M", num_2M_pages, .mode = 0444), - VM_STAT("largepages_1G", num_1G_pages, .mode = 0444), + { NULL } +}; + +struct stats_fs_value stats_fs_vm_entries[] = { +VM_STAT ("largepages_2M" , num_2M_pages, + .value_flag = STATS_FS_FLOATING_VALUE), +VM_STAT ("largepages_1G" , num_1G_pages, + .value_flag = STATS_FS_FLOATING_VALUE), {NULL } };
arch/powerpc/kvm/booke.c +6 -2
@@ -35,7 +35,12 @@unsigned long kvmppc_booke_handlers; -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vm_entries[] = { +VM_STAT ("remote_tlb_flush" , remote_tlb_flush), + {NULL } +}; + +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("mmio" , mmio_exits),VCPU_STAT ("sig" , signal_exits),VCPU_STAT ("itlb_r" , itlb_real_miss_exits),
@@ -54,7 +59,6 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("halt_wakeup" , halt_wakeup),VCPU_STAT ("doorbell" , dbell_exits),VCPU_STAT ("guest doorbell" , gdbell_exits), -VM_STAT ("remote_tlb_flush" , remote_tlb_flush), { NULL } };
arch/s390/kvm/Kconfig +1 -0
@@ -20,6 +20,7 @@ config KVM def_tristate y prompt "Kernel-based Virtual Machine (KVM) support" depends on HAVE_KVM + select STATS_FS_API select PREEMPT_NOTIFIERS select HAVE_KVM_CPU_RELAX_INTERCEPT select HAVE_KVM_VCPU_ASYNC_IOCTL
arch/s390/kvm/kvm-s390.c +10 -6
@@ -57,7 +57,16 @@#define VCPU_IRQS_MAX_BUF (sizeof(struct kvm_s390_irq) * \ (KVM_MAX_VCPUS + LOCAL_IRQS)) -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vm_entries[] = { +VM_STAT ("inject_float_mchk" , inject_float_mchk), +VM_STAT ("inject_io" , inject_io), +VM_STAT ("inject_pfault_done" , inject_pfault_done), +VM_STAT ("inject_service_signal" , inject_service_signal), +VM_STAT ("inject_virtio" , inject_virtio), + {NULL } +}; + +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("userspace_handled" , exit_userspace),VCPU_STAT ("exit_null" , exit_null),VCPU_STAT ("exit_validity" , exit_validity),
@@ -95,18 +104,13 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("inject_ckc" , inject_ckc),VCPU_STAT ("inject_cputm" , inject_cputm),VCPU_STAT ("inject_external_call" , inject_external_call), -VM_STAT ("inject_float_mchk" , inject_float_mchk),VCPU_STAT ("inject_emergency_signal" , inject_emergency_signal), -VM_STAT ("inject_io" , inject_io),VCPU_STAT ("inject_mchk" , inject_mchk), -VM_STAT ("inject_pfault_done" , inject_pfault_done),VCPU_STAT ("inject_program" , inject_program),VCPU_STAT ("inject_restart" , inject_restart), -VM_STAT ("inject_service_signal" , inject_service_signal),VCPU_STAT ("inject_set_prefix" , inject_set_prefix),VCPU_STAT ("inject_stop_signal" , inject_stop_signal),VCPU_STAT ("inject_pfault_init" , inject_pfault_init), -VM_STAT ("inject_virtio" , inject_virtio),VCPU_STAT ("instruction_epsw" , instruction_epsw),VCPU_STAT ("instruction_gs" , instruction_gs), VCPU_STAT("instruction_io_other", instruction_io_other),
arch/x86/include/asm/kvm_host.h +1 -1
@@ -35,7 +35,7 @@#include <asm/kvm_vcpu_regs.h> #include <asm/hyperv-tlfs.h> -#define __KVM_HAVE_ARCH_VCPU_DEBUGFS +#define __KVM_HAVE_ARCH_VCPU_STATS_FS#define KVM_MAX_VCPUS 288#define KVM_SOFT_MAX_VCPUS 240
arch/x86/kvm/Kconfig +1 -0
@@ -25,6 +25,7 @@ config KVM # for TASKSTATS/TASK_DELAY_ACCT: depends on NET && MULTIUSER depends on X86_LOCAL_APIC + select STATS_FS_API select PREEMPT_NOTIFIERS select MMU_NOTIFIER select HAVE_KVM_IRQCHIP
arch/x86/kvm/Makefile +1 -1
@@ -11,7 +11,7 @@ kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \ - hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o + hyperv.o stats_fs.o mmu/mmu.o mmu/page_track.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o
arch/x86/kvm/debugfs.c deleted +0 -64
64 lines not on this page · raw message
arch/x86/kvm/stats_fs.c new file +60 -0
60 lines not on this page · raw message
arch/x86/kvm/x86.c +8 -3
@@ -190,7 +190,7 @@ static u64 __read_mostly host_xss;u64 __read_mostly supported_xss;EXPORT_SYMBOL_GPL (supported_xss); -struct kvm_stats_debugfs_item debugfs_entries[] = { +struct stats_fs_value stats_fs_vcpu_entries[] = {VCPU_STAT ("pf_fixed" , pf_fixed),VCPU_STAT ("pf_guest" , pf_guest),VCPU_STAT ("tlb_flush" , tlb_flush),
@@ -217,6 +217,10 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VCPU_STAT ("nmi_injections" , nmi_injections),VCPU_STAT ("req_event" , req_event),VCPU_STAT ("l1d_flush" , l1d_flush), + { NULL } +}; + +struct stats_fs_value stats_fs_vm_entries[] = {VM_STAT ("mmu_shadow_zapped" , mmu_shadow_zapped),VM_STAT ("mmu_pte_write" , mmu_pte_write), VM_STAT("mmu_pte_updated", mmu_pte_updated),
@@ -226,8 +230,9 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {VM_STAT ("mmu_cache_miss" , mmu_cache_miss),VM_STAT ("mmu_unsync" , mmu_unsync),VM_STAT ("remote_tlb_flush" , remote_tlb_flush), - VM_STAT("largepages", lpages, .mode = 0444), - VM_STAT("nx_largepages_splitted", nx_lpage_splits, .mode = 0444), + VM_STAT("largepages", lpages, .value_flag = STATS_FS_FLOATING_VALUE), +VM_STAT ("nx_largepages_splitted" , nx_lpage_splits, + .value_flag = STATS_FS_FLOATING_VALUE),VM_STAT ("max_mmu_page_hash_collisions" , max_mmu_page_hash_collisions), { NULL } };
include/linux/kvm_host.h +14 -31
76 lines not on this page · raw message
virt/kvm/arm/arm.c +1 -1
@@ -140,7 +140,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)return ret; } -int kvm_arch_create_vcpu_debugfs (struct kvm_vcpu *vcpu) +int kvm_arch_create_vcpu_stats_fs (struct kvm_vcpu *vcpu) {return 0 ; }
virt/kvm/kvm_main.c +44 -274
440 lines not on this page · raw message
Signature · 1 line
2.25.4
PATCH v3 6/7 kvm: example of stats_fs_value show function
In reply to Emanuele Giuseppe EspositoAdd an example of the show function using the mp_state value.
mp_state is an enum that represents the VCPU state, so instead of displaying its integer representation, the show function takes care of translating the integer into a more meaningful string representation.
The VCPU status is shown in the kvm/<vmid>/vcpu<cpuid>/mp_state file
--- arch/x86/kvm/stats_fs.c | 54 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+)
arch/x86/kvm/stats_fs.c +54 -0
@@ -39,11 +39,65 @@ struct stats_fs_value stats_fs_vcpu_arch_tsc_frac[] = { { NULL } /* base is &kvm_tsc_scaling_ratio_frac_bits */ }; +char *stats_fs_vcpu_get_mpstate (uint64_t state) +{ +char *state_str; + + state_str =kzalloc (20 ,GFP_KERNEL ); +if (!state_str) +return ERR_PTR (-ENOMEM ); + +switch (state) { +case KVM_MP_STATE_RUNNABLE : +strcpy (state_str,"RUNNABLE" ); +break ; +case KVM_MP_STATE_UNINITIALIZED : +strcpy (state_str,"UNINITIALIZED" ); +break ; +case KVM_MP_STATE_INIT_RECEIVED : +strcpy (state_str,"INIT_RECEIVED" ); +break ; +case KVM_MP_STATE_HALTED : +strcpy (state_str,"HALTED" ); +break ; +case KVM_MP_STATE_SIPI_RECEIVED : +strcpy (state_str,"SIPI_RECEIVED" ); +break ; +case KVM_MP_STATE_STOPPED : +strcpy (state_str,"STOPPED" ); +break ; +case KVM_MP_STATE_CHECK_STOP : +strcpy (state_str,"CHECK_STOP" ); +break ; +case KVM_MP_STATE_OPERATING : +strcpy (state_str,"OPERATING" ); +break ; +case KVM_MP_STATE_LOAD : +strcpy (state_str,"LOAD" ); +break ; +default : +strcpy (state_str,"UNRECOGNIZED" ); +break ; + } + +return state_str; +} + +struct stats_fs_value stats_fs_vcpu_mp_state[] = { +VCPU_ARCH_STATS_FS ("mp_state" , kvm_vcpu_arch, mp_state, + .type = &stats_fs_type_u32, + .show = stats_fs_vcpu_get_mpstate), + {NULL } +}; +void kvm_arch_create_vcpu_stats_fs (struct kvm_vcpu *vcpu) {stats_fs_source_add_values (vcpu->stats_fs_src, stats_fs_vcpu_tsc_offset, &vcpu->arch,0 ); +stats_fs_source_add_values (vcpu->stats_fs_src, stats_fs_vcpu_mp_state, + &vcpu->arch,0 ); +if (lapic_in_kernel (vcpu)) stats_fs_source_add_values(vcpu->stats_fs_src, stats_fs_vcpu_arch_lapic_timer,
Signature · 1 line
2.25.4
PATCH v3 7/7 netstats: example use of stats_fs API
In reply to Emanuele Giuseppe EspositoApply stats_fs on the networking statistics subsystem.
Currently it only works with disabled network namespace (CONFIG_NET_NS=n), because multiple namespaces will have the same device name under the same root source that will cause a conflict in stats_fs.
--- include/linux/netdevice.h | 2 ++ net/Kconfig | 1 + net/core/dev.c | 66 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 69 insertions(+)
include/linux/netdevice.h +2 -0
@@ -48,6 +48,7 @@#include <uapi/linux/if_bonding.h> #include <uapi/linux/pkt_cls.h> #include <linux/hashtable.h> +#include <linux/stats_fs.h> struct netpoll_info ;struct device ;
@@ -2117,6 +2118,7 @@ struct net_device { unsigned wol_enabled:1;struct list_head net_notifier_list; +struct stats_fs_source *stats_fs_src;#if IS_ENABLED (CONFIG_MACSEC )/* MACsec management functions */
net/Kconfig +1 -0
@@ -8,6 +8,7 @@ menuconfig NET select NLATTR select GENERIC_NET_UTILS select BPF + select STATS_FS_API ---help--- Unless you really know what you are doing, you should say Y here. The reason is that some programs need kernel networking support even
net/core/dev.c +66 -0
102 lines not on this page · raw message
Signature · 1 line
2.25.4
Re: [PATCH v3 7/7] [not for merge] netstats: example use of stats_fs API
Quoted 6 lines · Emanuele Giuseppe Esposito
Apply
stats_fson the networking statistics subsystem.Currently it only works with disabled network namespace (
CONFIG_NET_NS=n), because multiple namespaces will have the same device name under the same root source that will cause a conflict instats_fs.
Hi Emanuele
How do you atomically get and display a group of statistics?
If you look at how the netlink socket works, you will see code like:
do {
start = u64_stats_fetch_begin_irq(&cpu_stats->syncp);
rx_packets = cpu_stats->rx_packets;
rx_bytes = cpu_stats->rx_bytes;
....
} while (u64_stats_fetch_retry_irq(&cpu_stats->syncp, start));It will ensure that rx_packets and rx_bytes are consistent with each other. If the value of the sequence counter changes while inside the loop, the loop so repeated until it does not change.
In general, hardware counters in NICs are the same. You tell it to take a snapshot of the statistics counters, and then read them all back, to give a consistent view across all the statistics.
I've not looked at this new code in detail, but it looks like you have one file per statistic, and assume each statistic is independent of every other statistic. This independence can limit how you use the values, particularly when debugging. The netlink interface we use does not have this limitation.
Andrew
Re: [PATCH v3 7/7] [not for merge] netstats: example use of stats_fs API
Hi Andrew
Quoted 19 lines
How do you atomically get and display a group of statistics?
If you look at how the netlink socket works, you will see code like:
do { start = u64_stats_fetch_begin_irq(&cpu_stats->syncp); rx_packets = cpu_stats->rx_packets; rx_bytes = cpu_stats->rx_bytes; .... } while (u64_stats_fetch_retry_irq(&cpu_stats->syncp, start));It will ensure that
rx_packetsandrx_bytesare consistent with each other. If the value of the sequence counter changes while inside the loop, the loop so repeated until it does not change.In general, hardware counters in NICs are the same. You tell it to take a snapshot of the statistics counters, and then read them all back, to give a consistent view across all the statistics.
Quoted 5 lines
I've not looked at this new code in detail, but it looks like you have one file per statistic, and assume each statistic is independent of every other statistic. This independence can limit how you use the values, particularly when debugging. The netlink interface we use does not have this limitation.
You're right, statistics are treated independently so what you describe is currently not supported.
In KVM the utilization is more qualitative, so there isn't such problem. But as long as the interface is based on file access, the possibility of snapshotting might not be useful; however, it could still be considered to be added later together with the binary access.
Jonathan, how is your metricfs handling this case?
Thank you,
Emanuele
Jakub Kicinski
maintainerQuoted 14 lines · On Tue, 26 May 2020 13:03:10 +0200 Emanuele Giuseppe Esposito wrote:
There is currently no common way for Linux kernel subsystems to expose statistics to userspace shared throughout the Linux kernel; subsystems have to take care of gathering and displaying statistics by themselves, for example in the form of files in debugfs. For example KVM has its own code section that takes care of this in
virt/kvm/kvm_main.c, where it sets up debugfs handlers for displaying values and aggregating them from various subfolders to obtain information about the system state (i.e. displaying the total number of exits, calculated by summing all exits of all cpus of all running virtual machines).Allowing each section of the kernel to do so has two disadvantages. First, it will introduce redundant code. Second, debugfs is anyway not the right place for statistics (for example it is affected by lockdown)
Quoted 6 lines
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
The file system is mounted on
/sys/kernel/statsand would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].
What's the direct motivation for this work? Moving KVM stats out of debugfs?
In my experience stats belong in the API used for creating/enumerating objects, statsfs sounds like going in the exact opposite direction - creating a parallel structure / hierarchy for exposing stats. I know nothing about KVM but are you sure all the info that has to be exposed will be stats?
In case of networking we have the basic stats in sysfs, under the netdevice's kobject. But since we're not using sysfs much any more for config, new stats are added in netlink APIs. Again - same APIs used for enumeration and config.
Quoted 7 lines
Quoted 4 lines
The file system is mounted on
/sys/kernel/statsand would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].What's the direct motivation for this work? Moving KVM stats out of debugfs?
There's many reasons: one of these is not using debugfs for statistics, but also (and mainly) to try and have a single tool that automatically takes care and displays them, instead of leaving each subsystem "on its own".
Sure, everyone gathers and processes stats in different ways, and the aim of this tool is to hopefully be extensible enough to cover all needs.
Quoted 3 lines
In my experience stats belong in the API used for creating/enumerating objects, statsfs sounds like going in the exact opposite direction - creating a parallel structure / hierarchy for exposing stats.
I know
Quoted 8 lines
nothing about KVM but are you sure all the info that has to be exposed will be stats?I don't understand, what do you mean here?
In case of networking we have the basic stats in sysfs, under the netdevice's kobject. But since we're not using sysfs much any more for config, new stats are added in netlink APIs. Again - same APIs used for enumeration and config.
I don't really know a lot about the networking subsystem, and as it was pointed out in another email on patch 7 by Andrew, networking needs to atomically gather and display statistics in order to make them consistent, and currently this is not supported by stats_fs but could be added in future.
In addition, right now it won't work properly if the networking namespaces are enabled. That is another issue to take into consideration. That's also why I marked patch 7 as "not for merge"
Regarding the config, as I said the idea is to gather multiple subsystems' statistics, therefore there wouldn't be a single configuration method like in netlink.
For example in kvm there are file descriptors for configuration, and creating them requires no privilege, contrary to the network interfaces.
Thank you,
Emanuele
Quoted 5 lines
I don't really know a lot about the networking subsystem, and as it was pointed out in another email on patch 7 by Andrew, networking needs to atomically gather and display statistics in order to make them consistent, and currently this is not supported by
stats_fsbut could be added in future.
Hi Emanuele
Do you have any idea how you will support atomic access? It does not seem easy to implement in a filesystem based model.
Andrew
Paolo Bonzini
maintainerQuoted 8 lines · Andrew Lunn
Quoted 5 lines
I don't really know a lot about the networking subsystem, and as it was pointed out in another email on patch 7 by Andrew, networking needs to atomically gather and display statistics in order to make them consistent, and currently this is not supported by
stats_fsbut could be added in future.Do you have any idea how you will support atomic access? It does not seem easy to implement in a filesystem based model.
Hi Andrew,
there are plans to support binary access. Emanuele and I don't really have a plan for how to implement it, but there are developers from Google that have ideas (because Google has a similar "metricfs" thing in-house).
I think atomic access would use some kind of "source_ops" struct containing create_snapshot and release_snapshot function pointers.
Paolo
Jakub Kicinski
maintainerQuoted 5 lines · On Wed, 27 May 2020 15:14:41 +0200 Emanuele Giuseppe Esposito wrote:
Regarding the config, as I said the idea is to gather multiple subsystems' statistics, therefore there wouldn't be a single configuration method like in netlink.
For example in kvm there are file descriptors for configuration, and creating them requires no privilege, contrary to the network interfaces.
Enumerating networking interfaces, addresses, and almost all of the configuration requires no extra privilege. In fact I'd hope that whatever daemon collects network stats doesn't run as root :)
I think enumerating objects is of primary importance, and statistics of those objects are subordinate.
Again, I have little KVM knowledge, but BPF also uses a fd-based API, and carries stats over the same syscall interface.
Paolo Bonzini
maintainerQuoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 27 May 2020 15:14:41 +0200 Emanuele Giuseppe Esposito wrote:
Regarding the config, as I said the idea is to gather multiple subsystems' statistics, therefore there wouldn't be a single configuration method like in netlink.
For example in kvm there are file descriptors for configuration, and creating them requires no privilege, contrary to the network interfaces.
Quoted 6 lines
Enumerating networking interfaces, addresses, and almost all of the configuration requires no extra privilege. In fact I'd hope that whatever daemon collects network stats doesn't run as root :)
I think enumerating objects is of primary importance, and statistics of those objects are subordinate.
I see what you meant now. statsfs can also be used to enumerate objects if one is so inclined (with the prototype in patch 7, for example, each network interface becomes a directory).
Quoted 2 lines
Again, I have little KVM knowledge, but BPF also uses a fd-based API, and carries stats over the same syscall interface.
Can BPF stats (for BPF scripts created by whatever process is running in the system) be collected by an external daemon that does not have access to the file descriptor? For KVM it's of secondary importance to gather stats in the program; it can be nice to have and we are thinking of a way to export the stats over the fd-based API, but it's less useful than system-wide monitoring. Perhaps this is a difference between the two.
Another case where stats and configuration are separate is CPUs, where CPU enumeration is done in sysfs but statistics are exposed in various procfs files such as /proc/interrupts and /proc/stats.
Thanks,
Paolo
Jakub Kicinski
maintainerQuoted 9 lines · On Wed, 27 May 2020 23:07:53 +0200 Paolo Bonzini wrote:
Quoted 2 lines
Again, I have little KVM knowledge, but BPF also uses a fd-based API, and carries stats over the same syscall interface.
Can BPF stats (for BPF scripts created by whatever process is running in the system) be collected by an external daemon that does not have access to the file descriptor? For KVM it's of secondary importance to gather stats in the program; it can be nice to have and we are thinking of a way to export the stats over the fd-based API, but it's less useful than system-wide monitoring. Perhaps this is a difference between the two.
Yes, check out bpftool prog list (bpftool code is under tools/bpf/ in the kernel tree). BPF statistics are under a static key, so you may not see any on your system. My system shows e.g.:
81: kprobe name abc tag cefaa9376bdaae75 gpl run_time_ns 80941 run_cnt 152
loaded_at 2020-05-26T13:00:24-0700 uid 0 xlated 512B jited 307B memlock 4096B map_ids 66,64 btf_id 16
In this example run_time_ns and run_cnt are stats.
The first number on the left is the program ID. BPF has an IDA, and each object gets an integer id. So admin (or CAP_BPF, I think) can iterate over the ids and open fds to objects of interest.
Quoted 3 lines
Another case where stats and configuration are separate is CPUs, where CPU enumeration is done in sysfs but statistics are exposed in various procfs files such as
/proc/interruptsand/proc/stats.
True, but I'm guessing everyone is just okay living with the legacy procfs format there. Otherwise I'd guess the stats would had been added to sysfs. I'd be curious to hear the full story there.
Paolo Bonzini
maintainerQuoted 20 lines · Jakub Kicinski
Quoted 9 lines · On Wed, 27 May 2020 23:07:53 +0200 Paolo Bonzini wrote:
Quoted 2 lines
Again, I have little KVM knowledge, but BPF also uses a fd-based API, and carries stats over the same syscall interface.
Can BPF stats (for BPF scripts created by whatever process is running in the system) be collected by an external daemon that does not have access to the file descriptor? For KVM it's of secondary importance to gather stats in the program; it can be nice to have and we are thinking of a way to export the stats over the fd-based API, but it's less useful than system-wide monitoring. Perhaps this is a difference between the two.
Yes, check out bpftool prog list (bpftool code is under
tools/bpf/in the kernel tree). BPF statistics are under a static key, so you may not see any on your system. My system shows e.g.:81: kprobe name abc tag
cefaa9376bdaae75gplrun_time_ns80941run_cnt152loaded_at 2020-05-26T13:00:24-0700 uid 0 xlated 512B jited 307B memlock 4096B map_ids 66,64 btf_id 16
Quoted 5 lines
In this example
run_time_nsandrun_cntare stats.The first number on the left is the program ID. BPF has an IDA, and each object gets an integer id. So admin (or
CAP_BPF, I think) can iterate over the ids and open fds to objects of interest.
Got it, thanks. But then "I'd hope that whatever daemon collects [BPF] stats doesn't run as root". :)
Quoted 7 lines
Quoted 3 lines
Another case where stats and configuration are separate is CPUs, where CPU enumeration is done in sysfs but statistics are exposed in various procfs files such as
/proc/interruptsand/proc/stats.True, but I'm guessing everyone is just okay living with the legacy procfs format there. Otherwise I'd guess the stats would had been added to sysfs. I'd be curious to hear the full story there.
Yeah, it's a chicken-and-egg problem in that there's no good place in sysfs to put statistics right now, which is part of what this filesystem is trying to solve (the other part is the API).
You can read more about Google's usecase at
http://lkml.iu.edu/hypermail/linux/kernel/2005.0/08056.html, it does include both network and interrupt stats and it's something that they've been using in production for quite some time. We'd like the statsfs API to be the basis for including something akin to that in Linux.
To be honest, it's unlikely that Emanuele (who has just finished his internship at Red Hat) and I will pursue the networking stats further than the demo patch at the end of this series. However, we're trying to make sure that the API is at least ready for that, and to probe whether any developers from other subsystems would be interested in using statsfs. So thanks for bringing your point of view!
Thanks,
Paolo
Quoted 3 lines · Paolo Bonzini
I see what you meant now. statsfs can also be used to enumerate objects if one is so inclined (with the prototype in patch 7, for example, each network interface becomes a directory).
there are many use cases that have 100's to 1000's have network devices. Having a sysfs entry per device already bloats memory usage for these use cases; another filesystem with an entry per device makes that worse. Really the wrong direction for large scale systems.
Paolo Bonzini
maintainerQuoted 9 lines · David Ahern
Quoted 3 lines · Paolo Bonzini
I see what you meant now. statsfs can also be used to enumerate objects if one is so inclined (with the prototype in patch 7, for example, each network interface becomes a directory).
there are many use cases that have 100's to 1000's have network devices. Having a sysfs entry per device already bloats memory usage for these use cases; another filesystem with an entry per device makes that worse. Really the wrong direction for large scale systems.
Hi David,
IMO the important part for now is having a flexible kernel API for exposing statistics across multiple subsystems, so that they can be harvested in an efficient way. The userspace API is secondary, and multiple APIs can be added to cater for different usecases.
For example, as of the first five patches the memory usage is the same as what is now in the mainline kernel, since all the patchset does is take existing debugfs inodes and move them to statsfs. I agree that, if the concept is extended to the whole kernel, scalability and memory usage becomes an issue; and indeed, the long-term plan is to support a binary format that is actually _more_ efficient than the status quo for large scale systems.
In the meanwhile, the new filesystem can be disabled (see the difference between "STATS_FS" and "STATS_FS_API") if it imposes undesirable overhead.
Thanks,
Paolo
Quoted 35 lines · Emanuele Giuseppe Esposito
There is currently no common way for Linux kernel subsystems to expose statistics to userspace shared throughout the Linux kernel; subsystems have to take care of gathering and displaying statistics by themselves, for example in the form of files in debugfs. For example KVM has its own code section that takes care of this in
virt/kvm/kvm_main.c, where it sets up debugfs handlers for displaying values and aggregating them from various subfolders to obtain information about the system state (i.e. displaying the total number of exits, calculated by summing all exits of all cpus of all running virtual machines).Allowing each section of the kernel to do so has two disadvantages. First, it will introduce redundant code. Second, debugfs is anyway not the right place for statistics (for example it is affected by lockdown)
In this patch series I introduce statsfs, a synthetic ram-based virtual filesystem that takes care of gathering and displaying statistics for the Linux kernel subsystems.
The file system is mounted on
/sys/kernel/statsand would be already used by kvm. Statsfs was initially introduced by Paolo Bonzini [1].Statsfs offers a generic and stable API, allowing any kind of directory/file organization and supporting multiple kind of aggregations (not only sum, but also average, max, min and
count_zero) and data types (boolean, unsigned/signed and custom types). The implementation, which is a generalization of KVM’s debugfs statistics code, takes care of gathering and displaying information at run time; users only need to specify the values to be included in each source.Statsfs would also be a different mountpoint from debugfs, and would not suffer from limited access due to the security lock down patches. Its main function is to display each statistics as a file in the desired folder hierarchy defined through the API. Statsfs files can be read, and possibly cleared if their file mode allows it.
Quoted 4 lines
Statsfs has two main components: the public API defined by
include/linux/statsfs.h, and the virtual file system which should end up in/sys/kernel/stats.
Hi Emanuele
Quoted 4 lines
The API has two main elements, values and sources. Kernel subsystems like KVM can use the API to create a source, add child sources/values/aggregates and register it to the root source (that on the virtual fs would be
/sys/kernel/statsfs).
Another issue i see with networking is that statistic counters can be dynamic. They can come and go. One of the drivers i work on has extra statistics available when a fibre interface is used, compared to a copper interface. And this happens at run time. The netlink API has no problems with this. It is a snapshot of what counters are currently available. There is no state in the API.
In my humble opinion, networking is unlikely to adopt your approach. You probably want to look around for other subsystems which have statistics, and see if you can cover their requirements, and get them on board.
Andrew