"statsfs" API design
Quoted 2 lines
statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in
/sys/kernel/stats
I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.
Quoted 4 lines
The first user of statsfs would be KVM, which is currently exposing its stats in debugfs
Google has KVM patches to gather statistics in a binary format
Which is a right thing to do.
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines
statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in
/sys/kernel/statsI think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.
Quoted 6 lines
Quoted 4 lines
The first user of statsfs would be KVM, which is currently exposing its stats in debugfs
Google has KVM patches to gather statistics in a binary format
Which is a right thing to do.
It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.
The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.
So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.
Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.
greg k-h
Quoted 24 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines
statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in
/sys/kernel/statsI think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.
Quoted 6 lines
Quoted 4 lines
The first user of statsfs would be KVM, which is currently exposing its stats in debugfs
Google has KVM patches to gather statistics in a binary format
Which is a right thing to do.
It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.
The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.
Quoted 6 lines
So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.
Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.
What about using a text format like YAML? Here's some benefits:
- The fields are self describing based on the key name.
- New fields can be easily added without breaking compatibility.
- Allows for a script to easily parse the contents while keeping human readability.
- Would work for systems that run busybox as their userspace without having to install additional tools.
- Allows for a nested data structure.
The downside is that the output would be larger than a binary interface but it's more maintainable in my opinion.
Brian
Quoted 34 lines · Brian Masney
Quoted 24 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines
statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in
/sys/kernel/statsI think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.
Quoted 6 lines
Quoted 4 lines
The first user of statsfs would be KVM, which is currently exposing its stats in debugfs
Google has KVM patches to gather statistics in a binary format
Which is a right thing to do.
It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.
The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.
Quoted 6 lines
So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.
Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.
What about using a text format like YAML? Here's some benefits:
Quoted 7 lines
- The fields are self describing based on the key name.
- New fields can be easily added without breaking compatibility.
- Allows for a script to easily parse the contents while keeping human readability.
- Would work for systems that run busybox as their userspace without having to install additional tools.
- Allows for a nested data structure.
varlink was created to solve the issues that people have had with YAML over time, so you might want to look into that :)
https://varlink.org/
Quoted 2 lines
The downside is that the output would be larger than a binary interface but it's more maintainable in my opinion.
binary interfaces are unmaintainable over time, especially when you do not control both sides of the interface (unlike Google and their use of this for KVM stats.)
thanks,
greg k-h
Quoted 43 lines · Greg KH
Quoted 34 lines · Brian Masney
Quoted 24 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines
statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in
/sys/kernel/statsI think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.
Quoted 6 lines
Quoted 4 lines
The first user of statsfs would be KVM, which is currently exposing its stats in debugfs
Google has KVM patches to gather statistics in a binary format
Which is a right thing to do.
It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.
The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.
Quoted 6 lines
So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.
Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.
What about using a text format like YAML? Here's some benefits:
Quoted 7 lines
- The fields are self describing based on the key name.
- New fields can be easily added without breaking compatibility.
- Allows for a script to easily parse the contents while keeping human readability.
- Would work for systems that run busybox as their userspace without having to install additional tools.
- Allows for a nested data structure.
Quoted 3 lines
varlink was created to solve the issues that people have had with YAML over time, so you might want to look into that :)
https://varlink.org/
Well, varlink really is JSON, but at least there's example code for userspace and kernelspace to pull from to knock up tests for people who want to look into it.
thanks,
greg k-h
Quoted 8 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines
statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in
/sys/kernel/statsI think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
The first user of statsfs would be KVM, which is currently exposing its stats in debugfs
Google has KVM patches to gather statistics in a binary format
Which is a right thing to do.
It's always "simpler" to just take binary data and suck it in.
Faster too!
Quoted 5 lines
That works for a year or so until another value needs to be supported. Or removed. Or features are backported.
The reason text values in individual files work is they are "self describable" and "self discoverable".
Untrue. Applications always knows what the data means, by definition:
"0x42"
What is this? 4-char NUL-terminated string? Or an integer 66? Or a 4-byte nonce blob for some kind of crypto algorithm.
In the other direction: describe every field of /proc/*/stat file without looking to the manpage:
$ cat /proc/self/stat 5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0
Quoted 3 lines
You "know" what the value is and
that it is supported because the file is there or not. With binary values in a single file you do not know any of that.
You _always_ know that.
Quoted 6 lines
So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.
Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.
Text doesn't work either. Why do you think /proc/*/maps have 1 space character at the end of anon mappings? Because someone fucked up. Here is how people use /proc in the field:
https://stackoverflow.com/questions/3596781/how-to-detect-if-the-current-process-is-being-run-by-gdb
open /proc/*/status
read
strstr("TracerPid:")
I'd humbly suggest to define the minimum amount of work for the task:
* some kind of percpu loop to gather stats
* some kind of accumulation code, possibly with min/max/avg
* write clear data
* copy_to_user
and realise that everything alse is a waste of electricity, namely,
* pathname allocation (4KB)
* VFS '/' split, lookups (/sys/kernel/.../" means 3+ lookups
* 192 bytes for each dentry
* 550+ bytes per inode
* 3 system calls per act of gathering statistics
userspace will be written in the most stupid way possible without openat() etc
* userspace snprintf() for pathname
* kernel space snprintf() somewhere
* multiple copying inside kernel (vsnprintf.c)
* general inability for userspace to estimate the amount of data in decimal (nobody does that), so nicely sized buffers of 4K or 1K or 16KB (bash) will be used which is a waste.
Quoted 5 lines · Alexey Dobriyan
In the other direction: describe every field of
/proc/*/statfile without looking to the manpage:$ cat /proc/self/stat 5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0
That's why this is not what I am proposing, and also not what Greg has mentioned.
Quoted 15 lines
and realise that everything alse is a waste of electricity, namely,
* pathname allocation (4KB)
* VFS '/' split, lookups (/sys/kernel/.../" means 3+ lookups
* 192 bytes for each dentry
* 550+ bytes per inode
* 3 system calls per act of gathering statistics
userspace will be written in the most stupid way possible withoutopenat()etc
* userspacesnprintf()for pathname
* kernel spacesnprintf()somewhere
* multiple copying inside kernel (vsnprintf.c)
* general inability for userspace to estimate the amount of data in decimal (nobody does that), so nicely sized buffers of 4K or 1K or 16KB (bash) will be used which is a waste.
Yeah, all of this is true but I know how much I use /sys/kernel/debug/kvm so backwards-compatibility with it is certainly a requirement for stats. Good thing, having a high-level stats API lets you also design something that targets different usecases than just quick "cat" or "watch". The somewhat wasteful sysfs interface to statsfs can even be hidden behind a kconfig symbol once there is an alternative. It also makes it possible to create inodes on demand if someone is so inclined.
So the good thing is that despite the disagreements, this can be considered an argument in favor of statsfs, and we agree on that. :)
Thanks,
Paolo
Quoted 9 lines · Paolo Bonzini
Quoted 5 lines · Alexey Dobriyan
In the other direction: describe every field of
/proc/*/statfile without looking to the manpage:$ cat /proc/self/stat 5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0That's why this is not what I am proposing, and also not what Greg has mentioned.
The argument was that text is somehow superior to binary. Experiment shows that userspace can make a mess of both modes therefore preferring one to another should be based on something else (preferably objective).
/proc have these two problems:
First, noticeably slow:
https://news.ycombinator.com/item?id=21414882
Second, overinstantiating inodes and dentries:
https://lore.kernel.org/lkml/[email protected]/
statfs maybe never get to that level but it is not hard to see what lies at the end of the tunnel.
Quoted 8 lines · Alexey Dobriyan
Quoted 7 lines · Paolo Bonzini
Quoted 5 lines · Alexey Dobriyan
In the other direction: describe every field of
/proc/*/statfile without looking to the manpage:$ cat /proc/self/stat 5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0
Quoted 6 lines
Quoted 2 lines
That's why this is not what I am proposing, and also not what Greg has mentioned.
The argument was that text is somehow superior to binary. Experiment shows that userspace can make a mess of both modes therefore preferring one to another should be based on something else (preferably objective).
No, that was NOT what my argument was.
My argument is that you have to have self-describing data somehow, otherwise you will always get out of sync with what the kernel is exporting and what userspace expects. The above crazy procfs example proves my point very well :)
sysfs "solves" this problem by requiring one value per file. If the file is not present, userspace "knows" that the value isn't there at all. It a simple solution for the problem that procfs has with multiple values in single files and is why we did it that way.
Now that's not to say this is the only way to solve the issue here, it's just the one that we decided to use at the time. statfs can choose to do it differently, but it can NOT just ignore the problem here, otherwise we end up with the old procfs problems as you show above.
Quoted 4 lines
/proc have these two problems:
First, noticeably slow:https://news.ycombinator.com/item?id=21414882
Yes, opening thousands of files is "slow", that's known :)
Quoted 3 lines
Second, overinstantiating inodes and dentries:
https://lore.kernel.org/lkml/[email protected]/
Exporting hundreds of thousands of files, what could go wrong? :)
Quoted 2 lines
statfs maybe never get to that level but it is not hard to see what lies at the end of the tunnel.
Those are nice things to think about while doing this, but I think we are a long ways off from these types of issues.
thanks,
greg k-h