Skip to contentSkip to conversation

"statsfs" API design

DISCUSSION Alexey Dobriyan 4 participants 9 messages Nov 9, 2019 – Nov 26, 2019

lore
Share & tools
lore
#0 lore raw
Quoted 2 lines

statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in /sys/kernel/stats

I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.

Quoted 4 lines

The first user of statsfs would be KVM, which is currently exposing its stats in debugfs

Google has KVM patches to gather statistics in a binary format

Which is a right thing to do.

#1 lore raw
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines

statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in /sys/kernel/stats

I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.

Quoted 6 lines
Quoted 4 lines

The first user of statsfs would be KVM, which is currently exposing its stats in debugfs

Google has KVM patches to gather statistics in a binary format

Which is a right thing to do.

It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.

The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.

So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.

Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.

greg k-h

#2 lore raw
Quoted 24 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines

statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in /sys/kernel/stats

I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.

Quoted 6 lines
Quoted 4 lines

The first user of statsfs would be KVM, which is currently exposing its stats in debugfs

Google has KVM patches to gather statistics in a binary format

Which is a right thing to do.

It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.

The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.

Quoted 6 lines

So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.

Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.

What about using a text format like YAML? Here's some benefits:

- The fields are self describing based on the key name.
- New fields can be easily added without breaking compatibility.
- Allows for a script to easily parse the contents while keeping human readability.
- Would work for systems that run busybox as their userspace without having to install additional tools.
- Allows for a nested data structure.

The downside is that the output would be larger than a binary interface but it's more maintainable in my opinion.

Brian

#3 lore raw
Quoted 34 lines · Brian Masney
Quoted 24 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines

statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in /sys/kernel/stats

I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.

Quoted 6 lines
Quoted 4 lines

The first user of statsfs would be KVM, which is currently exposing its stats in debugfs

Google has KVM patches to gather statistics in a binary format

Which is a right thing to do.

It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.

The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.

Quoted 6 lines

So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.

Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.

What about using a text format like YAML? Here's some benefits:

Quoted 7 lines

- The fields are self describing based on the key name.
- New fields can be easily added without breaking compatibility.
- Allows for a script to easily parse the contents while keeping human readability.
- Would work for systems that run busybox as their userspace without having to install additional tools.
- Allows for a nested data structure.

varlink was created to solve the issues that people have had with YAML over time, so you might want to look into that :)
https://varlink.org/

Quoted 2 lines

The downside is that the output would be larger than a binary interface but it's more maintainable in my opinion.

binary interfaces are unmaintainable over time, especially when you do not control both sides of the interface (unlike Google and their use of this for KVM stats.)

thanks,

greg k-h

#4 lore raw
Quoted 43 lines · Greg KH
Quoted 34 lines · Brian Masney
Quoted 24 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines

statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in /sys/kernel/stats

I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.

Quoted 6 lines
Quoted 4 lines

The first user of statsfs would be KVM, which is currently exposing its stats in debugfs

Google has KVM patches to gather statistics in a binary format

Which is a right thing to do.

It's always "simpler" to just take binary data and suck it in. That works for a year or so until another value needs to be supported. Or removed. Or features are backported.

The reason text values in individual files work is they are "self describable" and "self discoverable". You "know" what the value is and that it is supported because the file is there or not. With binary values in a single file you do not know any of that.

Quoted 6 lines

So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.

Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.

What about using a text format like YAML? Here's some benefits:

Quoted 7 lines

- The fields are self describing based on the key name.
- New fields can be easily added without breaking compatibility.
- Allows for a script to easily parse the contents while keeping human readability.
- Would work for systems that run busybox as their userspace without having to install additional tools.
- Allows for a nested data structure.

Quoted 3 lines

varlink was created to solve the issues that people have had with YAML over time, so you might want to look into that :)
https://varlink.org/

Well, varlink really is JSON, but at least there's example code for userspace and kernelspace to pull from to knock up tests for people who want to look into it.

thanks,

greg k-h

#5 lore raw
In reply to Greg KH
Quoted 8 lines · Greg KH
Quoted 7 lines · Alexey Dobriyan
Quoted 2 lines

statsfs is a proposal for a new Linux kernel synthetic filesystem, to be mounted in /sys/kernel/stats

I think /proc experiment teaches pretty convincingly that dressing things into a filesystem can be done but ultimately is a stupid idea. It adds so much overhead for small-to-medium systems.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The first user of statsfs would be KVM, which is currently exposing its stats in debugfs

Google has KVM patches to gather statistics in a binary format

Which is a right thing to do.

It's always "simpler" to just take binary data and suck it in.

Faster too!

Quoted 5 lines

That works for a year or so until another value needs to be supported. Or removed. Or features are backported.

The reason text values in individual files work is they are "self describable" and "self discoverable".

Untrue. Applications always knows what the data means, by definition:

	"0x42"

What is this? 4-char NUL-terminated string? Or an integer 66? Or a 4-byte nonce blob for some kind of crypto algorithm.

In the other direction: describe every field of /proc/*/stat file without looking to the manpage:

$ cat /proc/self/stat
5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0
Quoted 3 lines

You "know" what the value is and
that it is supported because the file is there or not. With binary values in a single file you do not know any of that.

You _always_ know that.

Quoted 6 lines

So you need some way of describing the data to userspace in order for this to work properly over the next 20+ years.

Maybe something like varlink which describes the data coming from the kernel in an easy-to-handle format? Or something else, but just using blobs does not work over the long-term, sorry.

Text doesn't work either. Why do you think /proc/*/maps have 1 space character at the end of anon mappings? Because someone fucked up. Here is how people use /proc in the field:

https://stackoverflow.com/questions/3596781/how-to-detect-if-the-current-process-is-being-run-by-gdb

open /proc/*/status
read
strstr("TracerPid:")

I'd humbly suggest to define the minimum amount of work for the task:
* some kind of percpu loop to gather stats
* some kind of accumulation code, possibly with min/max/avg
* write clear data
* copy_to_user

and realise that everything alse is a waste of electricity, namely,

* pathname allocation (4KB)
* VFS '/' split, lookups (/sys/kernel/.../" means 3+ lookups
* 192 bytes for each dentry
* 550+ bytes per inode
* 3 system calls per act of gathering statistics
userspace will be written in the most stupid way possible without openat() etc
* userspace snprintf() for pathname
* kernel space snprintf() somewhere
* multiple copying inside kernel (vsnprintf.c)
* general inability for userspace to estimate the amount of data in decimal (nobody does that), so nicely sized buffers of 4K or 1K or 16KB (bash) will be used which is a waste.

#6 lore raw
Quoted 5 lines · Alexey Dobriyan

In the other direction: describe every field of /proc/*/stat file without looking to the manpage:

$ cat /proc/self/stat
5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0

That's why this is not what I am proposing, and also not what Greg has mentioned.

Quoted 15 lines

and realise that everything alse is a waste of electricity, namely,

* pathname allocation (4KB)
* VFS '/' split, lookups (/sys/kernel/.../" means 3+ lookups
* 192 bytes for each dentry
* 550+ bytes per inode
* 3 system calls per act of gathering statistics
userspace will be written in the most stupid way possible without openat() etc
* userspace snprintf() for pathname
* kernel space snprintf() somewhere
* multiple copying inside kernel (vsnprintf.c)
* general inability for userspace to estimate the amount of data in decimal (nobody does that), so nicely sized buffers of 4K or 1K or 16KB (bash) will be used which is a waste.

Yeah, all of this is true but I know how much I use /sys/kernel/debug/kvm so backwards-compatibility with it is certainly a requirement for stats. Good thing, having a high-level stats API lets you also design something that targets different usecases than just quick "cat" or "watch". The somewhat wasteful sysfs interface to statsfs can even be hidden behind a kconfig symbol once there is an alternative. It also makes it possible to create inodes on demand if someone is so inclined.

So the good thing is that despite the disagreements, this can be considered an argument in favor of statsfs, and we agree on that. :)

Thanks,

Paolo

#7 lore raw
Quoted 9 lines · Paolo Bonzini
Quoted 5 lines · Alexey Dobriyan

In the other direction: describe every field of /proc/*/stat file without looking to the manpage:

$ cat /proc/self/stat
5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0

That's why this is not what I am proposing, and also not what Greg has mentioned.

The argument was that text is somehow superior to binary. Experiment shows that userspace can make a mess of both modes therefore preferring one to another should be based on something else (preferably objective).

/proc have these two problems:
First, noticeably slow:

	https://news.ycombinator.com/item?id=21414882

Second, overinstantiating inodes and dentries:

	https://lore.kernel.org/lkml/[email protected]/

statfs maybe never get to that level but it is not hard to see what lies at the end of the tunnel.

#8 lore raw
Quoted 8 lines · Alexey Dobriyan
Quoted 7 lines · Paolo Bonzini
Quoted 5 lines · Alexey Dobriyan

In the other direction: describe every field of /proc/*/stat file without looking to the manpage:

$ cat /proc/self/stat
5349 (cat) R 5342 5349 5342 34826 5349 4210688 91 0 0 0 0 0 0 0 20 0 1 0 864988 9183232 184 18446744073709551615 94352028622848 94352028651936 140733810522864 0 0 0 0 0 0 0 0 0 17 5 0 0 0 0 0 94352030751824 94352030753376 94352060055552 140733810527527 140733810527547 140733810527547 140733810532335 0
Quoted 6 lines
Quoted 2 lines

That's why this is not what I am proposing, and also not what Greg has mentioned.

The argument was that text is somehow superior to binary. Experiment shows that userspace can make a mess of both modes therefore preferring one to another should be based on something else (preferably objective).

No, that was NOT what my argument was.

My argument is that you have to have self-describing data somehow, otherwise you will always get out of sync with what the kernel is exporting and what userspace expects. The above crazy procfs example proves my point very well :)

sysfs "solves" this problem by requiring one value per file. If the file is not present, userspace "knows" that the value isn't there at all. It a simple solution for the problem that procfs has with multiple values in single files and is why we did it that way.

Now that's not to say this is the only way to solve the issue here, it's just the one that we decided to use at the time. statfs can choose to do it differently, but it can NOT just ignore the problem here, otherwise we end up with the old procfs problems as you show above.

Quoted 4 lines

/proc have these two problems:
First, noticeably slow:

	https://news.ycombinator.com/item?id=21414882

Yes, opening thousands of files is "slow", that's known :)

Quoted 3 lines

Second, overinstantiating inodes and dentries:

	https://lore.kernel.org/lkml/[email protected]/

Exporting hundreds of thousands of files, what could go wrong? :)

Quoted 2 lines

statfs maybe never get to that level but it is not hard to see what lies at the end of the tunnel.

Those are nice things to think about while doing this, but I think we are a long ways off from these types of issues.

thanks,

greg k-h