RFE: 'btrfs' tools machine readable output
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
It would be really helpful if the btrfs tools had a machine-readable output.
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.
With machine-readable output, there'd be a flag which would change the output. eg:
$ btrfs filesystem show
Label: 'ROOT' uuid: af471cfc-421a-4d51-8811-ce969f76828a
Total devices 1 FS bytes used 5.29MiB
devid 1 size 767.97MiB used 92.00MiB path /dev/sda2would become:
$ btrfs --json filesystem show
{
"devices": {
"Label": "ROOT",
"uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
/// etc
}
}By this example I don't mean that JSON has to be the format -- in fact it's a terrible format with all sorts of problems -- any format which is parseable with C libraries would do for us.
Rich.
Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Hello Richard,
Quoted 6 lines · On Montag, 16. Mai 2016 13:14:56 CEST Richard W.M. Jones wrote:
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
It would be really helpful if the btrfs tools had a machine-readable output.
Quoted 8 lines
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.With machine-readable output, there'd be a flag which would change the output. eg:
I wonder whether parsing a text based output is really the most elegant method here.
How about a libbtrfs so that other tools can benefit from btrfs tools functionality? This was also desktop environments wishing to make use of snapshot functionality or advanced disk usage reporting for example can easily make use of it without calling external commands.
Of course it would likely me more effort than to implement structured output.
Thanks,
Signature · 1 line
Martin
Quoted 21 lines · Martin Steigerwald
Hello Richard,
Quoted 6 lines · On Montag, 16. Mai 2016 13:14:56 CEST Richard W.M. Jones wrote:
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
It would be really helpful if the btrfs tools had a machine-readable output.
Quoted 8 lines
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.With machine-readable output, there'd be a flag which would change the output. eg:
I wonder whether parsing a text based output is really the most elegant method here.
Quoted 6 lines
How about a libbtrfs so that other tools can benefit from btrfs tools functionality? This was also desktop environments wishing to make use of snapshot functionality or advanced disk usage reporting for example can easily make use of it without calling external commands.
Of course it would likely me more effort than to implement structured output.
Yes, a similar situation happened with qemu actually. We've been talking for over half a decade about putting the qemu block layer into a library, and it's still not happened. But we got `qemu-img info' to have JSON output in a few weeks, and that is now how we query the properties of VM disk images:
$ qemu-img info --output=json ./builder/fedora.qcow2
{
"virtual-size": 1073741824,
"filename": "./builder/fedora.qcow2",
"cluster-size": 65536,
"format": "qcow2",
"actual-size": 7933952,
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false
}
},
"dirty-flag": false
}Rich.
Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Quoted 15 lines · On Monday 16 May 2016 14:21:07 Martin Steigerwald wrote:
Hello Richard,
Quoted 12 lines · On Montag, 16. Mai 2016 13:14:56 CEST Richard W.M. Jones wrote:
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
It would be really helpful if the btrfs tools had a machine-readable output.
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.
Quoted 8 lines
Quoted 2 lines
With machine-readable output, there'd be a flag which would change the output. eg:
I wonder whether parsing a text based output is really the most elegant method here.
How about a libbtrfs so that other tools can benefit from btrfs tools functionality?
btrfs-progs is GPL v2 only, and that may be an issue for consumers (the libguestfs daemon, the component interacting with devices and filesystems inside the libguestfs appliance, is GPL v2+ for example).
Quoted 1 line
Of course it would likely me more effort than to implement structured output.
There's this of course, which is not entirely negligible.
Signature · 1 line
Pino Toscano
Quoted 31 lines · Richard W.M. Jones
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
It would be really helpful if the btrfs tools had a machine-readable output.
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.With machine-readable output, there'd be a flag which would change the output. eg:
$ btrfs filesystem show Label: 'ROOT' uuid: af471cfc-421a-4d51-8811-ce969f76828a Total devices 1 FS bytes used 5.29MiB devid 1 size 767.97MiB used 92.00MiB path /dev/sda2would become:
$ btrfs --json filesystem show { "devices": { "Label": "ROOT", "uuid": "af471cfc-421a-4d51-8811-ce969f76828a", /// etc } }
Quoted 3 lines
By this example I don't mean that JSON has to be the format -- in fact it's a terrible format with all sorts of problems -- any format which is parseable with C libraries would do for us.
I would love to see something like this myself, as it would make integration with monitoring tools so much easier. I'd vote for YAML as the output format though, as it's easily human readable as well as machine parseable while still being space efficient. Output from your example above might look something like this:
---
- filesystems
- label: 'ROOT'
uuid: af471cfc-421a-4d51-8811-ce969f76828a
devices: 1
- devid: 1
size: 767.97 MB
used 92.00 MB
path: /dev/sda2
used: 5.29 MB...
Although of course the numbers would probably just be in bytes instead of 'human' values.
Quoted 21 lines · Austin S. Hemmelgarn
Quoted 5 lines · Richard W.M. Jones
It would be really helpful if the btrfs tools had a machine-readable output.
With machine-readable output, there'd be a flag which would change the output. eg:Quoted 8 lines
$ btrfs --json filesystem show { "devices": { "Label": "ROOT", "uuid": "af471cfc-421a-4d51-8811-ce969f76828a", /// etc } }I would love to see something like this myself, as it would make integration with monitoring tools so much easier. I'd vote for YAML as the output format though, as it's easily human readable as well as machine parseable while still being space efficient.
---
- filesystems
- label: 'ROOT'
Quoted 8 lines
uuid: af471cfc-421a-4d51-8811-ce969f76828a devices: 1 - devid: 1 size: 767.97 MB used 92.00 MB path: /dev/sda2 used: 5.29 MB...
Rather than adding language-specific output/extensions to btrfs-progs, what does everyone think of an interface which allows arbitrary ordering similar to the date command, with arbitrary delimiter support?
eg: date +%a\ %d\ \%B\ %H:%m
with something like: %D (device), %L (label), %V (volume), %v (subvolume), %U (volume UUID), %u (subvol UUID), et al.
'avoids formatting the output to any particular convention, and the order of what is queried can be in any order...but I think the [unit to operate on] [named unit] bit would necessarily need to come first. We already have that in btrfs [sub/fi/dev] [name] :-)
Or should it be a separate command... Something like "btrfs-machine [unit to operate on] [named unit] [action eg: print] [cryptic line with user's choice of delimiter and formatting]". And then there might also be the option to "btrfs-machine [unit to operate on] [named unit] return [scrub status, balance status, etc.]" which returns a single integer value along the lines of idle|in progress|success|failure. And at some point in the future btrfs-machine [unit to operate on] [named unit] progress [operation] which functions like "fsck -C [fd]".
Now that, I imagine, would be something libvirt users would like, whether for web interface, GUI, or wrapper script for for a slick NAS web interfaces.
For now, maybe start with a proof of concept with only the "print" action implemented? And eventually merge this into the main btrfs command something like this: "btrfs -m unit name action args" or "btrfs --machine unit name action args"
Quoted 2 lines · Martin Steigerwald
How about a libbtrfs so that other tools can benefit from btrfs tools functionality?
Does much does /usr/lib/x86_64-linux-gnu/libbtrfs.so expose?
What is the path of minimum duplication of work, and minimum long-term maintenance? If my assumptions are correct, this is a 2000-level programming course textbook challenge on writing a program using a library to query some values...and that might be something I could manage :-)
Cheers,
Nicholas
Quoted 56 lines · Nicholas D Steeves
Quoted 21 lines · Austin S. Hemmelgarn
Quoted 5 lines · Richard W.M. Jones
It would be really helpful if the btrfs tools had a machine-readable output.
With machine-readable output, there'd be a flag which would change the output. eg:Quoted 8 lines
$ btrfs --json filesystem show { "devices": { "Label": "ROOT", "uuid": "af471cfc-421a-4d51-8811-ce969f76828a", /// etc } }I would love to see something like this myself, as it would make integration with monitoring tools so much easier. I'd vote for YAML as the output format though, as it's easily human readable as well as machine parseable while still being space efficient.
---
- filesystems
- label: 'ROOT'Quoted 8 lines
uuid: af471cfc-421a-4d51-8811-ce969f76828a devices: 1 - devid: 1 size: 767.97 MB used 92.00 MB path: /dev/sda2 used: 5.29 MB...
Rather than adding language-specific output/extensions to btrfs-progs, what does everyone think of an interface which allows arbitrary ordering similar to the date command, with arbitrary delimiter support?
eg: date +%a\ %d\ \%B\ %H:%m
with something like: %D (device), %L (label), %V (volume), %v (subvolume), %U (volume UUID), %u (subvol UUID), et al.
'avoids formatting the output to any particular convention, and the order of what is queried can be in any order...but I think the [unit to operate on] [named unit] bit would necessarily need to come first. We already have that in btrfs [sub/fi/dev] [name] :-)
Or should it be a separate command... Something like "btrfs-machine [unit to operate on] [named unit] [action eg: print] [cryptic line with user's choice of delimiter and formatting]". And then there might also be the option to "btrfs-machine [unit to operate on] [named unit] return [scrub status, balance status, etc.]" which returns a single integer value along the lines of idle|in progress|success|failure. And at some point in the future btrfs-machine [unit to operate on] [named unit] progress [operation] which functions like "fsck -C [fd]".
Quoted 8 lines
Now that, I imagine, would be something libvirt users would like, whether for web interface, GUI, or wrapper script for for a slick NAS web interfaces.
For now, maybe start with a proof of concept with only the "print" action implemented? And eventually merge this into the main btrfs command something like this: "btrfs -m unit name action args" or "btrfs --machine unit name action args"
I'm not sure how practical this would be, the format language would have to be pretty complex, and would need to be easily extensible at the same time. We have a functionally infinite upper bound on what can be reported by a number of commands (fi df for example is bounded only by the number of supported profiles), and trying to sanely handle that gets complicated.
Also, once we have _some_ kind of machine parseable output, it becomes much easier to use other tools to pull out the exact info you want.
Quoted 11 lines · Martin Steigerwald
Quoted 2 lines
How about a libbtrfs so that other tools can benefit from btrfs tools functionality?
Does much does
/usr/lib/x86_64-linux-gnu/libbtrfs.soexpose?What is the path of minimum duplication of work, and minimum long-term maintenance? If my assumptions are correct, this is a 2000-level programming course textbook challenge on writing a program using a library to query some values...and that might be something I could manage :-)
Based on the fact that our CLI interface is really the only API people are using, I don't think there's almost anything exposed at a high-level in libbtrfs.
Quoted 2 lines · Richard W.M. Jones
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
Neither do have I, but agree with the idea and the proposed way. Here are my notes
https://github.com/kdave/drafts/blob/master/btrfs/progs-output-format.txt
Quoted 8 lines
It would be really helpful if the btrfs tools had a machine-readable output.
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.
The error messages haven't been touched for a long time and they get fixed or rephrased incrementally. I'm sorry you have to deal with the breakage but please understand that we don't have resources to do it at once or to implement complete machine readable output. At the moment the messages are aimed for a human consumer.
Wiring the various formats into the codebase would be pretty intrusive, but doable. We can do some commands first as a prototype, the 'fi show' seems like a good candidate.
Quoted 19 lines
With machine-readable output, there'd be a flag which would change the output. eg:
$ btrfs filesystem show Label: 'ROOT' uuid: af471cfc-421a-4d51-8811-ce969f76828a Total devices 1 FS bytes used 5.29MiB devid 1 size 767.97MiB used 92.00MiB path /dev/sda2would become:
$ btrfs --json filesystem show { "devices": { "Label": "ROOT", "uuid": "af471cfc-421a-4d51-8811-ce969f76828a", /// etc } }
Quoted 3 lines
By this example I don't mean that JSON has to be the format -- in fact it's a terrible format with all sorts of problems -- any format which is parseable with C libraries would do for us.
My ideas was to allow multiple formats, with slight change to the syntax:
$ btrfs --format=json ... $ btrfs --format=plain ... $ btrfs --format=yaml ...
Quoted 23 lines · David Sterba
Quoted 2 lines · Richard W.M. Jones
I don't have time to implement this right now, so I'm just posting this as a suggestion/request ...
Neither do have I, but agree with the idea and the proposed way. Here are my notes
https://github.com/kdave/drafts/blob/master/btrfs/progs-output-format.txtQuoted 8 lines
It would be really helpful if the btrfs tools had a machine-readable output.
Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs comes along it changes the output slightly, and we end up having to add all sorts of hacks.The error messages haven't been touched for a long time and they get fixed or rephrased incrementally. I'm sorry you have to deal with the breakage but please understand that we don't have resources to do it at once or to implement complete machine readable output. At the moment the messages are aimed for a human consumer.
Quoted 3 lines
Wiring the various formats into the codebase would be pretty intrusive, but doable. We can do some commands first as a prototype, the 'fi show' seems like a good candidate.
General list off the top of my head that would be good candidates:
* fi show
* fi df
* fi usage
* scrub status
* subvolume list
* subvolume show
Those cover about 90% of the regular maintenance scenario's that I can think of which need more data than just whether or not something succeeded.
Quoted 24 lines
Quoted 20 lines
With machine-readable output, there'd be a flag which would change the output. eg:
$ btrfs filesystem show Label: 'ROOT' uuid: af471cfc-421a-4d51-8811-ce969f76828a Total devices 1 FS bytes used 5.29MiB devid 1 size 767.97MiB used 92.00MiB path /dev/sda2would become:
$ btrfs --json filesystem show { "devices": { "Label": "ROOT", "uuid": "af471cfc-421a-4d51-8811-ce969f76828a", /// etc } }Quoted 3 lines
By this example I don't mean that JSON has to be the format -- in fact it's a terrible format with all sorts of problems -- any format which is parseable with C libraries would do for us.
Quoted 6 lines
My ideas was to allow multiple formats, with slight change to the syntax:
$ btrfs --format=json ... $ btrfs --format=plain ... $ btrfs --format=yaml ...
I like this idea, with one specific caveat, it would be _really_ nice to be able to control at compile time what formats are supported. Somebody's eventually going to want XML support, and I'd rather not have to depend on expat or something similar in a tool that I regularly build statically (I'm willing to tolerate YAML because that's what I'd use, but I would probably compile out the JSON support too).
Quoted 10 lines · Austin S. Hemmelgarn
Quoted 10 lines
Quoted 3 lines
By this example I don't mean that JSON has to be the format -- in fact it's a terrible format with all sorts of problems -- any format which is parseable with C libraries would do for us.
My ideas was to allow multiple formats, with slight change to the syntax:
$ btrfs --format=json ... $ btrfs --format=plain ... $ btrfs --format=yaml ...
Quoted 6 lines
I like this idea, with one specific caveat, it would be _really_ nice to be able to control at compile time what formats are supported. Somebody's eventually going to want XML support, and I'd rather not have to depend on expat or something similar in a tool that I regularly build statically (I'm willing to tolerate YAML because that's what I'd use, but I would probably compile out the JSON support too).
I hope we can avoid any 3rd party library dependencies, we're just generating the output not parsing it. Either copy a version of the libraries or copy the quoting code, the rest can be simply implemented as a printf with enhanced format stings.
Quoted 18 lines · David Sterba
Quoted 10 lines · Austin S. Hemmelgarn
Quoted 10 lines
Quoted 3 lines
By this example I don't mean that JSON has to be the format -- in fact it's a terrible format with all sorts of problems -- any format which is parseable with C libraries would do for us.
My ideas was to allow multiple formats, with slight change to the syntax:
$ btrfs --format=json ... $ btrfs --format=plain ... $ btrfs --format=yaml ...Quoted 6 lines
I like this idea, with one specific caveat, it would be _really_ nice to be able to control at compile time what formats are supported. Somebody's eventually going to want XML support, and I'd rather not have to depend on expat or something similar in a tool that I regularly build statically (I'm willing to tolerate YAML because that's what I'd use, but I would probably compile out the JSON support too).
Quoted 5 lines
I hope we can avoid any 3rd party library dependencies, we're just generating the output not parsing it. Either copy a version of the libraries or copy the quoting code, the rest can be simply implemented as a printf with enhanced format stings.
As much as I hate the possibility of having 3rd party library dependencies in btrfs-progs, I'm a lot more nervous about having the equivalent stuff open-coded or having a local copy of the library, as both have significant potential for bugs. A significant majority of Linux systems have Python, and therefore have at least some XML library (I forget which one Python prefers to use). Many also have at least one JSON library too. If we make such dependencies optional, and also only load the library if we need it at runtime (so we would only load the JSON library if we are asked to print JSON output), then we could still avoid the overhead for most people.
The other possibility would be to just pick one output format, open code that, and be done with it. It's not hard to write s simple script to convert between YAML, JSON, and other similar formats. If we do go this way, I'd still suggest YAML as it's more human friendly, a lot easier to parse using scripts due to the line oriented nature of it, and is also probably easier to code without any external dependencies.
As I said I don't have any particular favourite format. However I'll just point you to the code that qemu uses (basically open-coding) to write json:
Hopefully this is useful as a data point.
Rich.
Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Quoted 7 lines · Richard W.M. Jones
As I said I don't have any particular favourite format. However I'll just point you to the code that qemu uses (basically open-coding) to write json:
Hopefully this is useful as a data point.
It is, thanks, this amount of code to do json export matches my expectations of a local version of the sources.