RFC: btrfs-progs json formatting
Hi,
I'd like to get some feedback on the json output, the overall structure of the information and naming.
Code: git://github.com/kdave/btrfs-progs.git preview-json
The one example command that supports it is
$ ./btrfs --format json subvolume show /mnt
The test tests/cli-tests/011-* can demonstrate the capabilities. The json formatter and example use are in the top commits, otherwise the branch contains lot of unrelated changes.
Example output:
{
"__header": {
"version": "1"
},
"subvolume-show": {
"path": "subv2",
"name": "subv2",
"uuid": "9178604e-e961-ef40-a5f0-2c109126b209",
"parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209",
"received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209",
"otime": "2019-06-25 17:15:28 +0200",
"subvolume_id": "256",
"generation": "6",
"parent_id": "5",
"toplevel_id": "5",
"flags": "-",
"snapshots": [
],
"quota-group": {
"qgroupid": "0/256",
"qgroup-limit-referenced": "-",
"qgroup-limit-exclusive": "-",
"qgroup-usage-referenced": "16.00KiB",
"qgroup-usage-exclusive": "16.00KiB"
} }
}
The current proposal:
- the toplevel group contains 2 maps, one is some generic header, to identify version of the format or the version of progs that produces it or whatever could be useful and I did not think of it
- the 2nd map is named according to the command that generated the output, this is to be able to merge outputs from several commands, or to somehow define the contents of the map
Compare that to a simple unnamed group with bunch of values:
{
"path": "subv2",
"name": "subv2",
"uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256",
"generation": "6",
"parent_id": "5",
"toplevel_id": "5",
"flags": "-",
"snapshots": [
],
"quota-group": {
"qgroupid": "0/256",
"qgroup-limit-referenced": "-",
"qgroup-limit-exclusive": "-",
"qgroup-usage-referenced": "16.00KiB",
"qgroup-usage-exclusive": "16.00KiB" }
}
That's a bit shorter but makes validation harder. I assume that the output would be accessed programatically like (python pseudo-code)
---
rawjson = output_of_command("btrfs --format json subvolume show /mnt") json = Json.from_string(rawjson)
if json["subvolume-show"]["flags"] != "-":
print("Subvolume %s has flags", json["subvolume-show"]["name"])
---
so the latter is subset of the former, without one map reference.
Once the json format is settled, more commands can be easily converted.
Hi,
Quoted 10 lines · David Sterba
Hi,
I'd like to get some feedback on the json output, the overall structure of the information and naming.
Code: git://github.com/kdave/btrfs-progs.git preview-jsonThe one example command that supports it is
$ ./btrfs --format json subvolume show /mnt
I think for deciding what the structure will look like, it's very interesting to look at what tools, utils, use cases, etc.. will use this and how. Taking the bike for a ride, instead of painting the shed.
E.g.
- Does the user expect the same text as in a current btrfs sub list,
but while being able to quickly use specific fields instead of using awk to split the current output? -> a field flags with contents like "foo,bar" or "FOO|BAR" might be expected instead of an integer.
- Does the user want to write another tool that actually
programmatically does more? In that case the user might prefer having the number instead of the text of flags which needs to be parsed back again.
Quoted 36 lines
The test tests/cli-tests/011-* can demonstrate the capabilities. The json formatter and example use are in the top commits, otherwise the branch contains lot of unrelated changes.
Example output:
{
"__header": { "version": "1" }, "subvolume-show": { "path": "subv2", "name": "subv2", "uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256", "generation": "6", "parent_id": "5", "toplevel_id": "5", "flags": "-", "snapshots": [ ], "quota-group": { "qgroupid": "0/256", "qgroup-limit-referenced": "-", "qgroup-limit-exclusive": "-", "qgroup-usage-referenced": "16.00KiB", "qgroup-usage-exclusive": "16.00KiB" }}
}The current proposal:
Quoted 7 lines
- the toplevel group contains 2 maps, one is some generic header, to identify version of the format or the version of progs that produces it or whatever could be useful and I did not think of it
- the 2nd map is named according to the command that generated the output, this is to be able to merge outputs from several commands, or to somehow define the contents of the map
In what example use case would I be combining output of several commands?
Quoted 22 lines
Compare that to a simple unnamed group with bunch of values:
{
"path": "subv2",
"name": "subv2",
"uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256",
"generation": "6",
"parent_id": "5",
"toplevel_id": "5",
"flags": "-",
"snapshots": [
],"quota-group": { "qgroupid": "0/256", "qgroup-limit-referenced": "-", "qgroup-limit-exclusive": "-", "qgroup-usage-referenced": "16.00KiB", "qgroup-usage-exclusive": "16.00KiB"
Quoted 5 lines
}
}That's a bit shorter but makes validation harder. I assume that the output would be accessed programatically like (python pseudo-code)
What about having structures that are result-data-oriented instead of command-oriented?
E.g. something that shows a subvol or lists them could have a bunch of "subvolumes" in the result:
{
"__header": {
[...]
},
"subvolumes": {
5: {
}
256: {
},
260: {
},
1337: {
} }
}
Then subvol show 260 could return the same format, with just one of them:
{
"__header": {
[...]
},
"subvolumes": {
260: {
}, }
}
Now I can also combine the result of a few targeted subvol show commands:
subvols = sub_show_260_json["subvolumes"]subvols.update(sub_show_1337_json["subvolumes"])
list(subvols.keys())
-> [260, 1337]
Quoted 4 lines
---
rawjson =output_of_command("btrfs --format json subvolume show /mnt")json =Json.from_string(rawjson)
Quoted 8 lines
if json["subvolume-show"]["flags"] != "-":
print("Subvolume %s has flags", json["subvolume-show"]["name"])
---so the latter is subset of the former, without one map reference.
Once the json format is settled, more commands can be easily converted.
Just some random thoughts.
Hans
Quoted 10 lines · Hans van Kranenburg
Hi,
Quoted 7 lines · David Sterba
Hi,
I'd like to get some feedback on the json output, the overall structure of the information and naming.
Code: git://github.com/kdave/btrfs-progs.git preview-json
Quoted 7 lines
Quoted 3 lines
The one example command that supports it is
$ ./btrfs --format json subvolume show /mntI think for deciding what the structure will look like, it's very interesting to look at what tools, utils, use cases, etc.. will use this and how. Taking the bike for a ride, instead of painting the shed.
Geez bike shedding mentioned in 2nd sentence?
Quoted 5 lines
E.g.
- Does the user expect the same text as in a current btrfs sub list,
but while being able to quickly use specific fields instead of using awk to split the current output? -> a field flags with contents like "foo,bar" or "FOO|BAR" might be expected instead of an integer.
The idea is that the default output is for humans, but if there's need to parse the output, json is the format for that. Cutting out the information using 'awk' is not expected here.
So the information provided by the text and json in a single command is almost the same, with json some enhancements are possible to allow linking to other commands, eg. not just the subvolume path but also the id. Something that you suggested below.
Quoted 3 lines
- Does the user want to write another tool that actually
programmatically does more? In that case the user might prefer having the number instead of the text of flags which needs to be parsed back again.
Ok, so bitfield values can be exported in two ways, one as a raw number and one as the string.
Quoted 40 lines
Quoted 40 lines
The test tests/cli-tests/011-* can demonstrate the capabilities. The json formatter and example use are in the top commits, otherwise the branch contains lot of unrelated changes.
Example output:
{
"__header": { "version": "1" }, "subvolume-show": { "path": "subv2", "name": "subv2", "uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256", "generation": "6", "parent_id": "5", "toplevel_id": "5", "flags": "-", "snapshots": [ ], "quota-group": { "qgroupid": "0/256", "qgroup-limit-referenced": "-", "qgroup-limit-exclusive": "-", "qgroup-usage-referenced": "16.00KiB", "qgroup-usage-exclusive": "16.00KiB" }}
}The current proposal:
- the toplevel group contains 2 maps, one is some generic header, to identify version of the format or the version of progs that produces it or whatever could be useful and I did not think of it
Quoted 5 lines
Quoted 3 lines
- the 2nd map is named according to the command that generated the output, this is to be able to merge outputs from several commands, or to somehow define the contents of the map
In what example use case would I be combining output of several commands?
Well, that's always an issue with usecases. I don't have immediate use for that but as the format will be set to stone, additional flexibility can pay off in the future.
The util-linux tools support json output and that's what I'm trying to copy for sake of some consistency, as the 'btrfs' tool is sort of low-level administration tool very close in purpose.
Check eg 'lsblk --json' or 'lscpu --json', that print "blockdevices" or "lscpu" as the name of the map.
Quoted 25 lines
Quoted 25 lines
Compare that to a simple unnamed group with bunch of values:
{
"path": "subv2",
"name": "subv2",
"uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256",
"generation": "6",
"parent_id": "5",
"toplevel_id": "5",
"flags": "-",
"snapshots": [
],"quota-group": { "qgroupid": "0/256", "qgroup-limit-referenced": "-", "qgroup-limit-exclusive": "-", "qgroup-usage-referenced": "16.00KiB", "qgroup-usage-exclusive": "16.00KiB"}
}
Quoted 5 lines
Quoted 2 lines
That's a bit shorter but makes validation harder. I assume that the output would be accessed programatically like (python pseudo-code)
What about having structures that are result-data-oriented instead of command-oriented?
The commands give different level of details and retrieving some of that can take longer. Eg. the quota information query the tree for the limits, so this might not be suitable for the 'list' command.
Quoted 31 lines
E.g. something that shows a subvol or lists them could have a bunch of "subvolumes" in the result:
{
"__header": { [...] }, "subvolumes": { 5: { } 256: { }, 260: { }, 1337: { }}
}Then subvol show 260 could return the same format, with just one of them:
{
"__header": { [...] }, "subvolumes": { 260: { },}
}
Quoted 6 lines
Now I can also combine the result of a few targeted subvol show commands:
subvols = sub_show_260_json["subvolumes"]
subvols.update(sub_show_1337_json["subvolumes"])
list(subvols.keys())
-> [260, 1337]
I think I see what you mean, perhaps more commands need to be explored to see if we can find some common structures to avoid the per-command output. Something like lightweight-subvolume and full-details-subvolume, with some way distinguish that programatically or just leave it to try/catch.
Quoted 13 lines · David Sterba
Quoted 12 lines · Hans van Kranenburg
Hi,
Quoted 9 lines · David Sterba
Hi,
I'd like to get some feedback on the json output, the overall structure of the information and naming.
Code: git://github.com/kdave/btrfs-progs.git preview-jsonThe one example command that supports it is
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
$ ./btrfs --format json subvolume show /mntI think for deciding what the structure will look like, it's very interesting to look at what tools, utils, use cases, etc.. will use this and how. Taking the bike for a ride, instead of painting the shed.
Geez bike shedding mentioned in 2nd sentence?
Sorry, I didn't mean it like that. :)
What I was trying to say is that I'm genuinely interested in hearing the actual use cases from actual users who asked for this feature in the past, because I think it will be the best help to improve the output format (as opposed to looking at the structures from an aesthetical viewpoint).
If you still know who requested the feature in the past, then maybe Cc them on the thread? Might be missed easily, otherwise.
Quoted 152 lines
Quoted 5 lines
E.g.
- Does the user expect the same text as in a current btrfs sub list,
but while being able to quickly use specific fields instead of using awk to split the current output? -> a field flags with contents like "foo,bar" or "FOO|BAR" might be expected instead of an integer.The idea is that the default output is for humans, but if there's need to parse the output, json is the format for that. Cutting out the information using 'awk' is not expected here.
So the information provided by the text and json in a single command is almost the same, with json some enhancements are possible to allow linking to other commands, eg. not just the subvolume path but also the id. Something that you suggested below.
Quoted 3 lines
- Does the user want to write another tool that actually
programmatically does more? In that case the user might prefer having the number instead of the text of flags which needs to be parsed back again.Ok, so bitfield values can be exported in two ways, one as a raw number and one as the string.
Quoted 40 lines
Quoted 40 lines
The test tests/cli-tests/011-* can demonstrate the capabilities. The json formatter and example use are in the top commits, otherwise the branch contains lot of unrelated changes.
Example output:
{
"__header": { "version": "1" }, "subvolume-show": { "path": "subv2", "name": "subv2", "uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256", "generation": "6", "parent_id": "5", "toplevel_id": "5", "flags": "-", "snapshots": [ ], "quota-group": { "qgroupid": "0/256", "qgroup-limit-referenced": "-", "qgroup-limit-exclusive": "-", "qgroup-usage-referenced": "16.00KiB", "qgroup-usage-exclusive": "16.00KiB" }}
}The current proposal:
- the toplevel group contains 2 maps, one is some generic header, to identify version of the format or the version of progs that produces it or whatever could be useful and I did not think of it
Quoted 5 lines
Quoted 3 lines
- the 2nd map is named according to the command that generated the output, this is to be able to merge outputs from several commands, or to somehow define the contents of the map
In what example use case would I be combining output of several commands?
Well, that's always an issue with usecases. I don't have immediate use for that but as the format will be set to stone, additional flexibility can pay off in the future.
The util-linux tools support json output and that's what I'm trying to copy for sake of some consistency, as the 'btrfs' tool is sort of low-level administration tool very close in purpose.
Check eg 'lsblk --json' or 'lscpu --json', that print "blockdevices" or "lscpu" as the name of the map.
Quoted 25 lines
Quoted 25 lines
Compare that to a simple unnamed group with bunch of values:
{
"path": "subv2",
"name": "subv2",
"uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "parent_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "received_uuid": "9178604e-e961-ef40-a5f0-2c109126b209", "otime": "2019-06-25 17:15:28 +0200", "subvolume_id": "256",
"generation": "6",
"parent_id": "5",
"toplevel_id": "5",
"flags": "-",
"snapshots": [
],"quota-group": { "qgroupid": "0/256", "qgroup-limit-referenced": "-", "qgroup-limit-exclusive": "-", "qgroup-usage-referenced": "16.00KiB", "qgroup-usage-exclusive": "16.00KiB"}
}Quoted 5 lines
Quoted 2 lines
That's a bit shorter but makes validation harder. I assume that the output would be accessed programatically like (python pseudo-code)
What about having structures that are result-data-oriented instead of command-oriented?
The commands give different level of details and retrieving some of that can take longer. Eg. the quota information query the tree for the limits, so this might not be suitable for the 'list' command.
Quoted 31 lines
E.g. something that shows a subvol or lists them could have a bunch of "subvolumes" in the result:
{
"__header": { [...] }, "subvolumes": { 5: { } 256: { }, 260: { }, 1337: { }}
}Then subvol show 260 could return the same format, with just one of them:
{
"__header": { [...] }, "subvolumes": { 260: { },}
}Quoted 6 lines
Now I can also combine the result of a few targeted subvol show commands:
subvols = sub_show_260_json["subvolumes"]
subvols.update(sub_show_1337_json["subvolumes"])
list(subvols.keys())
-> [260, 1337]
Quoted 6 lines
I think I see what you mean, perhaps more commands need to be explored to see if we can find some common structures to avoid the per-command output. Something like lightweight-subvolume and full-details-subvolume, with some way distinguish that programatically or just leave it to try/catch.
Hans