test if a subvolume is a snapshot?

How can I test if a subvolume is a snapshot?

Example:

/test/.snapshot/2017-09-08_1037.single is a snapshot (of /test) /test/data is a regular subvolume

I know this, because I have created them with suitable names :-)

But how can I see/test it?

root@fex:~/bin# btrfs subvol show /test/.snapshot/2017-09-08_1037.single
/test/.snapshot/2017-09-08_1037.single
        Name:                   2017-09-08_1037.single
        UUID:                   eff1cebb-b885-6b47-ae69-36a7c3f266eb
        Parent UUID:            -
        Received UUID:          -
        Creation time:          2017-09-08 10:37:04 +0200
        Subvolume ID:           354
        Generation:             237
        Gen at creation:        237
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):

root@fex:~/bin# btrfs subvol show /test/data
/test/data
        Name:                   data
        UUID:                   b32a5949-dfd6-ef45-8616-34ae4cdf6fb8
        Parent UUID:            -
        Received UUID:          -
        Creation time:          2017-09-06 18:30:33 +0200
        Subvolume ID:           257
        Generation:             224
        Gen at creation:        8
        Parent ID:              5
        Top level ID:           5
        Flags:                  -

root@fex:~/bin# btrfs subvol show /test
/test is toplevel subvolume

Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[email protected]>

This question is based on the assumption that "snapshot" is a distinct type of subvolume and not just an operation that creates a subvolume with reflinked contents.

Unfortunately Btrfs does indeed make snapshots a distinct type of subvolume... In my 4.4 kernel/progs version of Btrfs it seems that the 'Parent UUID' is that of the source of the snapshot, and the source of a snapshot somehow comes with a list to all the snapshots taken from it:

# ls /fs/sda7

  =        @170826  @170829  @170901  @170903  @170905  @170907
  @170825  @170828  @170830  @170902  @170904  @170906  lost+found

# btrfs subvolume list /fs/sda7
ID 431 gen 532441 top level 5 path =
ID 1619 gen 524915 top level 5 path @170825
ID 1649 gen 524915 top level 5 path @170826
ID 1651 gen 524915 top level 5 path @170828
ID 1652 gen 524915 top level 5 path @170829
ID 1654 gen 524915 top level 5 path @170830
ID 1655 gen 523316 top level 5 path @170901
ID 1656 gen 524034 top level 5 path @170902
ID 1658 gen 525628 top level 5 path @170903
ID 1659 gen 527121 top level 5 path @170904
ID 1660 gen 528719 top level 5 path @170905
ID 1665 gen 530565 top level 5 path @170906
ID 1666 gen 532217 top level 5 path @170907

# btrfs subvolume show /fs/sda7/= | egrep 'UUID|Parent|Top level|Snap|@'

	  UUID:                   cb99579f-64e5-e94c-b22c-41dcc397c37f
	  Parent UUID:            -
	  Received UUID:          -
	  Parent ID:              5
	  Top level ID:           5
	  Snapshot(s):
				  @170825
				  @170826
				  @170828
				  @170829
				  @170830
				  @170901
				  @170902
				  @170903
				  @170904
				  @170905
				  @170906
				  @170907

# btrfs subvolume show /fs/sda7/@170901 | egrep 'UUID|Parent|Top level|Snap|@' /fs/sda7/@170901

	  Name:                   @170901
	  UUID:                   851f8ef3-c2af-4b46-89af-0193fd4e6fc4
	  Parent UUID:            cb99579f-64e5-e94c-b22c-41dcc397c37f
	  Received UUID:          -
	  Parent ID:              5
	  Top level ID:           5

Snapshot(s):

Note that with typical Btrfs consistency "Parent UUID" is that the source of the snapshot, while "Parent ID" is that of the upper level subvolume, and in the "flat" layout for this volume the snapshot parent is '/fs/sda7/=' and the upper level is '/fs/sda7' instead.

The different results that you get make me suspect that the top-level subvolume is "special".

The inode number is 256 on a btrfs filesystem:

if [ stat -f --format=%T $path = btrfs -a stat --format=%i $path = 256 ]; ...

The directory that's result of snapshotting a subvolume, also called EMPTY_SUBVOL has inode number 2, but that's not considered a normal subvolume.

This oneliner shows how much really basic btrfs tools commands syntax is broken by design :(
Looking on how so freakishly overcomplicated btrfs command syntax is that command like above is completely unintuitive and unreadable sometimes I'm really thinking about start rewrite btrfs-progs to make btrfs basic tools syntax as similar as it is only possible to ZFS zfs, zpool and zdb commands on using which in +90% cases you can guess how necessary syntax must look like without looking on man pages.

Any volunteers want to join to help implement something like this? Maybe someone already started doing this?

kloczek

Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

This is nothing to do with btrfs tooling. The two commands involved
here are test (aka "[") and stat.

The main complaint that can be directed at the btrfs command is
that its output is rarely machine-processable. It would therefore make sense to have a "--table" or "--structured" mode for output, which would be more trivially parsable by shell tools.

Hugo.

Hugo Mills             | Ceci est un travail pour l'Australien.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                 Louison, Delicatessen

On 8 September 2017 at 16:38, Hugo Mills <[email protected]> wrote:
[..]

Output of the btrfs command it is coooooompletely different pair of shoes. On making btrfs tools similar to ZFS analogues *obviously* output should be as same similar.
By this would possible to solve complains about unreadable output in one go.

For example zfs command parseable output is possible to generate by add -p switch in those subcommands where it is needed (no --tables or --structures .. just one switch).

Instead reinventing the wheel just please try to look first how it is already done in completely predictable/guessable way:
https://www.freebsd.org/cgi/man.cgi?query=zfs&sektion=8
https://www.freebsd.org/cgi/man.cgi?query=zpool&sektion=8

kloczek

Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

--tables _is_ one switch.

What in what I said was reinventing a wheel? Literally the *only*
thing I was suggesting was adding some option to make the btrfs tool output more machine-parsable.

Call the option whatever you like. However, note that there are
probably very few single-letter options which are not used in at least one of the btrfs tool subcommands.

Hugo.

Hugo Mills             | How do you become King? You stand in the marketplace
hugo@... carfax.org.uk | and announce you're going to tax everyone. If you
http://carfax.org.uk/  | get out alive, you're King.
PGP: E2AB1DE4          |                                        Harry Harrison

My plan is to introduce a global options to set various this, also the output format, eg.

 $ btrfs --format=json subvolume list

that would dump the list in json obviously, more formats could follow. This requires to switch all printf to a wrapper that would select the format based on global config. Some of the code is there, eg. the global option parser and the config structure. The printf transitions can be done incrementally. All of that is easy IMHO, somebody just needs to do it. I work on that when there are no other more pressing things to do.

On 8 September 2017 at 17:39, David Sterba <[email protected]> wrote:
[..]

What in the future json will be abandoned and some new/fancy parseable output will be most desirable?
Hmm ??
Seems some people never saw or already forgot some old jokes.
http://www.webaugur.com/bazaar/53-what-if-operating-systems-were-airlines.html

"Unix Airlines

Each passenger brings a piece of the airplane and a box of tools to the airport. They gather on the tarmac*, arguing constantly about what kind of plane they want to build and how to put it together. Eventually, *they build several different aircraft*, but give them all the same name. Some passengers actually *reach their destination*"

So how to apply above to generate json output out of simple list? Try below command on your linux:

$ awk 'BEGIN {print "{\"data\":["; ORS=""} {if (NR!=1) {print ",\n"};
print "{\"{#DISK}\":\"" $3 "\"}"} END {print "\n]}\n"}'
/proc/diskstats

***This is why no one is asking to transform Linux kernel procfs to provide json output** as standard or an option.
(try to think few seconds on this)

As long as something is able to produce list with comma or spaces separated fields to process such list and provide some usable json output you need only:

$  (echo "foo:foo"; echo "bar:bar") | awk -F: 'BEGIN {print
"{\"data\":["; ORS=""} {if (NR!=1) {print ",\n"}; print
"{\"{#FOO}\":{\""$1"\",\""$2"\"}}"} END {print "\n]}\n"}'
{"data":[
{"{#FOO}":{"foo","foo"}},
{"{#FOO}":{"bar","bar"}}
]}

If you don't like awk you can use jq, sed, perl, python, ruby or whatever you have/like/want.
However on writing fs tools all what you really *must do* is define *as simple as it is only possible >stable< interface*. Only this and nothing more.

Really on writing something so simple like fs tools you don't need to think about future possibility to solve problem of famine on Earth (using your code).

kloczek

Then we'll add a new format of the day to --format=yaml, as far as possible we'll use existing code infrastructure, ie. there will be no changes on the string emmiting side, we'll just change the "backend".

Json was an example because it's available in the util-linux and this is also output style and capabilities I'm most likely going to copy (see eg. lsblk --help). There's plain ascii, list, json outputs, or column selection, sorting. The global option namespace in 'btrfs' is completely empty now, so we can add one letter options.

The tools' output could consumed by humans and scripts, so the idea is to make the defaults human readlabe but provide a simple (as a commandline option) way for scripts to parse just the data.

The libsmartcols that provides the columns and output formatting for util-linux is rich enough to get the same or very similaro output you get by the awk script above. And that's not supposed to be the only way. Say I ask the tool to give me all data it has and parse it in the script.

Stability of the output is a nice goal, hard to achieve in practice. We dont't know the future needs but can't freeze the output forever. The scripting tools you mention are powerful and no fancy formatting we could add to btrfs will replace them. So if there's a terse plain text output, the chances are high that script writer will be able to get the information he wants.

I don't disagree with you and don't see what else is there to argue about.

No, the primary reason that nobody is asking to transform procfs to spit out JSON is that it's a pain to parse (in a shell, you need excessive stuff like the above command, in almost any other language you have to write a special parser or use a separate library) when compared to the simple space and newline delimited stuff already in proc (in Python for example, you just need str.split() and str.splitlines(), both of which are part of the core language), and it's insanely more efficient to just spit out the data as-is without all the extra stuff required by JSON (it's not as bad as XML, but it's still mostly pointless). The only thing that adding JSON support would give people is a way to avoid having to use extra tools or kernel documentation to figure out what everything is, which is not seen as a significant benefit since almost nobody interacts directly with anything in /proc other than developers and people poking at sysctls.

And which command is more readable? Something like the above, or:
btrfs device stats --format=json /

Almost anybody is going to say that the second case (with a simple option) is more readable, and it's also almost always going to be more efficient too since you don't have to fork() an extra time and deal with another pipe. Even using the other tools you mentioned, none of them is going to be as trivially easy for people to understand what's going on as just having an extra option to control the output format.

With the added constraint that your output should be self consistent and sensibly arranged. The current output from various btrfs commands is designed to be human readable, and does not consistently conform to the same formatting. It's also a serious pain to parse without using a language with proper string operations (take a look at the output of btrfs device stats for an example, it uses runs of spaces to separate and align fields, which is ridiculous to try and parse in a shell script), and the lack of a standard output format means that anything interacting with it is inherently fragile.

This is a case though where having something that's more easily machine parseable is a significant benefit, because the command-line is the API unless you're willing to fight with ioctls.

On 8 September 2017 at 20:06, Austin S. Hemmelgarn <[email protected]> wrote:
[..]

How many people you know who are communicating in json or using comma separated lines of text?
Maybe it is not obvious but such interface is not for the humans.

kloczek

Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

Directly between people? Probably none other than CS students and developers. JSON is one of the few reliably unambiguous ways of representing tables of data in a language agnostic manner while still having them be easily machine parseable (and in fact, many programming languages use a similar format for defining tables).

Indirectly between people? Tens of millions at least. At minimum, almost everyone who uses social media is using JSON.

When dealing with administering a computer? No idea, but at least as many people as use LVM and a number of widely used web application frameworks (and web applications built from them). There is quite a lot of software that uses JSON files for configuration because it's easy to parse and also pretty trivial for most people to understand (and allows for certain particularly useful things that the INI format doesn't).

But the code communicating using those formats _is_ for humans. One of the most basic rules of software development is to write code that is easily maintainable, which in turn means easily readable. My comment on readability was not about the output, but about the means of generating that output, which humans (more specifically, programmers) _do_ care about.

Put slightly differently, here's a comparison of getting info about device error counters in Python with and without JSON support in btrfs-progs:

Without JSON support (and without any error handling:

```
import subprocess

raw = subprocess.check_output([
          '/sbin/btrfs',
          'device',
          'stats',
          '/'
      ])
raw = stats.splitlines()
stats = dict()
for i in range(0, len(raw)):
    line = raw[i].split()
    value = int(line[1])
    device = line[0].split('.')[0].lstrip('[').rstrip(']')
    counter = line[0].split('.')[1]
    if not device in stats.keys():
        stats[device] = dict()
    stats[device][counter] = value
```

With JSON support in btrfs-progs (and also without error handling):

```
import subprocess
import json

raw = subprocess.check_output([
          '/sbin/btrfs',
          'device',
          'stats',
          '--format=json',
          '/'
      ])
stats = json.loads(raw)
```

Notice that the second form is exponentially easier to follow (assuming you already know that json.loads() parses a string as JSON, but if you're working with Python you probably already know that), and will probably run faster too.

I could have been more explicit that it's a shell snippet that implements the check, but the original mail contained some commands so I expected that replying in shell is appropriate.

Do you have examples of overcomplicated commands? The command hierarchy is similar to what I know about /bin/zfs, sometimes it's deeper that adds some typing but the shortcuts are allowed so it's IMO not that bad. Also the features do not map 1:1.

A rewrite gives a lot of space to do the interface right, but effectly means end of backward compatibility. In the past new commands have been put anywhere the authors saw fit, now it's the opposite when new commands do not appear.

Starting a new tool could be useful to gather feedback what commands are actually desired, though not everything could be implemented in progs.

This is not true.

root@fex:~# mount | grep /mnt/test
/dev/sdc1 on /mnt/test type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

root@fex:~# btrfs subvol show /mnt/test
/mnt/test is toplevel subvolume

root@fex:~# btrfs subvolume snapshot -r /mnt/test /mnt/test/.snapshot/test
Create a readonly snapshot of '/mnt/test' in '/mnt/test/.snapshot/test'

root@fex:~# stat --format=%i /mnt/test
256

root@fex:~# stat --format=%i /mnt/test/.snapshot/test
256

No difference: both have inode=256

Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: [email protected]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[email protected]>

Oh I see, I've read the mail too quickly sorry, not just a subvolume but really a snapshot. Then the simple inode number check does not work of cours, the answer is more or less what Peter Grandi replied.