Skip to contentSkip to conversation

Announcing the btrfs-list tool

DISCUSSION Stéphane Lesimple 2 participants 3 messages Mar 2, 2019 – Mar 3, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi!

I've been using btrfs as my main NAS filesystem for a bunch of years now, and I always found that managing dozens of subvolumes having dozens of snapshots each was not really straightforward, especially when trying to get an overview of "what's taking space on my filesystem". I was missing the niceness of the "zfs list" command, which does that quite well.

So I've started to write a wrapper around btrfs qgroup show and btrfs fi df (then btrfs fi usage when it got out), to implement this broad overview of my filesystem I was missing so much. Adding some features along the way, I thought that after all it might also be useful to others than me!

As stdout is better than a thousand words, here's for example how it can show you a complex hierarchy of intricated snapshots:

root@nas:~# btrfs-list /mnt/a
NAME                                       TYPE     REFER      USED MOUNTPOINT
4582dacf                                     fs         -     0.00  (1002.88M free)
   [main]                               mainvol    16.00k    16.00k 
/mnt/a
   sub1                                  subvol    16.00k    16.00k
      sub1/.snap1                          snap    16.00k    16.00k
      sub1/.snap2                          snap    16.00k    16.00k
      sub1/subsub1                         snap    16.00k    16.00k
         sub1/subsub1/.snap3               snap    16.00k    16.00k
         sub1/subsub1/subsubsub1           snap    16.00k    16.00k
            sub1/subsub1/subsubsubsub1     snap    16.00k    16.00k

Or how it can help you find big snapshots that you might want to get rid of if you're running out of space:

root@nas:~# btrfs-list --snap-min-used 4G --snap-only /tank
NAME                                                          TYPE     REFER      USED MOUNTPOINT
      backups/.snaps/skyline/20130213_231649_lastskyline    rosnap    
22.52G    19.58G
      backups/.snaps/box/20171231_221207_monthly.12         rosnap    
88.73G     4.96G
      backups/.snaps/box/20180130_221209_monthly.11         rosnap    
91.25G     4.90G
      backups/.snaps/box/20180307_154215_monthly.10         rosnap    
96.28G    10.72G
      backups/.snaps/box/20190120_193004_weekly.3           rosnap    
56.45G     4.25G
      backups/.snaps/nasroot/20180122_091325_monthly.12     rosnap    
34.65G    10.79G
      backups/.snaps/nasroot/20180221_092311_monthly.11     rosnap    
31.96G     4.98G
      backups/.snaps/nasroot/20180323_092734_monthly.10     rosnap    
33.69G     7.05G
      backups/.snaps/nasroot/20180820_205559_monthly.5      rosnap    
31.74G     5.37G
      .syncthing-bkp                                        rosnap    
40.48G     8.15G

Want to give it a try? Here you are:
https://github.com/speed47/btrfs-list

There are a couple snapshots there too, to give you a rough idea of how it looks like (yes, it also has nice colors)

Happy btrfs weekend!

Signature · 1 line
Stéphane.
#1 lore raw

02.03.2019 22:06, Stéphane Lesimple пишет:

Quoted 17 lines

Hi!

I've been using btrfs as my main NAS filesystem for a bunch of years now, and I always found that managing dozens of subvolumes having dozens of snapshots each was not really straightforward, especially when trying to get an overview of "what's taking space on my filesystem". I was missing the niceness of the "zfs list" command, which does that quite well.

So I've started to write a wrapper around btrfs qgroup show and btrfs fi df (then btrfs fi usage when it got out), to implement this broad overview of my filesystem I was missing so much.
Adding some features along the way, I thought that after all it might also be useful to others than me!

As stdout is better than a thousand words, here's for example how it can show you a complex hierarchy of intricated snapshots:

Quoted 15 lines
root@nas:~# btrfs-list /mnt/a
NAME                                       TYPE     REFER      USED
MOUNTPOINT
4582dacf                                     fs         -     0.00 
(1002.88M free)
   [main]                               mainvol    16.00k    16.00k /mnt/a
   sub1                                  subvol    16.00k    16.00k
      sub1/.snap1                          snap    16.00k    16.00k
      sub1/.snap2                          snap    16.00k    16.00k
      sub1/subsub1                         snap    16.00k    16.00k
         sub1/subsub1/.snap3               snap    16.00k    16.00k
         sub1/subsub1/subsubsub1           snap    16.00k    16.00k
            sub1/subsub1/subsubsubsub1     snap    16.00k    16.00k

btrfs does not have anything resembling USED in zfs, so I'm afraid this will be confusing to anyone coming from zfs. In zfs datasets are hierarchical (including snapshots) so USED on every level is exactly the amount of space that will become available in parent dataset if this one is destroyed. This does not matter much without quota (as all datasets share one common pool of free space) but becomes quite visible as soon as you restrict dataset space consumption. In btrfs every subvolume is independent; what you show as USED is exclusive space; deleting subvolume does not mean its exclusively consumed space will be now accounted as available in another subvolume.

Showing hierarchical list with USED column creates false expectations; I'd rather call it EXCL to make its meaning clear.

That said, tool that lists qgroups with volume names is surely welcome.

Quoted 35 lines

Or how it can help you find big snapshots that you might want to get rid of if you're running out of space:

root@nas:~# btrfs-list --snap-min-used 4G --snap-only /tank
NAME                                                          TYPE    
REFER      USED MOUNTPOINT
      backups/.snaps/skyline/20130213_231649_lastskyline    rosnap   
22.52G    19.58G
      backups/.snaps/box/20171231_221207_monthly.12         rosnap   
88.73G     4.96G
      backups/.snaps/box/20180130_221209_monthly.11         rosnap   
91.25G     4.90G
      backups/.snaps/box/20180307_154215_monthly.10         rosnap   
96.28G    10.72G
      backups/.snaps/box/20190120_193004_weekly.3           rosnap   
56.45G     4.25G
      backups/.snaps/nasroot/20180122_091325_monthly.12     rosnap   
34.65G    10.79G
      backups/.snaps/nasroot/20180221_092311_monthly.11     rosnap   
31.96G     4.98G
      backups/.snaps/nasroot/20180323_092734_monthly.10     rosnap   
33.69G     7.05G
      backups/.snaps/nasroot/20180820_205559_monthly.5      rosnap   
31.74G     5.37G
      .syncthing-bkp                                        rosnap   
40.48G     8.15G

Want to give it a try? Here you are:
https://github.com/speed47/btrfs-list

There are a couple snapshots there too, to give you a rough idea of how it looks like (yes, it also has nice colors)

Happy btrfs weekend!

#2 lore raw
Quoted 43 lines · Le 2 mars 2019 21:28:29 Andrei Borzenkov <[email protected]> a écrit :

02.03.2019 22:06, Stéphane Lesimple пишет:

Quoted 17 lines

Hi!

I've been using btrfs as my main NAS filesystem for a bunch of years now, and I always found that managing dozens of subvolumes having dozens of snapshots each was not really straightforward, especially when trying to get an overview of "what's taking space on my filesystem". I was missing the niceness of the "zfs list" command, which does that quite well.

So I've started to write a wrapper around btrfs qgroup show and btrfs fi df (then btrfs fi usage when it got out), to implement this broad overview of my filesystem I was missing so much.
Adding some features along the way, I thought that after all it might also be useful to others than me!

As stdout is better than a thousand words, here's for example how it can show you a complex hierarchy of intricated snapshots:

Quoted 13 lines
root@nas:~# btrfs-list /mnt/a
NAME                                       TYPE     REFER      USED
MOUNTPOINT
4582dacf                                     fs         -     0.00
(1002.88M free)
  [main]                               mainvol    16.00k    16.00k /mnt/a
  sub1                                  subvol    16.00k    16.00k
     sub1/.snap1                          snap    16.00k    16.00k
     sub1/.snap2                          snap    16.00k    16.00k
     sub1/subsub1                         snap    16.00k    16.00k
        sub1/subsub1/.snap3               snap    16.00k    16.00k
        sub1/subsub1/subsubsub1           snap    16.00k    16.00k
           sub1/subsub1/subsubsubsub1     snap    16.00k    16.00k

btrfs does not have anything resembling USED in zfs, so I'm afraid this will be confusing to anyone coming from zfs. In zfs datasets are hierarchical (including snapshots) so USED on every level is exactly the amount of space that will become available in parent dataset if this one is destroyed. This does not matter much without quota (as all datasets share one common pool of free space) but becomes quite visible as soon as you restrict dataset space consumption. In btrfs every subvolume is independent; what you show as USED is exclusive space; deleting subvolume does not mean its exclusively consumed space will be now accounted as available in another subvolume.

Quoted 2 lines

Showing hierarchical list with USED column creates false expectations; I'd rather call it EXCL to make its meaning clear.

You're right. I personally just use the quota feature to track the readonly snapshots that I use as backups, hence I have only the auto-generated 0/ level qgroups on my filesystem.

As USED could be confusing in more complex setups I've replaced this terminology by EXCL as you mentioned, and adjusted the help text accordingly. Thanks!

Quoted 38 lines

That said, tool that lists qgroups with volume names is surely welcome.

Quoted 34 lines

Or how it can help you find big snapshots that you might want to get rid of if you're running out of space:

root@nas:~# btrfs-list --snap-min-used 4G --snap-only /tank
NAME                                                          TYPE
REFER      USED MOUNTPOINT
     backups/.snaps/skyline/20130213_231649_lastskyline    rosnap
22.52G    19.58G
     backups/.snaps/box/20171231_221207_monthly.12         rosnap
88.73G     4.96G
     backups/.snaps/box/20180130_221209_monthly.11         rosnap
91.25G     4.90G
     backups/.snaps/box/20180307_154215_monthly.10         rosnap
96.28G    10.72G
     backups/.snaps/box/20190120_193004_weekly.3           rosnap
56.45G     4.25G
     backups/.snaps/nasroot/20180122_091325_monthly.12     rosnap
34.65G    10.79G
     backups/.snaps/nasroot/20180221_092311_monthly.11     rosnap
31.96G     4.98G
     backups/.snaps/nasroot/20180323_092734_monthly.10     rosnap
33.69G     7.05G
     backups/.snaps/nasroot/20180820_205559_monthly.5      rosnap
31.74G     5.37G
     .syncthing-bkp                                        rosnap
40.48G     8.15G

Want to give it a try? Here you are:
https://github.com/speed47/btrfs-list

There are a couple snapshots there too, to give you a rough idea of how it looks like (yes, it also has nice colors)

Happy btrfs weekend!