Skip to contentSkip to conversation

How to find/reclaim missing space in volume

DISCUSSION Marc MERLIN 6 participants 13 messages Jun 5, 2023 – Jun 7, 2023 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I have this:
sauron [mc]# df -h .

Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2

sauron [mc]# btrfs fi show .
Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
Total devices 1 FS bytes used 847.89GiB
devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
sauron [mc]# btrfs fi df .
Data, single: total=878.00GiB, used=843.85GiB
System, DUP: total=8.00MiB, used=128.00KiB
Metadata, DUP: total=6.00GiB, used=4.04GiB GlobalReserve, single: total=512.00MiB, used=0.00B

sauron:/mnt/btrfs_pool2# du -sh *
599G	varchange2
598G	varchange2_ggm_daily_ro.20230605_07:57:43
4.0K	varchange2_last
599G	varchange2_ro.20230605_08:01:30
599G	varchange2_ro.20230605_09:01:43

I'm confused, the volumes above are snapshots with mostly the same data (made within the last 2 hours) and I didn't delete any data in the FS (they are mostly identical and used for btfrs send/receive)

Why do they add up ot 600GB, but btrfs says 847FB is used?

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#1 lore raw
Quoted 5 lines · On Mon, 5 Jun 2023 09:26:36 -0700 Marc MERLIN <[email protected]> wrote:

I'm confused, the volumes above are snapshots with mostly the same data (made within the last 2 hours) and I didn't delete any data in the FS (they are mostly identical and used for btfrs send/receive)

Why do they add up ot 600GB, but btrfs says 847FB is used?

- deleted snapshots that are still in the process of deletion

- deleted files that are still open by running apps

- files in subvolumes other than the currently mounted one (assuming mounting a non-root subvolume)

- a huge amount of no longer used metadata chunks (does not apply here, considering the fi df output)

- the infamous extent booking peculiarity of Btrfs, assuming you have large files that are partially rewritten in-place (such as VM images or databases)

Signature · 2 lines
With respect,
Roman
#2 lore raw
In reply to Marc MERLIN
Quoted 4 lines · Marc MERLIN

I have this:
sauron [mc]# df -h .

Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2
Quoted 10 lines

sauron [mc]# btrfs fi show .
Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
Total devices 1 FS bytes used 847.89GiB
devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
sauron [mc]# btrfs fi df .
Data, single: total=878.00GiB, used=843.85GiB
System, DUP: total=8.00MiB, used=128.00KiB
Metadata, DUP: total=6.00GiB, used=4.04GiB GlobalReserve, single: total=512.00MiB, used=0.00B

btrfs filesystem usage -T is usually more useful than both the above commands.

Quoted 8 lines
sauron:/mnt/btrfs_pool2# du -sh *
599G	varchange2
598G	varchange2_ggm_daily_ro.20230605_07:57:43
4.0K	varchange2_last
599G	varchange2_ro.20230605_08:01:30
599G	varchange2_ro.20230605_09:01:43
Quoted 6 lines

I'm confused, the volumes above are snapshots with mostly the same data (made within the last 2 hours) and I didn't delete any data in the FS (they are mostly identical and used for btfrs send/receive)

Why do they add up ot 600GB, but btrfs says 847FB is used?

Each subvolume references 600G but it does not mean they are the same 600G. If quota is enabled, "btrfs quota show" may provide some more information, otherwise "btrfs filesystem du" shows shared and exclusive space (you need to pass all subvolumes in question to correctly compute shared vs exclusive).

#3 lore raw
Quoted 16 lines · Andrei Borzenkov
Quoted 4 lines · Marc MERLIN

I have this:
sauron [mc]# df -h .

Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2
Quoted 10 lines

sauron [mc]# btrfs fi show .
Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
Total devices 1 FS bytes used 847.89GiB
devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
sauron [mc]# btrfs fi df .
Data, single: total=878.00GiB, used=843.85GiB
System, DUP: total=8.00MiB, used=128.00KiB
Metadata, DUP: total=6.00GiB, used=4.04GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Quoted 2 lines

btrfs filesystem usage -T is usually more useful than both the above commands.

sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
    Device size:		   1.04TiB
    Device allocated:		 890.02GiB
    Device unallocated:		 177.73GiB
    Device missing:		     0.00B
    Used:			 851.85GiB
    Free (estimated):		 211.93GiB	(min: 123.07GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data Metadata System
Id Path single DUP DUP Unallocated
-- ----------------- --------- -------- --------- -----------
1 /dev/mapper/pool2 878.00GiB 12.00GiB 16.00MiB 177.73GiB
-- ----------------- --------- -------- --------- -----------

   Total             878.00GiB  6.00GiB   8.00MiB   177.73GiB
   Used              843.79GiB  4.03GiB 128.00KiB      
Quoted 11 lines
Quoted 11 lines
sauron:/mnt/btrfs_pool2# du -sh *
599G	varchange2
598G	varchange2_ggm_daily_ro.20230605_07:57:43
4.0K	varchange2_last
599G	varchange2_ro.20230605_08:01:30
599G	varchange2_ro.20230605_09:01:43

I'm confused, the volumes above are snapshots with mostly the same data (made within the last 2 hours) and I didn't delete any data in the FS (they are mostly identical and used for btfrs send/receive)

Quoted 8 lines
Quoted 2 lines

Why do they add up ot 600GB, but btrfs says 847FB is used?

Each subvolume references 600G but it does not mean they are the same 600G. If quota is enabled, "btrfs quota show" may provide some more information, otherwise "btrfs filesystem du" shows shared and exclusive space (you need to pass all subvolumes in question to correctly compute shared vs exclusive).

Right, I did check/know that the snapshots shared the same data, but it doens't hurt to confirm:

sauron:/mnt/btrfs_pool2# btrfs filesystem du -s *
     Total   Exclusive  Set shared  Filename
 597.57GiB    20.00KiB   588.75GiB  varchange2
 597.57GiB     4.00KiB   588.75GiB  varchange2_ggm_daily_ro.20230605_09:59:26
 597.57GiB       0.00B   588.75GiB  varchange2_last
 597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:30:33
 597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_17:35:32
 597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_17:40:32
 597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:45:32
 597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:50:32
 597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:55:32
 597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_18:00:32
 597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_18:05:32
 597.57GiB     8.00KiB   588.75GiB  varchange2_minly.20230605_18:10:32
 597.57GiB    16.00KiB   588.75GiB  varchange2_ro.20230605_10:01:40
 597.57GiB    12.00KiB   588.75GiB  varchange2_ro.20230605_11:01:31
 597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_13:01:28
 597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_14:01:30
 597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_15:01:29
 597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_16:01:32
 597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_17:01:31
 597.57GiB       0.00B   588.75GiB  varchange2_ro.20230605_18:02:02
sauron:/mnt/btrfs_pool2# df -h .
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2
Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#4 lore raw
Quoted 79 lines · Marc MERLIN
Quoted 16 lines · Andrei Borzenkov
Quoted 4 lines · Marc MERLIN

I have this:
sauron [mc]# df -h .

Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2
Quoted 10 lines

sauron [mc]# btrfs fi show .
Label: 'btrfs_pool2' uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a
Total devices 1 FS bytes used 847.89GiB
devid 1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2
sauron [mc]# btrfs fi df .
Data, single: total=878.00GiB, used=843.85GiB
System, DUP: total=8.00MiB, used=128.00KiB
Metadata, DUP: total=6.00GiB, used=4.04GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Quoted 2 lines

btrfs filesystem usage -T is usually more useful than both the above commands.

sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
     Device size:		   1.04TiB
     Device allocated:		 890.02GiB
     Device unallocated:		 177.73GiB
     Device missing:		     0.00B
     Used:			 851.85GiB
     Free (estimated):		 211.93GiB	(min: 123.07GiB)
     Data ratio:			      1.00
     Metadata ratio:		      2.00
     Global reserve:		 512.00MiB	(used: 0.00B)

Data Metadata System
Id Path single DUP DUP Unallocated
-- ----------------- --------- -------- --------- -----------
1 /dev/mapper/pool2 878.00GiB 12.00GiB 16.00MiB 177.73GiB
-- ----------------- --------- -------- --------- -----------

    Total             878.00GiB  6.00GiB   8.00MiB   177.73GiB
    Used              843.79GiB  4.03GiB 128.00KiB
Quoted 11 lines
Quoted 11 lines
sauron:/mnt/btrfs_pool2# du -sh *
599G	varchange2
598G	varchange2_ggm_daily_ro.20230605_07:57:43
4.0K	varchange2_last
599G	varchange2_ro.20230605_08:01:30
599G	varchange2_ro.20230605_09:01:43

I'm confused, the volumes above are snapshots with mostly the same data (made within the last 2 hours) and I didn't delete any data in the FS (they are mostly identical and used for btfrs send/receive)

Quoted 8 lines
Quoted 2 lines

Why do they add up ot 600GB, but btrfs says 847FB is used?

Each subvolume references 600G but it does not mean they are the same 600G. If quota is enabled, "btrfs quota show" may provide some more information, otherwise "btrfs filesystem du" shows shared and exclusive space (you need to pass all subvolumes in question to correctly compute shared vs exclusive).

Right, I did check/know that the snapshots shared the same data, but it doens't hurt to confirm:

sauron:/mnt/btrfs_pool2# btrfs filesystem du -s *
      Total   Exclusive  Set shared  Filename
  597.57GiB    20.00KiB   588.75GiB  varchange2
  597.57GiB     4.00KiB   588.75GiB  varchange2_ggm_daily_ro.20230605_09:59:26
  597.57GiB       0.00B   588.75GiB  varchange2_last
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:30:33
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_17:35:32
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_17:40:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:45:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:50:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:55:32
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_18:00:32
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_18:05:32
  597.57GiB     8.00KiB   588.75GiB  varchange2_minly.20230605_18:10:32
  597.57GiB    16.00KiB   588.75GiB  varchange2_ro.20230605_10:01:40
  597.57GiB    12.00KiB   588.75GiB  varchange2_ro.20230605_11:01:31
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_13:01:28
Quoted 10 lines
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_14:01:30
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_15:01:29
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_16:01:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_17:01:31
  597.57GiB       0.00B   588.75GiB  varchange2_ro.20230605_18:02:02
sauron:/mnt/btrfs_pool2# df -h .
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2

Well, I have had it once, there were deleted but not freed subvolumes

https://lore.kernel.org/linux-btrfs/[email protected]/T/

#5 lore raw
Quoted 7 lines · Andrei Borzenkov
Quoted 3 lines
sauron:/mnt/btrfs_pool2# df -h .
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2

Well, I have had it once, there were deleted but not freed subvolumes

https://lore.kernel.org/linux-btrfs/[email protected]/T/

This sounds like it could be the same, thanks.

I started with

sauron:/mnt/btrfs_pool2# btrfs subvolume sync `pwd`

Unfortunately it's been stuck overnight.

Looks like I may have to reboot first (not ideal either)

Qu, any thing I should capture before reboot? Running 6.2.8 rigt now.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#6 lore raw
Quoted 6 lines · On Tue, 6 Jun 2023 09:41:39 -0700 Marc MERLIN <[email protected]> wrote:

This sounds like it could be the same, thanks.

I started with

sauron:/mnt/btrfs_pool2# btrfs subvolume sync `pwd`

Unfortunately it's been stuck overnight.

So it does not look the same after all.

Do you see a lot of IO to the device (iostat, iotop)? Is it a fast/slow one?

Anything in dmesg?

Signature · 2 lines
With respect,
Roman
#12 lore raw
In reply to Andrei Borzenkov
Quoted 63 lines · Andrei Borzenkov
Quoted 62 lines · Marc MERLIN
Quoted 16 lines · Andrei Borzenkov
Quoted 5 lines · Marc MERLIN

I have this:
sauron [mc]# df -h .
Filesystem         Size  Used Avail Use% Mounted on /dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2 sauron [mc]# btrfs fi show .

Quoted 9 lines

Label: 'btrfs_pool2'  uuid: fde3da31-67e9-4f88-b90d-6c3f6becd56a Total devices 1 FS bytes used 847.89GiB
    devid    1 size 1.04TiB used 890.02GiB path /dev/mapper/pool2 sauron [mc]# btrfs fi df .
Data, single: total=878.00GiB, used=843.85GiB
System, DUP: total=8.00MiB, used=128.00KiB
Metadata, DUP: total=6.00GiB, used=4.04GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Quoted 2 lines

btrfs filesystem usage -T is usually more useful than both the above commands.

sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
     Device size:           1.04TiB
     Device allocated:         890.02GiB
     Device unallocated:         177.73GiB
     Device missing:             0.00B
     Used:             851.85GiB
     Free (estimated):         211.93GiB    (min: 123.07GiB)
     Data ratio:                  1.00
     Metadata ratio:              2.00
     Global reserve:         512.00MiB    (used: 0.00B)

                      Data      Metadata System
Id Path              single    DUP      DUP       Unallocated -- ----------------- --------- -------- --------- ----------- 1 /dev/mapper/pool2 878.00GiB 12.00GiB  16.00MiB   177.73GiB -- ----------------- --------- -------- --------- ----------- Total             878.00GiB  6.00GiB   8.00MiB   177.73GiB Used              843.79GiB  4.03GiB 128.00KiB

Quoted 22 lines
Quoted 7 lines
sauron:/mnt/btrfs_pool2# du -sh *
599G    varchange2
598G    varchange2_ggm_daily_ro.20230605_07:57:43
4.0K    varchange2_last
599G    varchange2_ro.20230605_08:01:30
599G    varchange2_ro.20230605_09:01:43
Quoted 6 lines

I'm confused, the volumes above are snapshots with mostly the same data (made within the last 2 hours) and I didn't delete any data in the FS (they are mostly identical and used for btfrs send/receive)

Why do they add up ot 600GB, but btrfs says 847FB is used?

Each subvolume references 600G but it does not mean they are the same 600G.
If quota is enabled, "btrfs quota show" may provide some more information,
otherwise "btrfs filesystem du" shows shared and exclusive space (you need
to pass all subvolumes in question to correctly compute shared vs exclusive).

Quoted 2 lines
Quoted 2 lines

Right, I did check/know that the snapshots shared the same data, but it doens't hurt to confirm:

But your output doesn't confirm that. The problem is that the concept of "shared" is not useful as it tells you nothing about how many times each file is shared. The files in the last snapshot could all be completely different from the first and you would still see that output.

You need to do pairwise comparisons. A check between the first and the last would tell you if there was much change.

My "extents-lists" scripts (https://github.com/GrahamCobb/extents-lists)
were my hack to do such comparisons - there may be better ways to do it nowadays. If you use those tools you could try:

extents-expr -s /mnt/btrfs_pool2/varchange2_ro.20230605_10:01:40 ^ /mnt/btrfs_pool2/varchange2_ro.20230605_18:02:02

That would show you the amount of data shared between the two specified directories.

Graham

Quoted 34 lines
Quoted 19 lines
sauron:/mnt/btrfs_pool2# btrfs filesystem du -s *
      Total   Exclusive  Set shared  Filename
  597.57GiB    20.00KiB   588.75GiB  varchange2
  597.57GiB     4.00KiB   588.75GiB 
varchange2_ggm_daily_ro.20230605_09:59:26
  597.57GiB       0.00B   588.75GiB  varchange2_last
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:30:33
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_17:35:32
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_17:40:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:45:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:50:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_minly.20230605_17:55:32
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_18:00:32
  597.57GiB       0.00B   588.75GiB  varchange2_minly.20230605_18:05:32
  597.57GiB     8.00KiB   588.75GiB  varchange2_minly.20230605_18:10:32
  597.57GiB    16.00KiB   588.75GiB  varchange2_ro.20230605_10:01:40
  597.57GiB    12.00KiB   588.75GiB  varchange2_ro.20230605_11:01:31
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_13:01:28
Quoted 10 lines
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_14:01:30
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_15:01:29
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_16:01:32
  597.57GiB     4.00KiB   588.75GiB  varchange2_ro.20230605_17:01:31
  597.57GiB       0.00B   588.75GiB  varchange2_ro.20230605_18:02:02
sauron:/mnt/btrfs_pool2# df -h .
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  853G  212G  81% /mnt/btrfs_pool2

Well, I have had it once, there were deleted but not freed subvolumes

https://lore.kernel.org/linux-btrfs/[email protected]/T/

#7 lore raw
In reply to Roman Mamedov
Quoted 8 lines · Roman Mamedov
Quoted 5 lines · On Tue, 6 Jun 2023 09:41:39 -0700 Marc MERLIN <[email protected]> wrote:

This sounds like it could be the same, thanks.

I started with

sauron:/mnt/btrfs_pool2# btrfs subvolume sync `pwd`
Quoted 8 lines
Quoted 1 line

Unfortunately it's been stuck overnight.

So it does not look the same after all.

Do you see a lot of IO to the device (iostat, iotop)? Is it a fast/slow one?

Anything in dmesg?

For the ghost subvolumes, it can be confirmed with "btrfs ins dump-tree -t root", then looking for ROOT_ITEMs with "refs 0".

I'm not sure if it is the case.

Another possible cause is extent bookends, this needs something like btrfs quota to confirm, and not much we can do if there are snapshots. (If no snapshots, it's possible to defrag and free up such bookend extents).

Thanks,
Qu

#8 lore raw
Quoted 8 lines · Qu Wenruo

For the ghost subvolumes, it can be confirmed with "btrfs ins dump-tree -t root", then looking for ROOT_ITEMs with "refs 0".

I'm not sure if it is the case.

Another possible cause is extent bookends, this needs something like btrfs quota to confirm, and not much we can do if there are snapshots. (If no snapshots, it's possible to defrag and free up such bookend extents).

unfortunately the system rebooted overnight, so I lost the state where sync was hung.
On the plus side, this seems to have fixed the issue:

sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
    Device size:		   1.04TiB
    Device allocated:		 857.02GiB
    Device unallocated:		 210.73GiB
    Device missing:		     0.00B
    Used:			 589.30GiB
    Free (estimated):		 470.78GiB	(min: 365.42GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data Metadata System
Id Path single DUP DUP Unallocated
-- ----------------- --------- -------- --------- -----------
1 /dev/mapper/pool2 845.00GiB 12.00GiB 16.00MiB 210.73GiB
-- ----------------- --------- -------- --------- -----------

   Total             845.00GiB  6.00GiB   8.00MiB   210.73GiB
   Used              584.95GiB  2.18GiB 112.00KiB            
sauron:/mnt/btrfs_pool2# df -h .
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  590G  471G  56% /mnt/btrfs_pool2

I know it's hard to say after the fact, but have you seen issues where snapshot delete space cleanup would get stuck and
btrfs subvolume sync pwd
would hang forever (until reboot) ?

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#9 lore raw

Hi,

Quoted 6 lines · Marc MERLIN
Quoted 5 lines · Qu Wenruo

For the ghost subvolumes, it can be confirmed with "btrfs ins dump-tree -t root", then looking for ROOT_ITEMs with "refs 0".

I'm not sure if it is the case.

Quoted 7 lines
Quoted 3 lines

Another possible cause is extent bookends, this needs something like btrfs quota to confirm, and not much we can do if there are snapshots. (If no snapshots, it's possible to defrag and free up such bookend extents).

unfortunately the system rebooted overnight, so I lost the state where sync was hung.
On the plus side, this seems to have fixed the issue:

Just a random hint... One possible situation in which a deleted subvolume can't be freed up for real yet, is when there is a process that still has an open file in it.

Quoted 23 lines
sauron:/mnt/btrfs_pool2# btrfs fi usage -T .
Overall:
    Device size:		   1.04TiB
    Device allocated:		 857.02GiB
    Device unallocated:		 210.73GiB
    Device missing:		     0.00B
    Used:			 589.30GiB
    Free (estimated):		 470.78GiB	(min: 365.42GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data Metadata System
Id Path single DUP DUP Unallocated
-- ----------------- --------- -------- --------- -----------
1 /dev/mapper/pool2 845.00GiB 12.00GiB 16.00MiB 210.73GiB
-- ----------------- --------- -------- --------- -----------

   Total             845.00GiB  6.00GiB   8.00MiB   210.73GiB
   Used              584.95GiB  2.18GiB 112.00KiB            
sauron:/mnt/btrfs_pool2# df -h .
Filesystem         Size  Used Avail Use% Mounted on
/dev/mapper/pool2  1.1T  590G  471G  56% /mnt/btrfs_pool2
Quoted 4 lines

I know it's hard to say after the fact, but have you seen issues where snapshot delete space cleanup would get stuck and
btrfs subvolume sync pwd
would hang forever (until reboot) ?

Hans

#10 lore raw
Quoted 5 lines · Hans van Kranenburg
Quoted 1 line

On the plus side, this seems to have fixed the issue:

Just a random hint... One possible situation in which a deleted subvolume can't be freed up for real yet, is when there is a process that still has an open file in it.

this is a fair guess.
Too late now, but is this something that would show up in lsof -n | grep volume ?

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#11 lore raw
Quoted 10 lines · Marc MERLIN
Quoted 5 lines · Hans van Kranenburg
Quoted 1 line

On the plus side, this seems to have fixed the issue:

Just a random hint... One possible situation in which a deleted subvolume can't be freed up for real yet, is when there is a process that still has an open file in it.

this is a fair guess.
Too late now, but is this something that would show up in lsof -n | grep volume ?

Yes, it should.

I just quickly tested this, by doing something like...

btrfs sub create foo
cat >> foo/bar
.. and then while the file is still open ..
btrfs sub del foo/

-$ lsof -n |grep foo
cat 577355 knorrie 1w REG 0,0 32 257 /home/knorrie/p/foo/bar

So it doesn't show as DEL, but still as REG, even.

Hans