Skip to contentSkip to conversation

support request: btrfs df reports drive is out of space, cannot find what occupies it

DISCUSSION Skirnir Torvaldsson 2 participants 5 messages Apr 19, 2024 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Dear btrfs experts,

Could you please help me sort out the following situation:

btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):

root@next:/home/support# btrfs fi df /
Data, single: total=82.00GiB, used=78.23GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.70MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
root@next:/home/support# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        96G   79G   16G  84% /

However  when I try to locate files to delete with du that's what I get:

|root||@next||:/home/support# du -hd1 /|
|70M     /boot|
|0| |/dev|
|2||.2G /.snapshots|
|14M     /bin|
|4||.5M /etc|
|2||.5M /home|
|348M    /lib|
|4||.0K /lib64|
|0| |/media|
|0| |/mnt|
|0| |/opt|
|0| |/proc|
|40K     /root|
|2||.7M /run|
|12M     /sbin|
|0| |/srv|
|0| |/sys|
|0| |/tmp|
|566M    /usr|
|5||.0G /var|
|29G     /next|
|38G     /|
|
|
|I.e. almost 40Gb just gone somewhere. Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.|
|
|
|+++++++++++++++++++++++++++++++++++++|
|root@next:~||#  uname -a|
|Linux next 5.10.0-28-amd64 ||#1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux|
|root@next:~||#  btrfs --version|
|btrfs-progs v5.10.1|
|root@next:~||#  btrfs fi show|
|Label: ||'NEXT_ROOTFS'| |uuid: abc71bdb-c570-461d-a28a-54294a646089|
|||Total devices 1 FS bytes used 78.37GiB|
|||devid    1 size 95.46GiB used 84.06GiB path ||/dev/sda3|
|root@next:~||#  btrfs fi df /|
|Data, single: total=82.00GiB, used=78.22GiB|
|System, DUP: total=32.00MiB, used=16.00KiB|
|Metadata, DUP: total=1.00GiB, used=153.64MiB|
|GlobalReserve, single: total=68.45MiB, used=0.00B|
|
|
#1 lore raw
Quoted 42 lines · Skirnir Torvaldsson

Dear btrfs experts,

Could you please help me sort out the following situation:

btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):

root@next:/home/support# btrfs fi df /
Data, single: total=82.00GiB, used=78.23GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.70MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
root@next:/home/support# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        96G   79G   16G  84% /

However when I try to locate files to delete with du that's what I get:

root@next:/home/support# du -hd1 /
70M     /boot
0       /dev
2.2G    /.snapshots
14M     /bin
4.5M    /etc
2.5M    /home
348M    /lib
4.0K    /lib64
0       /media
0       /mnt
0       /opt
0       /proc
40K     /root
2.7M    /run
12M     /sbin
0       /srv
0       /sys
0       /tmp
566M    /usr
5.0G    /var
29G     /next
38G     /
Quoted 1 line

I.e. almost 40Gb just gone somewhere.

Huh?

2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?

If you have some other mount points, you could start with explaining your storage layout first.

Quoted 18 lines

Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.

+++++++++++++++++++++++++++++++++++++

root@next:~#  uname -a
Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
root@next:~#  btrfs --version
btrfs-progs v5.10.1
root@next:~#  btrfs fi show
Label: 'NEXT_ROOTFS'  uuid: abc71bdb-c570-461d-a28a-54294a646089
        Total devices 1 FS bytes used 78.37GiB
        devid    1 size 95.46GiB used 84.06GiB path /dev/sda3
root@next:~#  btrfs fi df /
Data, single: total=82.00GiB, used=78.22GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.64MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
#2 lore raw

With all due respect, 38G is a grand total, so it is 38G out of 78G reported data. There is one thing I've noticed that troubles me:

root@next:/home/support/btrfs-list-2.3# ./btrfs-list
NAME                      TYPE    REFER     EXCL  MOUNTPOINT
NEXT_ROOTFS                 fs       -    78.37G (single/dup, 15.03G/95.46G free, 15.74%)
   [main]              mainvol   16.00k   16.00k /
   @System              subvol   16.00k   16.00k /.snapshots
   @Logs                subvol   16.00k   16.00k /next/logs/.snapshots
   @Logs/current        subvol    4.70G    4.70G /next/logs
   @AppData             subvol   16.00k   16.00k /next/appdata/.snapshots
   @AppData/current     subvol  370.14M  370.14M /next/appdata
   @AppData/var         subvol   16.00k   16.00k
   @Databases           subvol   16.00k   16.00k /next/databases/.snapshots
   @Databases/current   subvol  754.95M  754.95M /next/databases
   @MessageBus          subvol   16.00k   16.00k /next/mbus/.snapshots
   @MessageBus/current  subvol   67.70G   67.70G /next/mbus
   @Updates             subvol   16.00k   16.00k /next/updates/.snapshots
   @Updates/current     subvol    1.81G    1.81G /next/updates
   @SystemData          subvol   16.00k   16.00k /next/systemdata/.snapshots
   @SystemData/current  subvol    1.21G    1.21G /next/systemdata
   @System/prev         subvol    1.48G    1.48G
   @System/current      subvol  443.27M  443.27M /
root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus
0       /next/mbus/.snapshots
1.4G    /next/mbus/redpanda
1.4G    /next/mbus

So, the MessageBus subvolume is occupying 67Gb (?), however I fail to understand how come this space is not accounted for by du and how I can clean it and limit it in future.

Quoted 70 lines
Quoted 41 lines · Skirnir Torvaldsson

Dear btrfs experts,

Could you please help me sort out the following situation:

btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):

root@next:/home/support# btrfs fi df /
Data, single: total=82.00GiB, used=78.23GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.70MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
root@next:/home/support# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        96G   79G   16G  84% /

However when I try to locate files to delete with du that's what I get:

root@next:/home/support# du -hd1 /
70M     /boot
0       /dev
2.2G    /.snapshots
14M     /bin
4.5M    /etc
2.5M    /home
348M    /lib
4.0K    /lib64
0       /media
0       /mnt
0       /opt
0       /proc
40K     /root
2.7M    /run
12M     /sbin
0       /srv
0       /sys
0       /tmp
566M    /usr
5.0G    /var
29G     /next
38G     /
Quoted 1 line

I.e. almost 40Gb just gone somewhere.

Huh?

2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?

If you have some other mount points, you could start with explaining your storage layout first.

Quoted 18 lines

Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.

+++++++++++++++++++++++++++++++++++++

root@next:~#  uname -a
Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
root@next:~#  btrfs --version
btrfs-progs v5.10.1
root@next:~#  btrfs fi show
Label: 'NEXT_ROOTFS'  uuid: abc71bdb-c570-461d-a28a-54294a646089
         Total devices 1 FS bytes used 78.37GiB
         devid    1 size 95.46GiB used 84.06GiB path /dev/sda3
root@next:~#  btrfs fi df /
Data, single: total=82.00GiB, used=78.22GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.64MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
#3 lore raw
Quoted 32 lines · Skirnir Torvaldsson

With all due respect, 38G is a grand total, so it is 38G out of 78G reported data.
There is one thing I've noticed that troubles me:

root@next:/home/support/btrfs-list-2.3# ./btrfs-list
NAME                      TYPE    REFER     EXCL  MOUNTPOINT
NEXT_ROOTFS                 fs       -    78.37G (single/dup,
15.03G/95.46G free, 15.74%)
    [main]              mainvol   16.00k   16.00k /
    @System              subvol   16.00k   16.00k /.snapshots
    @Logs                subvol   16.00k   16.00k /next/logs/.snapshots
    @Logs/current        subvol    4.70G    4.70G /next/logs
    @AppData             subvol   16.00k   16.00k /next/appdata/.snapshots
    @AppData/current     subvol  370.14M  370.14M /next/appdata
    @AppData/var         subvol   16.00k   16.00k
    @Databases           subvol   16.00k   16.00k /next/databases/.snapshots
    @Databases/current   subvol  754.95M  754.95M /next/databases
    @MessageBus          subvol   16.00k   16.00k /next/mbus/.snapshots
    @MessageBus/current  subvol   67.70G   67.70G /next/mbus
    @Updates             subvol   16.00k   16.00k /next/updates/.snapshots
    @Updates/current     subvol    1.81G    1.81G /next/updates
    @SystemData          subvol   16.00k   16.00k
/next/systemdata/.snapshots
    @SystemData/current  subvol    1.21G    1.21G /next/systemdata
    @System/prev         subvol    1.48G    1.48G
    @System/current      subvol  443.27M  443.27M /
root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus
0       /next/mbus/.snapshots
1.4G    /next/mbus/redpanda
1.4G    /next/mbus
Quoted 4 lines

So, the MessageBus subvolume is occupying 67Gb (?), however I fail to understand how come this space is not accounted for by du and how I can clean it and limit it in future.

Could be variation of

https://lore.kernel.org/linux-btrfs/[email protected]/

Quoted 71 lines
Quoted 70 lines
Quoted 41 lines · Skirnir Torvaldsson

Dear btrfs experts,

Could you please help me sort out the following situation:

btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):

root@next:/home/support# btrfs fi df /
Data, single: total=82.00GiB, used=78.23GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.70MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
root@next:/home/support# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        96G   79G   16G  84% /

However when I try to locate files to delete with du that's what I get:

root@next:/home/support# du -hd1 /
70M     /boot
0       /dev
2.2G    /.snapshots
14M     /bin
4.5M    /etc
2.5M    /home
348M    /lib
4.0K    /lib64
0       /media
0       /mnt
0       /opt
0       /proc
40K     /root
2.7M    /run
12M     /sbin
0       /srv
0       /sys
0       /tmp
566M    /usr
5.0G    /var
29G     /next
38G     /
Quoted 1 line

I.e. almost 40Gb just gone somewhere.

Huh?

2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?

If you have some other mount points, you could start with explaining your storage layout first.

Quoted 18 lines

Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.

+++++++++++++++++++++++++++++++++++++

root@next:~#  uname -a
Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
root@next:~#  btrfs --version
btrfs-progs v5.10.1
root@next:~#  btrfs fi show
Label: 'NEXT_ROOTFS'  uuid: abc71bdb-c570-461d-a28a-54294a646089
         Total devices 1 FS bytes used 78.37GiB
         devid    1 size 95.46GiB used 84.06GiB path /dev/sda3
root@next:~#  btrfs fi df /
Data, single: total=82.00GiB, used=78.22GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.64MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
#4 lore raw
Quoted 33 lines
Quoted 31 lines · Skirnir Torvaldsson

With all due respect, 38G is a grand total, so it is 38G out of 78G reported data.
There is one thing I've noticed that troubles me:

root@next:/home/support/btrfs-list-2.3# ./btrfs-list
NAME                      TYPE    REFER     EXCL  MOUNTPOINT
NEXT_ROOTFS                 fs       -    78.37G (single/dup,
15.03G/95.46G free, 15.74%)
     [main]              mainvol   16.00k   16.00k /
     @System              subvol   16.00k   16.00k /.snapshots
     @Logs                subvol   16.00k   16.00k /next/logs/.snapshots
     @Logs/current        subvol    4.70G    4.70G /next/logs
     @AppData             subvol   16.00k   16.00k /next/appdata/.snapshots
     @AppData/current     subvol  370.14M  370.14M /next/appdata
     @AppData/var         subvol   16.00k   16.00k
     @Databases           subvol   16.00k   16.00k /next/databases/.snapshots
     @Databases/current   subvol  754.95M  754.95M /next/databases
     @MessageBus          subvol   16.00k   16.00k /next/mbus/.snapshots
     @MessageBus/current  subvol   67.70G   67.70G /next/mbus
     @Updates             subvol   16.00k   16.00k /next/updates/.snapshots
     @Updates/current     subvol    1.81G    1.81G /next/updates
     @SystemData          subvol   16.00k   16.00k
/next/systemdata/.snapshots
     @SystemData/current  subvol    1.21G    1.21G /next/systemdata
     @System/prev         subvol    1.48G    1.48G
     @System/current      subvol  443.27M  443.27M /
root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus
0       /next/mbus/.snapshots
1.4G    /next/mbus/redpanda
1.4G    /next/mbus
Quoted 7 lines
Quoted 4 lines

So, the MessageBus subvolume is occupying 67Gb (?), however I fail to understand how come this space is not accounted for by du and how I can clean it and limit it in future.

Could be variation of

https://lore.kernel.org/linux-btrfs/[email protected]/

Thank you so much, this seems to be my case:

root@next:/next/mbus/redpanda/data# compsize .
Processed 3490 files, 3489 regular extents (3489 refs), 1 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       99%       67G          67G         209M
none       100%       67G          67G         177M
zstd         8%           4.0K          48K          48K
prealloc   100%       32M          32M          31M

And these 67Gb of data are reported as "unreachable" by btdu. However, manual defragmentation has no effect.  Is there anything else I could try short of deleting these files completely?

Quoted 72 lines
Quoted 72 lines
Quoted 72 lines
Quoted 41 lines · Skirnir Torvaldsson

Dear btrfs experts,

Could you please help me sort out the following situation:

btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):

root@next:/home/support# btrfs fi df /
Data, single: total=82.00GiB, used=78.23GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.70MiB
GlobalReserve, single: total=68.45MiB, used=0.00B
root@next:/home/support# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        96G   79G   16G  84% /

However when I try to locate files to delete with du that's what I get:

root@next:/home/support# du -hd1 /
70M     /boot
0       /dev
2.2G    /.snapshots
14M     /bin
4.5M    /etc
2.5M    /home
348M    /lib
4.0K    /lib64
0       /media
0       /mnt
0       /opt
0       /proc
40K     /root
2.7M    /run
12M     /sbin
0       /srv
0       /sys
0       /tmp
566M    /usr
5.0G    /var
29G     /next
38G     /
Quoted 1 line

I.e. almost 40Gb just gone somewhere.

Huh?

2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?

If you have some other mount points, you could start with explaining your storage layout first.

Quoted 18 lines

Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.

+++++++++++++++++++++++++++++++++++++

root@next:~#  uname -a
Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
root@next:~#  btrfs --version
btrfs-progs v5.10.1
root@next:~#  btrfs fi show
Label: 'NEXT_ROOTFS'  uuid: abc71bdb-c570-461d-a28a-54294a646089
          Total devices 1 FS bytes used 78.37GiB
          devid    1 size 95.46GiB used 84.06GiB path /dev/sda3
root@next:~#  btrfs fi df /
Data, single: total=82.00GiB, used=78.22GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=153.64MiB
GlobalReserve, single: total=68.45MiB, used=0.00B