support request: btrfs df reports drive is out of space, cannot find what occupies it
Dear btrfs experts,
Could you please help me sort out the following situation:
btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
root@next:/home/support# btrfs fi df / Data, single: total=82.00GiB, used=78.23GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.70MiB GlobalReserve, single: total=68.45MiB, used=0.00B root@next:/home/support# df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda3 96G 79G 16G 84% /
However when I try to locate files to delete with du that's what I get:
|root||@next||:/home/support# du -hd1 /| |70M /boot| |0| |/dev| |2||.2G /.snapshots| |14M /bin| |4||.5M /etc| |2||.5M /home| |348M /lib| |4||.0K /lib64| |0| |/media| |0| |/mnt| |0| |/opt| |0| |/proc| |40K /root| |2||.7M /run| |12M /sbin| |0| |/srv| |0| |/sys| |0| |/tmp| |566M /usr| |5||.0G /var| |29G /next| |38G /| | | |I.e. almost 40Gb just gone somewhere. Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.| | | |+++++++++++++++++++++++++++++++++++++| |root@next:~||# uname -a| |Linux next 5.10.0-28-amd64 ||#1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux| |root@next:~||# btrfs --version| |btrfs-progs v5.10.1| |root@next:~||# btrfs fi show| |Label: ||'NEXT_ROOTFS'| |uuid: abc71bdb-c570-461d-a28a-54294a646089| |||Total devices 1 FS bytes used 78.37GiB| |||devid 1 size 95.46GiB used 84.06GiB path ||/dev/sda3| |root@next:~||# btrfs fi df /| |Data, single: total=82.00GiB, used=78.22GiB| |System, DUP: total=32.00MiB, used=16.00KiB| |Metadata, DUP: total=1.00GiB, used=153.64MiB| |GlobalReserve, single: total=68.45MiB, used=0.00B| | |
Quoted 42 lines · Skirnir Torvaldsson
Dear btrfs experts,
Could you please help me sort out the following situation:
btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
root@next:/home/support# btrfs fi df / Data, single: total=82.00GiB, used=78.23GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.70MiB GlobalReserve, single: total=68.45MiB, used=0.00B root@next:/home/support# df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda3 96G 79G 16G 84% /However when I try to locate files to delete with du that's what I get:
root@next:/home/support# du -hd1 / 70M /boot 0 /dev 2.2G /.snapshots 14M /bin 4.5M /etc 2.5M /home 348M /lib 4.0K /lib64 0 /media 0 /mnt 0 /opt 0 /proc 40K /root 2.7M /run 12M /sbin 0 /srv 0 /sys 0 /tmp 566M /usr 5.0G /var 29G /next 38G /
Quoted 1 line
I.e. almost 40Gb just gone somewhere.
Huh?
2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?
If you have some other mount points, you could start with explaining your storage layout first.
Quoted 18 lines
Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
+++++++++++++++++++++++++++++++++++++
root@next:~# uname -a Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux root@next:~# btrfs --version btrfs-progs v5.10.1 root@next:~# btrfs fi show Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089 Total devices 1 FS bytes used 78.37GiB devid 1 size 95.46GiB used 84.06GiB path /dev/sda3root@next:~# btrfs fi df / Data, single: total=82.00GiB, used=78.22GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.64MiB GlobalReserve, single: total=68.45MiB, used=0.00B
With all due respect, 38G is a grand total, so it is 38G out of 78G reported data. There is one thing I've noticed that troubles me:
root@next:/home/support/btrfs-list-2.3# ./btrfs-list NAME TYPE REFER EXCL MOUNTPOINT NEXT_ROOTFS fs - 78.37G (single/dup, 15.03G/95.46G free, 15.74%) [main] mainvol 16.00k 16.00k / @System subvol 16.00k 16.00k /.snapshots @Logs subvol 16.00k 16.00k /next/logs/.snapshots @Logs/current subvol 4.70G 4.70G /next/logs @AppData subvol 16.00k 16.00k /next/appdata/.snapshots @AppData/current subvol 370.14M 370.14M /next/appdata @AppData/var subvol 16.00k 16.00k @Databases subvol 16.00k 16.00k /next/databases/.snapshots @Databases/current subvol 754.95M 754.95M /next/databases @MessageBus subvol 16.00k 16.00k /next/mbus/.snapshots @MessageBus/current subvol 67.70G 67.70G /next/mbus @Updates subvol 16.00k 16.00k /next/updates/.snapshots @Updates/current subvol 1.81G 1.81G /next/updates @SystemData subvol 16.00k 16.00k /next/systemdata/.snapshots @SystemData/current subvol 1.21G 1.21G /next/systemdata @System/prev subvol 1.48G 1.48G @System/current subvol 443.27M 443.27M / root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus 0 /next/mbus/.snapshots 1.4G /next/mbus/redpanda 1.4G /next/mbus
So, the MessageBus subvolume is occupying 67Gb (?), however I fail to understand how come this space is not accounted for by du and how I can clean it and limit it in future.
Quoted 70 lines
Quoted 41 lines · Skirnir Torvaldsson
Dear btrfs experts,
Could you please help me sort out the following situation:
btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
root@next:/home/support# btrfs fi df / Data, single: total=82.00GiB, used=78.23GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.70MiB GlobalReserve, single: total=68.45MiB, used=0.00B root@next:/home/support# df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda3 96G 79G 16G 84% /However when I try to locate files to delete with du that's what I get:
root@next:/home/support# du -hd1 / 70M /boot 0 /dev 2.2G /.snapshots 14M /bin 4.5M /etc 2.5M /home 348M /lib 4.0K /lib64 0 /media 0 /mnt 0 /opt 0 /proc 40K /root 2.7M /run 12M /sbin 0 /srv 0 /sys 0 /tmp 566M /usr 5.0G /var 29G /next 38G /Quoted 1 line
I.e. almost 40Gb just gone somewhere.
Huh?
2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?
If you have some other mount points, you could start with explaining your storage layout first.
Quoted 18 lines
Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
+++++++++++++++++++++++++++++++++++++
root@next:~# uname -a Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux root@next:~# btrfs --version btrfs-progs v5.10.1 root@next:~# btrfs fi show Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089 Total devices 1 FS bytes used 78.37GiB devid 1 size 95.46GiB used 84.06GiB path /dev/sda3root@next:~# btrfs fi df / Data, single: total=82.00GiB, used=78.22GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.64MiB GlobalReserve, single: total=68.45MiB, used=0.00B
Quoted 32 lines · Skirnir Torvaldsson
With all due respect, 38G is a grand total, so it is 38G out of 78G reported data.
There is one thing I've noticed that troubles me:root@next:/home/support/btrfs-list-2.3# ./btrfs-list NAME TYPE REFER EXCL MOUNTPOINT NEXT_ROOTFS fs - 78.37G (single/dup, 15.03G/95.46G free, 15.74%) [main] mainvol 16.00k 16.00k / @System subvol 16.00k 16.00k /.snapshots @Logs subvol 16.00k 16.00k /next/logs/.snapshots @Logs/current subvol 4.70G 4.70G /next/logs @AppData subvol 16.00k 16.00k /next/appdata/.snapshots @AppData/current subvol 370.14M 370.14M /next/appdata @AppData/var subvol 16.00k 16.00k @Databases subvol 16.00k 16.00k /next/databases/.snapshots @Databases/current subvol 754.95M 754.95M /next/databases @MessageBus subvol 16.00k 16.00k /next/mbus/.snapshots @MessageBus/current subvol 67.70G 67.70G /next/mbus @Updates subvol 16.00k 16.00k /next/updates/.snapshots @Updates/current subvol 1.81G 1.81G /next/updates @SystemData subvol 16.00k 16.00k /next/systemdata/.snapshots @SystemData/current subvol 1.21G 1.21G /next/systemdata @System/prev subvol 1.48G 1.48G @System/current subvol 443.27M 443.27M / root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus 0 /next/mbus/.snapshots 1.4G /next/mbus/redpanda 1.4G /next/mbus
Quoted 4 lines
So, the MessageBus subvolume is occupying 67Gb (?), however I fail to understand how come this space is not accounted for by du and how I can clean it and limit it in future.
Could be variation of
https://lore.kernel.org/linux-btrfs/[email protected]/
Quoted 71 lines
Quoted 70 lines
Quoted 41 lines · Skirnir Torvaldsson
Dear btrfs experts,
Could you please help me sort out the following situation:
btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
root@next:/home/support# btrfs fi df / Data, single: total=82.00GiB, used=78.23GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.70MiB GlobalReserve, single: total=68.45MiB, used=0.00B root@next:/home/support# df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda3 96G 79G 16G 84% /However when I try to locate files to delete with du that's what I get:
root@next:/home/support# du -hd1 / 70M /boot 0 /dev 2.2G /.snapshots 14M /bin 4.5M /etc 2.5M /home 348M /lib 4.0K /lib64 0 /media 0 /mnt 0 /opt 0 /proc 40K /root 2.7M /run 12M /sbin 0 /srv 0 /sys 0 /tmp 566M /usr 5.0G /var 29G /next 38G /Quoted 1 line
I.e. almost 40Gb just gone somewhere.
Huh?
2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?
If you have some other mount points, you could start with explaining your storage layout first.
Quoted 18 lines
Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
+++++++++++++++++++++++++++++++++++++
root@next:~# uname -a Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux root@next:~# btrfs --version btrfs-progs v5.10.1 root@next:~# btrfs fi show Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089 Total devices 1 FS bytes used 78.37GiB devid 1 size 95.46GiB used 84.06GiB path /dev/sda3root@next:~# btrfs fi df / Data, single: total=82.00GiB, used=78.22GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.64MiB GlobalReserve, single: total=68.45MiB, used=0.00B
Quoted 33 lines
Quoted 31 lines · Skirnir Torvaldsson
With all due respect, 38G is a grand total, so it is 38G out of 78G reported data.
There is one thing I've noticed that troubles me:root@next:/home/support/btrfs-list-2.3# ./btrfs-list NAME TYPE REFER EXCL MOUNTPOINT NEXT_ROOTFS fs - 78.37G (single/dup, 15.03G/95.46G free, 15.74%) [main] mainvol 16.00k 16.00k / @System subvol 16.00k 16.00k /.snapshots @Logs subvol 16.00k 16.00k /next/logs/.snapshots @Logs/current subvol 4.70G 4.70G /next/logs @AppData subvol 16.00k 16.00k /next/appdata/.snapshots @AppData/current subvol 370.14M 370.14M /next/appdata @AppData/var subvol 16.00k 16.00k @Databases subvol 16.00k 16.00k /next/databases/.snapshots @Databases/current subvol 754.95M 754.95M /next/databases @MessageBus subvol 16.00k 16.00k /next/mbus/.snapshots @MessageBus/current subvol 67.70G 67.70G /next/mbus @Updates subvol 16.00k 16.00k /next/updates/.snapshots @Updates/current subvol 1.81G 1.81G /next/updates @SystemData subvol 16.00k 16.00k /next/systemdata/.snapshots @SystemData/current subvol 1.21G 1.21G /next/systemdata @System/prev subvol 1.48G 1.48G @System/current subvol 443.27M 443.27M / root@next:/home/support/btrfs-list-2.3# du -hd1 /next/mbus 0 /next/mbus/.snapshots 1.4G /next/mbus/redpanda 1.4G /next/mbus
Quoted 7 lines
Quoted 4 lines
So, the MessageBus subvolume is occupying 67Gb (?), however I fail to understand how come this space is not accounted for by du and how I can clean it and limit it in future.
Could be variation of
Thank you so much, this seems to be my case:
root@next:/next/mbus/redpanda/data# compsize . Processed 3490 files, 3489 regular extents (3489 refs), 1 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 99% 67G 67G 209M none 100% 67G 67G 177M zstd 8% 4.0K 48K 48K prealloc 100% 32M 32M 31M
And these 67Gb of data are reported as "unreachable" by btdu. However, manual defragmentation has no effect. Is there anything else I could try short of deleting these files completely?
Quoted 72 lines
Quoted 72 lines
Quoted 72 lines
Quoted 41 lines · Skirnir Torvaldsson
Dear btrfs experts,
Could you please help me sort out the following situation:
btrfs df reports my 100Gb device is almost out of space (which agrees with the results produced by the standard "df"):
root@next:/home/support# btrfs fi df / Data, single: total=82.00GiB, used=78.23GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.70MiB GlobalReserve, single: total=68.45MiB, used=0.00B root@next:/home/support# df -h / Filesystem Size Used Avail Use% Mounted on /dev/sda3 96G 79G 16G 84% /However when I try to locate files to delete with du that's what I get:
root@next:/home/support# du -hd1 / 70M /boot 0 /dev 2.2G /.snapshots 14M /bin 4.5M /etc 2.5M /home 348M /lib 4.0K /lib64 0 /media 0 /mnt 0 /opt 0 /proc 40K /root 2.7M /run 12M /sbin 0 /srv 0 /sys 0 /tmp 566M /usr 5.0G /var 29G /next 38G /Quoted 1 line
I.e. almost 40Gb just gone somewhere.
Huh?
2.2G + 5.0G + 29G + 38G == 75.2G out of 78G reported for DATA. What 40G are you talking about?
If you have some other mount points, you could start with explaining your storage layout first.
Quoted 18 lines
Am I doing something wrong? Is there a problem or a piece of theory I'm missing? Kindly advice.
+++++++++++++++++++++++++++++++++++++
root@next:~# uname -a Linux next 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux root@next:~# btrfs --version btrfs-progs v5.10.1 root@next:~# btrfs fi show Label: 'NEXT_ROOTFS' uuid: abc71bdb-c570-461d-a28a-54294a646089 Total devices 1 FS bytes used 78.37GiB devid 1 size 95.46GiB used 84.06GiB path /dev/sda3root@next:~# btrfs fi df / Data, single: total=82.00GiB, used=78.22GiB System, DUP: total=32.00MiB, used=16.00KiB Metadata, DUP: total=1.00GiB, used=153.64MiB GlobalReserve, single: total=68.45MiB, used=0.00B