Skip to contentSkip to conversation

100% disk usage reported by "df", 60% disk usage reported by "btrfs fi usage" - this breaks userspace behaviour

DISCUSSION Tomasz Chmielewski 2 participants 5 messages Dec 6, 2019 – Dec 7, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I know that "df -h" is not to be trusted with btrfs, but some userspace tools (i.e. apt, monitoring checks) rely on it.

Sometimes, "df -h" will report 100% disk usage on a btrfs filesystem, while there is still space available and writes are possible.

Linux 5.4.1, LXD containers located in /data:

# df -h
/dev/nvme1n1               1000G  571G     0 100% /data
# btrfs fi usage /data
Overall:
    Device size:                1000.00GiB
    Device allocated:            580.02GiB
    Device unallocated:          419.98GiB
    Device missing:                  0.00B
    Used:                        570.06GiB
    Free (estimated):            429.45GiB      (min: 429.45GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:578.01GiB, Used:568.54GiB
/dev/nvme1n1 578.01GiB

Metadata,single: Size:2.01GiB, Used:1.53GiB
/dev/nvme1n1 2.01GiB

System,single: Size:4.00MiB, Used:80.00KiB
/dev/nvme1n1 4.00MiB

Unallocated:
/dev/nvme1n1 419.98GiB

From container point of view, it looks like:

container# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 1000G 571G 0 100% /

Let's see if we can still write? Yes we can:

container# dd if=/dev/urandom of=/bigfile.img bs=1M count=1024 1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.28306 s, 147 MB/s container# sync
container# rm /bigfile.img

Let's try to use apt - unfortunately it says that there is not enough space in /var/cache/apt/archives/:

container# apt install lzop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
liblzo2-2
The following NEW packages will be installed:
liblzo2-2 lzop
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 88.5 kB of archives.
After this operation, 282 kB of additional disk space will be used.
E: You don't have enough free space in /var/cache/apt/archives/.

So let's try to write 1 GB to /var/cache/apt/archives/:

container# dd if=/dev/urandom of=/var/cache/apt/archives/bigfile.img bs=1M count=1024 sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.66032 s, 140 MB/s

container# sync
container# rm /var/cache/apt/archives/bigfile.img

This IMO is a filesystem bug?

Some more info:

# btrfs fi df /data
Data, single: total=580.01GiB, used=569.78GiB
System, single: total=4.00MiB, used=80.00KiB
Metadata, single: total=2.01GiB, used=1.53GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /data
Label: 'data'  uuid: 38fc3739-8695-49f1-bf14-946d0c0869c5
        Total devices 1 FS bytes used 571.30GiB
        devid    1 size 1000.00GiB used 582.02GiB path /dev/nvme1n1

Tomasz Chmielewski
https://lxadm.com

#1 lore raw
Quoted 12 lines · Tomasz Chmielewski

I know that "df -h" is not to be trusted with btrfs, but some userspace tools (i.e. apt, monitoring checks) rely on it.

Sometimes, "df -h" will report 100% disk usage on a btrfs filesystem, while there is still space available and writes are possible.

Linux 5.4.1, LXD containers located in /data:

# df -h
/dev/nvme1n1               1000G  571G     0 100% /data
Quoted 11 lines
# btrfs fi usage /data
Overall:
     Device size:                1000.00GiB
     Device allocated:            580.02GiB
     Device unallocated:          419.98GiB
     Device missing:                  0.00B
     Used:                        570.06GiB
     Free (estimated):            429.45GiB      (min: 429.45GiB)
     Data ratio:                       1.00
     Metadata ratio:                   1.00
     Global reserve:              512.00MiB      (used: 0.00B)

It does look like a bug. But I'm not sure if this is expected, I thought such significant disagreements were long since fixed.

What version of coreutils?

Maybe attach a strace of df? (I'm not sure of the list attach size limit but it's preferred, but something like a pastebin is OK also)

And maybe this will be useful for a dev?

# grep -r . /sys/fs/btrfs/<fs-uuid>/allocation/

You'll need to use the correct fs uuid for the btrfs in question.

Quoted 2 lines

Data,single: Size:578.01GiB, Used:568.54GiB

    /dev/nvme1n1  578.01GiB

df seems to be treating only unallocated space as free, rather than both unallocated and allocated that is unused. Color me surprised.

Signature · 1 line
Chris Murphy
#2 lore raw
Quoted 1 line · Chris Murphy

What version of coreutils?

This is Ubuntu 18.04 LTS, with coreutils 8.28.

Quoted 2 lines

Maybe attach a strace of df? (I'm not sure of the list attach size limit but it's preferred, but something like a pastebin is OK also)

"Unfortunately" I did run a balance, which "recovered" the space for df.

Tomasz

#3 lore raw
Quoted 5 lines · Tomasz Chmielewski
Quoted 1 line · Chris Murphy

What version of coreutils?

Quoted 7 lines

This is Ubuntu 18.04 LTS, with coreutils 8.28.

Quoted 2 lines

Maybe attach a strace of df? (I'm not sure of the list attach size limit but it's preferred, but something like a pastebin is OK also)

"Unfortunately" I did run a balance, which "recovered" the space for df.

Still an interesting data point.

Signature · 1 line
Chris Murphy
#4 lore raw
Quoted 9 lines · Chris Murphy
Quoted 8 lines · Tomasz Chmielewski
Quoted 1 line · Chris Murphy

What version of coreutils?

This is Ubuntu 18.04 LTS, with coreutils 8.28.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Maybe attach a strace of df? (I'm not sure of the list attach size limit but it's preferred, but something like a pastebin is OK also)

"Unfortunately" I did run a balance, which "recovered" the space for df.

Still an interesting data point.

Well, there is some crap there... it went read-only.

[164625.770369] BTRFS info (device nvme1n1): found 683 extents
[164630.127647] BTRFS info (device nvme1n1): relocating block group 345766887424 flags data
[164650.186985] BTRFS info (device nvme1n1): found 4268 extents
[164658.075685] BTRFS info (device nvme1n1): found 4266 extents
[164663.979192] BTRFS info (device nvme1n1): relocating block group 344693145600 flags data
[164686.646073] BTRFS info (device nvme1n1): found 3721 extents
[164696.815655] BTRFS: error (device nvme1n1) in btrfs_drop_snapshot:5402: errno=-4 unknown
[164696.821925] BTRFS info (device nvme1n1): forced readonly
[164696.834151] BTRFS warning (device nvme1n1): could not allocate space for delete; will truncate on mount
[164696.834162] BTRFS info (device nvme1n1): balance: ended with status: -4
[164697.468475] BTRFS info (device nvme1n1): delayed_refs has NO entry

Tomasz Chmielewski
https://lxadm.com