understanding btrfs fi df

Hello,

I am struggling to understand the output of btrfs fi df:

btrfs fi df /mnt/__Complete_Disk/
Data, RAID5: total=3.85TiB, used=3.85TiB System, RAID5: total=32.00MiB, used=576.00KiB Metadata, RAID5: total=6.46GiB, used=5.14GiB GlobalReserve, single: total=512.00MiB, used=0.00B

I have three disks:

btrfs fi show
Label: none uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
Total devices 3 FS bytes used 3.85TiB
devid 1 size 2.73TiB used 2.73TiB path /dev/sdf
devid 2 size 2.73TiB used 2.24TiB path /dev/sdd
devid 3 size 2.73TiB used 2.73TiB path /dev/sde

So I would expect around 5.4TiB to be available. In fact I am currently using 4.4TiB and the drive seems to be full (I cannot store more data).

I went through Marcs Blog post on this, but the hints there did not help. In particular:
btrfs balance start -dusage=95 /mnt/__Complete_Disk/ (started with 55)

What's wrong here?

Regards,
Hendrik

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus

Here's your problem -- you've got a RAID 5 filesystem, which has a
minimum allocation of 2 devices, but only one device has free space on it for allocation, so no more chunks can be allocated. I'm not sure how it ended up in this situation, but that's what's happened.

2) convert a few chunks of the existing FS, on devices 1 and 3, to single, which should free up some space across all the devices, and then run a full balance, and finally convert the single back to RAID 5:

# btrfs balance start -dconvert=single,devid=1,limit=4/mnt/__Complete_Disk
# btrfs balance start -dconvert=single,devid=3,limit=4/mnt/__Complete_Disk
# btrfs balance start -dprofiles=raid5/mnt/__Complete_Disk
# btrfs balance start -dconvert=raid5,soft/mnt/__Complete_Disk

Hello Hugo,

thanks for your hint.

Quoted 10 lines · Hugo Mills

Here's your problem -- you've got a RAID 5 filesystem, which has a
minimum allocation of 2 devices, but only one device has free space on it for allocation, so no more chunks can be allocated. I'm not sure how it ended up in this situation, but that's what's happened.

2) convert a few chunks of the existing FS, on devices 1 and 3, to single, which should free up some space across all the devices, and then run a full balance, and finally convert the single back to RAID 5:

Quoted 4 lines

# btrfs balance start -dconvert=single,devid=1,limit=4/mnt/__Complete_Disk
# btrfs balance start -dconvert=single,devid=3,limit=4/mnt/__Complete_Disk
# btrfs balance start -dprofiles=raid5/mnt/__Complete_Disk
# btrfs balance start -dconvert=raid5,soft/mnt/__Complete_Disk

I did that. In order to run the first two commands I had to free some space though, which was no problem.

Now the output is:

root@homeserver:/media# btrfs fi df /mnt/__Complete_Disk

Data, RAID5: total=3.79TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=416.00KiB Metadata, RAID5: total=6.46GiB, used=4.85GiB GlobalReserve, single: total=512.00MiB, used=0.00B

root@homeserver:/media# btrfs fi show
Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
        Total devices 3 FS bytes used 3.79TiB
        devid    1 size 2.73TiB used 2.43TiB path /dev/sdf
        devid    2 size 2.73TiB used 2.43TiB path /dev/sdd
        devid    3 size 2.73TiB used 1.38TiB path /dev/sde

How can only 1.38TiB be used on devid 3?

Quoted 15 lines

Now the output is:

root@homeserver:/media# btrfs fi df /mnt/__Complete_Disk

Data, RAID5: total=3.79TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=416.00KiB Metadata, RAID5: total=6.46GiB, used=4.85GiB GlobalReserve, single: total=512.00MiB, used=0.00B

root@homeserver:/media# btrfs fi show
Label: none  uuid: a8af3832-48c7-4568-861f-e80380dd7e0b
         Total devices 3 FS bytes used 3.79TiB
         devid    1 size 2.73TiB used 2.43TiB path /dev/sdf
         devid    2 size 2.73TiB used 2.43TiB path /dev/sdd
         devid    3 size 2.73TiB used 1.38TiB path /dev/sde

Quoted 1 line

How can only 1.38TiB be used on devid 3?

It shouldn't happen, as I understand how the process works. Can you
show the output of "btrfs fi df /mnt/__Complete_Disk"? Let's just check that everything is indeed RAID-5 still.

Quoted 3 lines

It shouldn't happen, as I understand how the process works. Can you
show the output of "btrfs fi df /mnt/__Complete_Disk"? Let's just check that everything is indeed RAID-5 still.

Here we go:

btrfs fi df /mnt/__Complete_Disk

Data, RAID5: total=3.79TiB, used=3.78TiB System, RAID5: total=32.00MiB, used=416.00KiB Metadata, RAID5: total=6.46GiB, used=4.85GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Quoted 2 lines

After this, we're probably going to have to look at the device and
chunk trees to work out what's going on.