Understanding "Used" in df
I have:
% LANG=en df -hT /home
Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/sata-home btrfs 300G 175G 123G 59% /home
And:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 173.91GiB
devid 1 size 300.00GiB used 223.03GiB path /dev/mapper/sata-home
devid 2 size 300.00GiB used 223.03GiB path /dev/mapper/msata-home
merkaba:~> btrfs fi df /home
Data, RAID1: total=218.00GiB, used=171.98GiB System, RAID1: total=32.00MiB, used=64.00KiB Metadata, RAID1: total=5.00GiB, used=1.94GiB GlobalReserve, single: total=490.48MiB, used=0.00B
As well as:
merkaba:~> btrfs fi usage -T /home
Overall:
Device size: 600.00GiB
Device allocated: 446.06GiB
Device unallocated: 153.94GiB Device missing: 0.00B
Used: 347.82GiB
Free (estimated): 123.00GiB (min: 123.00GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 490.45MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- ---------------------- --------- -------- -------- -----------
1 /dev/mapper/sata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB 2 /dev/mapper/msata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB -- ---------------------- --------- -------- -------- ----------- Total 218.00GiB 5.00GiB 32.00MiB 153.94GiB Used 171.97GiB 1.94GiB 64.00KiB
I think I understand all of it, including just 123G instead of 300 - 175 = 125 GiB "Avail" in df -hT.
But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs fi sh') is allocated *within* the block group / chunks?
Does this have something to do with that global reserve thing?
Thank you,
Signature · 1 line
Martin
Quoted 48 lines · Martin Steigerwald
I have:
% LANG=en df -hT /home
Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/sata-home btrfs 300G 175G 123G 59% /homeAnd:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 173.91GiB
devid 1 size 300.00GiB used 223.03GiB path/dev/mapper/sata-home
devid 2 size 300.00GiB used 223.03GiB path/dev/mapper/msata-homemerkaba:~> btrfs fi df /home
Data, RAID1: total=218.00GiB, used=171.98GiB System, RAID1: total=32.00MiB, used=64.00KiB Metadata, RAID1: total=5.00GiB, used=1.94GiB GlobalReserve, single: total=490.48MiB, used=0.00BAs well as:
merkaba:~> btrfs fi usage -T /home
Overall:Device size: 600.00GiB Device allocated: 446.06GiB Device unallocated: 153.94GiBDevice missing: 0.00B
Used: 347.82GiB
Free (estimated): 123.00GiB (min: 123.00GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 490.45MiB (used: 0.00B)
Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- ---------------------- --------- -------- -------- -----------1 /dev/mapper/sata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB 2 /dev/mapper/msata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB -- ---------------------- --------- -------- -------- ----------- Total 218.00GiB 5.00GiB 32.00MiB 153.94GiB Used 171.97GiB 1.94GiB 64.00KiBI think I understand all of it, including just 123G instead of 300 - 175 = 125 GiB "Avail" in df -hT.
Quoted 6 lines
But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs fi sh') is allocated *within* the block group / chunks?
Does this have something to do with that global reserve thing?
Thank you,
Hi,
I do not think global reserve should be counted in 'df' output, but it does count all unallocated space, even if some of that space will be used up by metadata chunks. Isn't the discrepancy the metadata allocated vs used?
I made a post a little while ago on this subject, on how to improve the space calculations if any one if interested in picking this up? I believe the space calculation is wrong in all of the df/btrfs df/btrfs us tools.
Quoted 45 lines · Martin Steigerwald
I have:
% LANG=en df -hT /home
Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/sata-home btrfs 300G 175G 123G 59% /homeAnd:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 173.91GiB
devid 1 size 300.00GiB used 223.03GiB path/dev/mapper/sata-home
devid 2 size 300.00GiB used 223.03GiB path/dev/mapper/msata-homemerkaba:~> btrfs fi df /home
Data, RAID1: total=218.00GiB, used=171.98GiB System, RAID1: total=32.00MiB, used=64.00KiB Metadata, RAID1: total=5.00GiB, used=1.94GiB GlobalReserve, single: total=490.48MiB, used=0.00BAs well as:
merkaba:~> btrfs fi usage -T /home
Overall:Device size: 600.00GiB Device allocated: 446.06GiB Device unallocated: 153.94GiBDevice missing: 0.00B
Used: 347.82GiB
Free (estimated): 123.00GiB (min: 123.00GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 490.45MiB (used: 0.00B)
Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- ---------------------- --------- -------- -------- -----------1 /dev/mapper/sata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB 2 /dev/mapper/msata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB -- ---------------------- --------- -------- -------- ----------- Total 218.00GiB 5.00GiB 32.00MiB 153.94GiB Used 171.97GiB 1.94GiB 64.00KiB
Quoted 5 lines
I think I understand all of it, including just 123G instead of 300 - 175 = 125 GiB "Avail" in df -hT.
But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs fi sh') is allocated *within* the block group / chunks?
statvfs (the 'df' syscall) does not report a "used" number, only total and available btrfs data blocks (no metadata blocks are counted). 'df' computes "used" by subtracting f_blocks - f_bavail.
122.99875 = 300 - 171.97 - 5 - .03125 df_free = total - data_used - metadata_allocated - system_allocated
Inline files count as metadata instead of data, so even when you are out of data blocks (zero blocks free in df), you can sometimes still write small files. Sometimes, when you write one small file, 1GB of available space disappears as a new metadata block group is allocated.
'df' doesn't take metadata or data sharing into account at all, or the space required to store csums, or bursty metadata usage workloads. 'df' can't predict these events, so its accuracy is limited to no better than about 0.5% of the size of the filesystem or +/- 1GB, whichever is larger.
Quoted 1 line
Does this have something to do with that global reserve thing?
'df' currently tells you nothing about metadata (except in kernels before 5.6, when you run too low on metadata space, f_bavail is abruptly set to zero). That's about the only impact global reserve has on 'df'.
Global reserve is metadata allocated-but-unused space, and all metadata is not visible to df. The reserve ensures that critical btrfs metadata operations can complete without running out of space, by forcing non-critical long-running operations to commit transactions when no metadata space is available outside the reserved pool. It mostly works, though there are still a few bugs left that lead to EROFS when metadata runs low.
Quoted 6 lines
Thank you,
Signature · 1 line
Martin
Zygo Blaxell - 23.07.20, 06:51:06 CEST:
Quoted 59 lines · Martin Steigerwald
Quoted 49 lines
I have:
% LANG=en df -hT /home
Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/sata-home btrfs 300G 175G 123G 59% /homeAnd:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]Total devices 2 FS bytes used 173.91GiB devid 1 size 300.00GiB used 223.03GiB path /dev/mapper/sata-home devid 2 size 300.00GiB used 223.03GiB path /dev/mapper/msata-homemerkaba:~> btrfs fi df /home
Data, RAID1: total=218.00GiB, used=171.98GiB System, RAID1: total=32.00MiB, used=64.00KiB Metadata, RAID1: total=5.00GiB, used=1.94GiB GlobalReserve, single: total=490.48MiB, used=0.00BAs well as:
merkaba:~> btrfs fi usage -T /home
Overall:
Device size: 600.00GiB Device allocated: 446.06GiB Device unallocated: 153.94GiBDevice missing: 0.00B
Used: 347.82GiB
Free (estimated): 123.00GiB (min: 123.00GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 490.45MiB (used: 0.00B)
Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- ---------------------- --------- -------- -------- -----------1 /dev/mapper/sata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB 2 /dev/mapper/msata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB-- ---------------------- --------- -------- -------- -----------
Quoted 8 lines
Total 218.00GiB 5.00GiB 32.00MiB 153.94GiB Used 171.97GiB 1.94GiB 64.00KiBI think I understand all of it, including just 123G instead of 300 - 175 = 125 GiB "Avail" in df -hT.
But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs fi sh') is allocated *within* the block group / chunks?
Quoted 7 lines
statvfs (the 'df' syscall) does not report a "used" number, only total and available btrfs data blocks (no metadata blocks are counted). 'df' computes "used" by subtracting
f_blocks-f_bavail.122.99875 = 300 - 171.97 - 5 - .03125 df_free = total - data_used - metadata_allocated - system_allocated
I get that one. That is for what is still free.
But I do not understand "Used" in df as.
1) It it would be doing 300 GiB - what is still available, it would do 300-122.99 = 177.01
2) If it would add together all allocated within a chunk…
171.98 GiB used in data + 64 KiB used in system + 1,94 GiB used in metadata ~= 174 GiB
3) It may consider all allocated system and metadata chunks as lost for writing data:
171.98 used in date + 32 MiB allocated in system + 5 GiB allocated in metadata ~= 176.98 GiB
4) It may consider 2 of those 5 GiB chunks for metadata as reclaimable and then it would go like this:
171.98 used in date + 32 MiB allocated in system + 3 GiB metadata ~= 116.98 GiB = 174.98 GiB
That would be about right, but also as unpredictable as it can get.
Quoted 10 lines
Inline files count as metadata instead of data, so even when you are out of data blocks (zero blocks free in df), you can sometimes still write small files. Sometimes, when you write one small file, 1GB of available space disappears as a new metadata block group is allocated.
'df' doesn't take metadata or data sharing into account at all, or the space required to store csums, or bursty metadata usage workloads. 'df' can't predict these events, so its accuracy is limited to no better than about 0.5% of the size of the filesystem or +/- 1GB, whichever is larger.
So just assume that df output can be +/- 1 GiB off?
I am just wondering cause I aimed to explaining this to participants of my Linux courses… and for now I have the honest answer that I have no clue why df displays "175 GiB" as used.
Quoted 6 lines
Quoted 1 line
Does this have something to do with that global reserve thing?
'df' currently tells you nothing about metadata (except in kernels before 5.6, when you run too low on metadata space,
f_bavailis abruptly set to zero). That's about the only impact global reserve has on 'df'.
But it won't claim used or even just allocated metadata space as available for writing data?
Quoted 7 lines
Global reserve is metadata allocated-but-unused space, and all metadata is not visible to df. The reserve ensures that critical btrfs metadata operations can complete without running out of space, by forcing non-critical long-running operations to commit transactions when no metadata space is available outside the reserved pool. It mostly works, though there are still a few bugs left that lead to EROFS when metadata runs low.
Hmmm, thanks.
But as far as I understood also from the other post, Global Reserve is reserved but not reported as used in df?
I am not sure whether I am getting it though.
Best,
Signature · 1 line
Martin
27.07.2020 14:38, Martin Steigerwald пишет:
Quoted 66 lines
Zygo Blaxell - 23.07.20, 06:51:06 CEST:
Quoted 65 lines · Martin Steigerwald
Quoted 49 lines
I have:
% LANG=en df -hT /home
Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/sata-home btrfs 300G 175G 123G 59% /homeAnd:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]Total devices 2 FS bytes used 173.91GiB devid 1 size 300.00GiB used 223.03GiB path /dev/mapper/sata-home devid 2 size 300.00GiB used 223.03GiB path /dev/mapper/msata-homemerkaba:~> btrfs fi df /home
Data, RAID1: total=218.00GiB, used=171.98GiB System, RAID1: total=32.00MiB, used=64.00KiB Metadata, RAID1: total=5.00GiB, used=1.94GiB GlobalReserve, single: total=490.48MiB, used=0.00BAs well as:
merkaba:~> btrfs fi usage -T /home
Overall:
Device size: 600.00GiB Device allocated: 446.06GiB Device unallocated: 153.94GiBDevice missing: 0.00B
Used: 347.82GiB
Free (estimated): 123.00GiB (min: 123.00GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 490.45MiB (used: 0.00B)
Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- ---------------------- --------- -------- -------- -----------1 /dev/mapper/sata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB 2 /dev/mapper/msata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB-- ---------------------- --------- -------- -------- -----------
Quoted 8 lines
Total 218.00GiB 5.00GiB 32.00MiB 153.94GiB Used 171.97GiB 1.94GiB 64.00KiBI think I understand all of it, including just 123G instead of 300 - 175 = 125 GiB "Avail" in df -hT.
But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs fi sh') is allocated *within* the block group / chunks?
statvfs (the 'df' syscall) does not report a "used" number, only total and available btrfs data blocks (no metadata blocks are counted). 'df' computes "used" by subtracting
f_blocks-f_bavail.122.99875 = 300 - 171.97 - 5 - .03125
Quoted 8 lines
Quoted 1 line
df_free= total -data_used-metadata_allocated-system_allocatedI get that one. That is for what is still free.
But I do not understand "Used" in df as.
1) It it would be doing 300 GiB - what is still available, it would do 300-122.99 = 177.01
df "Used" is computed as "total" - "free", where "free" is reported by filesystem. btrfs free is 76.97GiB + 49.12GiB. I suppose btrfs does internally round at least the first number to the full chunk size which gives us close to 125GiB. To
Quoted 7 lines
2) If it would add together all allocated within a chunk…
171.98 GiB used in data + 64 KiB used in system + 1,94 GiB used in metadata ~= 174 GiB
3) It may consider all allocated system and metadata chunks as lost for writing data:
Quoted 7 lines
171.98 used in date + 32 MiB allocated in system + 5 GiB allocated in metadata ~= 176.98 GiB
4) It may consider 2 of those 5 GiB chunks for metadata as reclaimable and then it would go like this:
171.98 used in date + 32 MiB allocated in system + 3 GiB metadata ~= 116.98 GiB = 174.98 GiB
"df" does not know anything about data vs. metadata vs. system reserve. It has only two values filesystem returns - free and avail. And yes, they are computed independently.
"used" + "free" == "total", but do not expect "avail" to have any direct relation to other metrics.
Unfortunately, "df" does not display "free" (I was wrong in other post). But using stat ...
$ LANGUAGE=en stat -f . ... Block size: 4096 Fundamental block size: 4096 Blocks: Total: 115164174 Free: 49153062 Available: 43297293
$ LANGUAGE=en df -B 4K . Filesystem 4K-blocks Used Available Use% Mounted on /dev/sda4 115164174 66011112 43297293 61% /
115164174 - 49153062 == 66011112
But there is no way you can compute Available from other values - it is whatever filesystem returns.
Quoted 46 lines
That would be about right, but also as unpredictable as it can get.
Quoted 10 lines
Inline files count as metadata instead of data, so even when you are out of data blocks (zero blocks free in df), you can sometimes still write small files. Sometimes, when you write one small file, 1GB of available space disappears as a new metadata block group is allocated.
'df' doesn't take metadata or data sharing into account at all, or the space required to store csums, or bursty metadata usage workloads. 'df' can't predict these events, so its accuracy is limited to no better than about 0.5% of the size of the filesystem or +/- 1GB, whichever is larger.
So just assume that df output can be +/- 1 GiB off?
I am just wondering cause I aimed to explaining this to participants of my Linux courses… and for now I have the honest answer that I have no clue why df displays "175 GiB" as used.
Quoted 6 lines
Quoted 1 line
Does this have something to do with that global reserve thing?
'df' currently tells you nothing about metadata (except in kernels before 5.6, when you run too low on metadata space,
f_bavailis abruptly set to zero). That's about the only impact global reserve has on 'df'.But it won't claim used or even just allocated metadata space as available for writing data?
Quoted 7 lines
Global reserve is metadata allocated-but-unused space, and all metadata is not visible to df. The reserve ensures that critical btrfs metadata operations can complete without running out of space, by forcing non-critical long-running operations to commit transactions when no metadata space is available outside the reserved pool. It mostly works, though there are still a few bugs left that lead to EROFS when metadata runs low.
Hmmm, thanks.
But as far as I understood also from the other post, Global Reserve is reserved but not reported as used in df?
I am not sure whether I am getting it though.
Best,
Quoted 10 lines · Andrei Borzenkov
Unfortunately, "df" does not display "free" (I was wrong in other post). But using stat ...
$ LANGUAGE=en stat -f . ... Block size: 4096 Fundamental block size: 4096 Blocks: Total: 115164174 Free: 49153062 Available: 43297293
Quoted 9 lines
$ LANGUAGE=en df -B 4K . Filesystem 4K-blocks Used Available Use% Mounted on /dev/sda4 115164174 66011112 43297293 61% /115164174 - 49153062 == 66011112
But there is no way you can compute Available from other values - it is whatever filesystem returns.
It's definitely goofy in the odd device raid1 case. If I fallocate a file equal to Avail, Avail is not zero. The fallocated file + new Avail != old Avail, which is just not at all correct behavior. So I keep fallocating files to get to 0 Avail, and instead I'm only chasing zero Avail condition. Once I'm close enough, and delete all six fallocated files, I end up for a short period of time, an Avail that seems reasonable but is way bigger than it was at the start. And then a minute later, without any additional action on my part, returns to reporting nonsense.
It's pretty confusing.
Signature · 1 line
Chris Murphy
27.07.2020 22:30, Chris Murphy пишет:
Quoted 15 lines · Andrei Borzenkov
Quoted 14 lines
Unfortunately, "df" does not display "free" (I was wrong in other post). But using stat ...
$ LANGUAGE=en stat -f . ... Block size: 4096 Fundamental block size: 4096 Blocks: Total: 115164174 Free: 49153062 Available: 43297293$ LANGUAGE=en df -B 4K . Filesystem 4K-blocks Used Available Use% Mounted on /dev/sda4 115164174 66011112 43297293 61% /
Quoted 7 lines
Quoted 5 lines
115164174 - 49153062 == 66011112
But there is no way you can compute Available from other values - it is whatever filesystem returns.
It's definitely goofy in the odd device raid1 case.
Well, I already explained why it happens. Yes, it looks like a bug, the question is how to do better estimation without performing exhaustive single-chunk allocation every time. Three equal size devices looks simple, but consider general case of multiple devices of different size or different amount of free space.
Quoted 2 lines
If I fallocate a
file equal to Avail, Avail is not zero.
Yes, because real allocator breaks large file into chunks and estimation in statfs "allocates" large size as single operation. So end result is different.
Quoted 12 lines
The fallocated file + new
Avail != old Avail, which is just not at all correct behavior. So I keep fallocating files to get to 0 Avail, and instead I'm only chasing zero Avail condition. Once I'm close enough, and delete all six fallocated files, I end up for a short period of time, an Avail that seems reasonable but is way bigger than it was at the start. And then a minute later, without any additional action on my part, returns to reporting nonsense.It's pretty confusing.
Quoted 22 lines · Andrei Borzenkov
27.07.2020 22:30, Chris Murphy пишет:
Quoted 21 lines · Andrei Borzenkov
Quoted 19 lines
Unfortunately, "df" does not display "free" (I was wrong in other post). But using stat ...
$ LANGUAGE=en stat -f . ... Block size: 4096 Fundamental block size: 4096 Blocks: Total: 115164174 Free: 49153062 Available: 43297293$ LANGUAGE=en df -B 4K . Filesystem 4K-blocks Used Available Use% Mounted on /dev/sda4 115164174 66011112 43297293 61% /115164174 - 49153062 == 66011112
But there is no way you can compute Available from other values - it is whatever filesystem returns.
Quoted 7 lines
Quoted 1 line
It's definitely goofy in the odd device raid1 case.
Well, I already explained why it happens. Yes, it looks like a bug, the question is how to do better estimation without performing exhaustive single-chunk allocation every time. Three equal size devices looks simple, but consider general case of multiple devices of different size or different amount of free space.
There's an O(n^2) algorithm in the number of devices. It's what I
used to implement the online space checker[1]. I've put up a write up of the process at [2]. Without proof -- I wasn't able to work it out -- but nobody's been able to catch it out yet.
Hugo.
[1] https://carfax.org.uk/btrfs-usage/
[2] https://carfax.org.uk/files/temp/btrfs-allocator.pdf
Signature · 4 lines
Hugo Mills | >squeek< *POP* hugo@... carfax.org.uk | gluglugluglug http://carfax.org.uk/ | <pause> PGP: E2AB1DE4 | gluglugluglug!
Quoted 64 lines · Martin Steigerwald
Zygo Blaxell - 23.07.20, 06:51:06 CEST:
Quoted 63 lines · Martin Steigerwald
Quoted 49 lines
I have:
% LANG=en df -hT /home
Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/sata-home btrfs 300G 175G 123G 59% /homeAnd:
merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]Total devices 2 FS bytes used 173.91GiB devid 1 size 300.00GiB used 223.03GiB path /dev/mapper/sata-home devid 2 size 300.00GiB used 223.03GiB path /dev/mapper/msata-homemerkaba:~> btrfs fi df /home
Data, RAID1: total=218.00GiB, used=171.98GiB System, RAID1: total=32.00MiB, used=64.00KiB Metadata, RAID1: total=5.00GiB, used=1.94GiB GlobalReserve, single: total=490.48MiB, used=0.00BAs well as:
merkaba:~> btrfs fi usage -T /home
Overall:
Device size: 600.00GiB Device allocated: 446.06GiB Device unallocated: 153.94GiBDevice missing: 0.00B
Used: 347.82GiB
Free (estimated): 123.00GiB (min: 123.00GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 490.45MiB (used: 0.00B)
Multiple profiles: noData Metadata System
Id Path RAID1 RAID1 RAID1 Unallocated
-- ---------------------- --------- -------- -------- -----------1 /dev/mapper/sata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB 2 /dev/mapper/msata-home 218.00GiB 5.00GiB 32.00MiB 76.97GiB-- ---------------------- --------- -------- -------- -----------
Quoted 8 lines
Total 218.00GiB 5.00GiB 32.00MiB 153.94GiB Used 171.97GiB 1.94GiB 64.00KiBI think I understand all of it, including just 123G instead of 300 - 175 = 125 GiB "Avail" in df -hT.
But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs fi sh') is allocated *within* the block group / chunks?
statvfs (the 'df' syscall) does not report a "used" number, only total and available btrfs data blocks (no metadata blocks are counted). 'df' computes "used" by subtracting
f_blocks-f_bavail.
Quoted 7 lines
Quoted 3 lines
122.99875 = 300 - 171.97 - 5 - .03125 df_free = total - data_used - metadata_allocated - system_allocatedI get that one. That is for what is still free.
But I do not understand "Used" in df as.
df "used" is f_blocks - f_bfree. f_bfree is f_blocks - total_used, where total_used is all space used in the filesystem. f_bavail is the sum of all free space in existing data block groups plus a simulation of how much space would be available if all unallocated block groups were converted to the current data profile. f_blocks is the total number of blocks in the raw filesystem divided by the RAID redundancy of the last data block group.
These values count different, partially overlapping things, potentially in different units, so no relation holds between f_blocks, f_bavail and f_bfree.
Quoted 6 lines
1) It it would be doing 300 GiB - what is still available, it would do 300-122.99 = 177.01
2) If it would add together all allocated within a chunk…
171.98 GiB used in data + 64 KiB used in system + 1,94 GiB used in metadata ~= 174 GiB
Quoted 7 lines
3) It may consider all allocated system and metadata chunks as lost for writing data:
171.98 used in date + 32 MiB allocated in system + 5 GiB allocated in metadata ~= 176.98 GiB
4) It may consider 2 of those 5 GiB chunks for metadata as reclaimable and then it would go like this:
df f_bavail doesn't consider reclaiming anything. It's strictly "how many data blocks could we theoretically fallocate in this instant" (and even that is wrong since fallocate isn't atomic wrt space allocation). It doesn't run a full simulation of the real allocator behavior.
Quoted 9 lines
171.98 used in date + 32 MiB allocated in system + 3 GiB metadata ~= 116.98 GiB = 174.98 GiB
That would be about right, but also as unpredictable as it can get.
Quoted 5 lines
Inline files count as metadata instead of data, so even when you are out of data blocks (zero blocks free in df), you can sometimes still write small files. Sometimes, when you write one small file, 1GB of available space disappears as a new metadata block group is allocated.
Quoted 7 lines
Quoted 5 lines
'df' doesn't take metadata or data sharing into account at all, or the space required to store csums, or bursty metadata usage workloads. 'df' can't predict these events, so its accuracy is limited to no better than about 0.5% of the size of the filesystem or +/- 1GB, whichever is larger.
So just assume that df output can be +/- 1 GiB off?
If anything, +/- 1GiB is a _lower_ bound on df estimation error.
df avail is an estimation of how much of a specific type of space resource (data blocks) is available, based on currently available information, so some error is expected. Some error components change over time and cannot be predicted. Some of these errors are percentages of free space, others are multiples of constant values (1GB per disk). If df says you have 300GB free, in theory you can write between 284GB and 299GB of data to that filesystem based on various factors (not including dedupe or compression). In practice there may be other error components outside of the limited data model presented here (e.g. system block group allocation).
If we write data with csums there's a 0.1-0.9% extra overhead depending on csum algorithm and write sizes. 0.3% of 300GB is 1GB, so filesystem with 300GB free can only hold 299GB of data with xxhash csums.
A metadata block group is normally 1 GB, and one could be allocated at any time. If df reports we have 1.1GB of space, we either have 1.1GB or 100MB of space for data in reality, and we won't find out which it is until after we do more than 100MB of writes (i.e. did we run out of space while doing the write, or not?).
If we write a million 1K files, df available space won't change, as they'll all be inline files, inline files are stored in metadata, and statvfs doesn't count metadata in f_bavail. If we write the next file, 1 GB or more may be allocated for a new metadata block group (depends on disk layout). The average size of each file is 1K (multiplied by metadata redundancy factor) but each individual file consumes either 0 or 1 GB of space if you compare df available space before and after writing the file.
If the metadata is raid0 or raid10 (or raid5 or raid6, but don't use those) then it's possible for a block group to contain more than 1GB of data. This multiplies the "1 GB" number across the board, so you can jump from e.g. 5.1 GB to 0.1 GB at any time on a 5 disk array with raid0 metadata, or adding 4K to a file could rarely but randomly allocate 5 GB of disk space for metadata.
If we write a million 4K blocks to a single file in random order or with fsync() between each block, every 75 4K data blocks will come with an extra 16K metadata block, 213MB total, 5.3%. If we do that 4 times, we'll use an extra GB of metadata and probably trigger a metadata block group allocation. If we write 4GB of file all at once, there will be 0 or 64K of metadata blocks due to rounding, and df's prediction will be a closer match to reality (but even that 64K could still trigger a 1GB allocation). Most workloads fall somewhere in between these extremes.
There are also prediction errors in the other direction. If data is overwritten or deleted during a transaction, it is not freed until the end of the transaction, or even much later with async discard. Balances and scrubs can mark entire block groups as temporarily full. These events will cause df to temporarily underreport a block group's worth of free space. This imposes additional requirements on free-space-triggered daemons like automatic file reapers--they can overdelete due to temporary conditions, so they either need hysteresis, long time delay averaging, or big error tolerances (i.e. they don't care if they're a GB or two above target). In these cases, df is not inaccurate--pinned extents really do make disk space unavailable temporarily--but the reported value from df is low compared to the number of data blocks that could actually be allocated.
statvfs f_bfree is a better estimation of how much space is available if all possible optimizations are performed, since it's "size of all disks minus size of everything in use." You'll never get exactly that much space out of a filesystem, though, due to minimum block group sizes, global reservations, data copy on write, fragmentation, metadata page unsharing, and other assorted inefficiencies.
Quoted 3 lines
I am just wondering cause I aimed to explaining this to participants of my Linux courses… and for now I have the honest answer that I have no clue why df displays "175 GiB" as used.
As a rule of thumb, a filesystem with less than 1GB free per disk is effectively full, mostly due to that "may allocate extra GB at any time" behavior. Even filesystems with less than 10GB or even 100GB per disk require extra attention to avoid sudden ENOSPC in some cases and performance degradation due to fragmentation in others.
If the filesytem has more than 100GB unallocated on every disk, no special attention is required. Every half-percent of writes consumes an extra GB, no big deal.
There are some scale factors in btrfs that change in smaller filesystems. The block group allocation unit gets smaller, so e.g. for a filesystem that is under 16GB the rule of thumb operates in units of 128MB per disk instead of 1GB per disk.
Mixed block groups are recommended for tiny filesystems. Mixed block groups don't have separate metadata and data, so the 'df' behavior is much more linear and predictive.
Quoted 9 lines
Quoted 6 lines
Quoted 1 line
Does this have something to do with that global reserve thing?
'df' currently tells you nothing about metadata (except in kernels before 5.6, when you run too low on metadata space,
f_bavailis abruptly set to zero). That's about the only impact global reserve has on 'df'.But it won't claim used or even just allocated metadata space as available for writing data?
Correct. Metadata space is never counted in df as available for data (f_bavail). It is counted in f_bfree (which df uses to calculate "used"), but f_bfree is a fairly meaningless internal statistic.
If a metadata block group is deleted (e.g. by resize, device delete, balance, or if it becomes completely empty) then its space will be counted as available, but not before. Balances may change the available space in df by changing on-disk layout so that future allocations become possible that are not possible with the present on-disk layout (e.g. when a new empty disk is added to a full raid5 array, initially none of the free space can be used). Converting balances can change the filesystem size in df too, which will change "used" since df used = size - free, sometimes in strange ways if there are block groups with multiple profiles present.
Quoted 8 lines
Quoted 7 lines
Global reserve is metadata allocated-but-unused space, and all metadata is not visible to df. The reserve ensures that critical btrfs metadata operations can complete without running out of space, by forcing non-critical long-running operations to commit transactions when no metadata space is available outside the reserved pool. It mostly works, though there are still a few bugs left that lead to EROFS when metadata runs low.
Quoted 4 lines
Hmmm, thanks.
But as far as I understood also from the other post, Global Reserve is reserved but not reported as used in df?
Global reserve overlaps with metadata. Some of global reserve inevitably gets counted in f_bfree (or more correctly f_blocks - f_bfree) because global reserve blocks are also metadata blocks, and f_bfree is affected by the number of metadata blocks in use. Changes in global reserve will therefore correlate with changes in df "used" output some of the time.
Global reserve isn't a physical region on disk like a metadata or system block group. Global reserve is just a block counter in the kernel that deducts future metadata allocations from the amount of free space that appears in metadata block groups in the present, so that later allocations required to commit the transaction are guaranteed to succeed. The usage numbers for global reserve and metadata cannot be meaningfully separated.
Quoted 7 lines
I am not sure whether I am getting it though.
Best,
Signature · 1 line
Martin