Unable to allocate for space usage in particular btrfs volume
Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported 887GB used and 'df -h' 920GB for this particular volume - I went on #btrfs for any suggestions, and balancing + defraging made no difference. It had no subvolumes/snapshots etc, I basically used it like a checksumed ext4fs.
Since the volume was converted from ext4, I redid it from scratch (so made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and the problem went away.
After a couple of months, df reports 907GB used, whereas du says 884GB - I currently have 8 large (1-5.5TB volumes) btrfs volumes in use, storage-1 is the only SSD volume and the only one with this problem.
No balancing or defraging this time, it didn't make a difference before and this is a relatively new volume.
Are there any sysadmin-level ways I can account for the ~23GB lost space?
Thanks for any help.
=========================================================
$ uname -a
Linux omega1 4.2.0-1-amd64 #1 SMP Debian 4.2.5-1 (2015-10-27) x86_64 GNU/Linux
$ btrfs --version
btrfs-progs v4.2.2
$ sudo btrfs fi usage /mnt/storage-1
Overall:
Device size: 953.87GiB
Device allocated: 932.04GiB
Device unallocated: 21.83GiB
Device missing: 0.00B
Used: 906.10GiB
Free (estimated): 45.35GiB (min: 34.43GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)Data,single: Size:925.01GiB, Used:901.50GiB
/dev/sdb 925.01GiB
Metadata,single: Size:8.00MiB, Used:0.00B
/dev/sdb 8.00MiB
Metadata,DUP: Size:3.50GiB, Used:2.30GiB
/dev/sdb 7.00GiB
System,single: Size:4.00MiB, Used:0.00B
/dev/sdb 4.00MiB
System,DUP: Size:8.00MiB, Used:128.00KiB
/dev/sdb 16.00MiB
Unallocated:
/dev/sdb 21.83GiB
$ sudo btrfs-show-super /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum 0x7f6b70be [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 27430475-c49a-4e3f-8f8d-be5c14be59db
label storage-1
generation 114344
root 683413471232sys_array_size 226chunk_root_generation 114251root_level 1chunk_root 21004288chunk_root_level 1log_root 683413979136log_root_transid 0log_root_level 0total_bytes 1024209543168bytes_used 971565568000
sectorsize 4096
nodesize 16384
leafsize 16384
stripesize 4096root_dir 6num_devices 1compat_flags 0x0compat_ro_flags 0x0incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )csum_type 0csum_size 4cache_generation 114344uuid_tree_generation 114344
dev_item.uuid c6b32341-6300-4f21-8c3b-3d7d458c3668
dev_item.fsid 27430475-c49a-4e3f-8f8d-be5c14be59db [match]
dev_item.type 0
dev_item.total_bytes 1024209543168
dev_item.bytes_used 1000765128704
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
=========================================================
dmesg contains a lot of information which is superfluous to btrfs and personal, I can filter on a regex and report if necessary.
Quoted 10 lines · OmegaPhil
Back in September I noticed that 'sudo du -chs
/mnt/storage-1' reported 887GB used and 'df -h' 920GB for this particular volume - I went on #btrfs for any suggestions, and balancing + defraging made no difference. It had no subvolumes/snapshots etc, I basically used it like a checksumed ext4fs.Since the volume was converted from ext4, I redid it from scratch (so made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and the problem went away.
Quoted 8 lines
After a couple of months, df reports 907GB used, whereas du says 884GB - I currently have 8 large (1-5.5TB volumes) btrfs volumes in use, storage-1 is the only SSD volume and the only one with this problem.
No balancing or defraging this time, it didn't make a difference before and this is a relatively new volume.
Are there any sysadmin-level ways I can account for the ~23GB lost space?
There's an issue where replacing blocks in the middle of an
existing extent won't split the extent, and thus the "old" blocks aren't freed up, because they're held by the original extent (even though not actually referenced by any existing file). This might be what you're seeing.
I'm not sure how to confirm this theory, or what to do about it if
it's true. (Defrag the file? Copy it elsewhere? Other?)
Two other cases for df > du are orphaned files, although 23 GiB of
orphans is large; and missing out the dot-files in the directory that du is run from (if doing, say, "du *" rather than "du ."). I've been bitten by both of those in the past.
Hugo.
Quoted 103 lines
Thanks for any help.
=========================================================
$ uname -aLinux omega1 4.2.0-1-amd64 #1 SMP Debian 4.2.5-1 (2015-10-27)
x86_64GNU/Linux$ btrfs --versionbtrfs-progs v4.2.2
$ sudo btrfs fi usage /mnt/storage-1Overall:
Device size: 953.87GiB Device allocated: 932.04GiB Device unallocated: 21.83GiB Device missing: 0.00B Used: 906.10GiB Free (estimated): 45.35GiB (min: 34.43GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B)Data,single: Size:925.01GiB, Used:901.50GiB
/dev/sdb925.01GiBMetadata,single: Size:8.00MiB, Used:0.00B
/dev/sdb8.00MiBMetadata,DUP: Size:3.50GiB, Used:2.30GiB
/dev/sdb7.00GiBSystem,single: Size:4.00MiB, Used:0.00B
/dev/sdb4.00MiBSystem,DUP: Size:8.00MiB, Used:128.00KiB
/dev/sdb16.00MiBUnallocated:
/dev/sdb 21.83GiB$ sudo btrfs-show-super /dev/sdbsuperblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum0x7f6b70be[match]
bytenr 65536
flags0x1
( WRITTEN )
magic_BHRfS_M[match]
fsid 27430475-c49a-4e3f-8f8d-be5c14be59db
label storage-1
generation 114344
root 683413471232sys_array_size226chunk_root_generation114251root_level1chunk_root21004288chunk_root_level1log_root683413979136log_root_transid0log_root_level0total_bytes1024209543168bytes_used971565568000
sectorsize 4096
nodesize 16384
leafsize 16384
stripesize 4096root_dir6num_devices1compat_flags0x0compat_ro_flags0x0incompat_flags0x161
(MIXED_BACKREF|
BIG_METADATA|
EXTENDED_IREF|
SKINNY_METADATA)csum_type0csum_size4cache_generation114344uuid_tree_generation114344
dev_item.uuid c6b32341-6300-4f21-8c3b-3d7d458c3668
dev_item.fsid 27430475-c49a-4e3f-8f8d-be5c14be59db [match]
dev_item.type 0
dev_item.total_bytes 1024209543168
dev_item.bytes_used 1000765128704
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0=========================================================
dmesg contains a lot of information which is superfluous to btrfs and personal, I can filter on a regex and report if necessary.
Signature · 4 lines
Hugo Mills | There are three things you should never see being hugo@... carfax.org.uk | made: laws, standards, and sausages. http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 29 lines · Hugo Mills
Quoted 10 lines · OmegaPhil
Back in September I noticed that 'sudo du -chs
/mnt/storage-1' reported 887GB used and 'df -h' 920GB for this particular volume - I went on #btrfs for any suggestions, and balancing + defraging made no difference. It had no subvolumes/snapshots etc, I basically used it like a checksumed ext4fs.Since the volume was converted from ext4, I redid it from scratch (so made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and the problem went away.
Quoted 8 lines
After a couple of months, df reports 907GB used, whereas du says 884GB - I currently have 8 large (1-5.5TB volumes) btrfs volumes in use, storage-1 is the only SSD volume and the only one with this problem.
No balancing or defraging this time, it didn't make a difference before and this is a relatively new volume.
Are there any sysadmin-level ways I can account for the ~23GB lost space?
There's an issue where replacing blocks in the middle of an
existing extent won't split the extent, and thus the "old" blocks aren't freed up, because they're held by the original extent (even though not actually referenced by any existing file). This might be what you're seeing.I'm not sure how to confirm this theory, or what to do about it if
it's true. (Defrag the file? Copy it elsewhere? Other?)
Quoted 6 lines
Two other cases for df > du are orphaned files, although 23 GiB of
orphans is large; and missing out the dot-files in the directory that du is run from (if doing, say, "du *" rather than "du ."). I've been bitten by both of those in the past.Hugo.
The volume doesn't change hugely over time, so it really ought not to have broken so quickly - a quick rundown of the storage usage:
36% general (small files, some smallish videos)
24% music
23% pr0n
17% VMs
But in terms of 'large files changing', it could be the VM disks perhaps
- I'll move them out, balance, and then back in again, hopefully that'd be a meaningful test.
du-wise was direct on the root directory, any idea how I could audit orphan files?
Thanks
OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
Quoted 10 lines
The volume doesn't change hugely over time, so it really ought not to have broken so quickly - a quick rundown of the storage usage:
36% general (small files, some smallish videos)
24% music 23% pr0n 17% VMsBut in terms of 'large files changing', it could be the VM disks perhaps -
I'll move them out, balance, and then back in again, hopefully that'd be a meaningful test.
VM image files (and large database files, for the same reason) are a bit of a problem on btrfs, and indeed, any COW-based filesystem, since the random rewrite pattern matching that use-case is pretty much the absolute worst-case match for a COW-based filesystem there is.
And that would be the worst-case in terms of the unsplit extents issue Hugo was talking about as well. So they may well be the problem, indeed.
Since you're not doing snapshotting (which conflicts with this option, with an imperfect workaround), setting nocow on those files may well eliminate the problem, but be aware if you aren't already that (1) nocow does turn off checksumming as well, in ordered to avoid a race that could easily lead to data corruption, and (2) you can't just activate nocow on the existing file and expect it to work, the procedure is a bit more complicated than that, since nocow is only guaranteed to work if it's set at file creation. Detailed instructions for #2 skipped as they've been posted many times, but if you are interested and haven't seen them, ask.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 21 lines · Duncan
OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
Quoted 10 lines
The volume doesn't change hugely over time, so it really ought not to have broken so quickly - a quick rundown of the storage usage:
36% general (small files, some smallish videos)
24% music 23% pr0n 17% VMsBut in terms of 'large files changing', it could be the VM disks perhaps -
I'll move them out, balance, and then back in again, hopefully that'd be a meaningful test.VM image files (and large database files, for the same reason) are a bit of a problem on btrfs, and indeed, any COW-based filesystem, since the random rewrite pattern matching that use-case is pretty much the absolute worst-case match for a COW-based filesystem there is.
And that would be the worst-case in terms of the unsplit extents issue Hugo was talking about as well. So they may well be the problem, indeed.
Quoted 9 lines
Since you're not doing snapshotting (which conflicts with this option, with an imperfect workaround), setting nocow on those files may well eliminate the problem, but be aware if you aren't already that (1) nocow does turn off checksumming as well, in ordered to avoid a race that could easily lead to data corruption, and (2) you can't just activate nocow on the existing file and expect it to work, the procedure is a bit more complicated than that, since nocow is only guaranteed to work if it's set at file creation. Detailed instructions for #2 skipped as they've been posted many times, but if you are interested and haven't seen them, ask.
Thankyou Hugo, Duncan - moving VMs out, then:
=====================================================
sudo chattr +C '/mnt/storage-1/Virtual Machines' sudo btrfs balance start -mconvert=dup /mnt/storage-1 sudo btrfs fi defrag -rv /mnt/storage-1
=====================================================
And after moving VMs back, du reports 884GB used,
=====================================================
$ sudo btrfs fi usage /mnt/storage-1
Overall:
Device size: 953.87GiB
Device allocated: 924.07GiB
Device unallocated: 29.80GiB
Device missing: 0.00B
Used: 886.11GiB
Free (estimated): 65.72GiB (min: 50.82GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)Data,single: Size:918.01GiB, Used:882.09GiB
/dev/sdb 918.01GiB
Metadata,DUP: Size:3.00GiB, Used:2.01GiB
/dev/sdb 6.00GiB
System,DUP: Size:32.00MiB, Used:128.00KiB
/dev/sdb 64.00MiB
Unallocated:
/dev/sdb 29.80GiB
=====================================================
So a couple of gig still unaccountable but irrelevant. Thanks, problem solved! Although hopefully checksumming will be allowed on nocow files in the future as thats currently 17% of all data unprotected and will get worse...
Quoted 4 lines · OmegaPhil
So a couple of gig still unaccountable but irrelevant. Thanks, problem solved! Although hopefully checksumming will be allowed on nocow files in the future as thats currently 17% of all data unprotected and will get worse...
It's not likely to happen. If you're modifying data in-place (as is
the case on nodatacow files), then there's a window of vulnerability between modifying the data and modifying the checksum records during which the checksums are incorrect. This breaks the consistency guarantees of the FS.
Hugo.
Signature · 4 lines
Hugo Mills | Nostalgia isn't what it used to be. hugo@... carfax.org.uk | http://carfax.org.uk/ | PGP: E2AB1DE4 |
Quoted 19 lines · OmegaPhil
Quoted 9 lines · Duncan
OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
VM image files (and large database files, for the same reason) are a bit
of a problem on btrfs, and indeed, any COW-based filesystem, since the
random rewrite pattern matching that use-case is pretty much the absolute
worst-case match for a COW-based filesystem there is. Since you're not doing snapshotting (which conflicts with thisQuoted 8 lines
option,
with an imperfect workaround), setting nocow on those files may well
eliminate the problem, but be aware if you aren't already that (1) nocow
does turn off checksumming as well, in ordered to avoid a race that could
easily lead to data corruption, and (2) you can't just activate
Quoted 6 lines
So a couple of gig still unaccountable but irrelevant. Thanks, problem
solved! Although hopefully checksumming will be allowed on nocow files
in the future as thats currently 17% of all data unprotected and will get worse...
There's actually an interesting workaround to this: Although the VM disk images aren't checksummed on the host filesystem, you can use btrfs *inside* the VMs and enable checksumming there. The downside is that you can only verify the VM data by booting the VM and running a scrub from inside.
This of course doesn't help if your VMs are Windows or legacy versions of Linux without btrfs support. On BSD you could try ZFS.
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 27 lines · Calvin Walton
Quoted 19 lines · OmegaPhil
Quoted 9 lines · Duncan
OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
VM image files (and large database files, for the same reason) are a bit
of a problem on btrfs, and indeed, any COW-based filesystem, since the
random rewrite pattern matching that use-case is pretty much the absolute
worst-case match for a COW-based filesystem there is. Since you're not doing snapshotting (which conflicts with thisQuoted 8 lines
option,
with an imperfect workaround), setting nocow on those files may well
eliminate the problem, but be aware if you aren't already that (1) nocow
does turn off checksumming as well, in ordered to avoid a race that could
easily lead to data corruption, and (2) you can't just activateQuoted 6 lines
So a couple of gig still unaccountable but irrelevant. Thanks, problem
solved! Although hopefully checksumming will be allowed on nocow files
in the future as thats currently 17% of all data unprotected and will get worse...
Quoted 5 lines
There's actually an interesting workaround to this: Although the VM disk images aren't checksummed on the host filesystem, you can use btrfs *inside* the VMs and enable checksumming there. The downside is that you can only verify the VM data by booting the VM and running a scrub from inside.
Actually, by using a combination of loop devices and kpartx, it's fully possible to mount the FS and verify it without booting the VM. Of course, doing this usually requires root access on the host system, but for most people I know, that's usually not an issue. I do this on occasion when I need to pull a file off of one of my VM disks on my laptop and don't have the time to spin up the VM itself.
Another option if you're doing a direct boot of the kernel (for example, when using a fully paravirtualized domain on Xen, or using some of the QEMU ARM systems) is to just do the volume management (partitioning and such) on the host, and expose each filesystem to the guest as a separate disk. I do this with most of my Linux VM's on my Xen system where I use LVM as the back-end storage for the virtual disk images, as it allows me to easily directly mount the VM's filesystems on the host if need be (and let's you do all kinds of cool things like using a cluster-aware filesystem for the VM's root so that you can mount it from the host safely while the VM is still online).