Skip to contentSkip to conversation

Unable to allocate for space usage in particular btrfs volume

DISCUSSION OmegaPhil 5 participants 8 messages Nov 4, 2015 – Nov 6, 2015 Btrfs file system

lore
Share & tools
lore

OmegaPhil

author
#0 lore raw

Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported 887GB used and 'df -h' 920GB for this particular volume - I went on #btrfs for any suggestions, and balancing + defraging made no difference. It had no subvolumes/snapshots etc, I basically used it like a checksumed ext4fs.

Since the volume was converted from ext4, I redid it from scratch (so made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and the problem went away.

After a couple of months, df reports 907GB used, whereas du says 884GB - I currently have 8 large (1-5.5TB volumes) btrfs volumes in use, storage-1 is the only SSD volume and the only one with this problem.

No balancing or defraging this time, it didn't make a difference before and this is a relatively new volume.

Are there any sysadmin-level ways I can account for the ~23GB lost space?

Thanks for any help.

=========================================================

$ uname -a

Linux omega1 4.2.0-1-amd64 #1 SMP Debian 4.2.5-1 (2015-10-27) x86_64 GNU/Linux

$ btrfs --version

btrfs-progs v4.2.2

$ sudo btrfs fi usage /mnt/storage-1

Overall:

    Device size:		 953.87GiB
    Device allocated:		 932.04GiB
    Device unallocated:		  21.83GiB
    Device missing:		     0.00B
    Used:			 906.10GiB
    Free (estimated):		  45.35GiB	(min: 34.43GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:925.01GiB, Used:901.50GiB
/dev/sdb 925.01GiB

Metadata,single: Size:8.00MiB, Used:0.00B
/dev/sdb 8.00MiB

Metadata,DUP: Size:3.50GiB, Used:2.30GiB
/dev/sdb 7.00GiB

System,single: Size:4.00MiB, Used:0.00B
/dev/sdb 4.00MiB

System,DUP: Size:8.00MiB, Used:128.00KiB
/dev/sdb 16.00MiB

Unallocated:
/dev/sdb 21.83GiB

$ sudo btrfs-show-super /dev/sdb

superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum 0x7f6b70be [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 27430475-c49a-4e3f-8f8d-be5c14be59db
label storage-1
generation 114344
root 683413471232
sys_array_size 226
chunk_root_generation 114251
root_level 1
chunk_root 21004288
chunk_root_level 1
log_root 683413979136
log_root_transid 0
log_root_level 0
total_bytes 1024209543168
bytes_used 971565568000
sectorsize 4096
nodesize 16384
leafsize 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
csum_type 0
csum_size 4
cache_generation 114344
uuid_tree_generation 114344
dev_item.uuid c6b32341-6300-4f21-8c3b-3d7d458c3668
dev_item.fsid 27430475-c49a-4e3f-8f8d-be5c14be59db [match]
dev_item.type 0
dev_item.total_bytes 1024209543168
dev_item.bytes_used 1000765128704
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

=========================================================

dmesg contains a lot of information which is superfluous to btrfs and personal, I can filter on a regex and report if necessary.

#1 lore raw
Quoted 10 lines · OmegaPhil

Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported 887GB used and 'df -h' 920GB for this particular volume - I went on #btrfs for any suggestions, and balancing + defraging made no difference. It had no subvolumes/snapshots etc, I basically used it like a checksumed ext4fs.

Since the volume was converted from ext4, I redid it from scratch (so made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and the problem went away.

Quoted 8 lines

After a couple of months, df reports 907GB used, whereas du says 884GB - I currently have 8 large (1-5.5TB volumes) btrfs volumes in use, storage-1 is the only SSD volume and the only one with this problem.

No balancing or defraging this time, it didn't make a difference before and this is a relatively new volume.

Are there any sysadmin-level ways I can account for the ~23GB lost space?

There's an issue where replacing blocks in the middle of an
existing extent won't split the extent, and thus the "old" blocks aren't freed up, because they're held by the original extent (even though not actually referenced by any existing file). This might be what you're seeing.

I'm not sure how to confirm this theory, or what to do about it if
it's true. (Defrag the file? Copy it elsewhere? Other?)

Two other cases for df > du are orphaned files, although 23 GiB of
orphans is large; and missing out the dot-files in the directory that du is run from (if doing, say, "du *" rather than "du ."). I've been bitten by both of those in the past.

Hugo.

Quoted 103 lines

Thanks for any help.

=========================================================

$ uname -a

Linux omega1 4.2.0-1-amd64 #1 SMP Debian 4.2.5-1 (2015-10-27) x86_64 GNU/Linux

$ btrfs --version

btrfs-progs v4.2.2

$ sudo btrfs fi usage /mnt/storage-1

Overall:

    Device size:		 953.87GiB
    Device allocated:		 932.04GiB
    Device unallocated:		  21.83GiB
    Device missing:		     0.00B
    Used:			 906.10GiB
    Free (estimated):		  45.35GiB	(min: 34.43GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:925.01GiB, Used:901.50GiB
/dev/sdb 925.01GiB

Metadata,single: Size:8.00MiB, Used:0.00B
/dev/sdb 8.00MiB

Metadata,DUP: Size:3.50GiB, Used:2.30GiB
/dev/sdb 7.00GiB

System,single: Size:4.00MiB, Used:0.00B
/dev/sdb 4.00MiB

System,DUP: Size:8.00MiB, Used:128.00KiB
/dev/sdb 16.00MiB

Unallocated:

   /dev/sdb	  21.83GiB
$ sudo btrfs-show-super /dev/sdb

superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum 0x7f6b70be [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 27430475-c49a-4e3f-8f8d-be5c14be59db
label storage-1
generation 114344
root 683413471232
sys_array_size 226
chunk_root_generation 114251
root_level 1
chunk_root 21004288
chunk_root_level 1
log_root 683413979136
log_root_transid 0
log_root_level 0
total_bytes 1024209543168
bytes_used 971565568000
sectorsize 4096
nodesize 16384
leafsize 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
( MIXED_BACKREF |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
csum_type 0
csum_size 4
cache_generation 114344
uuid_tree_generation 114344
dev_item.uuid c6b32341-6300-4f21-8c3b-3d7d458c3668
dev_item.fsid 27430475-c49a-4e3f-8f8d-be5c14be59db [match]
dev_item.type 0
dev_item.total_bytes 1024209543168
dev_item.bytes_used 1000765128704
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

=========================================================

dmesg contains a lot of information which is superfluous to btrfs and personal, I can filter on a regex and report if necessary.

Signature · 4 lines
Hugo Mills             | There are three things you should never see being
hugo@... carfax.org.uk | made: laws, standards, and sausages.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |
author
#2 lore raw
Quoted 29 lines · Hugo Mills
Quoted 10 lines · OmegaPhil

Back in September I noticed that 'sudo du -chs /mnt/storage-1' reported 887GB used and 'df -h' 920GB for this particular volume - I went on #btrfs for any suggestions, and balancing + defraging made no difference. It had no subvolumes/snapshots etc, I basically used it like a checksumed ext4fs.

Since the volume was converted from ext4, I redid it from scratch (so made with kernel v4.1.3 or v4.1.6 on this Debian Testing machine), and the problem went away.

Quoted 8 lines

After a couple of months, df reports 907GB used, whereas du says 884GB - I currently have 8 large (1-5.5TB volumes) btrfs volumes in use, storage-1 is the only SSD volume and the only one with this problem.

No balancing or defraging this time, it didn't make a difference before and this is a relatively new volume.

Are there any sysadmin-level ways I can account for the ~23GB lost space?

There's an issue where replacing blocks in the middle of an
existing extent won't split the extent, and thus the "old" blocks aren't freed up, because they're held by the original extent (even though not actually referenced by any existing file). This might be what you're seeing.

I'm not sure how to confirm this theory, or what to do about it if
it's true. (Defrag the file? Copy it elsewhere? Other?)

Quoted 6 lines

Two other cases for df > du are orphaned files, although 23 GiB of
orphans is large; and missing out the dot-files in the directory that du is run from (if doing, say, "du *" rather than "du ."). I've been bitten by both of those in the past.

Hugo.

The volume doesn't change hugely over time, so it really ought not to have broken so quickly - a quick rundown of the storage usage:

36% general (small files, some smallish videos)
24% music
23% pr0n
17% VMs

But in terms of 'large files changing', it could be the VM disks perhaps
- I'll move them out, balance, and then back in again, hopefully that'd be a meaningful test.

du-wise was direct on the root directory, any idea how I could audit orphan files?

Thanks

#3 lore raw

OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:

Quoted 10 lines

The volume doesn't change hugely over time, so it really ought not to have broken so quickly - a quick rundown of the storage usage:

36% general (small files, some smallish videos)
24% music 23% pr0n 17% VMs

But in terms of 'large files changing', it could be the VM disks perhaps -
I'll move them out, balance, and then back in again, hopefully that'd be a meaningful test.

VM image files (and large database files, for the same reason) are a bit of a problem on btrfs, and indeed, any COW-based filesystem, since the random rewrite pattern matching that use-case is pretty much the absolute worst-case match for a COW-based filesystem there is.

And that would be the worst-case in terms of the unsplit extents issue Hugo was talking about as well. So they may well be the problem, indeed.

Since you're not doing snapshotting (which conflicts with this option, with an imperfect workaround), setting nocow on those files may well eliminate the problem, but be aware if you aren't already that (1) nocow does turn off checksumming as well, in ordered to avoid a race that could easily lead to data corruption, and (2) you can't just activate nocow on the existing file and expect it to work, the procedure is a bit more complicated than that, since nocow is only guaranteed to work if it's set at file creation. Detailed instructions for #2 skipped as they've been posted many times, but if you are interested and haven't seen them, ask.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
author
#4 lore raw
Quoted 21 lines · Duncan

OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:

Quoted 10 lines

The volume doesn't change hugely over time, so it really ought not to have broken so quickly - a quick rundown of the storage usage:

36% general (small files, some smallish videos)
24% music 23% pr0n 17% VMs

But in terms of 'large files changing', it could be the VM disks perhaps -
I'll move them out, balance, and then back in again, hopefully that'd be a meaningful test.

VM image files (and large database files, for the same reason) are a bit of a problem on btrfs, and indeed, any COW-based filesystem, since the random rewrite pattern matching that use-case is pretty much the absolute worst-case match for a COW-based filesystem there is.

And that would be the worst-case in terms of the unsplit extents issue Hugo was talking about as well. So they may well be the problem, indeed.

Quoted 9 lines

Since you're not doing snapshotting (which conflicts with this option, with an imperfect workaround), setting nocow on those files may well eliminate the problem, but be aware if you aren't already that (1) nocow does turn off checksumming as well, in ordered to avoid a race that could easily lead to data corruption, and (2) you can't just activate nocow on the existing file and expect it to work, the procedure is a bit more complicated than that, since nocow is only guaranteed to work if it's set at file creation. Detailed instructions for #2 skipped as they've been posted many times, but if you are interested and haven't seen them, ask.

Thankyou Hugo, Duncan - moving VMs out, then:

=====================================================

sudo chattr +C '/mnt/storage-1/Virtual Machines' sudo btrfs balance start -mconvert=dup /mnt/storage-1 sudo btrfs fi defrag -rv /mnt/storage-1

=====================================================

And after moving VMs back, du reports 884GB used,

=====================================================

$ sudo btrfs fi usage /mnt/storage-1

Overall:

    Device size:		 953.87GiB
    Device allocated:		 924.07GiB
    Device unallocated:		  29.80GiB
    Device missing:		     0.00B
    Used:			 886.11GiB
    Free (estimated):		  65.72GiB	(min: 50.82GiB)
    Data ratio:			      1.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:918.01GiB, Used:882.09GiB
/dev/sdb 918.01GiB

Metadata,DUP: Size:3.00GiB, Used:2.01GiB
/dev/sdb 6.00GiB

System,DUP: Size:32.00MiB, Used:128.00KiB
/dev/sdb 64.00MiB

Unallocated:
/dev/sdb 29.80GiB

=====================================================

So a couple of gig still unaccountable but irrelevant. Thanks, problem solved! Although hopefully checksumming will be allowed on nocow files in the future as thats currently 17% of all data unprotected and will get worse...

#5 lore raw
Quoted 4 lines · OmegaPhil

So a couple of gig still unaccountable but irrelevant. Thanks, problem solved! Although hopefully checksumming will be allowed on nocow files in the future as thats currently 17% of all data unprotected and will get worse...

It's not likely to happen. If you're modifying data in-place (as is
the case on nodatacow files), then there's a window of vulnerability between modifying the data and modifying the checksum records during which the checksums are incorrect. This breaks the consistency guarantees of the FS.

Hugo.

Signature · 4 lines
Hugo Mills             | Nostalgia isn't what it used to be.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |
#6 lore raw
In reply to OmegaPhil
Quoted 19 lines · OmegaPhil
Quoted 9 lines · Duncan

OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
VM image files (and large database files, for the same reason) are a bit
of a problem on btrfs, and indeed, any COW-based filesystem, since the
random rewrite pattern matching that use-case is pretty much the absolute
worst-case match for a COW-based filesystem there is. Since you're not doing snapshotting (which conflicts with this

Quoted 8 lines

option,
with an imperfect workaround), setting nocow on those files may well
eliminate the problem, but be aware if you aren't already that (1) nocow
does turn off checksumming as well, in ordered to avoid a race that could
easily lead to data corruption, and (2) you can't just activate

Quoted 6 lines

So a couple of gig still unaccountable but irrelevant. Thanks, problem
solved! Although hopefully checksumming will be allowed on nocow files
in the future as thats currently 17% of all data unprotected and will get worse...

There's actually an interesting workaround to this: Although the VM disk images aren't checksummed on the host filesystem, you can use btrfs *inside* the VMs and enable checksumming there. The downside is that you can only verify the VM data by booting the VM and running a scrub from inside.

This of course doesn't help if your VMs are Windows or legacy versions of Linux without btrfs support. On BSD you could try ZFS.

Signature · 1 line
Calvin Walton <[email protected]>
#7 lore raw
Quoted 27 lines · Calvin Walton
Quoted 19 lines · OmegaPhil
Quoted 9 lines · Duncan

OmegaPhil posted on Wed, 04 Nov 2015 21:53:09 +0000 as excerpted:
VM image files (and large database files, for the same reason) are a bit
of a problem on btrfs, and indeed, any COW-based filesystem, since the
random rewrite pattern matching that use-case is pretty much the absolute
worst-case match for a COW-based filesystem there is. Since you're not doing snapshotting (which conflicts with this

Quoted 8 lines

option,
with an imperfect workaround), setting nocow on those files may well
eliminate the problem, but be aware if you aren't already that (1) nocow
does turn off checksumming as well, in ordered to avoid a race that could
easily lead to data corruption, and (2) you can't just activate

Quoted 6 lines

So a couple of gig still unaccountable but irrelevant. Thanks, problem
solved! Although hopefully checksumming will be allowed on nocow files
in the future as thats currently 17% of all data unprotected and will get worse...

Quoted 5 lines

There's actually an interesting workaround to this: Although the VM disk images aren't checksummed on the host filesystem, you can use btrfs *inside* the VMs and enable checksumming there. The downside is that you can only verify the VM data by booting the VM and running a scrub from inside.

Actually, by using a combination of loop devices and kpartx, it's fully possible to mount the FS and verify it without booting the VM. Of course, doing this usually requires root access on the host system, but for most people I know, that's usually not an issue. I do this on occasion when I need to pull a file off of one of my VM disks on my laptop and don't have the time to spin up the VM itself.

Another option if you're doing a direct boot of the kernel (for example, when using a fully paravirtualized domain on Xen, or using some of the QEMU ARM systems) is to just do the volume management (partitioning and such) on the host, and expose each filesystem to the guest as a separate disk. I do this with most of my Linux VM's on my Xen system where I use LVM as the back-end storage for the virtual disk images, as it allows me to easily directly mount the VM's filesystems on the host if need be (and let's you do all kinds of cool things like using a cluster-aware filesystem for the VM's root so that you can mount it from the host safely while the VM is still online).