Big disk space usage difference, even after defrag, on identical data

Linux mozwell 3.19.0-trunk-amd64 #1 SMP Debian 3.19.1-1~exp1 (2015-03-08) x86_64 GNU/Linux
btrfs-progs v3.19.1

I have a btrfs volume that's been in use for a week or 2. It has about ~560G of uncompressible data (video files, tar.xz, git repos, ...) and ~200G of data that compresses 2:1 with LZO (PostgreSQL db).

It's split into 2 subvolumes:
ID 257 gen 6550 top level 5 path @db
ID 258 gen 6590 top level 5 path @large

and mounted like this:
/dev/sdc /srv/db btrfs rw,noatime,compress=lzo,space_cache 0 0 /dev/sdc /srv/large btrfs rw,noatime,compress=lzo,space_cache 0 0

du -skh /srv
768G /srv

df -h

/dev/sdc        1.4T  754G  641G  55% /srv/db
/dev/sdc        1.4T  754G  641G  55% /srv/large

btrfs fi df /srv/large
Data, single: total=808.01GiB, used=749.36GiB System, DUP: total=8.00MiB, used=112.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=3.50GiB, used=1.87GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B

So that's a bit bigger than perhaps expected (~750G instead of ~660G+metadata). I thought it might've been related to compress bailing out too easily, but I've done a
btrfs fi defragment -r -v -clzo /srv/db /srv/large
and this doesn't change anything.

I recently copied this data to a new, bigger disk, and the result looks worrying:

mount options:
/dev/sdd /mnt/large btrfs rw,noatime,compress=lzo,space_cache 0 0 /dev/sdd /mnt/db btrfs rw,noatime,compress=lzo,space_cache 0 0

btrfs fi df
Data, single: total=684.00GiB, used=683.00GiB System, DUP: total=8.00MiB, used=96.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=3.50GiB, used=2.04GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B

df

/dev/sdd        3.7T  688G  3.0T  19% /mnt/large
/dev/sdd        3.7T  688G  3.0T  19% /mnt/db

du
767G /mnt

That's a 66G difference for the same data with the same compress option. The used size here is much more in line with what I'd have expected given the nature of the data.

I would think that compression differences or things like fragmentation or bookending for modified files shouldn't affect this, because the first filesystem has been defragmented/recompressed and didn't shrink.

So what can explain this? Where did the 66G go?

GCP

There are a few places: the kernel may have decided your files are not compressible and disabled compression on them (some older kernels did this with great enthusiasm); your files might have preallocated space from the fallocate system call (which disables compression and allocates
contiguous space, so defrag will not touch it). 'filefrag -v' can
tell you if this is happening to your files.

In practice database files take about double the amount of space they appear to because of extent shingling.

Suppose we have a defragmented file with one extent "A" like this:

0 MB AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 1MB

Now we overwrite about half of the blocks:

0 MB BBBBBBBBBBBBBBBBAAAAAAAAAAAAAAAA 1MB

btrfs tracks references to the entire extent, so what is on disk now is this:

0 MB aaaaaaaaaaaaaaaaAAAAAAAAAAAAAAAA 1MB original extent
0 MB BBBBBBBBBBBBBBBB 1MB new extent

The "a" are blocks from the original extent that are not visible in the file, but remain present on disk. In other words, this 1MB file is now taking up 1.5MB of space.

This continues as long as any blocks of partially overwritten extents are visible in any file (including snapshots, dedup, and clones), with the worst case being something like this:

0 MB BBBBBBBBBBBBBCCCCCCCCCCCCCCDDDDA 1MB

which could be like this on disk:

0 MB aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaA 1MB first extent
0 MB BBBBBBBBBBBBBbbb 1MB second extent
0 MB CCCCCCCCCCCCCCcccc 1MB third extent
0 MB DDDD 1MB fourth extent

This 1MB file takes up a little over 2MB of disk space, and there are parts of extents A, B, and C which persist on disk but are no longer part of any file's content.

In this case, if we wrote the last 4K of the file, we would free 1MB of disk space by doing so:

(extent A now deleted)
0 MB BBBBBBBBBBBBBbbb 1MB second extent
0 MB CCCCCCCCCCCCCCcccc 1MB third extent
0 MB DDDD 1MB fourth extent
0 MB E 1MB fifth extent

Similarly to free the "B" extent we have to overwrite all the visible blocks, i.e. from 0 to the beginning of the "C" extent, before the last visible block from "B" is destroyed and the entire "B" extent can be freed.

The worst case is pretty bad: with the worst possible overwrite pattern, a file can occupy the square of its size on disk divided by the block size (4K) divided by two. That's a little under 128MB for a 1MB file, or 128TB for a 1GB file. Above 1GB, the scaling is linear instead of quadratic because the extent size limit (1G) has been reached and single-extent files are no longer possible (so a worst-case 2GB file takes only 256TB of space instead of 512TB).

Defragmenting the files helps free space temporarily; however, space usage will quickly grow again until it returns to the steady state around 2x the file size.

A database ends up maxing out at about a factor of two space usage because it tends to write short uniform-sized bursts of pages randomly, so we get a pattern a bit like bricks in a wall:

0 MB AA BB CC DD EE FF GG HH II JJ KK 1 MB half the extents 0 MB LL MM NN OO PP QQ RR SS TT UU V 1 MB the other half

0 MB ALLBMMCNNDOOEPPFQQGRRHSSITTJUUKV 1 MB what the file looks like

Fixing this is non-trivial (it may require an incompatible disk format change). Until this is fixed, the most space-efficient approach seems to be to force compression (so the maximum extent is 128K instead of 1GB) and never defragment database files ever.

Zygo Blaxell posted on Mon, 13 Apr 2015 00:04:36 -0400 as excerpted:

... Or set the database file nocow at creation, and don't snapshot it, so overwrites are always in-place. (Btrfs compression and checksumming get turned off with nocow, but as we've seen, compression isn't all that effective on random-rewrite-pattern files anyway, and databases generally have their own data integrity handling, so neither one is a huge loss, and the in-place rewrite makes for better performance and a more predictable steady-state.)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

As stated in the previous mail, this is 3.19.1. Moreover, the data is either uniformly compressible or not at all. Lastly, note that the *exact same* mount options are being used on *the exact same kernel* with *the exact same data*. Getting a different compressible decision given the same inputs would point to bugs.

So defrag -clzo or -czlib won't actually re-compress mostly-continuous files? That's evil. I have no idea whether PostgreSQL allocates files that way, though.

Not sure how to interpret that. Without "-v", I see most of the (DB) data has 2-5 extents per Gigabyte. A few have 8192 extents per Gigabyte.

Comparing to the copy that takes 66G less, there every (compressible) file has about 8192 extents per Gigabyte, and the others 5 or 6.

So you may be right that some DB files are "wedged" in a format that btrfs can't compress. I forced the files to be rewritten (VACUUM FULL) and that "fixed" the problem.

This is what I called "bookending" in the original mail, I didn't know the correct name, but I understand doing updates can result in N^2/2 or thereabouts disk space usage, however:

As stated in the original mail, the filesystem was *freshly defragmented* so that can't have been the cause.

Would that fix the problem with fallocated() files?

GCP

Gian-Carlo Pascutto posted on Sat, 11 Apr 2015 21:59:50 +0200 as excerpted:

Out of curiosity, does a balance on the actively used btrfs help?

You mentioned defrag -v -r -clzo, but didn't use the -f (flush) or -t (minimum size file) options. Does adding -f -t1 help?

You aren't doing btrfs snapshots of either subvolume, are you?

I'm not sure this is related to the answer to your question, since you did defrag, but it might be, and it's good to know when dealing with database files on btrfs in any case.

Btrfs is in general a copy-on-write (COW) based filesystem. Random rewrite pattern files, database and VM image files being prime examples, typically HEAVILY fragment on COW filesystems, since any rewrite forces a copy of the rewritten data block elsewhere. The often rather large original extents get holes, but remained pinned by the existing data still remaining in them that hasn't been rewritten. This is analogous to the way databases often rewrite records but leave holes behind that aren't immediately cleaned up, only it's occurring at the filesystem extent level. Only after all the data in an extent has been rewritten, can the extent itself be unpinned and returned to the free space pool.

Defrag should force the rewrite of entire files and take care of this, but obviously it's not returning to "clean" state. I forgot what the default minimum file size is if -t isn't set, maybe 128 MiB? But a -t1 will force it to defrag even small files, and I recall at least one thread here where the poster said it made all the difference for him, so try that. And the -f should force a filesystem sync afterward, so you know the numbers from any report you run afterward match the final state.

Meanwhile, you may consider using the nocow attribute on those database files. It will disable compression on them, but rewrites should then occur in-place, so you don't get the fragmentation and extent usage holes and duplication that you'd have otherwise. It'll also disable btrfs checksumming, but mature databases already have their own error detection and correction system, since they don't normally run on filesystems that provide that sort of service like btrfs does. While initial usage will be higher due to the lack of compression, as you've discovered, over time, on an actively updated database, compression isn't all that effective anyway. And while usage may be a bit higher at least originally, it should be stable, but for expanding the actual size of the database, anyway.

But there's a couple of caveats to nocow. First, in ordered to be properly effective, it needs to be set on a file while it's still empty. The most effective way to do this is to set nocow on the empty parent directory, then copy the nocow-target files into it so they inherit the nocow attribute as they are created, before they actually have any data.

The second pertains to btrfs snapshots. Snapshots lock the existing file in place, effectively making an otherwise nocow file cow1 -- the first write to an existing file block will cow it, but after that, further writes to the same block will rewrite in-place... until the next snapshot, of course. So try to minimize the number of snapshots done to nocow files, and if you do snapshot them, defrag them once in awhile as well.

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Unfortunately I can no longer try this, see the other reply why. But the problem turned out to be some 1G-sized files, written using 3-5 extents, that for whatever reason defrag was not touching.

No :-) I should've mentioned that.

Reading the corresponding manual, the -t explanation says that "any extent bigger than this size will be considered already defragged". So I guess setting -t1 might've fixed the problem too...but after checking the source, I'm not so sure.

I didn't find the -t default in the manpages - after browsing through the source, the default is in the kernel:
https://github.com/torvalds/linux/blob/4f671fe2f9523a1ea206f63fe60a7c7b3a56d5c7/fs/btrfs/ioctl.c#L1268
(Not sure what units those are.)

I wonder if this is relevant:
https://github.com/torvalds/linux/blob/4f671fe2f9523a1ea206f63fe60a7c7b3a56d5c7/fs/btrfs/ioctl.c#L2572

This seems to reset the -t flag if compress (-c) is set? This looks a bit fishy?

I'm using btrfs specifically to get compression, so this isn't an option.

I don't see why. If you're referring to the additional overhead of continuously compressing and decompressing everything - yes, of course. But in my case I have a mostly-append workload to a huge amount of fairly compressible data that's on magnetic storage, so compression is a win in disk space and perhaps even in performance.

I'm well aware of the many caveats in using btrfs for databases - they're well documented and although I much appreciate your extended explanation, it wasn't new to me.

It turns out that if your dataset isn't update heavy (so it doesn't fragment to begin with), or has to be queried via indexed access (i.e. mostly via random seeks), the fragmentation doesn't matter much anyway. Conversely, btrfs appears to have better sync performance with multiple threads, and allows one to disable part of the partial-page-write protection logic in the database (full_page_writes=off for PostgreSQL), because btrfs is already doing the COW to ensure those can't actually happen [1].

The net result is a *boost* from about 40 tps (ext4) to 55 tps (btrfs), which certainly is contrary to popular wisdom. Maybe btrfs would fall off eventually as fragementation does set in gradually, but given that there's an offline defragmentation tool that can run in the background, I don't care.

[1] I wouldn't be too surprised if database COW, which consists of journal-writing a copy of the data out of band, then rewriting it again in the original place, is actually functionally equivalent do disabling COW in the database and running btrfs + defrag. Obviously you shouldn't keep COW enabled in btrfs *AND* the DB, requiring all data to be copied around at least 3 times...which I'm afraid almost everyone does because it's the default...

GCP

There are several corner cases that defrag won't touch by default. It's designed to be conservative and favor speed over size.

Also when the kernel decides you're not getting enough compression, it seems to disable compression on the file _forever_ even if future writes are compressible again. mount -o compress-force works around that.

read-only snapshots: yet another thing defrag won't touch.

Short writes won't compress--not just well, but at all--because btrfs won't look at adjacent already-written blocks. If you write a file at less than 4K/minute, there will be no compression, as each new extent (or replacement extent for overwritten data) is already minimum-sized.

If you write in bursts of 128K or more, consecutively, then you can get compression benefit.

There has been talk of teaching autodefrag to roll up the last few dozen extents of files that grow slowly so they can be compressed.

I've found the performance of PostgreSQL to be wildly variable on btrfs. It may be OK at first, but watch it for a week or two to admire the full four-orders-of-magnitude swing (100 tps to 0.01 tps). :-O

Journalling writes all the data twice: once to the journal, once to update the origin page after the journal (though PostgreSQL will omit some of those duplicate writes in cases where there is no origin page to overwrite).

COW writes all the new and updated data only once.

In the event of a crash, if the log tree is not recoverable (and it's a rich source of btrfs bugs, so it's often not), you lose everything that happened to the database in the last 30 seconds. If you were already using async commit in PostgreSQL anyway then that's not much of a concern (and not having to call fsync 100 times a second _really_ helps performance!) but if you really need sync commit then btrfs is not the filesystem for you.

Gian-Carlo Pascutto posted on Mon, 13 Apr 2015 16:06:39 +0200 as excerpted:

Oops! You are correct. There was an on-list discussion of that before that I had forgotten. The "make sure everything gets defragged" magic setting is -t 1G or higher, *not* the -t 1 I was trying to tell you previously (which will end up skipping everything, instead of defragging everything).

Thanks for spotting the inconsistency and calling me on it! =:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman