efficiency of btrfs cow
I have a backup volume on an ext4 filesystem that is using rsync and it's --link-dest option to create "hard-linked incremental" backups. I am sure everyone here is familiar with the technique but in case anyone isn't basically it's effectively doing (each backup):
# cp -al /backup/previous-backup/ /backup/current-backup # rsync -aAHX ... --exclude /backup / /backup/current-backup
The shortcoming of this of course is that it just takes 1 byte in a (possibly huge) file to require that the whole file be recopied to the backup.
btrfs and it's CoW capability to the rescue -- again, no surprise to anyone here.
So I replicated a few of the directories in my backup volume to a btrfs volume using snapshots for each backup to take advantage of CoW and with any luck, avoid entire file duplication where only some subset of the file has changed.
Overall, it seems that I saw success. Most backups on btrfs were smaller than their source, and overall, for all of the backups replicated, the use was less. There were some however that were significantly larger. Here's the analysis:
Backup btrfs ext4
------ ----- ----
monthly.22: 112GiB 113GiB 98%
monthly.21: 14GiB 14GiB 95%
monthly.20: 19GiB 20GiB 94%
monthly.19: 12GiB 13GiB 94%
monthly.18: 5GiB 6GiB 87%
monthly.17: 11GiB 12GiB 92%
monthly.16: 8GiB 10GiB 82%
monthly.15: 16GiB 11GiB 146%
monthly.14: 19GiB 20GiB 94%
monthly.13: 21GiB 22GiB 96%
monthly.12: 61GiB 67GiB 91%
monthly.11: 24GiB 22GiB 106%
monthly.10: 22GiB 19GiB 114%
monthly.9: 12GiB 13GiB 90%
monthly.8: 15GiB 17GiB 91%
monthly.7: 9GiB 11GiB 87%
monthly.6: 8GiB 9GiB 85%
monthly.5: 16GiB 18GiB 91%
monthly.4: 13GiB 15GiB 89%
monthly.3: 11GiB 19GiB 62%
monthly.2: 29GiB 22GiB 134%
monthly.1: 23GiB 24GiB 94%
monthly.0: 5GiB 5GiB 94%
Total: 497GiB 512GiB 96%btrfs use is a calculation of the "df" value of the fileystem before and after each backup. ext4 (rsync, really) use is calculated with "du -xks" on the whole backup volume, which as you know only counts a multiply hard-linked file's space use once.
So as you can see, for the most part, btrfs and CoW was more efficient, but in some cases (i.e. monthly.15, monthly.11, monthly.10, monthly.2) it was less efficient.
Taking the biggest anomaly, monthly.15, a du of just that directory on both the btrfs and ext4 filesystems shows results I would expect:
btrfs: 136,876,580 monthly.15 ext4: 142,153,928 monthly.15
Yet the before and after "df" results show the btrfs usage higher than ext4. Is there some "periodic" jump in "overhead" used by btrfs that would account for this mysterious increased usage in some of the copies?
Any other ideas for the anomalous results?
Cheers,
b.
Quoted 6 lines · Brian J. Murrell
# cp -al /backup/previous-backup/ /backup/current-backup # rsync -aAHX ... --exclude /backup / /backup/current-backupThe shortcoming of this of course is that it just takes 1 byte in a (possibly huge) file to require that the whole file be recopied to the backup.
If you have snapshots anyway, why not :
- create a snapshot before each backup run
- use the same directory (e.g. just /backup), no need to "cp" anything
- add "--inplace" to rsync
Signature · 1 line
Fajar
Quoted 5 lines · Fajar A. Nugraha
If you have snapshots anyway, why not :
- create a snapshot before each backup run
- use the same directory (e.g. just /backup), no need to "cp" anything
- add "--inplace" to rsync
Which is exactly what I am doing. There is no "cp" involved in making the btrfs copies of the existing backup. It's simply "rsync -aAXH ... --inplace" from the existing backup archive to the new, btrfs archive.
Cheers,
b.
Quoted 4 lines · Fajar A. Nugraha
Quoted 3 lines · Brian J. Murrell
# cp -al /backup/previous-backup/ /backup/current-backup # rsync -aAHX ... --exclude /backup / /backup/current-backup
Quoted 8 lines
Quoted 3 lines
The shortcoming of this of course is that it just takes 1 byte in a (possibly huge) file to require that the whole file be recopied to the backup.
If you have snapshots anyway, why not :
- create a snapshot before each backup run
- use the same directory (e.g. just /backup), no need to "cp" anything
- add "--inplace" to rsync
To add a bit to this: if you *do not* use the --inplace option on rsync, rsync will rewrite the entire file, instead of updating the existing file!
This of course negates some of the benefits of btrfs's COW support when doing incremental backups.
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 4 lines · Calvin Walton
To add a bit to this: if you *do not* use the --inplace option on rsync, rsync will rewrite the entire file, instead of updating the existing file!
Of course. As I mentioned to Fajar previously, I am indeed using --inplace when copying from the existing archive to the new btrfs archive.
Quoted 2 lines
This of course negates some of the benefits of btrfs's COW support when doing incremental backups.
Absolutely.
b.
Quoted 4 lines · Fajar A. Nugraha
Quoted 3 lines · Brian J. Murrell
# cp -al /backup/previous-backup/ /backup/current-backup # rsync -aAHX ... --exclude /backup / /backup/current-backup
Quoted 8 lines
Quoted 3 lines
The shortcoming of this of course is that it just takes 1 byte in a (possibly huge) file to require that the whole file be recopied to the backup.
If you have snapshots anyway, why not :
- create a snapshot before each backup run
- use the same directory (e.g. just /backup), no need to "cp" anything
- add "--inplace" to rsync
You may also want to test with/without --no-whole-file as well. That's most useful when the two filesystems are on the same system and should reduce the amount of data copied around, as it forces rsync to only use file deltas. This is very much a win on ZFS, which is also CoW, so it should be a win on Btrfs.
Signature · 2 lines
Freddie Cash [email protected]
Quoted 18 lines · Brian J. Murrell
I have a backup volume on an ext4 filesystem that is using rsync and it's --link-dest option to create "hard-linked incremental" backups. I am sure everyone here is familiar with the technique but in case anyone isn't basically it's effectively doing (each backup):
So I replicated a few of the directories in my backup volume to a btrfs volume using snapshots for each backup to take advantage of CoW and with any luck, avoid entire file duplication where only some subset of the file has changed.
Overall, it seems that I saw success. Most backups on btrfs were smaller than their source, and overall, for all of the backups replicated, the use was less. There were some however that were significantly larger. Here's the analysis:
Taking the biggest anomaly, monthly.15, a du of just that directory on both the btrfs and ext4 filesystems shows results I would expect:
Quoted 6 lines
btrfs: 136,876,580 monthly.15 ext4: 142,153,928 monthly.15Yet the before and after "df" results show the btrfs usage higher than ext4. Is there some "periodic" jump in "overhead" used by btrfs that would account for this mysterious increased usage in some of the copies?
There actually is such a periodic jump in overhead, caused by the way which btrfs dynamically allocates space for metadata as needed by the creation of new files, which it does whenever the free metadata space ratio reaches a threshold (it's probably more complicated than that, but close enough for now).
To see exactly what's going on, you should use the "btrfs filesystem df" command to see how space is being allocated for data and metadata separately:
ayu ~ # btrfs fi df /
Data: total=266.01GB, used=249.35GB
System, DUP: total=8.00MB, used=36.00KB
Metadata, DUP: total=3.62GB, used=1.93GB
ayu ~ # df -h /
Filesystem Size Used Avail Use% Mounted on /dev/sda4 402G 254G 145G 64% /
If you use the btrfs tool's df command to account for space in your testing, you should get much more accurate results.
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 2 lines · Calvin Walton
There actually is such a periodic jump in overhead,
Ahh. So my instincts were correct.
Quoted 5 lines
caused by the way
which btrfs dynamically allocates space for metadata as needed by the creation of new files, which it does whenever the free metadata space ratio reaches a threshold (it's probably more complicated than that, but close enough for now).
Sounds fair enough.
Quoted 9 lines
To see exactly what's going on, you should use the "btrfs filesystem df" command to see how space is being allocated for data and metadata separately:
ayu ~ # btrfs fi df /
Data: total=266.01GB, used=249.35GB
System, DUP: total=8.00MB, used=36.00KB
Metadata, DUP: total=3.62GB, used=1.93GB
ayu ~ # df -h /
Quoted 5 lines
Filesystem Size Used Avail Use% Mounted on /dev/sda4 402G 254G 145G 64% /If you use the btrfs tool's df command to account for space in your testing, you should get much more accurate results.
Indeed! Unfortunately that tool seems to be completely silent on my system:
# btrfs filesystem df /mnt/btrfs-test/ # btrfs filesystem df /mnt/btrfs-test
Where /mnt/btrfs-test is where I have the device that I created the btrfs filesystem on mounted. i.e.:
# grep btrfs /proc/mounts /dev/mapper/btrfs--test-btrfs--test /mnt/btrfs-test btrfs rw,relatime 0 0
My btrfs-tools appears to be from 20101101. The changelog says:
* Merging upstream version 0.19+20101101.
Cheers,
b.
Quoted 4 lines · Calvin Walton
To see exactly what's going on, you should use the "btrfs filesystem df" command to see how space is being allocated for data and metadata separately:
OK. So with an empty filesystem, before my first copy (i.e. the base on which the next copy will CoW from) df reports:
Filesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/btrfs--test-btrfs--test
922746880 56 922746824 1% /mnt/btrfs-test
and btrfs fi df reports:
Data: total=8.00MB, used=0.00
Metadata: total=1.01GB, used=24.00KB
System: total=12.00MB, used=4.00KB
after the first copy df and btrfs fi df report:
Filesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/btrfs--test-btrfs--test
922746880 121402328 801344552 14% /mnt/btrfs-test
root@linux:/mnt/btrfs-test# cat .snapshots/monthly.22/metadata/btrfs_df-stop Data: total=110.01GB, used=109.26GB Metadata: total=5.01GB, used=3.26GB System: total=12.00MB, used=24.00KB
So it's clear that total usage (as reported by df) was 121,402,328KB but Metadata has two values:
Metadata: total=5.01GB, used=3.26GB
What's the difference between total and used? And for that matter, what's the difference between the total and used for Data (total=110.01GB, used=109.26GB)?
Even if I take the largest values (i.e. the total values) for Data and Metadata (each converted to KB first) and add them up they are:
120,607,211.52 which is not quite the 121,402,328 that df reports. There is a 795,116.48KB discrepancy.
In any case, which value from a btrfs df fi should I be subtracting from df's accounting to get a real accounting of the amount of data used?
Cheers,
b.
I'm not a developer, but I think it goes something like this:
btrfs doesn't write the filesystem on the entire device/partition at format time, rather, it dynamically increases the size of the filesystem as data is used. That's why formating a disk in btrfs can be so fast.
Quoted 4 lines · Brian J. Murrell
Quoted 2 lines · Calvin Walton
To see exactly what's going on, you should use the "btrfs filesyste=
m df"
Quoted 4 lines
Quoted 2 lines
command to see how space is being allocated for data and metadata separately:
OK. =A0So with an empty filesystem, before my first copy (i.e. the ba=
se on
Quoted 3 lines
which the next copy will CoW from) df reports:
Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available Us=
e% Mounted on
Quoted 2 lines
/dev/mapper/btrfs--test-btrfs--test
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 922746880 =A0 =A0 =A0 =A056 9=
22746824 =A0 1% /mnt/btrfs-test
Quoted 10 lines
and btrfs fi df reports:
Data: total=3D8.00MB, used=3D0.00
Metadata: total=3D1.01GB, used=3D24.00KB
System: total=3D12.00MB, used=3D4.00KBafter the first copy df and btrfs fi df report:
Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available Us=
e% Mounted on
Quoted 2 lines
/dev/mapper/btrfs--test-btrfs--test
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 922746880 121402328 801344552=
=A014% /mnt/btrfs-test
Quoted 2 lines
root@linux:/mnt/btrfs-test# cat .snapshots/monthly.22/metadata/btrfs_=
df-stop
Quoted 5 lines
Data: total=3D110.01GB, used=3D109.26GB
Metadata: total=3D5.01GB, used=3D3.26GB
System: total=3D12.00MB, used=3D24.00KBSo it's clear that total usage (as reported by df) was 121,402,328KB =
but
Quoted 9 lines
Metadata has two values:
Metadata: total=3D5.01GB, used=3D3.26GB
What's the difference between total and used? =A0And for that matter, what's the difference between the total and used for Data (total=3D110.01GB, used=3D109.26GB)?
Even if I take the largest values (i.e. the total values) for Data an=
d
Quoted 5 lines
Metadata (each converted to KB first) and add them up they are:
120,607,211.52 which is not quite the 121,402,328 that df reports. There is a 795,116.48KB discrepancy.In any case, which value from a btrfs df fi should I be subtracting f=
rom
Quoted 5 lines
df's accounting to get a real accounting of the amount of data used?
Cheers,
b.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Chester
I'm not a developer, but I think it goes something like this:
btrfs doesn't write the filesystem on the entire device/partition at format time, rather, it dynamically increases the size of the filesystem as data is used. That's why formating a disk in btrfs can be so fast.
Indeed, this much is understood, which is why I am using btrfs fi df to try to determine how much of the increase in raw device usage is the dynamic allocation of metadata.
Cheers,
b.
Quoted 1 line
So it's clear that total usage (as reported by df) was 121,402,328KB =
but
Quoted 5 lines
Metadata has two values:
Metadata: total=3D5.01GB, used=3D3.26GB
What's the difference between total and used? =C2=A0And for that matt=
er,
Quoted 3 lines
what's the difference between the total and used for Data (total=3D110.01GB, used=3D109.26GB)?
total is the space allocated (reserved) for a kind usage (metadata or data) the space allocated for a kind of usage can't be used for something else. The used value is the space that is used from the space that has been allocated for a kind of usage.
The wiki gives you a overview how to interpret the values:
https://btrfs.wiki.kernel.org/index.php/FAQ#btrfs_filesystem_df_.2Fmoun=
tpoint
cheers Kolja.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html