Why so much "btrfs send" data for "cp -a --reflink"?
Hallöchen!
I have two subvolumes A and B. „A“ contains 50GB data, B is empty. None is a snapshot of the other. Now, I copy all data from A to B with "cp -a --reflink A/* B". This copying takes less than a second. So apparently, no bulk data was duplicated. "diff -rq A B" is empty. So far, so good.
However, it surprises me that
btrfs send -p A B | wc -c
reports 12GB. I would have hoped for very few data (say, a couple of MBs). Or, the whole 50GB (because A is not a real parent of B, and never has been). An additional "-c A" does not change anything.
Why is this? In other words, what comprises those 12GB?
It may be insignificant, but the 50GB are almost fully a single VirtualBox .vdi file, somewhat fragmented (filefrag says 16000).
Regards,
Torsten.
Signature · 1 line
Torsten Bronger
Hallöchen!
Quoted 11 lines · Torsten Bronger
I have two subvolumes A and B. „A“ contains 50GB data, B is empty. None is a snapshot of the other. Now, I copy all data from A to B with "cp -a --reflink A/* B". This copying takes less than a second. So apparently, no bulk data was duplicated. "diff -rq A B" is empty. So far, so good.
However, it surprises me that
btrfs send -p A B | wc -c
reports 12GB.
At <https://wilson.bronger.org/btrfs-receive.dump.xz>, I have uploaded the output of "btrfs receive --dump". Apparently, there are many "write". Why doesn’t btrfs detect that all those extents are present in the parent?
Regards,
Torsten.
Signature · 1 line
Torsten Bronger
Quoted 13 lines · Torsten Bronger
Hallöchen!
Quoted 8 lines · Torsten Bronger
I have two subvolumes A and B. „A“ contains 50GB data, B is empty. None is a snapshot of the other. Now, I copy all data from A to B with "cp -a --reflink A/* B". This copying takes less than a second. So apparently, no bulk data was duplicated. "diff -rq A B" is empty. So far, so good.
However, it surprises me that
Quoted 8 lines
Quoted 3 lines
btrfs send -p A B | wc -c
reports 12GB.
At <https://wilson.bronger.org/btrfs-receive.dump.xz>, I have uploaded the output of "btrfs receive --dump". Apparently, there are many "write". Why doesn’t btrfs detect that all those extents are present in the parent?
I haven't looked at the dump, and don't have the time to do so right now.
But since the files are VM images, very likely what you are seeing are writes full of zero bytes.
This is because the current send protocol does not have support holes, instead it issues write operations with a bunch of zeros. And holes are very common in VM images in general.
Just look at a few write commands, grab the file name, offset and length, then check if after you read the corresponding file range you get only zeros.
If so, then it has nothing to do with deduplication/reflinks, just the lack of support for hole punching commands in the send stream.
Cheers.
Quoted 6 lines
Regards,
Torsten.--
Torsten Bronger
Signature · 3 lines
Filipe David Manana, “Whether you think you can, or you think you can't — you're right.”
Hallöchen!
Quoted 7 lines · Filipe Manana
[...]
But since the files are VM images, very likely what you are seeing are writes full of zero bytes. This is because the current send protocol does not have support holes, instead it issues write operations with a bunch of zeros. And holes are very common in VM images in general.
Thank you for your answer! It is plausible: I created a 10G file with random content, reflink-copied it, and the "btrfs send" is almost nothing.
Quoted 3 lines
Just look at a few write commands, grab the file name, offset and length, then check if after you read the corresponding file range you get only zeros.
I haven’t tested it, but the "btrfs send" produces 12G, 1.5G of which are zeroes, and the source file contains 6.8GB of zeroes. Can this be the holes nevertheless?
Regards,
Torsten.
Signature · 1 line
Torsten Bronger