Skip to contentSkip to conversation

Why so much "btrfs send" data for "cp -a --reflink"?

DISCUSSION Torsten Bronger 2 participants 4 messages Oct 5, 2020 – Oct 6, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hallöchen!

I have two subvolumes A and B. „A“ contains 50GB data, B is empty. None is a snapshot of the other. Now, I copy all data from A to B with "cp -a --reflink A/* B". This copying takes less than a second. So apparently, no bulk data was duplicated. "diff -rq A B" is empty. So far, so good.

However, it surprises me that

btrfs send -p A B | wc -c

reports 12GB. I would have hoped for very few data (say, a couple of MBs). Or, the whole 50GB (because A is not a real parent of B, and never has been). An additional "-c A" does not change anything.

Why is this? In other words, what comprises those 12GB?

It may be insignificant, but the 50GB are almost fully a single VirtualBox .vdi file, somewhat fragmented (filefrag says 16000).

Regards,
Torsten.

Signature · 1 line
Torsten Bronger
#1 lore raw

Hallöchen!

Quoted 11 lines · Torsten Bronger

I have two subvolumes A and B. „A“ contains 50GB data, B is empty. None is a snapshot of the other. Now, I copy all data from A to B with "cp -a --reflink A/* B". This copying takes less than a second. So apparently, no bulk data was duplicated. "diff -rq A B" is empty. So far, so good.

However, it surprises me that

btrfs send -p A B | wc -c

reports 12GB.

At <https://wilson.bronger.org/btrfs-receive.dump.xz>, I have uploaded the output of "btrfs receive --dump". Apparently, there are many "write". Why doesn’t btrfs detect that all those extents are present in the parent?

Regards,
Torsten.

Signature · 1 line
Torsten Bronger
#2 lore raw
Quoted 13 lines · Torsten Bronger

Hallöchen!

Quoted 8 lines · Torsten Bronger

I have two subvolumes A and B. „A“ contains 50GB data, B is empty. None is a snapshot of the other. Now, I copy all data from A to B with "cp -a --reflink A/* B". This copying takes less than a second. So apparently, no bulk data was duplicated. "diff -rq A B" is empty. So far, so good.

However, it surprises me that

Quoted 8 lines
Quoted 3 lines

btrfs send -p A B | wc -c

reports 12GB.

At <https://wilson.bronger.org/btrfs-receive.dump.xz>, I have uploaded the output of "btrfs receive --dump". Apparently, there are many "write". Why doesn’t btrfs detect that all those extents are present in the parent?

I haven't looked at the dump, and don't have the time to do so right now.

But since the files are VM images, very likely what you are seeing are writes full of zero bytes.
This is because the current send protocol does not have support holes, instead it issues write operations with a bunch of zeros. And holes are very common in VM images in general.

Just look at a few write commands, grab the file name, offset and length, then check if after you read the corresponding file range you get only zeros.
If so, then it has nothing to do with deduplication/reflinks, just the lack of support for hole punching commands in the send stream.

Cheers.

Quoted 6 lines

Regards,
Torsten.

--
Torsten Bronger

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#3 lore raw

Hallöchen!

Quoted 7 lines · Filipe Manana

[...]

But since the files are VM images, very likely what you are seeing are writes full of zero bytes. This is because the current send protocol does not have support holes, instead it issues write operations with a bunch of zeros. And holes are very common in VM images in general.

Thank you for your answer! It is plausible: I created a 10G file with random content, reflink-copied it, and the "btrfs send" is almost nothing.

Quoted 3 lines

Just look at a few write commands, grab the file name, offset and length, then check if after you read the corresponding file range you get only zeros.

I haven’t tested it, but the "btrfs send" produces 12G, 1.5G of which are zeroes, and the source file contains 6.8GB of zeroes. Can this be the holes nevertheless?

Regards,
Torsten.

Signature · 1 line
Torsten Bronger