btrfs: obtain block checksums from user space
Hello,
I would like to read the list of the checksums for the specific file stored onto btrfs filesystem. I think I could use the checksums in the manner like rsync does, but safe both CPU (because csums are already calculated for the file) and I/O (because I don't need to reread all the file from the hard drive).
I've looked through linux kernel sources and not found appropriate ioctl to do this. Frankly speaking, I've not found good documentations for all available btrfs ioctls.
Quoted 8 lines · Matwey V. Kornilov
Hello,
I would like to read the list of the checksums for the specific file stored onto btrfs filesystem. I think I could use the checksums in the manner like rsync does, but safe both CPU (because csums are already calculated for the file) and I/O (because I don't need to reread all the file from the hard drive).
As of right now, there is no way to do this from userspace without just directly parsing the on-disk format (which isn't safe or reliable if the filesystem is mounted). It has been discussed before, but the discussions haven't really gotten anywhere.
It's worth noting that the way btrfs does checksums isn't per-file, it's per-block. This means that:
a. I think (I'm not 100% certain about this) that the checksum in btrfs includes the padding up to the end of the block for blocks that aren't full.
b. Files that get stored in-line in their metadata block won't have a checksum just for the file data (because the checksum will cover the whole metadata block).
c. While it is possible with some checksum algorithms (if I remember right, CRC32c is one such algorithm, and that is what btrfs uses for it's checksums) to combine the checksums from a group of data blocks to get the checksum for data as a whole, this in and of itself takes a significant amount of CPU time for large amounts of data.
All in all, this means that if you just want a checksum of the contents of the file, it's almost certainly better to just do it in userspace. If you're trying to figure out what changed, using send/receive and snapshots is more efficient (usually).
Quoted 4 lines
I've looked through linux kernel sources and not found appropriate ioctl to do this. Frankly speaking, I've not found good documentations for all available btrfs ioctls.
I agree that this documentation really needs to be improved (if you want to take the time to figure out how it all works, patches for the documentation would be greatly appreciated).
2015-09-24 21:35 GMT+03:00 Austin S Hemmelgarn <[email protected]>:
Quoted 29 lines · Matwey V. Kornilov
Quoted 9 lines
Hello,
I would like to read the list of the checksums for the specific file stored onto btrfs filesystem. I think I could use the checksums in the manner like rsync does, but safe both CPU (because csums are already calculated for the file) and I/O (because I don't need to reread all the file from the hard drive).
As of right now, there is no way to do this from userspace without just directly parsing the on-disk format (which isn't safe or reliable if the filesystem is mounted). It has been discussed before, but the discussions haven't really gotten anywhere.
It's worth noting that the way btrfs does checksums isn't per-file, it's per-block. This means that:
a. I think (I'm not 100% certain about this) that the checksum in btrfs includes the padding up to the end of the block for blocks that aren't full. b. Files that get stored in-line in their metadata block won't have a checksum just for the file data (because the checksum will cover the whole metadata block).
c. While it is possible with some checksum algorithms (if I remember right, CRC32c is one such algorithm, and that is what btrfs uses for it's checksums) to combine the checksums from a group of data blocks to get the checksum for data as a whole, this in and of itself takes a significant amount of CPU time for large amounts of data.
Quoted 4 lines
All in all, this means that if you just want a checksum of the contents of the file, it's almost certainly better to just do it in userspace. If you're trying to figure out what changed, using send/receive and snapshots is more efficient (usually).
I want the checksums of the every block of the file to see which part has been changed.
I cannot use send/receive because my other file replica is on the remote host but not on the same filesystem. Compare with how rsync works. It calculates checksums of the chunks of both versions of the file and then syncs different chunks over the network. I just want to utilize the fact that btrfs already has the data I need to calculate.
Quoted 10 lines
Quoted 5 lines
I've looked through linux kernel sources and not found appropriate ioctl to do this. Frankly speaking, I've not found good documentations for all available btrfs ioctls.
I agree that this documentation really needs to be improved (if you want to take the time to figure out how it all works, patches for the documentation would be greatly appreciated).
Signature · 4 lines
With best regards, Matwey V. Kornilov http://blog.matwey.name xmpp://[email protected]
Quoted 35 lines · Matwey V. Kornilov
2015-09-24 21:35 GMT+03:00 Austin S Hemmelgarn <[email protected]>:
Quoted 29 lines · Matwey V. Kornilov
Quoted 9 lines
Hello,
I would like to read the list of the checksums for the specific file stored onto btrfs filesystem. I think I could use the checksums in the manner like rsync does, but safe both CPU (because csums are already calculated for the file) and I/O (because I don't need to reread all the file from the hard drive).
As of right now, there is no way to do this from userspace without just directly parsing the on-disk format (which isn't safe or reliable if the filesystem is mounted). It has been discussed before, but the discussions haven't really gotten anywhere.
It's worth noting that the way btrfs does checksums isn't per-file, it's per-block. This means that:
a. I think (I'm not 100% certain about this) that the checksum in btrfs includes the padding up to the end of the block for blocks that aren't full. b. Files that get stored in-line in their metadata block won't have a checksum just for the file data (because the checksum will cover the whole metadata block).
c. While it is possible with some checksum algorithms (if I remember right, CRC32c is one such algorithm, and that is what btrfs uses for it's checksums) to combine the checksums from a group of data blocks to get the checksum for data as a whole, this in and of itself takes a significant amount of CPU time for large amounts of data.Quoted 4 lines
All in all, this means that if you just want a checksum of the contents of the file, it's almost certainly better to just do it in userspace. If you're trying to figure out what changed, using send/receive and snapshots is more efficient (usually).
Quoted 7 lines
I want the checksums of the every block of the file to see which part has been changed.
I cannot use send/receive because my other file replica is on the remote host but not on the same filesystem. Compare with how rsync works. It calculates checksums of the chunks of both versions of the file and then syncs different chunks over the network. I just want to utilize the fact that btrfs already has the data I need to calculate.
On current versions of btrfs-progs, btrfs send has a mode that will just spit out the metadata, which can then be parsed to figure out what has changed. The parsing is of course non-trivial, but should still be faster than checksumming everything, and I'm relatively sure (although I may be wrong) that the send stream format is well documented.
Quoted 7 lines · Matwey V. Kornilov
2015-09-24 21:35 GMT+03:00 Austin S Hemmelgarn <[email protected]> :
Quoted 5 lines · Matwey V. Kornilov
It's worth noting that the way btrfs does checksums isn't per-file, it's
per-block. This means that:
[...]
Quoted 6 lines
Quoted 5 lines
All in all, this means that if you just want a checksum of the contents of
the file, it's almost certainly better to just do it in userspace. If you're trying to figure out what changed, using send/receive and snapshots is more efficient (usually).
Quoted 7 lines
I want the checksums of the every block of the file to see which part has been changed.
I cannot use send/receive because my other file replica is on the remote host but not on the same filesystem. Compare with how rsync works. It calculates checksums of the chunks of both versions of the file and then syncs different chunks over the network. I just want to utilize the fact that btrfs already has the data I need to calculate.
The problem with trying to use btrfs checksums to compare two different files is that the blocks might not match up, if only due to fragmentation. E.g., the same 1gb file might be stored like this on one machine:
[ 256MB ][ 512 MB ][ 256MB ]
And like this on the other:
[ 512MB ][ 512MB ]
Since the checksums are per block, and the blocks can be different arrangements on different machines, they're not really all that useful for doing comparisons like you want.
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 28 lines · Calvin Walton
Quoted 7 lines · Matwey V. Kornilov
2015-09-24 21:35 GMT+03:00 Austin S Hemmelgarn <[email protected]> :
Quoted 5 lines · Matwey V. Kornilov
It's worth noting that the way btrfs does checksums isn't per-file, it's
per-block. This means that:[...]
Quoted 6 lines
Quoted 5 lines
All in all, this means that if you just want a checksum of the contents of
the file, it's almost certainly better to just do it in userspace. If you're trying to figure out what changed, using send/receive and snapshots is more efficient (usually).Quoted 7 lines
I want the checksums of the every block of the file to see which part has been changed.
I cannot use send/receive because my other file replica is on the remote host but not on the same filesystem. Compare with how rsync works. It calculates checksums of the chunks of both versions of the file and then syncs different chunks over the network. I just want to utilize the fact that btrfs already has the data I need to calculate.The problem with trying to use btrfs checksums to compare two different files is that the blocks might not match up, if only due to fragmentation. E.g., the same 1gb file might be stored like this on one machine:
Quoted 8 lines
[ 256MB ][ 512 MB ][ 256MB ]
And like this on the other:
[ 512MB ][ 512MB ]Since the checksums are per block, and the blocks can be different arrangements on different machines, they're not really all that useful for doing comparisons like you want.
No, those are extents, not blocks, and the FS doesn't checksum in
extents. :) Blocks are 4 KiB in size.
Hugo.
Signature · 4 lines
Hugo Mills | SCSI is usually fixed by remembering that it needs hugo@... carfax.org.uk | three terminations: One at each end of the chain. http://carfax.org.uk/ | And the goat. PGP: E2AB1DE4 | Andrew McDonald, HantsLUG
Quoted 15 lines · Hugo Mills
Quoted 13 lines · Calvin Walton
The problem with trying to use btrfs checksums to compare two different
files is that the blocks might not match up, if only due to fragmentation. E.g., the same 1gb file might be stored like this on one
machine:[ 256MB ][ 512 MB ][ 256MB ]
And like this on the other:
[ 512MB ][ 512MB ]
Quoted 7 lines
Quoted 4 lines
Since the checksums are per block, and the blocks can be different arrangements on different machines, they're not really all that useful
for doing comparisons like you want.No, those are extents, not blocks, and the FS doesn't checksum in extents. :) Blocks are 4 KiB in size.
Oh, wow, btrfs is storing a lot more checksums than I thought it was...
Thanks for clearing up my misconception.
Signature · 1 line
Calvin Walton <[email protected]>
Quoted 9 lines · Matwey V. Kornilov
I would like to read the list of the checksums for the specific file stored onto btrfs filesystem. I think I could use the checksums in the manner like rsync does, but safe both CPU (because csums are already calculated for the file) and I/O (because I don't need to reread all the file from the hard drive).
I've looked through linux kernel sources and not found appropriate ioctl to do this. Frankly speaking, I've not found good documentations for all available btrfs ioctls.
I have the ioctl docs in my todo and barely started it. There's no single ioctl for getting the per-block checksums but it is possible using several calls to the SEARCH_TREE. I have a prototype in my gits in branch dev/block-csum-search (2nd patch from the top), I'll try to find some time to make it a proper subcommand, so far the command UI is not righ enough and the output is quite rough.