ntfs: add code to make FIBMAP ioctl work
Subsystems (2)
PATCH 1/1 ntfs: add code to make FIBMAP ioctl work
Implement ntfs_bmap() function for the "bmap" address space operation. Now user space programs can send FIBMAP ioctl to get files' placement on NTFS the same way they can do it on a number of other linux filesystems.
The result is returned in FIGETBSZ units, which is equal to sector size in ntfs driver. An error value zero is returned for resident, compressed, encrypted files, and for holes in sparse files.
Tested on drives with different sector sizes and different cluster sizes.
--- fs/ntfs/aops.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
fs/ntfs/aops.c +31 -0
@@ -1538,6 +1538,36 @@ err_out:#endif /* NTFS_RW */ +static sector_t ntfs_bmap (struct address_space *mapping,sector_t block) +{ +struct inode *vi = mapping->host; +ntfs_inode *ni =NTFS_I (vi); +ntfs_volume *vol = ni->vol; +const unsigned clust2sect_bits = vol->cluster_size_bits - + vol->sector_size_bits; +const sector_t clust2sect_mask = (1 << clust2sect_bits) -1 ; +LCN lcn; + +BUG_ON (S_ISDIR (vi->i_mode)); +ntfs_debug ("Requested LCN for block %llu." , (unsigned long long )block); +if (!NInoNonResident (ni) ||NInoCompressed (ni) ||NInoEncrypted (ni)) +return 0 ; + +down_read (&ni->runlist.lock); + lcn =ntfs_attr_vcn_to_lcn_nolock (ni, block >> clust2sect_bits, false); +up_read (&ni->runlist.lock); + +if (lcn <0 ) { +if (lcn ==LCN_HOLE ) +ntfs_warning (vol->sb,"Cannot get LCN for a hole." ); +else +ntfs_error (vol->sb,"Error getting LCN: %lld." , lcn); +return 0 ; + } + +return (lcn << clust2sect_bits) + (block & clust2sect_mask); +} +/** * ntfs_aops - general address space operations for inodes and attributes */
@@ -1546,6 +1576,7 @@ const struct address_space_operations ntfs_aops = {#ifdef NTFS_RW .writepage = ntfs_writepage, /* Write dirty page to disk. */#endif /* NTFS_RW */ + .bmap = ntfs_bmap, /* Map file offset to volume. */ .migratepage = buffer_migrate_page, /* Move a page cache page fromone physical page to an other. */
Signature · 1 line
2.1.2
Quoted 7 lines · Sergei Antonov
Implement
ntfs_bmap()function for the "bmap" address space operation. Now user space programs can send FIBMAP ioctl to get files' placement on NTFS the same way they can do it on a number of other linux filesystems.The result is returned in FIGETBSZ units, which is equal to sector size in ntfs driver. An error value zero is returned for resident, compressed, encrypted files, and for holes in sparse files.
Any thoughts about implementing FIEMAP for NTFS? This allows byte offset granularity, and reduces the number of system calls significantly.
Cheers, Andreas
Quoted 64 lines
Tested on drives with different sector sizes and different cluster sizes.
Cc Anton Altaparmakov <[email protected]>Signed-off-by Sergei Antonov <[email protected]>---
fs/ntfs/aops.c| 31 +++++++++++++++++++++++++++++++1 file changed, 31 insertions(+)fs/ntfs/aops.c +31
-0@@ -1538,6 +1538,36 @@ err_out:#endif /* NTFS_RW */ +static sector_t ntfs_bmap (struct address_space *mapping,sector_t block) +{ +struct inode *vi = mapping->host; +ntfs_inode *ni =NTFS_I (vi); +ntfs_volume *vol = ni->vol; +const unsigned clust2sect_bits = vol->cluster_size_bits - + vol->sector_size_bits; +const sector_t clust2sect_mask = (1 << clust2sect_bits) -1 ; +LCN lcn; + +BUG_ON (S_ISDIR (vi->i_mode)); +ntfs_debug ("Requested LCN for block %llu." , (unsigned long long )block); +if (!NInoNonResident (ni) ||NInoCompressed (ni) ||NInoEncrypted (ni)) +return 0 ; + +down_read (&ni->runlist.lock); + lcn =ntfs_attr_vcn_to_lcn_nolock (ni, block >> clust2sect_bits, false); +up_read (&ni->runlist.lock); + +if (lcn <0 ) { +if (lcn ==LCN_HOLE ) +ntfs_warning (vol->sb,"Cannot get LCN for a hole." ); +else +ntfs_error (vol->sb,"Error getting LCN: %lld." , lcn); +return 0 ; + } + +return (lcn << clust2sect_bits) + (block & clust2sect_mask); +} +/** * ntfs_aops - general address space operations for inodes and attributes */ @@ -1546,6 +1576,7 @@ const struct address_space_operations ntfs_aops = {#ifdef NTFS_RW .writepage = ntfs_writepage, /* Write dirty page to disk. */#endif /* NTFS_RW */ + .bmap = ntfs_bmap, /* Map file offset to volume. */ .migratepage = buffer_migrate_page, /* Move a page cache page fromone physical page to an other. */ Signature · 6 lines
2.1.2 -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas
Quoted 11 lines · Andreas Dilger
Quoted 7 lines · Sergei Antonov
Implement
ntfs_bmap()function for the "bmap" address space operation. Now user space programs can send FIBMAP ioctl to get files' placement on NTFS the same way they can do it on a number of other linux filesystems.The result is returned in FIGETBSZ units, which is equal to sector size in ntfs driver. An error value zero is returned for resident, compressed, encrypted files, and for holes in sparse files.
Any thoughts about implementing FIEMAP for NTFS? This allows byte offset granularity, and reduces the number of system calls significantly.
FIEMAP is good. But it is also less ubiquitous than FIBMAP (absent in fat, hfsplus, ...).
Hi,
Quoted 11 lines · Andreas Dilger
Quoted 7 lines · Sergei Antonov
Implement
ntfs_bmap()function for the "bmap" address space operation. Now user space programs can send FIBMAP ioctl to get files' placement on NTFS the same way they can do it on a number of other linux filesystems.The result is returned in FIGETBSZ units, which is equal to sector size in ntfs driver. An error value zero is returned for resident, compressed, encrypted files, and for holes in sparse files.
Any thoughts about implementing FIEMAP for NTFS? This allows byte offset granularity, and reduces the number of system calls significantly.
Sure, would be nice to have it. And I will get round to it one day but no customer has ever requested it so at least I haven't implemented it yet...
Best regards,
Anton
Quoted 59 lines
Cheers, Andreas
Quoted 57 lines
Tested on drives with different sector sizes and different cluster sizes.
Cc Anton Altaparmakov <[email protected]>Signed-off-by Sergei Antonov <[email protected]>---
fs/ntfs/aops.c| 31 +++++++++++++++++++++++++++++++1 file changed, 31 insertions(+)fs/ntfs/aops.c +31
-0@@ -1538,6 +1538,36 @@ err_out:#endif /* NTFS_RW */ +static sector_t ntfs_bmap (struct address_space *mapping,sector_t block) +{ +struct inode *vi = mapping->host; +ntfs_inode *ni =NTFS_I (vi); +ntfs_volume *vol = ni->vol; +const unsigned clust2sect_bits = vol->cluster_size_bits - + vol->sector_size_bits; +const sector_t clust2sect_mask = (1 << clust2sect_bits) -1 ; +LCN lcn; + +BUG_ON (S_ISDIR (vi->i_mode)); +ntfs_debug ("Requested LCN for block %llu." , (unsigned long long )block); +if (!NInoNonResident (ni) ||NInoCompressed (ni) ||NInoEncrypted (ni)) +return 0 ; + +down_read (&ni->runlist.lock); + lcn =ntfs_attr_vcn_to_lcn_nolock (ni, block >> clust2sect_bits, false); +up_read (&ni->runlist.lock); + +if (lcn <0 ) { +if (lcn ==LCN_HOLE ) +ntfs_warning (vol->sb,"Cannot get LCN for a hole." ); +else +ntfs_error (vol->sb,"Error getting LCN: %lld." , lcn); +return 0 ; + } + +return (lcn << clust2sect_bits) + (block & clust2sect_mask); +} +/** * ntfs_aops - general address space operations for inodes and attributes */ @@ -1546,6 +1576,7 @@ const struct address_space_operations ntfs_aops = {#ifdef NTFS_RW .writepage = ntfs_writepage, /* Write dirty page to disk. */#endif /* NTFS_RW */ + .bmap = ntfs_bmap, /* Map file offset to volume. */ .migratepage = buffer_migrate_page, /* Move a page cache page fromone physical page to an other. */
Signature · 3 lines
Anton Altaparmakov <anton at tuxera.com> (replace at with @) Senior Kernel Engineer, Tuxera Inc., http://www.tuxera.com/ Linux NTFS maintainer
Hi Sergei,
Quoted 9 lines · Sergei Antonov
Implement
ntfs_bmap()function for the "bmap" address space operation. Now user space programs can send FIBMAP ioctl to get files' placement on NTFS the same way they can do it on a number of other linux filesystems.The result is returned in FIGETBSZ units, which is equal to sector size in ntfs driver. An error value zero is returned for resident, compressed, encrypted files, and for holes in sparse files.
Tested on drives with different sector sizes and different cluster sizes.
Thanks for your patch. It skirts/looks over some limitations so instead of applying your patch I have open sourced the bmap implementation from Tuxera NTFS driver (with permission) and have pushed it to the open source NTFS tree at git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs.git.
I have sent a pull request to Linus so he will hopefully merge it into mainline.
You can compare it to yours but most important points are that you do not forbid non-data attributes and you completely ignore initialized size which leads to exposing stale on-disk data on read and to losing the written data on writes which is obviously bad. You also don't verify that the value to be returned isn't too large to fit in the return value size so you can potentially return truncated values on 32-bit kernels with 32-bit sector_t on a large volume where the sector number would need 64-bits which would then lead to disk corruption as the user would read/write random sectors on disk instead of the correct ones.
Best regards,
Anton
Quoted 55 lines
Cc Anton Altaparmakov <[email protected]>Signed-off-by Sergei Antonov <[email protected]>---
fs/ntfs/aops.c| 31 +++++++++++++++++++++++++++++++1 file changed, 31 insertions(+)fs/ntfs/aops.c +31
-0@@ -1538,6 +1538,36 @@ err_out:#endif /* NTFS_RW */ +static sector_t ntfs_bmap (struct address_space *mapping,sector_t block) +{ +struct inode *vi = mapping->host; +ntfs_inode *ni =NTFS_I (vi); +ntfs_volume *vol = ni->vol; +const unsigned clust2sect_bits = vol->cluster_size_bits - + vol->sector_size_bits; +const sector_t clust2sect_mask = (1 << clust2sect_bits) -1 ; +LCN lcn; + +BUG_ON (S_ISDIR (vi->i_mode)); +ntfs_debug ("Requested LCN for block %llu." , (unsigned long long )block); +if (!NInoNonResident (ni) ||NInoCompressed (ni) ||NInoEncrypted (ni)) +return 0 ; + +down_read (&ni->runlist.lock); + lcn =ntfs_attr_vcn_to_lcn_nolock (ni, block >> clust2sect_bits, false); +up_read (&ni->runlist.lock); + +if (lcn <0 ) { +if (lcn ==LCN_HOLE ) +ntfs_warning (vol->sb,"Cannot get LCN for a hole." ); +else +ntfs_error (vol->sb,"Error getting LCN: %lld." , lcn); +return 0 ; + } + +return (lcn << clust2sect_bits) + (block & clust2sect_mask); +} +/** * ntfs_aops - general address space operations for inodes and attributes */ @@ -1546,6 +1576,7 @@ const struct address_space_operations ntfs_aops = {#ifdef NTFS_RW .writepage = ntfs_writepage, /* Write dirty page to disk. */#endif /* NTFS_RW */ + .bmap = ntfs_bmap, /* Map file offset to volume. */ .migratepage = buffer_migrate_page, /* Move a page cache page fromone physical page to an other. */
Signature · 3 lines
Anton Altaparmakov <anton at tuxera.com> (replace at with @) Senior Kernel Engineer, Tuxera Inc., http://www.tuxera.com/ Linux NTFS maintainer
Quoted 7 lines · Anton Altaparmakov
Hi Sergei,
Quoted 4 lines · Sergei Antonov
Implement
ntfs_bmap()function for the "bmap" address space operation. Now user space programs can send FIBMAP ioctl to get files' placement on NTFS the same way they can do it on a number of other linux filesystems.
Quoted 7 lines
Quoted 5 lines
The result is returned in FIGETBSZ units, which is equal to sector size in ntfs driver. An error value zero is returned for resident, compressed, encrypted files, and for holes in sparse files.
Tested on drives with different sector sizes and different cluster sizes.
Thanks for your patch. It skirts/looks over some limitations so instead of applying your patch I have open sourced the bmap implementation from Tuxera NTFS driver (with permission) and have pushed it to the open source NTFS tree at
git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs.git.
Cool. Thanks.
It is good you liberate the code.
Quoted 3 lines
I have sent a pull request to Linus so he will hopefully merge it into mainline.
You can compare it to yours but most important points are that you do not forbid non-data attributes and you completely ignore initialized size which leads to exposing stale on-disk data on read and to losing the written data on writes which is obviously bad. You also don't verify that the value to be returned isn't too large to fit in the return value size so you can potentially return truncated values on 32-bit kernels with 32-bit
sector_ton a large volume where the sector number would need 64-bits which would then lead to disk corruption as the user would read/write random sectors on disk instead of the correct ones.
I looked at your code and do not quite understand the handling of initialized size. You read it into a local copy while holding the lock, but then after the lock is released it may change and, I'm afraid, cause the same problems you mentioned.
What is an example of a platform with 32-bit sector_t? Interesting to know.
Is it really possible that ntfs_bmap() is called for non-data attributes? In other words, what other attributes are accessible as files? Just curious.
Hi Sergei,
Quoted 4 lines · Sergei Antonov
I looked at your code and do not quite understand the handling of initialized size. You read it into a local copy while holding the lock, but then after the lock is released it may change and, I'm afraid, cause the same problems you mentioned.
Correct. That is why FIBMAP is a stupid interface! However initialized_size can only ever go up and not down so in the worst case you will get result zero, i.e. hole instead of the now real sector number you could access.
The only time it can shrink is when the file size shrinks and I am afraid if anyone is using FIBMAP against a file that can be truncated they will corrupt the file system!
Quoted 1 line
What is an example of a platform with 32-bit
sector_t? Interesting to know.
Any 32-bit system that does not configure the kernel for 64-bit sector_t. Sorry can't remember the .config option name at the moment. Would need to look it up.
Quoted 3 lines
Is it really possible that
ntfs_bmap()is called for non-data attributes? In other words, what other attributes are accessible as files? Just curious.
I admit in current kernel driver it cannot. It might in future - reparse points for example are very odd in many ways and though they are often indicated as directories rather than files they could not always be and what attributes a reparse point has is anyones guess - depends on the reparse point.
Then there are system files that do not have an unnamed $DATA attribute at all so in a way the check should probably be extended to ensure the attribute is $DATA _and_ unnamed but again anyone using FIBMAP on a system file is asking to cause problems...
Best regards,
Anton
Signature · 3 lines
Anton Altaparmakov <anton at tuxera.com> (replace at with @) Senior Kernel Engineer, Tuxera Inc., http://www.tuxera.com/ Linux NTFS maintainer
Quoted 11 lines · Anton Altaparmakov
Hi Sergei,
Quoted 4 lines · Sergei Antonov
I looked at your code and do not quite understand the handling of initialized size. You read it into a local copy while holding the lock, but then after the lock is released it may change and, I'm afraid, cause the same problems you mentioned.
Correct. That is why FIBMAP is a stupid interface! However
initialized_sizecan only ever go up and not down so in the worst case you will get result zero, i.e. hole instead of the now real sector number you could access.The only time it can shrink is when the file size shrinks and I am afraid if anyone is using FIBMAP against a file that can be truncated they will corrupt the file system!
I've tried to merge all your arguments in my head :) and concluded that a practical importance of the check against initialized_size is exaggerated :).
It is not a reliable protection (API safety issues) implemented in a way that it may not always work (limited locking scope). Can be an indicator of programmer's diligence, though - "you see, I care about it!". Sorry for this ramble. Your patch will do the job.
Quoted 3 lines
Quoted 1 line
What is an example of a platform with 32-bit
sector_t? Interesting to know.Any 32-bit system that does not configure the kernel for 64-bit
sector_t. Sorry can't remember the .config option name at the moment. Would need to look it up.
Thanks. Found it. It is CONFIG_LBDAF.
Quoted 7 lines
Quoted 3 lines
Is it really possible that
ntfs_bmap()is called for non-data attributes? In other words, what other attributes are accessible as files? Just curious.I admit in current kernel driver it cannot. It might in future - reparse points for example are very odd in many ways and though they are often indicated as directories rather than files they could not always be and what attributes a reparse point has is anyones guess - depends on the reparse point.
Then there are system files that do not have an unnamed $DATA attribute at all so in a way the check should probably be extended to ensure the attribute is $DATA _and_ unnamed but again anyone using FIBMAP on a system file is asking to cause problems...
Aren't those files handled by 'ntfs_mst_aops' (the one without a bmap function) rather than 'ntfs_normal_aops'?
Hi Sergei,
Quoted 7 lines · Sergei Antonov
Quoted 6 lines · Anton Altaparmakov
Quoted 4 lines · Sergei Antonov
I looked at your code and do not quite understand the handling of initialized size. You read it into a local copy while holding the lock, but then after the lock is released it may change and, I'm afraid, cause the same problems you mentioned.
Quoted 7 lines
Quoted 3 lines
Correct. That is why FIBMAP is a stupid interface! However
initialized_sizecan only ever go up and not down so in the worst case you will get result zero, i.e. hole instead of the now real sector number you could access.The only time it can shrink is when the file size shrinks and I am afraid if anyone is using FIBMAP against a file that can be truncated they will corrupt the file system!
I've tried to merge all your arguments in my head :) and concluded that a practical importance of the check against
initialized_sizeis exaggerated :).
I am afraid you are showing a serious lack of understanding of how NTFS works as a file system! I completely disagree with your statement. It is perfectly valid for an application to open() an UNCHANGING file and then run FIBMAP against it and then read from it directly (example applications: lilo and grub). It is completely NOT ok for the FIBMAP to return uninitialized sectors and the application to then read random garbage from disk instead of the zeroes that are meant to be there because you did not pay attention to the fact that initialized_size is less than file size and thus failed to return 0 instead of the sector number!
It has nothing to do with the fact that it might change. You must read zero beyond initialized_size. The whole point of it is that the area above it has never been written thus must be considered to be zero on disk no matter what old disk contents there might be from when the sectors were in use for something else/by some other file or directory or other metadata or even a previous file system that might have been there before.
Quoted 4 lines
It is not a reliable protection (API safety issues) implemented in a way that it may not always work (limited locking scope). Can be an indicator of programmer's diligence, though - "you see, I care about it!". Sorry for this ramble. Your patch will do the job.
The locking is there because initialized_size is 64-bit and this cannot be read on 32-bit without locking. Surely you knew that already? If not you need to do some more studying of 64-bit memory accesses on 32-bit systems... (-; E.g. have a look at i_size_read()/i_size_write() and inode_{get,add,sub,set}_bytes()...
And if you really don't understand it: a 32-bit CPU has to use two separate "read 32-bit memory location" instructions to read a 64-bit value so reading the lower 32-bits and higher 32-bits can happen a long time apart (if process is preempted between the two read instructions for example) and in the meantime a different process can write a new value in which case when the second read instruction finally executes it reads the high 32-bits from the new value and suddenly you have read random junk - the low 32-bits of old value and high 32-bits of new value and you think that is a valid 64-bit value which it is not. Thus you must lock to prevent this from happening hence why all 64-bit accesses of shared data are locked in Linux kernel (at least on 32-bit CPUs).
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What is an example of a platform with 32-bit
sector_t? Interesting to know.Any 32-bit system that does not configure the kernel for 64-bit
sector_t. Sorry can't remember the .config option name at the moment. Would need to look it up.Thanks. Found it. It is
CONFIG_LBDAF.
So it is, thanks for finding it. It is not an intuitive name!
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Is it really possible that
ntfs_bmap()is called for non-data attributes? In other words, what other attributes are accessible as files? Just curious.I admit in current kernel driver it cannot. It might in future - reparse points for example are very odd in many ways and though they are often indicated as directories rather than files they could not always be and what attributes a reparse point has is anyones guess - depends on the reparse point.
Then there are system files that do not have an unnamed $DATA attribute at all so in a way the check should probably be extended to ensure the attribute is $DATA _and_ unnamed but again anyone using FIBMAP on a system file is asking to cause problems...
Aren't those files handled by '
ntfs_mst_aops' (the one without a bmap function) rather than 'ntfs_normal_aops'?
Only directories and attributes containing B-trees are handled by mst_aops. Not all system files are B-trees and often there are both B-trees and data attributes and the latter are usually not mst protected thus normal_aops are used.
Best regards,
Anton
Signature · 3 lines
Anton Altaparmakov <anton at tuxera.com> (replace at with @) Senior Kernel Engineer, Tuxera Inc., http://www.tuxera.com/ Linux NTFS maintainer
Quoted 1 line · Anton Altaparmakov
Only directories and attributes containing B-trees are handled by
mst_aops. Not all system files are B-trees and often there are both B-trees and data attributes and the latter are usually not mst protected thusnormal_aopsare used.
Need to correct my above statement. $MFT and $MFTmirr are both $DATA attribute containing files that use mst_aops. But they are the only exceptions to the above.
Best regards,
Anton
Signature · 3 lines
Anton Altaparmakov <anton at tuxera.com> (replace at with @) Senior Kernel Engineer, Tuxera Inc., http://www.tuxera.com/ Linux NTFS maintainer