Skip to contentSkip to conversation

generic block based fiemap implementation

PATCH In review Mark Fasheh 4 participants 6 messages May 25, 2008 – May 29, 2008

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 5/5 generic block based fiemap implementation

From: Josef Bacik <[email protected]>

Here is my updated generic block based fiemap implementation with the fixes that Andreas suggested. Basically any block based fs (this patch includes ext3) just has to declare its own fiemap() function and then call this generic function with its own get_block_t. This works well for block based filesystems that will map multiple contiguous blocks at one time, but will work for filesystems that only map one block at a time, you will just end up with an "extent" for each block. One gotcha is this will not play nicely where there is hole+data after the EOF. This function will assume its hit the end of the data as soon as it hits a hole after the EOF, so if there is any data past that it will not pick that up. AFAIK no block based fs does this anyway, but its in the comments of the function anyway just in case. Thanks much,

---
 fs/ext3/file.c          |    1 +
 fs/ext3/inode.c         |    8 +++
 fs/ioctl.c              |  128 +++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/ext3_fs.h |    2 +
 include/linux/fs.h      |    3 +
 5 files changed, 142 insertions(+), 0 deletions(-)
fs/ext3/file.c +1 -0
index acc4913..3be1e06 100644
@@ -134,5 +134,6 @@ const struct inode_operations ext3_file_inode_operations = {
 	.removexattr	= generic_removexattr,
 #endif
 	.permission	= ext3_permission,
+	.fiemap		= ext3_fiemap,
 };
 
fs/ext3/inode.c +8 -0
index 6ae4ecf..878b6da 100644
@@ -36,6 +36,7 @@
 #include <linux/mpage.h>
 #include <linux/uio.h>
 #include <linux/bio.h>
+#include <linux/fiemap.h>
 #include "xattr.h"
 #include "acl.h"
 
@@ -981,6 +982,13 @@ out:
 	return ret;
 }
 
+int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		u64 start, u64 len)
+{
+	return generic_block_fiemap(inode, fieinfo, start, len,
+				    ext3_get_block);
+}
+
 /*
  * `handle' can be NULL if create is zero
  */
fs/ioctl.c +128 -0

140 lines not on this page · raw message

include/linux/ext3_fs.h +2 -0
index 36c5403..2dc0dbb 100644
@@ -836,6 +836,8 @@ extern void ext3_truncate (struct inode *);
 extern void ext3_set_inode_flags(struct inode *);
 extern void ext3_get_inode_flags(struct ext3_inode_info *);
 extern void ext3_set_aops(struct inode *inode);
+extern int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		       u64 start, u64 len);
 
 /* ioctl.c */
 extern int ext3_ioctl (struct inode *, struct file *, unsigned int,
include/linux/fs.h +3 -0
index ed90393..5945136 100644
@@ -1976,6 +1976,9 @@ extern int vfs_fstat(unsigned int, struct kstat *);
 
 extern int do_vfs_ioctl(struct file *filp, unsigned int fd, unsigned int cmd,
 		    unsigned long arg);
+extern int generic_block_fiemap(struct inode *inode,
+				struct fiemap_extent_info *fieinfo, u64 start,
+				u64 len, get_block_t *get_block);
 
 extern void get_filesystem(struct file_system_type *fs);
 extern void put_filesystem(struct file_system_type *fs);
Signature · 1 line
1.5.4.1
#1 lore raw

fs/ext3/inode.c out: · line 990

Quoted 6 lines · Mark Fasheh
fs/ext3/inode.c +6 -0
+int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		u64 start, u64 len)
+{
+	return generic_block_fiemap(inode, fieinfo, start, len,
+				    ext3_get_block);
+}

We can wire up ext2 in the same way, and likely most of the other filesystems.

#define GENERIC_FIEMAP_FLAG_SUPP (FIEMAP_FLAG_NUM_EXTENTS | FIEMAP_FLAG_SYNC |
FIEMAP_FLAG_HSM_READ | FIEMAP_FLAG_LUN_ORDER)

fs/ioctl.c out_bad_flags: · line 239

Quoted 5 lines
fs/ioctl.c +5 -0
+int generic_block_fiemap(struct inode *inode,
+			 struct fiemap_extent_info *fieinfo, u64 start,
+			 u64 len, get_block_t *get_block)
+{
+	int just_count = fieinfo->fi_flags & FIEMAP_FLAG_NUM_EXTENTS;

Don't need this anymore.

	if (fieinfo->fi_flags & ~GENERIC_FIEMAP_FLAG_SUPP) {
		fieinfo->fi_flags &= ~GENERIC_FIEMAP_FLAG_SUPP;
		return -EBADF;
	}

fs/ioctl.c out_bad_flags: · line 277

Quoted 12 lines
fs/ioctl.c +12 -0
+		/* HOLE */
+		if (!tmp.b_blocknr) {
+			if (length <= 0) {
+				if (just_count) {
+					fieinfo->fi_extents_mapped++;
+					break;
+				}
+				flags = FIEMAP_EXTENT_LAST;
+				ret = fiemap_fill_next_extent(fieinfo, logical,
+							      phys, size,
+							      flags, 0);
+				break;

Can remove the just_count check, fiemap_fill_next_extent() will handle it.

Quoted 14 lines
fs/ioctl.c +14 -0
+		} else {
+			if (length <= 0 && size && !just_count) {
+				ret = fiemap_fill_next_extent(fieinfo, logical,
+							      phys, size,
+							      flags, 0);
+				if (ret)
+					break;
+			}
+
+			logical = blk_to_logical(inode, start_blk);
+			phys = blk_to_logical(inode, tmp.b_blocknr);
+			size = tmp.b_size;
+			flags = 0;
+

fs/ioctl.c out_bad_flags: · line 310

Quoted 10 lines
fs/ioctl.c +10 -0
+			length -= tmp.b_size;
+			start_blk += logical_to_blk(inode, size);
+
+			if (just_count) {
+				fieinfo->fi_extents_mapped++;
+				if (fieinfo->fi_extents_mapped ==
+				    fieinfo->fi_extents_max)
+					break;
+				continue;
+			}

Can also remove this just_count check, which is incorrect in any case. The fi_extents_max is only when storing extents in the array, it isn't meaningful when only counting extents.

More important is the extent merging I proposed for fiemap_fill_next_extent() for filesystems that only return single blocks, so we want to always call it from the generic_block_fiemap() code.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#2 lore raw
In reply to Mark Fasheh
Quoted 13 lines · Mark Fasheh

From: Josef Bacik <[email protected]>

Here is my updated generic block based fiemap implementation with the fixes that Andreas suggested. Basically any block based fs (this patch includes ext3) just has to declare its own fiemap() function and then call this generic function with its own get_block_t.

fs/ioctl.c +5 -0
+int generic_block_fiemap(struct inode *inode,
+			 struct fiemap_extent_info *fieinfo, u64 start,
+			 u64 len, get_block_t *get_block)
+{
+	struct buffer_head tmp;

[ ... ]

Quoted 4 lines
fs/ioctl.c +4 -0
+	do {
+		/*
+		 * we set b_size to the total size we want so it will map as
+		 * many contiguous blocks as possible at once

fs/ioctl.c out_bad_flags: · line 262

Quoted 10 lines
fs/ioctl.c +10 -0
+		 */
+		memset(&tmp, 0, sizeof(struct buffer_head));
+		tmp.b_size = map_len;
+
+		ret = get_block(inode, start_blk, &tmp, 0);
+		if (ret)
+			break;
+
+		/* HOLE */
+		if (!tmp.b_blocknr) {

This should be if (!buffer_mapped(&tmp)).

You can't be sure that filesystems returning a hole will use a b_blocknr of zero, and you can't be sure that a block number of zero is a hole.

-chris

-chris

#3 lore raw
In reply to Mark Fasheh
Quoted 14 lines · Mark Fasheh

From: Josef Bacik <[email protected]>

Here is my updated generic block based fiemap implementation with the fixes that Andreas suggested. Basically any block based fs (this patch includes ext3) just has to declare its own fiemap() function and then call this generic function with its own get_block_t. This works well for block based filesystems that will map multiple contiguous blocks at one time, but will work for filesystems that only map one block at a time, you will just end up with an "extent" for each block. One gotcha is this will not play nicely where there is hole+data after the EOF. This function will assume its hit the end of the data as soon as it hits a hole after the EOF, so if there is any data past that it will not pick that up. AFAIK no block based fs does this anyway, but its in the comments of the function anyway just in case. Thanks much,

Here is my updated patch that should address Andreas's and Chris's concerns. I added ext2 support as well. Thank you,

fs/ext3/file.c +1 -0
Index: linux-2.6/fs/ext3/file.c
===================================================================
@@ -134,5 +134,6 @@ const struct inode_operations ext3_file_
 	.removexattr	= generic_removexattr,
 #endif
 	.permission	= ext3_permission,
+	.fiemap		= ext3_fiemap,
 };
 
fs/ext3/inode.c +8 -0
Index: linux-2.6/fs/ext3/inode.c
===================================================================
@@ -36,6 +36,7 @@
 #include <linux/mpage.h>
 #include <linux/uio.h>
 #include <linux/bio.h>
+#include <linux/fiemap.h>
 #include "xattr.h"
 #include "acl.h"
 
@@ -981,6 +982,13 @@ out:
 	return ret;
 }
 
+int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		u64 start, u64 len)
+{
+	return generic_block_fiemap(inode, fieinfo, start, len,
+				    ext3_get_block);
+}
+
 /*
  * `handle' can be NULL if create is zero
  */
include/linux/ext3_fs.h +2 -0
Index: linux-2.6/include/linux/ext3_fs.h
===================================================================
@@ -836,6 +836,8 @@ extern void ext3_truncate (struct inode 
 extern void ext3_set_inode_flags(struct inode *);
 extern void ext3_get_inode_flags(struct ext3_inode_info *);
 extern void ext3_set_aops(struct inode *inode);
+extern int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		       u64 start, u64 len);
 
 /* ioctl.c */
 extern int ext3_ioctl (struct inode *, struct file *, unsigned int,
fs/ioctl.c +114 -0

126 lines not on this page · raw message

include/linux/fs.h +3 -0
Index: linux-2.6/include/linux/fs.h
===================================================================
@@ -1976,6 +1976,9 @@ extern int vfs_fstat(unsigned int, struc
 
 extern int do_vfs_ioctl(struct file *filp, unsigned int fd, unsigned int cmd,
 		    unsigned long arg);
+extern int generic_block_fiemap(struct inode *inode,
+				struct fiemap_extent_info *fieinfo, u64 start,
+				u64 len, get_block_t *get_block);
 
 extern void get_filesystem(struct file_system_type *fs);
 extern void put_filesystem(struct file_system_type *fs);
fs/ext2/ext2.h +2 -0
Index: linux-2.6/fs/ext2/ext2.h
===================================================================
@@ -133,6 +133,8 @@ extern void ext2_truncate (struct inode 
 extern int ext2_setattr (struct dentry *, struct iattr *);
 extern void ext2_set_inode_flags(struct inode *inode);
 extern void ext2_get_inode_flags(struct ext2_inode_info *);
+extern int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		       u64 start, u64 len);
 int __ext2_write_begin(struct file *file, struct address_space *mapping,
 		loff_t pos, unsigned len, unsigned flags,
 		struct page **pagep, void **fsdata);
fs/ext2/file.c +1 -0
Index: linux-2.6/fs/ext2/file.c
===================================================================
@@ -86,4 +86,5 @@ const struct inode_operations ext2_file_
 #endif
 	.setattr	= ext2_setattr,
 	.permission	= ext2_permission,
+	.fiemap		= ext2_fiemap,
 };
fs/ext2/inode.c +8 -0
Index: linux-2.6/fs/ext2/inode.c
===================================================================
@@ -31,6 +31,7 @@
 #include <linux/writeback.h>
 #include <linux/buffer_head.h>
 #include <linux/mpage.h>
+#include <linux/fiemap.h>
 #include "ext2.h"
 #include "acl.h"
 #include "xip.h"
@@ -704,6 +705,13 @@ int ext2_get_block(struct inode *inode, 
 
 }
 
+int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+		u64 start, u64 len)
+{
+	return generic_block_fiemap(inode, fieinfo, start, len,
+				    ext2_get_block);
+}
+
 static int ext2_writepage(struct page *page, struct writeback_control *wbc)
 {
 	return block_write_full_page(page, ext2_get_block, wbc);
#4 lore raw
Quoted 13 lines · Josef Bacik

+/*
+ * @inode - the inode to map
+ * @arg - the pointer to userspace where we copy everything to
+ * @get_block - the fs's get_block function
+ *
+ * This does FIEMAP for block based inodes. Basically it will just loop
+ * through get_block until we hit the number of extents we want to map, or we
+ * go past the end of the file and hit a hole.
+ *
+ * If it is possible to have holes and data past EOF then please don't use
+ * this function, it will do wrong things.
+ */

I think it would be clearer to write at the end "If it is possible to have data blocks beyond a hole past @inode->i_size, then please do not use this function, it will stop at the first unmapped block beyond i_size."

Quoted 20 lines
fs/ioctl.c +20 -0
+int generic_block_fiemap(struct inode *inode,
+			 struct fiemap_extent_info *fieinfo, u64 start,
+			 u64 len, get_block_t *get_block)
+{
+	struct buffer_head tmp;
+	unsigned int start_blk;
+	long long length = 0, map_len = 0;
+	u64 logical = 0, phys = 0, size = 0;
+	u32 flags = 0;
+	int ret = 0;
+
+	start_blk = logical_to_blk(inode, start);
+
+	/* guard against change */
+	mutex_lock(&inode->i_mutex);
+
+	length = (long long)min_t(u64, len, i_size_read(inode));
+	map_len = length;
+
+	do {

fs/ioctl.c out_bad_flags: · line 258

Quoted 10 lines
fs/ioctl.c +10 -0
+		/*
+		 * we set b_size to the total size we want so it will map as
+		 * many contiguous blocks as possible at once
+		 */
+		memset(&tmp, 0, sizeof(struct buffer_head));
+		tmp.b_size = map_len;
+
+		ret = get_block(inode, start_blk, &tmp, 0);
+		if (ret)
+			break;

Shouldn't map_len be reduced as the file is traversed? Otherwise the filesystem may be doing a lot more work than it needs to for block mapping if the requested length is not the whole file. If, for example, the first 1GB of the file is requested (map_len = 2 << 30) and 1022MB are mapped, the last 1MB get_block() request will still request 1GB mapping.

It isn't a major problem (only a corner case really), and the map_len can't be exactly the same as "length", because it still needs to be positive after EOF so that blocks will continue to be mapped.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#5 lore raw
In reply to Josef Bacik

fs/ioctl.c out_bad_flags: · line 232

Quoted 4 lines · Josef Bacik
fs/ioctl.c +4 -0
+int generic_block_fiemap(struct inode *inode,
+			 struct fiemap_extent_info *fieinfo, u64 start,
+			 u64 len, get_block_t *get_block)
+{

This still doesn't do any validity checking on fieinfo->fi_flags. You can't assume that the check in ioctl_fiemap() is correct for this routine. As soon as some new flag is added (e.g. FIEMAP_FLAG_XATTR) this code would do the wrong thing.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.