ext4_fallocate

When calling fallocate on ext4 fs, ext4_fallocate does not initialize the extents.

The extents are allocated only when they are actually
written. This is causing a problem for us. Our programs create many
"write only once" files as large as 1G on ext4 very rapidly at times.
We thought fallocate would solve the problem. But it didnt.
If I change the EXT4_GET_BLOCKS_CREATE_UNINIT_EXT to
just EXT4_GET_BLOCKS_CREATE in the ext4_map_blocks in the ext4_fallocate call, the extents get created in fallocate call itself. This is helping us.

Now the write throughtput to the disk was close to 98%. When extents were not initialized, our disk throughput were only 70%.

Can this change be made to ext4_fallocate?

There was a recent patch series "ext4: add an io-tree to track block allocation" that may improve the performance for your case of overwrite of uninitialized files, but it hasn't landed yet.

Quoted 3 lines · Andreas Dilger

There was a recent patch series "ext4: add an io-tree to track block allocation" that may improve the performance for your case of overwrite of uninitialized files, but it hasn't landed yet.

I'm confused. Why is writing to uninitialized extents slow, and why would this help? If you have an uninitialized extent, then the blocks are already allocated, just flagged as containing uninitialized data.
Writing to them should be no different than writing to initialized
extents, save for the step of clearing the uninitialized flag.

Hello Ext4 developers,

When calling fallocate on ext4 fs, ext4_fallocate does not initialize the extents. The extents are allocated only when they are actually written. This is causing a problem for us. Our programs create many "write only once" files as large as 1G on ext4 very rapidly at times. We thought fallocate would solve the problem. But it didnt. If I change the EXT4_GET_BLOCKS_CREATE_UNINIT_EXT to
just EXT4_GET_BLOCKS_CREATE in the ext4_map_blocks in the ext4_fallocate call,
the extents get created in fallocate call itself. This is helping us. Now the write throughtput to the disk was close to 98%. When extents were not
initialized, our disk throughput were only 70%.

Can this change be made to ext4_fallocate?

Quoted 15 lines · Fredrick

Hello Ext4 developers,

When calling fallocate on ext4 fs, ext4_fallocate does not initialize the extents. The extents are allocated only when they are actually written. This is causing a problem for us. Our programs create many "write only once" files as large as 1G on ext4 very rapidly at times. We thought fallocate would solve the problem. But it didnt. If I change the EXT4_GET_BLOCKS_CREATE_UNINIT_EXT to
just EXT4_GET_BLOCKS_CREATE in the ext4_map_blocks in the ext4_fallocate call,
the extents get created in fallocate call itself. This is helping us. Now the write throughtput to the disk was close to 98%. When extents were not
initialized, our disk throughput were only 70%.

Quoted 1 line

Can this change be made to ext4_fallocate?

Hi Fredrick,

I think that this patch maybe can help you. :-)

Actually I want to send a url for you from linux mailing list archive but I cannot find it. After applying this patch, you can call ioctl(2) to enable expose_stale_data flag, and then when you call fallocate(2), ext4 create initialized extents for you. This patch cannot be merged into upstream kernel because it brings a huge security hole.

Regards,
Zheng

From: Zheng Liu <[email protected]>
Date: Wed, 6 Jun 2012 11:10:57 +0800
Subject: [RFC][PATCH v2] ext4: add expose_stale_data flag in fallocate

Here is the v2 of FALLOC_FL_NO_HIDE_STALE in fallocate. Now no new flag is added into vfs in order to reduce the impacts and avoid a huge security hole. The application cannot call fallocate with a new flag to create an unwritten extent. It needs to call ioctl to enable/disable this feature. Meanwhile, in ioctl, filesystem will check CAP_SYS_RAWIO to ensure that the user has a privilege to switch on/off it. Currently, I only implement it in ext4.

Even though I try to reduce its impact, this feature still brings a security hole. So the application must ensure that it initializes an unwritten extent by itself before reading it, and it is used in a limited environment.

v1 -> v2:
* remove FALLOC_FL_NO_HIDE_STALE flag in vfs
* add 'i_expose_stale_data' in ext4 to enable/disable it

Signed-off-by Zheng Liu <[email protected]>

---
  fs/ext4/ext4.h    |    5 +++++
  fs/ext4/extents.c |    6 +++++-
  fs/ext4/ioctl.c   |   43 +++++++++++++++++++++++++++++++++++++++++++
  fs/ext4/super.c   |    1 +
  4 files changed, 54 insertions(+), 1 deletions(-)

fs/ext4/ext4.h +5 -0

index cfc4e01..61da070 100644
@@ -606,6 +606,8 @@ enum {
  #define EXT4_IOC_ALLOC_DA_BLKS		_IO('f', 12)
  #define EXT4_IOC_MOVE_EXT		_IOWR('f', 15, struct move_extent)
  #define EXT4_IOC_RESIZE_FS		_IOW('f', 16, __u64)
+#define EXT4_IOC_GET_EXPOSE_STALE	_IOR('f', 17, int)
+#define EXT4_IOC_SET_EXPOSE_STALE	_IOW('f', 18, int)
 
  #if defined(__KERNEL__) && defined(CONFIG_COMPAT)
  /*

@@ -925,6 +927,9 @@ struct ext4_inode_info {
 
  	/* Precomputed uuid+inum+igen checksum for seeding inode checksums */
  	__u32 i_csum_seed;
+
+	/* expose stale data in creating a new extent */
+	int i_expose_stale_data;
  };
 
  /*

fs/ext4/extents.c +5 -1

index 91341ec..9ef883c 100644
@@ -4336,6 +4336,7 @@ static void ext4_falloc_update_inode(struct inode *inode,
  long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  {
  	struct inode *inode = file->f_path.dentry->d_inode;
+	struct ext4_inode_info *ei = EXT4_I(inode);
  	handle_t *handle;
  	loff_t new_size;
  	unsigned int max_blocks;

@@ -4379,7 +4380,10 @@ long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  		trace_ext4_fallocate_exit(inode, offset, max_blocks, ret);
  		return ret;
  	}
-	flags = EXT4_GET_BLOCKS_CREATE_UNINIT_EXT;
+	if (ei->i_expose_stale_data)
+		flags = EXT4_GET_BLOCKS_CREATE;
+	else
+		flags = EXT4_GET_BLOCKS_CREATE_UNINIT_EXT;
  	if (mode & FALLOC_FL_KEEP_SIZE)
  		flags |= EXT4_GET_BLOCKS_KEEP_SIZE;
  	/*

fs/ext4/ioctl.c +43 -0

index 8ad112a..fffb3eb 100644
@@ -445,6 +445,47 @@ resizefs_out:
  		return 0;
  	}
 
+	case EXT4_IOC_GET_EXPOSE_STALE: {
+		int enable;
+
+		/* security check */
+		if (!capable(CAP_SYS_RAWIO))
+			return -EPERM;
+
+		/*
+		 * currently only extent-based files support (pre)allocate with
+		 * EXPOSE_STALE_DATA flag
+		 */
+		if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+			return -EOPNOTSUPP;
+
+		enable = ei->i_expose_stale_data;
+
+		return put_user(enable, (int __user *) arg);
+	}
+
+	case EXT4_IOC_SET_EXPOSE_STALE: {
+		int enable;
+
+		/* security check */
+		if (!capable(CAP_SYS_RAWIO))
+			return -EPERM;
+
+		/*
+		 * currently only extent-based files support (pre)allocate with
+		 * EXPOSE_STALE_DATA flag
+		 */
+		if (!(ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
+			return -EOPNOTSUPP;
+
+		if (get_user(enable, (int __user *) arg))
+			return -EFAULT;
+
+		ei->i_expose_stale_data = enable;
+
+		return 0;
+	}
+
  	default:
  		return -ENOTTY;
  	}

@@ -508,6 +549,8 @@ long ext4_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
  	case EXT4_IOC_MOVE_EXT:
  	case FITRIM:
  	case EXT4_IOC_RESIZE_FS:
+	case EXT4_IOC_GET_EXPOSE_STALE:
+	case EXT4_IOC_SET_EXPOSE_STALE:
  		break;
  	default:
  		return -ENOIOCTLCMD;

fs/ext4/super.c +0 -0

index eb7aa3e..3654bb8 100644

Actually I want to send a url for you from linux mailing list archive but I cannot find it. After applying this patch, you can call ioctl(2) to enable expose_stale_data flag, and then when you call fallocate(2), ext4 create initialized extents for you. This patch cannot be merged into upstream kernel because it brings a huge security hole.

Quoted 6 lines · Zheng Liu

Actually I want to send a url for you from linux mailing list archive but I cannot find it. After applying this patch, you can call ioctl(2) to enable expose_stale_data flag, and then when you call fallocate(2), ext4 create initialized extents for you. This patch cannot be merged into upstream kernel because it brings a huge security hole.

This is what we're using internally inside Google.... this allows the security exposure to be restricted to those programs running with a specific group id (which is better than giving programs access to CAP_SYS_RAWIO). We also require the use of a specific fallocate flag so that programs have to explicitly ask for this feature.

Also note that I restrict the combination of NO_HIDE_STALE && KEEP_SIZE since it causes e2fsck to complain --- and if you're trying to avoid fs metadata I/O, you want to avoid the extra i_size update anyway, so it's not worth trying to make this work w/o causing e2fsck complaints.

This patch is versus the v3.3 kernel (as it happens, I was just in the middle of rebasing this patch from 2.6.34 :-)

- Ted

P.S. It just occurred to me that there are some patches being discussed that assign new fallocate flags for volatile data handling. So it would probably be a good idea to move the fallocate flag codepoint assignment up out of the way to avoid future conflicts.

commit 5f12f1bc2b0fb0866d52763a611b022780780f05

Author: Theodore Ts'o <[email protected]>
Date:   Fri Jun 22 17:19:53 2012 -0400

ext4: add an fallocate flag to mark newly allocated extents initialized

This commit adds a new flag to ext4's fallocate that allows new, uninitialized extents to be marked as initialized. This flag, FALLOC_FL_NO_HIDE_STALE requires that the nohide_stale_gid=<gid> mount option be used when the file system is mounted, and that the user is in the group <gid>.

The benefit is to a program fallocates a larger space, but then writes to that space in small increments. This option prevents ext4 from having to split the unallocated extent and merge the newly initialized extent with the extent to its left. Even though this usually happens in-memory, this option is useful for tight memory situations and for ext4 on flash. Note: This allows an application in ths hohide_stale group to see stale data on the filesystem.

Tested: Updated xfstests g002 to test a case where
fallocate:no-hide-stale is not allowed. The existing tests now pass because I added a remount with a group that user root is in.
Rebase-Tested-v3.3: same

Effort: fs/nohide-stale
Origin-2.6.34-SHA1: c3099bf61be1baf94bc91c481995bb0d77f05786 Origin-2.6.34-SHA1: 004dd33b9ebc5d860781c3435526658cc8aa8ccb
Change-Id: I0d2a7f2a4cf34443269acbcedb7b7074e0055e69

fs/ext4/ext4.h +3 -0

index aaaece6..ac7aa42 100644
@@ -1240,6 +1240,9 @@ struct ext4_sb_info {
  	unsigned long s_mb_last_group;
  	unsigned long s_mb_last_start;
 
+	/* gid that's allowed to see stale data via falloc flag. */
+	gid_t no_hide_stale_gid;
+
  	/* stats for buddy allocator */
  	atomic_t s_bal_reqs;	/* number of reqs with len > 1 */
  	atomic_t s_bal_success;	/* we found long enough chunks */

fs/ext4/extents.c +20 -1

index cb99346..cc57c85 100644
@@ -4375,6 +4375,7 @@ long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  	int retries = 0;
  	int flags;
  	struct ext4_map_blocks map;
+	struct ext4_sb_info *sbi;
  	unsigned int credits, blkbits = inode->i_blkbits;
 
  	/*

@@ -4385,12 +4386,28 @@ long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  		return -EOPNOTSUPP;
 
  	/* Return error if mode is not supported */
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE |
+		     FALLOC_FL_NO_HIDE_STALE))
+		return -EOPNOTSUPP;
+
+	/* The combination of NO_HIDE_STALE and KEEP_SIZE is not supported */
+	if ((mode & FALLOC_FL_NO_HIDE_STALE) &&
+	    (mode & FALLOC_FL_KEEP_SIZE))
  		return -EOPNOTSUPP;
 
  	if (mode & FALLOC_FL_PUNCH_HOLE)
  		return ext4_punch_hole(file, offset, len);
 
+	sbi = EXT4_SB(inode->i_sb);
+	/* Must have RAWIO to see stale data. */
+	if ((mode & FALLOC_FL_NO_HIDE_STALE) &&
+	    !in_egroup_p(sbi->no_hide_stale_gid))
+		return -EACCES;
+
+	/* preallocation to directories is currently not supported */
+	if (S_ISDIR(inode->i_mode))
+		return -ENODEV;
+
  	trace_ext4_fallocate_enter(inode, offset, len, mode);
  	map.m_lblk = offset >> blkbits;
  	/*

@@ -4429,6 +4446,8 @@ retry:
  			ret = PTR_ERR(handle);
  			break;
  		}
+		if (mode & FALLOC_FL_NO_HIDE_STALE)
+			flags &= ~EXT4_GET_BLOCKS_UNINIT_EXT;
  		ret = ext4_map_blocks(handle, inode, &map, flags);
  		if (ret <= 0) {
  #ifdef EXT4FS_DEBUG

fs/ext4/super.c +12 -0

index 5b443a8..d976ec1 100644
@@ -1175,6 +1175,8 @@ static int ext4_show_options(struct seq_file *seq, struct dentry *root)
  	if (test_opt2(sb, BIG_EXT))
  		seq_puts(seq, ",big_extent");
  #endif
+	if (sbi->no_hide_stale_gid != -1)
+		seq_printf(seq, ",nohide_stale_gid=%u", sbi->no_hide_stale_gid);
 
  	ext4_show_quota_options(seq, sb);
 

@@ -1353,6 +1355,7 @@ enum {
  #ifdef CONFIG_EXT4_BIG_EXTENT
  	Opt_big_extent, Opt_nobig_extent,
  #endif
+	Opt_nohide_stale_gid,
  };
 
  static const match_table_t tokens = {

@@ -1432,6 +1435,7 @@ static const match_table_t tokens = {
  	{Opt_big_extent, "big_extent"},
  	{Opt_nobig_extent, "nobig_extent"},
  #endif
+	{Opt_nohide_stale_gid, "nohide_stale_gid=%u"},
  	{Opt_err, NULL},
  };
 

@@ -1931,6 +1935,12 @@ set_qf_format:
  				return 0;
  			sbi->s_li_wait_mult = option;
  			break;
+		case Opt_nohide_stale_gid:
+			if (match_int(&args[0], &option))
+				return 0;
+			/* -1 for disabled, otherwise it's valid. */
+			sbi->no_hide_stale_gid = option;
+			break;
  		case Opt_noinit_itable:
  			clear_opt(sb, INIT_INODE_TABLE);
  			break;

@@ -3274,6 +3284,8 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
  #ifdef CONFIG_EXT4_BIG_EXTENT
  	sbi->s_min_big_ext_size = EXT4_DEFAULT_MIN_BIG_EXT_SIZE;
  #endif
+	/* Default to having no-hide-stale disabled. */
+	sbi->no_hide_stale_gid = -1;
 
  	if ((def_mount_opts & EXT4_DEFM_NOBARRIER) == 0)
  		set_opt(sb, BARRIER);

fs/open.c +3 -1

index 201431a..4edc0cd 100644
@@ -224,7 +224,9 @@ int do_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  		return -EINVAL;
 
  	/* Return error if mode is not supported */
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+	if (mode & ~(FALLOC_FL_KEEP_SIZE |
+		     FALLOC_FL_PUNCH_HOLE |
+		     FALLOC_FL_NO_HIDE_STALE))
  		return -EOPNOTSUPP;
 
  	/* Punch hole must have keep size set */

include/linux/falloc.h +0 -0

index 73e0b62..a2489ac 100644

Quoted 5 lines · Zheng Liu

Actually I want to send a url for you from linux mailing list archive but I cannot find it. After applying this patch, you can call ioctl(2) to enable expose_stale_data flag, and then when you call fallocate(2), ext4 create initialized extents for you. This patch cannot be merged into upstream kernel because it brings a huge security hole.

This is what we're using internally inside Google.... this allows the security exposure to be restricted to those programs running with a specific group id (which is better than giving programs access to CAP_SYS_RAWIO). We also require the use of a specific fallocate flag so that programs have to explicitly ask for this feature.

Also note that I restrict the combination of NO_HIDE_STALE && KEEP_SIZE since it causes e2fsck to complain --- and if you're trying to avoid fs metadata I/O, you want to avoid the extra i_size update anyway, so it's not worth trying to make this work w/o causing e2fsck complaints.

This patch is versus the v3.3 kernel (as it happens, I was just in the middle of rebasing this patch from 2.6.34 :-)

- Ted

P.S. It just occurred to me that there are some patches being discussed that assign new fallocate flags for volatile data handling. So it would probably be a good idea to move the fallocate flag codepoint assignment up out of the way to avoid future conflicts.

commit 5f12f1bc2b0fb0866d52763a611b022780780f05

Author: Theodore Ts'o <[email protected]>
Date:   Fri Jun 22 17:19:53 2012 -0400

ext4: add an fallocate flag to mark newly allocated extents initialized

This commit adds a new flag to ext4's fallocate that allows new, uninitialized extents to be marked as initialized. This flag, FALLOC_FL_NO_HIDE_STALE requires that the nohide_stale_gid=<gid> mount option be used when the file system is mounted, and that the user is in the group <gid>.

The benefit is to a program fallocates a larger space, but then writes to that space in small increments. This option prevents ext4 from having to split the unallocated extent and merge the newly initialized extent with the extent to its left. Even though this usually happens in-memory, this option is useful for tight memory situations and for ext4 on flash. Note: This allows an application in ths hohide_stale group to see stale data on the filesystem.

Tested: Updated xfstests g002 to test a case where
fallocate:no-hide-stale is not allowed. The existing tests now pass because I added a remount with a group that user root is in.
Rebase-Tested-v3.3: same

Effort: fs/nohide-stale
Origin-2.6.34-SHA1: c3099bf61be1baf94bc91c481995bb0d77f05786 Origin-2.6.34-SHA1: 004dd33b9ebc5d860781c3435526658cc8aa8ccb
Change-Id: I0d2a7f2a4cf34443269acbcedb7b7074e0055e69

fs/ext4/ext4.h +3 -0

index aaaece6..ac7aa42 100644
@@ -1240,6 +1240,9 @@ struct ext4_sb_info {
  	unsigned long s_mb_last_group;
  	unsigned long s_mb_last_start;
 
+	/* gid that's allowed to see stale data via falloc flag. */
+	gid_t no_hide_stale_gid;
+
  	/* stats for buddy allocator */
  	atomic_t s_bal_reqs;	/* number of reqs with len > 1 */
  	atomic_t s_bal_success;	/* we found long enough chunks */

fs/ext4/extents.c +20 -1

index cb99346..cc57c85 100644
@@ -4375,6 +4375,7 @@ long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  	int retries = 0;
  	int flags;
  	struct ext4_map_blocks map;
+	struct ext4_sb_info *sbi;
  	unsigned int credits, blkbits = inode->i_blkbits;
 
  	/*

@@ -4385,12 +4386,28 @@ long ext4_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  		return -EOPNOTSUPP;
 
  	/* Return error if mode is not supported */
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE |
+		     FALLOC_FL_NO_HIDE_STALE))
+		return -EOPNOTSUPP;
+
+	/* The combination of NO_HIDE_STALE and KEEP_SIZE is not supported */
+	if ((mode & FALLOC_FL_NO_HIDE_STALE) &&
+	    (mode & FALLOC_FL_KEEP_SIZE))
  		return -EOPNOTSUPP;
 
  	if (mode & FALLOC_FL_PUNCH_HOLE)
  		return ext4_punch_hole(file, offset, len);
 
+	sbi = EXT4_SB(inode->i_sb);
+	/* Must have RAWIO to see stale data. */
+	if ((mode & FALLOC_FL_NO_HIDE_STALE) &&
+	    !in_egroup_p(sbi->no_hide_stale_gid))
+		return -EACCES;
+
+	/* preallocation to directories is currently not supported */
+	if (S_ISDIR(inode->i_mode))
+		return -ENODEV;
+
  	trace_ext4_fallocate_enter(inode, offset, len, mode);
  	map.m_lblk = offset >> blkbits;
  	/*

@@ -4429,6 +4446,8 @@ retry:
  			ret = PTR_ERR(handle);
  			break;
  		}
+		if (mode & FALLOC_FL_NO_HIDE_STALE)
+			flags &= ~EXT4_GET_BLOCKS_UNINIT_EXT;
  		ret = ext4_map_blocks(handle, inode, &map, flags);
  		if (ret <= 0) {
  #ifdef EXT4FS_DEBUG

fs/ext4/super.c +12 -0

index 5b443a8..d976ec1 100644
@@ -1175,6 +1175,8 @@ static int ext4_show_options(struct seq_file *seq, struct dentry *root)
  	if (test_opt2(sb, BIG_EXT))
  		seq_puts(seq, ",big_extent");
  #endif
+	if (sbi->no_hide_stale_gid != -1)
+		seq_printf(seq, ",nohide_stale_gid=%u", sbi->no_hide_stale_gid);
 
  	ext4_show_quota_options(seq, sb);
 

@@ -1353,6 +1355,7 @@ enum {
  #ifdef CONFIG_EXT4_BIG_EXTENT
  	Opt_big_extent, Opt_nobig_extent,
  #endif
+	Opt_nohide_stale_gid,
  };
 
  static const match_table_t tokens = {

@@ -1432,6 +1435,7 @@ static const match_table_t tokens = {
  	{Opt_big_extent, "big_extent"},
  	{Opt_nobig_extent, "nobig_extent"},
  #endif
+	{Opt_nohide_stale_gid, "nohide_stale_gid=%u"},
  	{Opt_err, NULL},
  };
 

@@ -1931,6 +1935,12 @@ set_qf_format:
  				return 0;
  			sbi->s_li_wait_mult = option;
  			break;
+		case Opt_nohide_stale_gid:
+			if (match_int(&args[0], &option))
+				return 0;
+			/* -1 for disabled, otherwise it's valid. */
+			sbi->no_hide_stale_gid = option;
+			break;
  		case Opt_noinit_itable:
  			clear_opt(sb, INIT_INODE_TABLE);
  			break;

@@ -3274,6 +3284,8 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
  #ifdef CONFIG_EXT4_BIG_EXTENT
  	sbi->s_min_big_ext_size = EXT4_DEFAULT_MIN_BIG_EXT_SIZE;
  #endif
+	/* Default to having no-hide-stale disabled. */
+	sbi->no_hide_stale_gid = -1;
 
  	if ((def_mount_opts & EXT4_DEFM_NOBARRIER) == 0)
  		set_opt(sb, BARRIER);

fs/open.c +3 -1

index 201431a..4edc0cd 100644
@@ -224,7 +224,9 @@ int do_fallocate(struct file *file, int mode, loff_t offset, loff_t len)
  		return -EINVAL;
 
  	/* Return error if mode is not supported */
-	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
+	if (mode & ~(FALLOC_FL_KEEP_SIZE |
+		     FALLOC_FL_PUNCH_HOLE |
+		     FALLOC_FL_NO_HIDE_STALE))
  		return -EOPNOTSUPP;
 
  	/* Punch hole must have keep size set */

include/linux/falloc.h +1 -1

index 73e0b62..a2489ac 100644
@@ -3,6 +3,7 @@
 
  #define FALLOC_FL_KEEP_SIZE	0x01 /* default is extend size */
  #define FALLOC_FL_PUNCH_HOLE	0x02 /* de-allocates range */
+#define FALLOC_FL_NO_HIDE_STALE	0x04 /* default is hide stale data */
 
  #ifdef __KERNEL__
 
--

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Has anyone made progress digging into the performance impact of running without this patch? We should definitely see if there is some low hanging fruit there, especially given that XFS does not seem to suffer such a huge hit.

Opening this security exposure is still something that is clearly a hack and best avoided if we can fix the root cause :)

Quoted 5 lines · Ric Wheeler

Has anyone made progress digging into the performance impact of running without this patch? We should definitely see if there is some low hanging fruit there, especially given that XFS does not seem to suffer such a huge hit.

I just haven't had time, sorry. It's so much easier to run with the patch. :-)

Part of the problem certainly caused by the fact that ext4 is using physical block journaling instead of logical journalling. But we see the problem in no-journal mode as well. I think part of the problem is simply that many of the workloads where people are doing this, they also care about robustness after power failures, and if you are doing random writes into uninitialized space, with fsyncs in-between, you are basically guaranteed a 2x expansion in the number of writes you need to do to the system.

One other thing which we *have* seen is that we need to do a better job with extent merging; if you run without this patch, and you run with fio in AIO mode where you are doing tons and tons of random writes into uninitialized space, you can end up fragmenting the extent
tree very badly. So fixing this would certainly help.

Quoted 2 lines

Opening this security exposure is still something that is clearly a hack and best avoided if we can fix the root cause :)

See Linus's recent rant about how security arguments made by theoreticians very often end up getting trumped by practical matters. If you are running a daemon, whether it is a user-mode cluster file system, or a database server, where it is (a) fundamentally trusted, and (b) doing its own user-space checksuming and its own guarantees to never return uninitialized data, even if we fix all potential problems, we *still* can be reducing the number of random writes --- and on a fully loaded system, we're guaranteed to be seek-constrained, so each random write to update fs metadata means that you're burning 0.5% of your 200 seeks/second on your 3TB disk (where previously you had half a dozen 500gig disks each with 200 seeks/second).

I agree with you that it would be nice to look into this further, and optimizing our extent merging is definitely on the hot list of perofrmance improvements to look at. But people who are using ext4 as back-end database servers or cluster file system servers and who are interested in wringing out every last percentage of performace are going to be interested in this technique, no matter what we do. If you have Sagans and Sagans of servers all over the world, even a tenth of a percentage point performance improvement can easily translate into big dollars.

- Ted

Quoted 4 lines · Ric Wheeler

Has anyone made progress digging into the performance impact of running without this patch? We should definitely see if there is some low hanging fruit there, especially given that XFS does not seem to suffer such a huge hit.

I just haven't had time, sorry. It's so much easier to run with the patch. :-)

Part of the problem certainly caused by the fact that ext4 is using physical block journaling instead of logical journalling. But we see the problem in no-journal mode as well. I think part of the problem is simply that many of the workloads where people are doing this, they also care about robustness after power failures, and if you are doing random writes into uninitialized space, with fsyncs in-between, you are basically guaranteed a 2x expansion in the number of writes you need to do to the system.

One other thing which we *have* seen is that we need to do a better job with extent merging; if you run without this patch, and you run with fio in AIO mode where you are doing tons and tons of random writes into uninitialized space, you can end up fragmenting the extent
tree very badly. So fixing this would certainly help.

Quoted 3 lines

Opening this security exposure is still something that is clearly a hack and best avoided if we can fix the root cause :)

See Linus's recent rant about how security arguments made by theoreticians very often end up getting trumped by practical matters. If you are running a daemon, whether it is a user-mode cluster file system, or a database server, where it is (a) fundamentally trusted, and (b) doing its own user-space checksuming and its own guarantees to never return uninitialized data, even if we fix all potential problems, we *still* can be reducing the number of random writes --- and on a fully loaded system, we're guaranteed to be seek-constrained, so each random write to update fs metadata means that you're burning 0.5% of your 200 seeks/second on your 3TB disk (where previously you had half a dozen 500gig disks each with 200 seeks/second).

I agree with you that it would be nice to look into this further, and optimizing our extent merging is definitely on the hot list of perofrmance improvements to look at. But people who are using ext4 as back-end database servers or cluster file system servers and who are interested in wringing out every last percentage of performace are going to be interested in this technique, no matter what we do. If you have Sagans and Sagans of servers all over the world, even a tenth of a percentage point performance improvement can easily translate into big dollars.

- Ted

It would be interesting to see if simply not doing the preallocation would be easier and safer. Better yet, figure out how to leverage trim commands to safely allow us to preallocate and not expose other users' data (and not have to mark the extents as allocated but not written).

This is not a theory guy worry. I would not use any server/web service that knowingly enabled this hack in a multi-user machine and would not enable it for any enterprise customers.

We should be very, very careful not to strip away the usefulness of file system just to cater to some users.

Quoted 5 lines · Ric Wheeler

It would be interesting to see if simply not doing the preallocation would be easier and safer. Better yet, figure out how to leverage trim commands to safely allow us to preallocate and not expose other users' data (and not have to mark the extents as allocated but not written).

TRIM is applicable to SSD's and enterprise storage arrays. It's not applicable HDD's.

Quoted 4 lines

This is not a theory guy worry. I would not use any server/web service that knowingly enabled this hack in a multi-user machine and would not enable it for any enterprise customers.

Sure, but for single-user or for dedicated cluster file system server, it makes sense; it's not not going to be useful for all customers, sure, but for some customers it will make sense.

Quoted 3 lines

We should be very, very careful not to strip away the usefulness of file system just to cater to some users.

A mount option or a superblock flag does not "strip away the usefulness of the file system". It makes it more useful for some use cases.

Your alternate proposals (preinitializing the space with zeros, using trim, writing larger chunks) will work for some use cases, certainly. But it's not going to be sufficient for at least some use cases, or they will simply not work.

Fundamentally it sounds to me the biggest problem is that you don't trust your customers, and so you're afraid people will use the no-hide-stale feature if it becomes available, even if it's not needed or if it's needed but you're afraid that they don't understand the security tradeoffs.

- Ted

Hi Ted,

Has anyone made progress digging into the performance impact of running without this patch? We should definitely see if there is some low hanging fruit there, especially given that XFS does not seem to suffer such a huge hit.

I think that we need to get a good reproducer for the workload that causes the pain and start to dig into this.

Opening this security exposure is still something that is clearly a hack and best avoided if we can fix the root cause :)

Ric

Quoted 1 line

             7,413 ext4:ext4_mark_inode_dirty 

         1,310,806 ext4:ext4_ext_map_blocks_enter 

         1,310,806 ext4:ext4_ext_map_blocks_exit 

I had run perf stat on ext4 functions between two runs of our program writing data to a file for the first time and writing data to the file for the second time(where the extents are initialized).

From your mballoc differences, it sounds like you were comparing

Quoted 8 lines

Hi Ted,

Has anyone made progress digging into the performance impact of running without this patch? We should definitely see if there is some low hanging fruit there, especially given that XFS does not seem to suffer such a huge hit.

Quoted 9 lines

I think that we need to get a good reproducer for the workload that causes the pain and start to dig into this.

Opening this security exposure is still something that is clearly a hack and best avoided if we can fix the root cause :)

Ric

Quoted 1 line

Hi Ric,

I had run perf stat on ext4 functions between two runs of our program writing data to a file for the first time and writing data to the file for the second time(where the extents are initialized).
The amount of data written is same between the two runs.

left is first time
right is second time.

< 42 ext4:ext4_mb_bitmap_load
< 42 ext4:ext4_mb_buddy_bitmap_load
< 642 ext4:ext4_mb_new_inode_pa
< 645 ext4:ext4_mballoc_alloc
< 9,596 ext4:ext4_mballoc_prealloc
< 10,240 ext4:ext4_da_update_reserve_space
---

Quoted 1 line

             7,413 ext4:ext4_mark_inode_dirty

49d52
< 10,241 ext4:ext4_allocate_blocks
51d53
< 10,241 ext4:ext4_request_blocks
55d56
< 1,310,720 ext4:ext4_da_reserve_space
58,60c59,60

<          1,331,288 ext4:ext4_ext_map_blocks_enter
<          1,331,288 ext4:ext4_ext_map_blocks_exit
<          1,341,467 ext4:ext4_mark_inode_dirty

---

Quoted 2 lines

         1,310,806 ext4:ext4_ext_map_blocks_enter
         1,310,806 ext4:ext4_ext_map_blocks_exit

May be the mballocs have overhead.

I ll try to compare numbers on XFS during this week.

-Fredrick

Thanks! Eric is also running some tests to evaluate the impact of various techniques :)

ric

Quoted 3 lines · Ric Wheeler

Thanks! Eric is also running some tests to evaluate the impact of various techniques :)

ric

Yup forgive me for interjecting actual numbers into the discussion ;)

I tried running this fio recipe on v3.3, which I think does a decent job of emulating the situation (fallocate 1G, do random 1M writes into it, with fsyncs after each):

[test]
filename=testfile
rw=randwrite
size=1g
filesize=1g
bs=1024k
ioengine=sync
fallocate=1
fsync=1

Stock ext4 (3 tests w/ file remove & cache drop in between):

WRITE: io=1024.0MB, aggrb=16322KB/s, minb=16713KB/s, maxb=16713KB/s, mint=64243msec, maxt=64243msec
WRITE: io=1024.0MB, aggrb=16249KB/s, minb=16639KB/s, maxb=16639KB/s, mint=64528msec, maxt=64528msec
WRITE: io=1024.0MB, aggrb=16370KB/s, minb=16763KB/s, maxb=16763KB/s, mint=64052msec, maxt=64052msec

With the patch which exposes other users' data:

WRITE: io=1024.0MB, aggrb=17840KB/s, minb=18268KB/s, maxb=18268KB/s, mint=58776msec, maxt=58776msec
WRITE: io=1024.0MB, aggrb=17841KB/s, minb=18269KB/s, maxb=18269KB/s, mint=58773msec, maxt=58773msec
WRITE: io=1024.0MB, aggrb=17828KB/s, minb=18255KB/s, maxb=18255KB/s, mint=58816msec, maxt=58816msec

so about 10% faster than without.

XFS, FWIW:

WRITE: io=1024.0MB, aggrb=24008KB/s, minb=24584KB/s, maxb=24584KB/s, mint=43675msec, maxt=43675msec
WRITE: io=1024.0MB, aggrb=24069KB/s, minb=24647KB/s, maxb=24647KB/s, mint=43564msec, maxt=43564msec
WRITE: io=1024.0MB, aggrb=24054KB/s, minb=24632KB/s, maxb=24632KB/s, mint=43591msec, maxt=43591msec

which is 35% faster than ext4 with the risky patch.

Haven't yet tried overwrites or done any tracing or profiling, but I think the fio recipe is a decent demonstrator, I'll try the overwrites etc in a bit when I get a moment.

-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 1 line · Eric Sandeen

On 6/26/12 3:30 PM, Ric Wheeler wrote:

Quoted 4 lines

I tried running this fio recipe on v3.3, which I think does a decent job of emulating the situation (fallocate 1G, do random 1M writes into it, with fsyncs after each):

Quoted 9 lines

[test]
filename=testfile
rw=randwrite
size=1g
filesize=1g
bs=1024k
ioengine=sync
fallocate=1
fsync=1

Quoted 5 lines

I tried running this fio recipe on v3.3, which I think does a decent job of emulating the situation (fallocate 1G, do random 1M writes into it, with fsyncs after each):

Quoted 9 lines

[test]
filename=testfile
rw=randwrite
size=1g
filesize=1g
bs=1024k
ioengine=sync
fallocate=1
fsync=1

Quoted 5 lines · Eric Sandeen

Quoted 5 lines

I tried running this fio recipe on v3.3, which I think does a decent job of emulating the situation (fallocate 1G, do random 1M writes into it, with fsyncs after each):

Quoted 9 lines

Quoted 9 lines

[test]
filename=testfile
rw=randwrite
size=1g
filesize=1g
bs=1024k
ioengine=sync
fallocate=1
fsync=1

A better workload would be to use a blocksize of 4k. By using a blocksize of 1024k, it's not surprising that the metadata overhead is in the noise.

Try something like this; this will cause the extent tree overhead to be roughly equal to the data block I/O.

[global]
rw=randwrite
size=128m
filesize=1g
bs=4k
ioengine=sync
fallocate=1
fsync=1

[thread1]
filename=testfile

Quoted 30 lines · Theodore Ts'o

Quoted 4 lines · Eric Sandeen

Quoted 4 lines

I tried running this fio recipe on v3.3, which I think does a decent job of emulating the situation (fallocate 1G, do random 1M writes into it, with fsyncs after each):

Quoted 9 lines

Quoted 9 lines

[test]
filename=testfile
rw=randwrite
size=1g
filesize=1g
bs=1024k
ioengine=sync
fallocate=1
fsync=1

A better workload would be to use a blocksize of 4k. By using a blocksize of 1024k, it's not surprising that the metadata overhead is in the noise.

Try something like this; this will cause the extent tree overhead to be roughly equal to the data block I/O.

[global]
rw=randwrite
size=128m
filesize=1g
bs=4k
ioengine=sync
fallocate=1
fsync=1

Quoted 2 lines

[thread1]
filename=testfile

Well, ok ... TBH I changed it to size=16m to finish in under 20m.... so here are the results:

fallocate 1g, do 16m of 4k random IOs, sync after each:

# for I in a b c; do rm -f testfile; echo 3 > /proc/sys/vm/drop_caches; fio tytso.fio | grep 2>&1 WRITE; done

   WRITE: io=16384KB, aggrb=154KB/s, minb=158KB/s, maxb=158KB/s, mint=105989msec, maxt=105989msec
   WRITE: io=16384KB, aggrb=163KB/s, minb=167KB/s, maxb=167KB/s, mint=99906msec, maxt=99906msec
   WRITE: io=16384KB, aggrb=176KB/s, minb=180KB/s, maxb=180KB/s, mint=92791msec, maxt=92791msec

same, but overwrite pre-written 1g file (same as the expose-my-data option ;)

# dd if=/dev/zero of=testfile bs=1M count=1024
# for I in a b c; do echo 3 > /proc/sys/vm/drop_caches; fio tytso.fio | grep 2>&1 WRITE; done

   WRITE: io=16384KB, aggrb=164KB/s, minb=168KB/s, maxb=168KB/s, mint=99515msec, maxt=99515msec
   WRITE: io=16384KB, aggrb=164KB/s, minb=168KB/s, maxb=168KB/s, mint=99371msec, maxt=99371msec
   WRITE: io=16384KB, aggrb=164KB/s, minb=168KB/s, maxb=168KB/s, mint=99677msec, maxt=99677msec

so no great surprise, small synchronous 4k writes have terrible performance, but I'm still not seeing a lot of fallocate overhead.

xfs, FWIW:

# for I in a b c; do rm -f testfile; echo 3 > /proc/sys/vm/drop_caches; fio tytso.fio | grep 2>&1 WRITE; done

   WRITE: io=16384KB, aggrb=202KB/s, minb=207KB/s, maxb=207KB/s, mint=80980msec, maxt=80980msec
   WRITE: io=16384KB, aggrb=203KB/s, minb=208KB/s, maxb=208KB/s, mint=80508msec, maxt=80508msec
   WRITE: io=16384KB, aggrb=204KB/s, minb=208KB/s, maxb=208KB/s, mint=80291msec, maxt=80291msec

# dd if=/dev/zero of=testfile bs=1M count=1024
# for I in a b c; do echo 3 > /proc/sys/vm/drop_caches; fio tytso.fio | grep 2>&1 WRITE; done

   WRITE: io=16384KB, aggrb=197KB/s, minb=202KB/s, maxb=202KB/s, mint=82869msec, maxt=82869msec
   WRITE: io=16384KB, aggrb=203KB/s, minb=208KB/s, maxb=208KB/s, mint=80348msec, maxt=80348msec
   WRITE: io=16384KB, aggrb=202KB/s, minb=207KB/s, maxb=207KB/s, mint=80827msec, maxt=80827msec

Again, I think this is just a diabolical workload ;)

-Eric

We need to keep in mind what the goal of pre-allocation is (should be?) - spend a bit of extra time doing the allocation call so we get really good, contiguous layout on disk which ultimately will help in streaming read/write workloads.

If you have a reasonably small file, pre-allocation is probably simply a waste of time - you would be better off overwriting the maximum file size with all zeros (even a 1GB file would take only a few seconds).

If the file is large enough to be interesting, I think that we might want to think about a scheme that would bring small random IO's more into line with the 1MB results Eric saw.

One way to do that might be to have a minimum "chunk" that we would zero out for any IO to an allocated but unwritten extent. You write 4KB to the middle of said region, we pad up and zero out to the nearest MB with zeros.

Note for the target class of drives (S-ATA) that Ted mentioned earlier, doing a random 4KB write vs a 1MB write is not that much slower (you need to pay the head movement costs already). Of course, the sweet spot might turn out to be a bit smaller or larger.

There is already code for this in the ext4 uninit extent handling. Currently the limit is 15(?) blocks, to inflate the write size up to zero-fill a 64kB chunk on 4kB blocksize filesystems. I wouldn't object to increasing this to zero out a full 1MB (aligned) chunk under random IO cases.

fallocate 1g, do 16m of 4k random IOs, sync after each:

# for I in a b c; do rm -f testfile; echo 3 > /proc/sys/vm/drop_caches; fio tytso.fio | grep 2>&1 WRITE; done

WRITE: io=16384KB, aggrb=154KB/s, minb=158KB/s, maxb=158KB/s, mint=105989msec, maxt=105989msec
WRITE: io=16384KB, aggrb=163KB/s, minb=167KB/s, maxb=167KB/s, mint=99906msec, maxt=99906msec
WRITE: io=16384KB, aggrb=176KB/s, minb=180KB/s, maxb=180KB/s, mint=92791msec, maxt=92791msec

same, but overwrite pre-written 1g file (same as the expose-my-data option ;)

# dd if=/dev/zero of=testfile bs=1M count=1024
# for I in a b c; do echo 3 > /proc/sys/vm/drop_caches; fio tytso.fio | grep 2>&1 WRITE; done

WRITE: io=16384KB, aggrb=164KB/s, minb=168KB/s, maxb=168KB/s, mint=99515msec, maxt=99515msec
WRITE: io=16384KB, aggrb=164KB/s, minb=168KB/s, maxb=168KB/s, mint=99371msec, maxt=99371msec
WRITE: io=16384KB, aggrb=164KB/s, minb=168KB/s, maxb=168KB/s, mint=99677msec, maxt=99677msec

Quoted 16 lines · Theodore Ts'o

A better workload would be to use a blocksize of 4k. By using a blocksize of 1024k, it's not surprising that the metadata overhead is in the noise.

Try something like this; this will cause the extent tree overhead to be roughly equal to the data block I/O.

[global]
rw=randwrite
size=128m
filesize=1g
bs=4k
ioengine=sync
fallocate=1
fsync=1

Quoted 2 lines

[thread1]
filename=testfile

Quoted 17 lines · Eric Sandeen

Quoted 16 lines · Theodore Ts'o

A better workload would be to use a blocksize of 4k. By using a blocksize of 1024k, it's not surprising that the metadata overhead is in the noise.

Try something like this; this will cause the extent tree overhead to be roughly equal to the data block I/O.

[global]
rw=randwrite
size=128m
filesize=1g
bs=4k
ioengine=sync
fallocate=1
fsync=1

Quoted 2 lines

Quoted 2 lines

[thread1]
filename=testfile

Hi Eric,

Could you please run this test with 'journal_async_commit' flag. In my previous test, this feature can dramatically improve the performance of uninitialized extent conversion.

I have sent an email to do a similar test [1]. In that email, I do a similar test and the journal_async_commit flag quite can improve the performance.

1. http://comments.gmane.org/gmane.linux.file-systems/63569

Regards,
Zheng

Quoted 6 lines · Zheng Liu

Hi Eric,

Could you please run this test with 'journal_async_commit' flag. In my previous test, this feature can dramatically improve the performance of uninitialized extent conversion.

Quoted 3 lines

I have sent an email to do a similar test [1]. In that email, I do a similar test and the journal_async_commit flag quite can improve the performance.

I can try to find time for that, but so far I haven't actually seen any severe impact of conversion on a non-debug kernel. And didn't Jan think that journal_async_commit was fatally flawed?

Quoted 10 lines · Eric Sandeen

Quoted 9 lines · Zheng Liu

Hi Eric,

Could you please run this test with 'journal_async_commit' flag. In my previous test, this feature can dramatically improve the performance of uninitialized extent conversion.

I have sent an email to do a similar test [1]. In that email, I do a similar test and the journal_async_commit flag quite can improve the performance.

Quoted 3 lines

I can try to find time for that, but so far I haven't actually seen any severe impact of conversion on a non-debug kernel. And didn't Jan think that journal_async_commit was fatally flawed?

Yes, that option is broken and basically unfixable for data=ordered mode
(see http://comments.gmane.org/gmane.comp.file-systems.ext4/30727). For data=writeback it works fine AFAICT.

Honza

Definitely more interesting I think to try and do the MB size extent conversion, that should be generally a good technique to minimize the overhead.

workload frequently does a uninitialized extent conversion. Thus, now we set it to 256 (1MB chunk), and put it into super block in order to adjust it dynamically in sysfs.

Quoted 4 lines

workload frequently does a uninitialized extent conversion. Thus, now we set it to 256 (1MB chunk), and put it into super block in order to adjust it dynamically in sysfs.

It's a bit of a tangent, but this caught my eye.

Review

Quoted 12 lines

(quoted diff) +4 -2

+	/* If extent has less than 2*s_extent_zeroout_len zerout directly */
+	if (ee_len<= 2*sbi->s_extent_zeroout_len&&
 	(EXT4_EXT_MAY_ZEROOUT&  split_flag)) {
 
-		if (allocated<= EXT4_EXT_ZERO_LEN&&
+		if (allocated<= sbi->s_extent_zeroout_len&&
 		(EXT4_EXT_MAY_ZEROOUT&  split_flag)) {
 
 		} else if ((map->m_lblk - ee_block + map->m_len<
-			   EXT4_EXT_ZERO_LEN)&&
+			   sbi->s_extent_zeroout_len)&&

I'd be worried about having to verify that nothing bad happened if these sbi s_extent_zeroout_len references could see different values if they raced with a sysfs update. Can they do that?

Maybe avoid the risk all together and have an on-stack copy that's only referenced once at the start?

Yes, that option is broken and basically unfixable for data=ordered mode
(see http://comments.gmane.org/gmane.comp.file-systems.ext4/30727). For data=writeback it works fine AFAICT.

Quoted 3 lines · Jan Kara

Yes, that option is broken and basically unfixable for data=ordered mode
(see http://comments.gmane.org/gmane.comp.file-systems.ext4/30727). For data=writeback it works fine AFAICT.

The journal_async_commit option can be saved, but it requires changing how we handle stale data. What we need to do is to change things so that we update the metadata *after* the data has been written back. We do this already if the experimental dioread_nolock code is used, but currently it only works for 4k blocks.

The I/O tree work will give us the infrastructure we need so we can easily update the metadata after the data blocks have been written out when we are extending or filling in a sparse hole, even when the block size != page size. (This is why we can't currently make the dioread_nolock code path the default; it would cause things to break on 1k/2k file systems, as well as 4k file systems on Power.) But once this is done, it will allow us to subsume and rip out dioread_nolock code[path, and the distinction between ordered and writeback mode.

Also, the metadata checksum patches will fix the other potential problem with using journal_async_commit, which is that it adds fine-grained checksums in the journal, so we can recover more easily from a corrupted journal.

So once all of this is stable, we'll be able significantly simplify the ext4 code and our testing matrix, and get all of the benefits of data=writeback, dioread_nolock, and journal_async_commit, without any of their current drawbacks. Which is why I've kept on pestering Zheng about how the I/O tree work has been coming along on the ext4 calls; it's going to enable some really cool things. :-)

Yes, that option is broken and basically unfixable for data=ordered mode
(see http://comments.gmane.org/gmane.comp.file-systems.ext4/30727). For data=writeback it works fine AFAICT.

Quoted 3 lines · Jan Kara

Yes, that option is broken and basically unfixable for data=ordered mode
(see http://comments.gmane.org/gmane.comp.file-systems.ext4/30727). For data=writeback it works fine AFAICT.

Does data=writeback with barriers fix this fallocate problem?

I can see that writing small random bits into a very large uninitialized file would cause a lot of updates to the extent tree, but with a sufficiently large cache, shouldn't many of the small, random writes become coalesced into fewer, larger extent updates that are done at delayed flush time? Are you sure that the extent tree updates are not being done right away and blocking the writing application, instead of being delayed?

Actually the workload needs to flush the data after writting a small random bits. This workload is met in our product system at Taobao. Thus, the application has to wait this write to be done. Certainly if we don't flush the data, the problem won't happen but there is a risk that we could loss our data.

Ohh, I see now... you want lots of small, random, synchronous writes. Then I think the only way to avoid the metadata overhead is with the unsafe stale data patch. More importantly, this workload is going to have terrible performance no matter what the fs does, because even with all of the blocks initialized, you're still doing lots of seeking and not allowing the IO elevator to help. Maybe the application can be redesigned so that it does not generate such pathological IO?

Or is this another case of userspace really needing access to barriers rather than using the big hammer of fsync?

Is there any technical reason why a barrier flag can't be added to the aio interface?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 5 lines · Zheng Liu

Actually the workload needs to flush the data after writting a small random bits. This workload is met in our product system at Taobao. Thus, the application has to wait this write to be done. Certainly if we don't flush the data, the problem won't happen but there is a risk that we could loss our data.

Ohh, I see now... you want lots of small, random, synchronous writes. Then I think the only way to avoid the metadata overhead is with the unsafe stale data patch. More importantly, this workload is going to have terrible performance no matter what the fs does, because even with all of the blocks initialized, you're still doing lots of seeking and not allowing the IO elevator to help. Maybe the application can be redesigned so that it does not generate such pathological IO?

Or is this another case of userspace really needing access to barriers rather than using the big hammer of fsync?

Is there any technical reason why a barrier flag can't be added to the aio interface?

On 07/03/2012 11:06 PM, Phillip Susi wrote:
For reasonably sized files, you might just as well really write out the full file with "write" (do pre-allocation the old fashioned way).

Performance of course depends on the size of the file, but for a 1GB file you can do this in a few seconds and prevent fragmentation and totally eliminate the performance of flipping extents.

How large is the file you need to pre-allocate? How long does the job typically run (minutes? hours? days?) :)

Hello Ext4 developers,

When calling fallocate on ext4 fs, ext4_fallocate does not initialize the extents. The extents are allocated only when they are actually written. This is causing a problem for us. Our programs create many "write only once" files as large as 1G on ext4 very rapidly at times. We thought fallocate would solve the problem. But it didnt.

If I change the EXT4_GET_BLOCKS_CREATE_UNINIT_EXT to
just EXT4_GET_BLOCKS_CREATE in the ext4_map_blocks in the ext4_fallocate call, the extents get created in fallocate call itself. This is helping us. Now the write throughtput to the disk was close to 98%. When extents were not initialized, our disk throughput were only 70%.

Can this change be made to ext4_fallocate?

Thanks,
Fredrick