Add new "development flag" to the ext4 filesystem

As discussed on RFC, this flag is simply a generic "this is a crash/burn test filesystem" marker. If it is set, then filesystem code which is "in development" will be allowed to mount the filesystem. Filesystem code which is not considered ready for prime-time will check for this flag, and if it is not set, it will refuse to touch the filesystem.

As we start rolling ext4 out to distro's like Fedora, et. al, this makes it less likely that a user might accidentally start using ext4 on a production filesystem; a bad thing, since that will essentially make it be unfsckable until e2fsprogs catches up.

- Ted

index cf2f612..a8c599a 100644
@@ -1991,6 +1991,17 @@ static int ext4_fill_super (struct super_block *sb, void *data, int silent)
 		printk(KERN_WARNING
 		       "EXT4-fs warning: feature flags set on rev 0 fs, "
 		       "running e2fsck is recommended\n");
+
+	/*
+	 * Since ext4 is still considered development code, we require
+	 * that the TEST_FILESYS flag in s->flags be set.
+	 */
+	if (!(le32_to_cpu(es->s_flags) & EXT2_FLAGS_TEST_FILESYS)) {
+		printk(KERN_WARNING, "EXT4-fs: %s: test filesystem flag "
+		       "not set.\n");
+		goto failed_mount;
+	}
+
 	/*
 	 * Check feature flags regardless of the revision level, since we
 	 * previously didn't change the revision level when setting the flags,

index 98d1c3c..d203543 100644
@@ -547,6 +547,13 @@ do {									       \
 #define	EXT4_ORPHAN_FS			0x0004	/* Orphans being recovered */
 
 /*
+ * Misc. filesystem flags
+ */
+#define EXT2_FLAGS_SIGNED_HASH		0x0001  /* Signed dirhash in use */
+#define EXT2_FLAGS_UNSIGNED_HASH	0x0002  /* Unsigned dirhash in use */
+#define EXT2_FLAGS_TEST_FILESYS		0x0004	/* OK for use on development code */
+
+/*
  * Mount flags
  */
 #define EXT4_MOUNT_CHECK		0x00001	/* Do mount-time checks */

index b9ae14a..957ba5b 100644
@@ -147,11 +147,15 @@ static void print_super_flags(struct ext2_super_block * s, FILE *f)
 
 	fputs("Filesystem flags:         ", f);
 	if (s->s_flags & EXT2_FLAGS_SIGNED_HASH) {
-		fputs("signed directory hash ", f);
+		fputs("signed_directory_hash ", f);
 		flags_found++;
 	}
 	if (s->s_flags & EXT2_FLAGS_UNSIGNED_HASH) {
-		fputs("unsigned directory hash ", f);
+		fputs("unsigned_directory_hash ", f);
+		flags_found++;
+	}
+	if (s->s_flags & EXT2_FLAGS_TEST_FILESYS) {
+		fputs("test_filesystem ", f);
 		flags_found++;
 	}
 	if (flags_found)

index e096577..dd5e495 100644
@@ -467,12 +467,14 @@ struct ext2_inode_large {
  */
 #define EXT2_VALID_FS			0x0001	/* Unmounted cleanly */
 #define EXT2_ERROR_FS			0x0002	/* Errors detected */
+#define EXT4_ORPHAN_FS			0x0004	/* Orphans being recovered */
 
 /*
  * Misc. filesystem flags
  */
 #define EXT2_FLAGS_SIGNED_HASH		0x0001  /* Signed dirhash in use */
 #define EXT2_FLAGS_UNSIGNED_HASH	0x0002  /* Unsigned dirhash in use */
+#define EXT2_FLAGS_TEST_FILESYS		0x0004	/* OK for use on development code */
 
 /*
  * Mount flags

index 7a360ea..e754d6b 100644
@@ -849,6 +849,8 @@ static void parse_extended_opts(struct ext2_super_block *param,
 
 				param->s_reserved_gdt_blocks = rsv_gdb;
 			}
+		} else if (!strcmp(token, "test_fs")) {
+			param->s_flags |= EXT2_FLAGS_TEST_FILESYS;
 		} else
 			r_usage++;
 	}

@@ -859,7 +861,8 @@ static void parse_extended_opts(struct ext2_super_block *param,
 			"\tis set off by an equals ('=') sign.\n\n"
 			"Valid extended options are:\n"
 			"\tstride=<stride length in blocks>\n"
-			"\tresize=<resize maximum size in blocks>\n\n"));
+			"\tresize=<resize maximum size in blocks>\n"
+			"\ttest_fs\n"));
 		free(buf);
 		exit(1);
 	}

@@ -1556,6 +1559,9 @@ int main (int argc, char *argv[])
 		exit(1);
 	}
 
+	if (fs_param.s_flags & EXT2_FLAGS_TEST_FILESYS)
+		fs->super->s_flags |= EXT2_FLAGS_TEST_FILESYS;
+
 	/*
 	 * Wipe out the old on-disk superblock
 	 */

Overall, seems ok. One other question though, when ext4 is a fully-fledged production filesystem, and the flag requirement is gone, what stops ext3 filesystems from being silently mounted as ext4, just as happened with ext4dev today? Will users need to be sure everything explicitly mounts -t ext3 to avoid this?

+static void parse_extended_opts(ext2_filsys fs, const char *opts)

...

+ if (!strcmp(token, "test_fs")) {
+ fs->super->s_flags |= EXT2_FLAGS_TEST_FILESYS;
+ printf("Setting test filesystem flag\n");
+ ext2fs_mark_super_dirty(fs);
+ } else if (!strcmp(token, "production_fs")) {
+ fs->super->s_flags &= ~EXT2_FLAGS_TEST_FILESYS;
+ printf("Clearing test filesystem flag\n");
+ ext2fs_mark_super_dirty(fs);
+ } else
+ r_usage++;
+ }
+ if (r_usage) {
+ fprintf(stderr, _("\nBad options specified.\n\n"
+ "Extended options are separated by commas, "
+ "and may take an argument which\n"
+ "\tis set off by an equals ('=') sign.\n\n"
+ "Valid extended options are:\n"
+ "\ttestfs\n"));

help text doesn't match reality here, missing a "_"

-Eric

Nothing will prevent that, but in the long term we don't want ext4 to be automatically adding new features flags anyway. The only reason why we were doing that was to encourage more people to test out ext4, and for the convenience of the ABAT auto-testing.

So I'm assuming that before we remove this test, we will also be fixing some of the automatic enablement of extents, etc., because that sort of thing will be moved into e2fsprogs as part of "tune2fs -O ext4" or "mke2fs -O ext4" or "mkfs.ext4".

If we do that, then the only downside of having ext3 filesystems run under ext4 is the test matrix concern. Since I'm still hoping that some point in the future, fs/ext4 could subsume fs/ext3 so we don't have to worry about bug fixes going into fs/ext2 AND fs/ext3 AND fs/ext4, I have my own reasons for wanting that. But I do understand the concerns that maybe in the short term some distro's don't want to do that. So in that case I could see adding a "you must have extents" test into ext4, if I distro has specific support concerns. But for people who are running mainline kernel, I think it's actually a *good* thing if fs/ext4 can mount and read and write to an ext3 filesystem --- as long as it doesn't automatically turn on features behind the user's back.

Oops, thanks for catching that.

- Ted

Well, sure, the ability of ext4 code to mount,read,write ext3 filesystems is fine, esp. if ext4.ko stops doing things which makes it hard to go back to ext3. And, I do like the long-term plan of ext4 replacing ext3, it's a bit of a pain to keep this all in sync.

I just think that ext4.ko running ext3 filesystems needs to be under explicit control, and not something that happens, occasionally, accidentally, without the user/administrator requesting it. Least surprise, and all that...

-Eric

Well, most of the time that will happen, given that /etc/fstab explicitly states which filesystem to use; and if the user doesn't specify a filesystme type, mount will use blkid/vol_id, and if that says ext3, then ext3 will be used. The only time where it might not happen without an explicit administrator request is on the root filesystem automount.... and if you are using an initrd to mount the filesystem (as all of the enterprise distros now do anyway), that could be easily controlled form userspace as well.

- Ted

If any newbie kernel hacker wants a filesystem project, allowing ext4 to mount ext2 filesystems w/o a journal would be very useful. I suspect that a simple flag check in the ext4_journal_* wrappers of the jbd2 functions would be enough in many cases.

One of the reasons to keep ext2 around is that ext3 cannot mount the filesystem without a journal, and removing that limitation for ext4 would bring us one step closer to removing a ton of duplicate code. Another reason for ext2 vs. ext3 was overhead from journaling, and that could also be removed by allowing ext4 to mount ext2 filesystems w/o a journal.

Maybe a good proposal for a Google Summer-of-Code project.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Hi,

Actually, folding ext2 into ext3/4 isn't as easy as one would guess in
the beginning. For example ext2 on fsync() just sync's a single inode (and has to use private_list to track metadata buffers associated with the inode) while ext3 flushes the whole journal. Also in ext2, directory handling code is quite different. ext2 works in page cache of the directory while ext3 uses page cache of the underlying device via buffer heads - at least this second thing would be more or less mechanical thing to change and would make sence (we wouldn't have to reimplement readahead in ext3 directory handling code as we do now). I've looked at it once but then more urgent things came and ... you know it.

Honza

Jan Kara <[email protected]>
SuSE CR Labs

As for fsync(), we definitely need to preserve correct behaviour for the file itself, but there isn't a requirement that ext2 behave exactly like ext3 (it of course cannot). In the proposed ext4-mount-unjournaled-ext2 case, the superblock would be marked dirty as it is today and an e2fsck would need to be run at boot time. That is fine so long as the fsync() will cause the one file's data to be on disk before it returns.

I don't think it is a requirement that ext3 mounting a filesystem without a journal has to use page cache for directories. I wouldn't object to that being fixed. It definitely isn't a requirement for this to work, just an implementation difference.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Well, you have to also make sure that all indirect blocks are on disk
before fsync() returns. Otherwise there's not much point in the fact that data itself reached the disk. And for that you need something like private_list.

Yes, of course. I just wanted to point out that ext2 isn't a strict
subset of ext3 so there is some non-trivial work to be done before you can safely mount ext2 as ext3-without-journal.

Honza

Jan Kara <[email protected]>
SUSE Labs, CR

how can i set this "flag" on my filesystem? i've never set any flags before so i just removed the code from ext4-module to mount my filesystems, but setting the flag once would make it easier i think :)

If you have e2fsprogs 1.40.5, you set at mke2fs time via "mke2fs -E test_fs ...". If you haven't mounted the filesystem using ext4 yet, you can also convert an existing filesystem using "tune2fs -E test_fs /dev/hdXXX". At least with e2fsprogs 1.40.5, it won't work with a filesystem with extents enabled because tune2fs will refuse to touch such a filesystem. (And you don't want to use e2fsprogs 1.40.5 with ext4 anyway, since e2fsck in 1.40.5 doesn't understand extents yet.)

You can manually set the flag using debugfs:

    # debugfs -w /dev/hdXX
    debugfs: set_super_value s_flags 4
    debugfs: quit

(Or, if you're lazy, you can also type the shorthand "ssv flags 4" to debugfs.)

With e2fsprogs 1.40.5, if the test_fs flag is set, then blkid will return a filesystem type ext4dev, such that if your mount is configured to use blkid (instead of vol_id), it will automatically use ext4dev to mount a filesystem that has (a) a journal (since ext4 currently requires a journal), and (b) the test_fs flag set.

This makes it easier for people who are testing ext4, since they will just be able to do something like:

      # mke2fs -j -E test_fs /dev/mapper/lvmset-test
      # mount /dev/mapper/lvmset-test

.. and /dev/mapper/lvmset-test will automatically be mounted using ext4.

- Ted

thanks for your quick answer! i think i've learned something more about ext4 and those flags.